Add support for environment variables

[Bramzor]

As best practices today is to use environment variables as config parameters, it would make sense to use it for environment based variables.
For example in my use case, I want to have rcpt_to.in_host_list plugin to check for environment variable to be set if there is no file. So kind of a fall back method.

[smfreegard]

That doesn't make sense - how can you possible replace the host_list file (which contains a list of domains) with an environment variable?

[Bramzor]

Easy... by setting HARAKA_HOST_LIST = "domain1.com domain2.com domain3.com"

[KingNoosh]

Linked to haraka/Haraka#2098

Good point @Bramzor you can have even an array environments in BASH nowadays.

[KingNoosh]

I think what would be easier is to allow environment variables to be used within the config files.

Less complicated to get working and easier to use.

[Bramzor]

+1 for environment variables inside config files

I was thinking the ability to override variables for example from smtp.ini.
Example (notice the __ separator)

HARAKA__SMTP_INI__LISTEN="[::0]:2525"

[KingNoosh]

https://gist.github.com/KingNoosh/4461e2cacff887d55604d149e06745f0

Is what we have atm, I can easily take this since I already have this.

[Bramzor]

Advantage of using environment variables inside config files is that you still keep one place where this is defined instead of having multiple places which might easily be overlooked.

[msimerson]

by setting HARAKA_HOST_LIST = "domain1.com domain2.com domain3.com"

another way is using URIs:

HARAKA_HOST_LIST="redis://redis.example.com:6179/0/host_list" or similar.
HARAKA_HOST_LIST="mysql://mysql.example.com/haraka/host_list" or similar.

@msimerson love the idea of fetching stuff directly from DB! How about passwords? inline, config or another env?

[smfreegard]

@msimerson love the idea of fetching stuff directly from DB!

Unfortunately being as config is a synchronous API, getting config from a database like this is impossible unless it's only ever read at start-up.

@smfreegard maybe cron (timer or something) like wrapper to refetch specific configs in master?

[smfreegard]

@smfreegard maybe cron like wrapper to refetch specific configs in master?

Yeah - that could work as I do something similar here - config gets read from the database in init_master and init_child and then an interval is set to re-read the database for only changes and deletions (I have a deleted flag and last_update timestamp in the database table that is set by triggers to do this).

However - you either need to use something that supports database triggers or pub/sub for it to work without requiring a complete re-read of the whole table on each interval, so PostgreSQL (like in my case) or Redis. I don't think MySQL would work as IIRC it doesn't have triggers like PostgreSQL.

It would need a fair amount of thought put into it for this to work generically for everyone that uses Haraka and not something that just works for a specific case.

Redis pub/sub seems like a sensible idea being as we already support Redis in one way or another. Redis keys would work fine for simple values, Hashes for INI/JSON type files (via dot notation) and Lists for List/Data types. However for list types, I don't think you can subscribe to see individual elements (but I might be wrong), so you might have to re-read the whole list - which wouldn't be ideal.

It's do-able, but I guess it would require a way to add plugins to config.

@smfreegard there actually are some triggers in MySQL:

CREATE     
[DEFINER = { user | CURRENT_USER }]     
TRIGGER trigger_name     
trigger_time trigger_event     
ON tbl_name FOR EACH ROW     
trigger_body
trigger_time: { BEFORE | AFTER } 
trigger_event: { INSERT | UPDATE | DELETE }

Maybe introduce some new event to trigger the update into the API? So plugins could also listen for it.
Redis would be neat because it is pretty fast from my experience.