Emoncms self contained docker container and HomeAssistant Addon

Emoncms
24

@alexandrecuer - what is the default user? I am not getting the usual ‘register’ message on first use.

I must have imported or used my other credentials :man_facepalming:

[edit]

Wow - this just works - fantastic job. I am really excited by this!

A few feedback points and thoughts :slight_smile:

  1. Can you pick up the TZ from the main HA system?
  2. I think we should either disable feedwriter or provide an option to enable it (disable by default). It is there to mitigate a SD Card wear and considering the amount of data HA writes to disk, I think our data is somewhat minimal in the overall scheme. It just adds complexity and delays data writing.
  3. I’d like to be able to disable the MQTT if not using the local Broker. I’m inclined to think that for MQTT, most people are already using it and/or using the HA Add-on so I think it is worth being able to disable this.
  4. I think this should have HTTP and HTTPS lables - will the HTTPS actually work though?
    image
  5. I’d like the Apps module added
  6. Updating the base system didn’t totall work as there are local changes to some of the local files - this may be to accommodate HA.

MIght

@TrystanLea - this is fantastic and deserves some attention.

It really needs a way of knowing this is inside HA so certain things are done internally to account for that (so ‘Full Update’ not available for instance)

1 Like
25

Thanks for the feedback. Nice to see it can be usefull

  1. i dont know how they deal with TZ in HA. Have to check.

  2. if you disable feewriter, you will not be able to write to feeds with datas diffused by the mqtt broker, local or not…

  3. if the local broker stays active, is it embarrassing ? Did yu manage to make the emoncms Add-on work using the HA MQTT Add-on ?

  4. I’m going to look into the https issue. Dont know how they do it with the grafana Add-on

  5. it should be ok to add the app plugin but i dont know how it works, it is only a dashboard/graph visualization plugin, if i understand correctly ? If there is no install script, it should work without any pain like the graph and dashboard modules…

  6. for the update, normally HA tells there is a new version. You press update and docker pull the new container and the add-on configuration files (which are in the /data/addons/git folder of the supervisor container) are also pulled

26

Not exactly sure what you mean here, but feedwriter simply writes the data to a Redis buffer first, then once x seconds, writes the data in bulk to disk (This is Low-Write). Has nothing to do with MQTT.

Not embarrassing, just using resources needlessly :). I made it work with an external MQTT broker (as that is what I have). I note you cannot specify the port, but as the HA one uses the standard MQTT Port, it should work OK.

It is more the config labelling. This is the std MQTT addon.

image
I’ve not tried using it :slight_smile:

Yes :slight_smile:

I don’t think that is then pulling a new version of emoncms - i.e. what Admin → update does. I did try that, but I presume if I tried to do an HA update, it would overwrite the internal update :frowning:

I also noted that some of the components were modified from the git repo status.

The other thing to consider (longer term) is what happens when the update script changes something internally for a normal setup. Will that affect the HA version?

27

Yes you are correct

It should be possible but I have to build to maintain 2 different docker images in that case because the feedwriter is integrated as a s6 service at build time not at run time. At runtime, I am afraid you cannot choose to deactivate a service at runtime, thats why the local broker cannot easily be deactivated as you suggest…

1 Like
28

I have a feeling that if you set the buffer time to 0 that effectively disables it - @TrystanLea ?

So an option to set the buffer time in the configuration page could solve that :slight_smile:

from settings.ini

[redis]
enabled = true
prefix = ''

setting to false (or actually deleting it as default is false) would stop the buffering.

29

when setting the redis buffer to false in the emoncms settings.ini :

redisbuffer[enabled] = 0

the emoncms log gets filled with errors…so I think that if you want to run without the redis buffer I have to remove the feedwriter as a service in the dockerfile…

2023-10-04 11:44:44.-82|ERROR|feedwriter.php|Error: setting must be true: settings['feed']['redisbuffer']['enabled']
2023-10-04 11:44:45.-37|ERROR|feedwriter.php|Error: setting must be true: settings['feed']['redisbuffer']['enabled']
2023-10-04 11:44:46.-10|ERROR|feedwriter.php|Error: setting must be true: settings['feed']['redisbuffer']['enabled']
2023-10-04 11:44:48.017|ERROR|feedwriter.php|Error: setting must be true: settings['feed']['redisbuffer']['enabled']
2023-10-04 11:44:49.045|ERROR|feedwriter.php|Error: setting must be true: settings['feed']['redisbuffer']['enabled']
2023-10-04 11:44:50.071|ERROR|feedwriter.php|Error: setting must be true: settings['feed']['redisbuffer']['enabled']
2023-10-04 11:44:51.112|ERROR|feedwriter.php|Error: setting must be true: settings['feed']['redisbuffer']['enabled']
2023-10-04 11:44:52.138|ERROR|feedwriter.php|Error: setting must be true: settings['feed']['redisbuffer']['enabled']

cf https://github.com/emoncms/emoncms/blob/6e7bc775dcaf4dacad055a1b1ae25a545c3dd163/scripts/feedwriter.php#L19

30

I think this should be false or true not 0 or 1

31

with 0 it is working as excepted with false…

Anyway bulk writing is more efficient, so why disabling it ?

32

Because you don’t get instant data if you want it.

33

But the last values are in redis, if you want last value for input 1, you can have it :

/opt/openenergymonitor # redis-cli
127.0.0.1:6379> keys *
1) "input:lastvalue:1"
2) "user:inputs:1"
3) "input:1"
4) "user:1"
127.0.0.1:6379> type input:lastvalue:1
hash
127.0.0.1:6379> hgetall input:lastvalue:1
1) "value"
2) "43.67"
3) "time"
4) "1696421993"

And It is something similar for feeds

the graph module show the last values even if they are not written to disk, or do I misunderstand something…

BUT maybe something can be done with s6 if creating a ‘down’ file in the service dir :

An optional, empty, regular file named down. If such a file exists, the default state of the service is considered down, not up: s6-supervise will not automatically start it until it receives a s6-svc -u command. If no down file exists, the default state of the service is up.

cf s6: service directories

going to test :slight_smile:

1 Like
34

Nice work @alexandrecuer What can we do to highlight this to others who would find it useful?

35

Something - maybe a cross between ‘Learn’ and ‘Showcase’?

Somewhere to park (or index) developed projects so that they don’t get buried in the forum and lost because you can’t quite remember the exact search term.

I’d say it needs to be curated so as not to get too cluttered.

36

I think this needs a bit more work before we advertise too widely.

I do think having the repo under OEM GitHub would be the right move.

Key issue for me is to solve the updating of the emoncms system. It looks like @alexandrecuer has had to make some modifications to some of the modules to work (unsurprisingly) so we really need a means for those to be accounted for at runtime.

I’ve lost track if this is still a standalone container or HA specific (which will be my fault :slight_smile:). If they are different, is there an appetite to maintain both?

37

Yes the repo includes some files to make the thing work as a home-assistant add-on, but basically it works as a standalone container, which is the primary goal. The HA goal came in a second time, as there is no conflict at all between them…

If you dont have home-assistant, you can use the images, a simple docker pull will catch and download the correct image for your machine (x86, armv7, aarch64).

To resume, all the conf is done in the docker way through env vars…

The only detail that is really specific to HA is that the first one_shot service emoncms_pre.sh tries to read a /data/options.json file in the container. If this file exists (HA case), it takes the values coming from the HA add-on configuration panel. Otherwise, it takes the classic docker env parameters, which have default values in the dockerfile.

The dockerfile defines some args, which you can modulate at build time through a matrix, as everything is done by the github actions with the help of of QEMU in order to produce the multiarch images - emoncms/.github/workflows/emoncms.yml at main · Open-Building-Management/emoncms · GitHub
Some of these args are also redefined as env at the end of the Dockerfile as they are used by the one_shot services…

All the images are here :
https://hub.docker.com/r/alexjunk/emoncms/tags

From what I know, this is very classic docker technics and is not very complex to maintain, more easy than the Emonscripts for example.

Maybe the only thing which is not classic is that the dockerfile generates the s6 init structure. Generally people maintain the folders and files related to the s6-overlay aside, and they make a COPY into the image at build time (it is what they do at HA) but as it is a lot of files, often empty, I prefer to generate all those stuff dynamically within the dockerfile.

1 Like
38

Thanks :slight_smile:. I’m not well versed in Docker, so excuse my ignorance.

If @TrystanLea releases a new version of emoncms (or any of the separate Git repos) how does that flow into the docker image? Presumably you want to pull the new one in that way rather than do an internal update of the software?

How can we align changes you may have had to make to any of the modules - e.g. the backup module?

39

Very light mods…Basically removing the use of sudo. Most systemctl instructions included in the modules shell scripts will fail, as you will not have systemctl working in docker (there is an open war betwen the 2 :slight_smile: cf s6-rc: why?), anyway it does not hurt.

BUT the admin module is not designed at all to be run in a container.

1 Like
40

The Dockerfile fetches the latest stable emoncms release, same for modules. Right now it is true it is not possible to udpate the modules (backup, sync) whith codes modified by the Dockerfile, because they have local mods and git cannot go further…

Generally containers run with the restart policy called always. So if you switch off the computer, it is not destroyed but only stopped and waiting to be restarted : this is the docker mantra. When the computer restarts, the container restarts and if you have previously updated to a specific emoncms or module version, you’ve got your new version online. If you destroy manually the container (with a docker rm command), of course, you go back to the versions programmed by the dockerfile, just like if you destroy by error a file : you just have to restore the latest backup you have…

After a migration or a modification of the HA conf, docker cannot be aware that you were previously running custom versions, and something like a factory reset happens (but datas are of course preserved)

In theory, the new version brought by the migration comes with the latest modules/versions…the case of a modification of conf parameters is different and i have to check the behaviour…

I tried to look into the problem of tagging docker images and including the emoncms version. There are possibilities with github actions to produce a build as soon as a new version comes out, that kind of thing…but first things first… for the moment, the tags are just set to the Alpine versions and I ran them manually. I also have to manage to introduce a latest tag.

1 Like
41

And by the way, dont know if there are some windows users, but all this work very well on windows using the docker-desktop application. You just have to install the windows subsystem for linux (WSL2)

1 Like
42

Ah great. Perhaps the next step then is to make modifications to the modules such that they do not need modifying for the Docker environment, but simply cope with that? @TrystanLea?

Yes absolutely. I’m sure it will be possible :slight_smile:

This really is a great step forward IMHO.

43

@borpin : just produced a new version you can test with the app module and the possibility to disable the low write mode and the feewriter. It is indeed possible, not the way I thought but it works. Need to be tested as I more familiar with the low write mode.
You can also choose the emoncms log level…

1 Like