Emoncms self contained docker container and HomeAssistant Addon

Hi,

I have a pipeline producing docker images with the complete stack onboard : mariadb, redis, apache, php, mosquitto and the main modules (sync, postprocess, backup)

So it is a kind of virtual twin for the physical emonpi, but without emonhub…

for an ephemeral test :

docker pull alexjunk/emoncms
docker run --rm -it alexjunk/emoncms

To get the container IP address : docker network inspect bridge

To send mqtt datas from the host if 172.17.0.2 is the container IP : mosquitto_pub -h 172.17.0.2 -u "emonpi" -P "emonpimqtt2016" -t 'emon/test/t3' -m 15

To run on a local network, use docker run -p :

docker run --rm -it -p 8080:80 alexjunk/emoncms

if 192.168.1.13 is the host IP, then emoncms can be reached via http://192.168.1.13:8080

The previous commands use the latest tag but you can find various images on the HUB

It is easy to use a client container to run a client app using emoncms databases or sending datas to it.

You just have to share the same network namespace :

docker run --rm --name=emoncms -it alexjunk/emoncms

the Dockerfile for a python client could be something like that :

ARG BUILD_FROM=alpine:3.16

FROM $BUILD_FROM

RUN set -x;\
    apk update && apk upgrade;\
    apk add --no-cache python3 py3-pip;\
    python3 -m pip install --no-cache-dir mysql-connector-python redis paho-mqtt==1.6.1

RUN set -x;\
    wget https://raw.githubusercontent.com/dromotherm/sandbox/master/emoncms/client_dockerfile/test.py

build :

docker build -t emoncms_client .

use :

docker run --rm --network=container:emoncms -it emoncms_client

Once connected :

python3 test.py

More on https://docker-emoncms.github.io

[Revised on request of @alexandrecuer – Moderator (RW)]

5 Likes

Nice! thanks for sharing this @alexandrecuer! can see that being useful

2 Likes

This is potentially really useful. More so than each part running in separate containers. Not for me though, as I’m a PVE user not Docker.

1 Like

I think this single container is the way to go for the HomeAssistant add-on. The exception I’d suggest is the MQTT Broker as very often, those using HA have a separate Broker running.

I’m sure there was a discussion about this but I can’t seem to find it. @glyn.hudson did an initial add-on IIRC.

Just discovering HA, add-ons seem to be nothing more than containers…

HA uses the s6 overlay to run multiple services in the same container, i think it is a cleaner approach than supervisord

1 Like

@TrystanLea @borpin : I did some more work on the container approach.
I’ve explored the s6-overlay to run multiple services in the same container, which seems inspiring.

I went for multiarch build with alpine 3.16 and it is working quite well. Alpine is a bit different so you cannot use the debian tools like phpenmod, a2enmod, a2ensite. Moreover there is no www-data user only an apache one, and I decided to run all the workers (feedwriter, service-runner, emoncms_mqtt) with the apache user, dont know what you think of that ?

After everything is a matter of rights but finally the Dockerfile is quite simple, aside it written in my own style :wink:

I have to spend more time on the data persistence (none for the moment), which can be solved (I think) with a oneshot s6 service and will also be the occasion to solve the problem of the database initialisation, that I treated in a rather dirty way

so if you want to test :

docker pull alexjunk/emoncms:alpine3.16

it will work with aarch64/amd64/armv7

I’ve tested with homeassistant on a pi3, I could managed to make it work as an addon, with a very simple config.yml file, giving an image name so that the home-assistant fetches on docker and does not build locally :

name: emoncms
description: The webapp for energy monitoring and more
version: alpine3.16
image: alexjunk/emoncms
slug: emoncms
init: false
arch:
  - armv7
  - aarch64
  - amd64
ports:
  80/tcp: 8002
  1883/tcp: 9883

I’ve crafted a small repo with some Dockerfiles and github actions configured to produced the multiarch images…

1 Like

That sounds really promising. Being able to run as an Add-On to HA would be magic.

Not sure what you mean by this :slight_smile:

1 Like

Actually the docker solution to achieve data persistence is to use volumes.

So if you create volumes it is OK, one for mariadb, one for the timeseries datadirs…volumes are initially fed from the container file tree : If you start a container which creates a new volume, and the container has files or directories in the directory to be mounted such as /app/, Docker copies the directory’s contents into the volume.

But in home-assistant add-ons, there is no support for volume from what I’ve understood…

So I’ve read how they proceed for the mariadb addon :

They use the data folder :

and :

and also :

Yes, thought they might.

As I say, a standalone addon would be magic! With the ability to automate the backup, it would become a very attractive option. Potentially, I’d consider getting rid of my Proxmox setup.

There really isn’t any other option for collecting longer term data in HA, other than Grafana really which is a bit of a pest to then use (in comparison to emoncms IMHO - other opinions are available), so it could become quite popular (@TrystanLea @glyn.hudson).

2 Likes

You mean automating backup with docker tools and not use the backup module, that’s it ?

Actually, I could make work the import script of the module without major changes, just removing all the trailing sudo, but export script is not yet working

No, with HA tools either an automation or another addon such as Google Drive Backup (which I use).

It should be working fine now.

To install, add the https://github.com/Open-Building-Management/emoncms to the repositories of the add-on store of Home-Assistant :

then install :

@TrystanLea : I had to modify some of the modules scripts, in order they work on alpine, very light mods, mostly removing sudo :
cf https://github.com/Open-Building-Management/emoncms/blob/5c8fa84c10eab4d3aa40782310f1e7faed7f85a2/Dockerfile#L132C2-L138C71

3 Likes

That looks great! thanks @alexandrecuer!

@alexandrecuer - taken the plunge and installed it.

First thing is MQTT. Whilst strictly for a ‘standalone’ system it is required, in reality, I want to be able to point emoncms at a different broker.

Can the MQTT IP Address be configurable?

Any thoughts on exposing the settings file? For instance Log Level?

[edit]
Configure for SSL - I run my HA on SSL locally :frowning:

Yes it possible I think
We have to add first an option MQTT_HOST in the emoncms/config.yaml at main · Open-Building-Management/emoncms · GitHub
Then it requires to modify in consequence emoncms_pre.sh, when it opens /data/options.json (HA creates this file to gather all the options of the add-on)
emoncms/emoncms_pre.sh at 5c8fa84c10eab4d3aa40782310f1e7faed7f85a2 · Open-Building-Management/emoncms · GitHub
and add a line to inject host = '$MQTT_HOST' in the settings.ini generated at runtime

To modify log level is possible also, same manner

For SSL, I did it once adding crt_file, key_file, crtchain in the container at buildtime, but I guess it must be different approach in HA…must understand first how they do…

1 Like

@alexandrecuer - where are the data files created?

It is in the /data folder of the host. From what I’ve understood, you don’t have access to the whole system when you connect to HA in ssh. As they say, SSH access through the SSH add-on (which will give you SSH access through port 22) will not provide you with all the necessary privileges

Debugging the Home Assistant Operating System | Home Assistant Developer Docs

Haven’t tried all this for the moment…

1 Like

@borpin : I was actually wrong. You dont need the extra priviledges to access to datas. To what i’ve understood, the extra priviledges are required to access to Home Assistant Operating System
And data files are in the supervisor container.
So initiate an ssh access through port 22 with the ssh add-on, list the containers and enter the supervisor one…

docker container ls            
CONTAINER ID   IMAGE                                                        COMMAND               CREATED        STATUS                 PORTS                                       NAMES
7267832b8d69   local/armv7-addon-hello_world_1:1.3.1                        "/init /run.sh"       3 hours ago    Up 3 hours             0.0.0.0:8001->8000/tcp, :::8001->8000/tcp   addon_local_hello_world_1
c301f8809379   ghcr.io/hassio-addons/grafana/armv7:9.0.3                    "/init"               3 hours ago    Up 3 hours (healthy)                                               addon_a0d7b954_grafana
40f85009a0f3   ghcr.io/hassio-addons/ssh/armv7:15.0.7                       "/init"               3 hours ago    Up 3 hours                                                         addon_a0d7b954_ssh
bf4adf5360a0   ghcr.io/home-assistant/armv7-hassio-supervisor:latest        "/init"               3 hours ago    Up 3 hours                                                         hassio_supervisor
41bfeb70a868   ghcr.io/home-assistant/armv7-hassio-multicast:2023.06.2      "/init"               3 hours ago    Up 3 hours                                                         hassio_multicast
84593aac705f   ghcr.io/home-assistant/armv7-hassio-audio:2023.06.0          "/init"               3 hours ago    Up 3 hours                                                         hassio_audio
2a01713b73e2   ghcr.io/home-assistant/armv7-hassio-dns:2023.06.2            "/init"               3 hours ago    Up 3 hours                                                         hassio_dns
82f2aed0d080   ghcr.io/home-assistant/armv7-hassio-cli:2023.06.0            "/init"               3 hours ago    Up 3 hours                                                         hassio_cli
d2f4548123e4   ghcr.io/home-assistant/raspberrypi3-homeassistant:2023.8.2   "/init"               5 weeks ago    Up 3 hours                                                         homeassistant
870a2b79ec25   ghcr.io/home-assistant/armv7-hassio-observer:2023.06.0       "/usr/bin/observer"   2 months ago   Up 3 hours             0.0.0.0:4357->80/tcp, :::4357->80/tcp       hassio_observer

the supervisor is container bf4adf5360a0

docker container exec -it bf4adf5360a0 bash            
bash-5.1# ls
bin                    etc                    media                  proc                   srv                    tmp
command                home                   mnt                    root                   supervisor.sha256      usr
data                   init                   opt                    run                    supervisor.sha256.sig  var
dev                    lib                    package                sbin                   sys
bash-5.1# cd data
bash-5.1# ls
addons              audio.json          dns                 homeassistant.json  multicast.json      ssl
addons.json         backup              dns.json            ingress.json        observer.json       store.json
apparmor            cli.json            emergency           media               services.json       tmp
audio               config.json         homeassistant       mounts              share               updater.json
bash-5.1# cd addons
bash-5.1# ls
core   data   git    local

you will find emoncms datas in data folder
git folder contains the git sources for each add-on…

1 Like

I just produced a new version including the conf of mqtt log level plus container timezone…also added the configuration of mqtt host but not tested. @borpin, can yu test it please ?

1 Like

I’ll look later this week :slight_smile: