Emoncms self contained docker container

Hi,

Dont know if this was done before but I just arrange a dockerfile that permits to have
the complete stack (mariadb, redis, apache, php, mosquitto) and the main modules (sync, postprocess, backup) in a single docker container. So it is a kind of virtual twin for the physical emonpi…

for an ephemeral test :

docker pull alexjunk/emoncms:0.0.2
docker run --rm -it alexjunk:emoncms:0.0.2

To get the container IP address : docker network inspect bridge

To send mqtt datas from the host if 172.17.0.2 is the container IP : mosquitto_pub -h 172.17.0.2 -u "emonpi" -P "emonpimqtt2016" -t 'emon/test/t3' -m 15

if you want persistant sql and timeseries datas :

docker volume create mariadb_datas
docker volume create ts_datas
docker run --rm -v mariadb_datas:/var/lib/mysql -v ts_datas:/var/opt/emoncms -it alexjunk/emoncms:0.0.2

here is the dockerfile

best :slight_smile:

5 Likes

Nice! thanks for sharing this @alexandrecuer! can see that being useful

2 Likes

Just another version with backup import working correctly and running supervisord as the current user and not as root. Anyway the container start by launching the main services with sudo : mariadb, apache, mosquitto-server and redis-server…

2 builds produced : amd64 and arm64v8 (jetson nano)

docker pull alexjunk/emoncms:0.0.3
docker pull alexjunk/emoncms:0.0.3.arm64v8

no idea if this kind of image with multi services could be used in production. Anyway, it is very easy to deploy…

It is possible to use a client container to run a client application using emoncms databases or sending datas to it. You just have to share the same network namespace :

docker run --rm --name=emoncms -it alexjunk/emoncms:0.0.3

the Dockerfile for a python client could be something like that :

FROM ubuntu:20.04

RUN apt-get update && apt-get upgrade -y; \
    apt-get install -y --no-install-recommends --yes net-tools \
    nano redis-tools mariadb-client \
    python3.8 python3-pip; \
    python3 -m pip install pip --upgrade; \
    python3 -m pip install mysql-connector-python redis \
    pyserial \
    paho-mqtt \
    pymodbus \
    requests

build

docker build -t emoncms_client .

use

docker run --rm --network=container:emoncms -it emoncms_client

basic python script to add to the client to check that connectivity is OK :

1 Like

This is potentially really useful. More so than each part running in separate containers. Not for me though, as I’m a PVE user not Docker.

1 Like

yes it could be useful as a nice direct first approach for new users…

to run on a local network, use docker run -p :

docker run --rm -it -p 8080:80 alexjunk/emoncms:0.0.3.arm64v8
 * Starting MariaDB database server mysqld                                 [ OK ] 
 * Starting Apache httpd web server apache2                                        * 
 * Starting network daemon: mosquitto                                      [ OK ] 
Starting redis-server: redis-server.
2023-06-09 11:59:58,046 INFO RPC interface 'supervisor' initialized
2023-06-09 11:59:58,049 INFO supervisord started with pid 381
2023-06-09 11:59:59,059 INFO spawned: 'emoncms_mqtt' with pid 386
2023-06-09 11:59:59,075 INFO spawned: 'service-runner' with pid 387
2023-06-09 11:59:59,090 INFO spawned: 'feedwriter' with pid 388
2023-06-09 12:00:00,099 INFO success: emoncms_mqtt entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-06-09 12:00:00,100 INFO success: service-runner entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-06-09 12:00:00,101 INFO success: feedwriter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

if 192.168.1.13 is the host IP, then emoncms can be reached via http://192.168.1.13:8080

used on a pi3 with the latest ubuntu server for raspberry installed : Install Ubuntu on a Raspberry Pi | Ubuntu

4 Likes

Here is a link to an application server giving the ability to launch emoncms containers :slight_smile:

Easy to test new version of modules…

I think this single container is the way to go for the HomeAssistant add-on. The exception I’d suggest is the MQTT Broker as very often, those using HA have a separate Broker running.

I’m sure there was a discussion about this but I can’t seem to find it. @glyn.hudson did an initial add-on IIRC.

Just discovering HA, add-ons seem to be nothing more than containers…

HA uses the s6 overlay to run multiple services in the same container, i think it is a cleaner approach than supervisord

1 Like

@TrystanLea @borpin : I did some more work on the container approach.
I’ve explored the s6-overlay to run multiple services in the same container, which seems inspiring.

I went for multiarch build with alpine 3.16 and it is working quite well. Alpine is a bit different so you cannot use the debian tools like phpenmod, a2enmod, a2ensite. Moreover there is no www-data user only an apache one, and I decided to run all the workers (feedwriter, service-runner, emoncms_mqtt) with the apache user, dont know what you think of that ?

After everything is a matter of rights but finally the Dockerfile is quite simple, aside it written in my own style :wink:

I have to spend more time on the data persistence (none for the moment), which can be solved (I think) with a oneshot s6 service and will also be the occasion to solve the problem of the database initialisation, that I treated in a rather dirty way

so if you want to test :

docker pull alexjunk/emoncms:alpine3.16

it will work with aarch64/amd64/armv7

I’ve tested with homeassistant on a pi3, I could managed to make it work as an addon, with a very simple config.yml file, giving an image name so that the home-assistant fetches on docker and does not build locally :

name: emoncms
description: The webapp for energy monitoring and more
version: alpine3.16
image: alexjunk/emoncms
slug: emoncms
init: false
arch:
  - armv7
  - aarch64
  - amd64
ports:
  80/tcp: 8002
  1883/tcp: 9883

I’ve crafted a small repo with some Dockerfiles and github actions configured to produced the multiarch images…

1 Like

That sounds really promising. Being able to run as an Add-On to HA would be magic.

Not sure what you mean by this :slight_smile:

1 Like

Actually the docker solution to achieve data persistence is to use volumes.

So if you create volumes it is OK, one for mariadb, one for the timeseries datadirs…volumes are initially fed from the container file tree : If you start a container which creates a new volume, and the container has files or directories in the directory to be mounted such as /app/, Docker copies the directory’s contents into the volume.

But in home-assistant add-ons, there is no support for volume from what I’ve understood…

So I’ve read how they proceed for the mariadb addon :

They use the data folder :

and :

and also :

Yes, thought they might.

As I say, a standalone addon would be magic! With the ability to automate the backup, it would become a very attractive option. Potentially, I’d consider getting rid of my Proxmox setup.

There really isn’t any other option for collecting longer term data in HA, other than Grafana really which is a bit of a pest to then use (in comparison to emoncms IMHO - other opinions are available), so it could become quite popular (@TrystanLea @glyn.hudson).

2 Likes

You mean automating backup with docker tools and not use the backup module, that’s it ?

Actually, I could make work the import script of the module without major changes, just removing all the trailing sudo, but export script is not yet working

No, with HA tools either an automation or another addon such as Google Drive Backup (which I use).

It should be working fine now.

To install, add the https://github.com/Open-Building-Management/emoncms to the repositories of the add-on store of Home-Assistant :

then install :

@TrystanLea : I had to modify some of the modules scripts, in order they work on alpine, very light mods, mostly removing sudo :
cf https://github.com/Open-Building-Management/emoncms/blob/5c8fa84c10eab4d3aa40782310f1e7faed7f85a2/Dockerfile#L132C2-L138C71

3 Likes

That looks great! thanks @alexandrecuer!

@alexandrecuer - taken the plunge and installed it.

First thing is MQTT. Whilst strictly for a ‘standalone’ system it is required, in reality, I want to be able to point emoncms at a different broker.

Can the MQTT IP Address be configurable?

Any thoughts on exposing the settings file? For instance Log Level?

[edit]
Configure for SSL - I run my HA on SSL locally :frowning:

Yes it possible I think
We have to add first an option MQTT_HOST in the https://github.com/Open-Building-Management/emoncms/blob/main/config.yaml
Then it requires to modify in consequence emoncms_pre.sh, when it opens /data/options.json (HA creates this file to gather all the options of the add-on)
https://github.com/Open-Building-Management/emoncms/blob/5c8fa84c10eab4d3aa40782310f1e7faed7f85a2/emoncms_pre.sh#L28
and add a line to inject host = '$MQTT_HOST' in the settings.ini generated at runtime

To modify log level is possible also, same manner

For SSL, I did it once adding crt_file, key_file, crtchain in the container at buildtime, but I guess it must be different approach in HA…must understand first how they do…

1 Like

@alexandrecuer - where are the data files created?

It is in the /data folder of the host. From what I’ve understood, you don’t have access to the whole system when you connect to HA in ssh. As they say, SSH access through the SSH add-on (which will give you SSH access through port 22) will not provide you with all the necessary privileges

Debugging the Home Assistant Operating System | Home Assistant Developer Docs

Haven’t tried all this for the moment…

1 Like