Emoncms and emonhub on Debian VM or Container

lamixer · 22 August 2023 13:36

Hello:

My goal is to bring data from my new sensors (from emonTx4) into Home Assistant. I understand perhaps the easiest way from the HA side is to pull data in from EmonCMS, though perhaps there is a way to pull the data directly from emonTx into HA. Let me know if I should forgo EmonCMS and take that route.

I have a Proxmox host computer near the emonTX connected by USB. I have tried creating a Debian Linux Container (CT) to install EmonCMS. I used the script and the recommended changed for running on Non-Rpi systems. After installing a couple packages to help the script out, it seems to be working. I have the web interface.

However, the non-Rpi mods said to disable emonHub. But it seems I need emonHub to pull in the data over USB from the TX. I tried to run that installer script but it failed with warnings about locally-managed python and conflicts with PIP.

What should I do?

Would switching to an Ubuntu container work better? That’s easy to do. Or if containters will give me problems, I could go for a straight up virtual machine that could be either Ubuntu or Debian (latter preferred).

I could excercise the option to let PIP go ahead and possibly clobber system Python packages – is that how it’s done on the normal Rpi software image? Or should I set up a python virtual environment? I have no experince with that and I’m not sure if I would have to do a fair amount of patching the magic installer scripts to make it use this python virtual environment.

Thanks for any advice!

fruit-offbeat-dastar · 22 August 2023 19:19

I’ve just set up EmonCMS on an Ubuntu CT. Like you, I had to modify things including using apt to get various python- packages instead of pip. I also had to modify the the systemd overrides to get the feedwriter and demandshaper components to actually start.

Looking at /opt/openenergymonitor/EmonScripts/install/emonhub.sh it looks like it pulls emonhub from Github, runs the installer script and adds users to the sudoer file.

Looking at https://github.com/openenergymonitor/emonhub/blob/master/install.sh it should be possible to get it to install on Debian but I think you’ll have to figure out if the Debian packages can provide paho-mqtt, requests, pybluez, py-sds011, sdm_modbus and minimalmodbus modules. I wonder if you can get away without the custom rpi-rfm69 module if you’re going to be using USB.

fruit-offbeat-dastar · 22 August 2023 19:42

It looks like there are Debian packages for some of these so you could:

apt install python3-paho-mqtt python3-requests python3-bluez

However I couldn’t see packages for py-sds011, sdm_modbus and minimalmodbus, also the custom rpi-rfm69. It may be emonHub works ok without these modules (or you may need to hack the scripts to remove the functionality which relies on these modules as you won’t use it for emonTx v4 via USB).

Reading “Externally managed environments”: when PEP 668 breaks pip it appears a change has been made that obsoletes pip to install packages system-wide. It’s likely the openenergymonitor team will need to have a re-think about how they create solutions for this. The article suggests using pipx but I’m not sure that will work out of the box without you modifying the scripts.

borpin · 23 August 2023 07:07

Yes, Debian

Yes, the instructions do rather assume you are not going to use a local connection so emonhub is normally not used.

However, it should work - you just need to pass the USB through to the container.

I always find Ubuntu containers better. These scripts work really well to create the LXC Proxmox VE Helper-Scripts | Scripts for Streamlining Your Homelab with Proxmox VE

I’d start with a fresh Ubuntu container and use the scripts leaving emonhub in.

borpin · 23 August 2023 07:09

Interesting. Last time I did it, the scripts just worked.

Which version of Ununtu?

lamixer · 23 August 2023 13:46

Thanks Brian and Aaron for your help. I have started over with an Ubuntu 22_04 container. Main installer script does not install emonhub even if requested. Separately running the emonhub installer script mostly works, except this morsel:

Defaulting to user installation because normal site-packages is not writeable
Collecting paho-mqtt
  Downloading paho-mqtt-1.6.1.tar.gz (99 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.4/99.4 KB 2.8 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting requests
  Downloading requests-2.31.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 KB 25.7 MB/s eta 0:00:00
Collecting pybluez
  Downloading PyBluez-0.23.tar.gz (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.6/97.6 KB 11.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      error in PyBluez setup command: use_2to3 is invalid.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Defaulting to user installation because normal site-packages is not writeable

I had to manually-install python3-requests and python3-paho-mqtt. But I can’t get systemd to cooperate!

I manually created the log path and folder, and followed directions to get past the var-log.mount issue. If I execute the command myself emonhub does start running, even writes to the log file:

/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log

but systemd gives error:

Aug 23 13:35:40 emon-cms systemd[2253]: emonhub.service: Failed to determine user credentials: No such process
Aug 23 13:35:40 emon-cms systemd[2253]: emonhub.service: Failed at step USER spawning /usr/local/bin/emonhub/emonhub.py: No such process

my current systmd service file is:

[Unit]
Description=emonHub data multiplexer
# The config file lives in /etc/emonhub/emonhub.conf
# The log file lives in /var/log/emonhub/emonhub.log
#Requires=var-log.mount
#After=var-log.mount network.target
Requires=rsyslog.service
After=network.target

[Service]
Type=exec
ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
User=mysystemusername
Environment='USER=mysystemusername'
Environment='LOG_PATH=/var/log/emonhub'
PermissionsStartOnly=true
#ExecStartPre=/usr/bin/mkdir -p ${LOG_PATH}
#ExecStartPre=/usr/bin/chown ${USER} ${LOG_PATH}

Restart=always
RestartSec=5

SyslogIdentifier=emonhub

[Install]
WantedBy=multi-user.target

Thoughts? I know systemd sometimes can be a little funny on Linux Containers. Seems like there should be a solution though.

borpin · 23 August 2023 14:49

Did you create a user Pi?

lamixer · 23 August 2023 15:23

No, I made a user, one of my normal usernames I like to use, and adapted the script with that name. Do you think I’m hitting some hard-coded pi user issue?

lamixer · 23 August 2023 15:26

Oh… demandshaper is not running because wants to run as Pi. Maybe I should start over once more!

lamixer · 23 August 2023 16:19

Got demandshaper running. Even got access to /dev/ttyUSB0 for the emonTX4 in the container. Tantalizingly close to working. Still the systemd problem for emonhub.service.

fruit-offbeat-dastar · 23 August 2023 21:00

Ubuntu 23.04 (GNU/Linux 6.2.16-8-pve x86_64) on a Proxmox CT container. Created from ubuntu-23.04-standard_23.04-1_amd64.tar.zst

I’ve just gone through the whole process again. I have found two issues:

Line 25 of redis.sh attempts to run

sudo pip3 install redis

This causes this error:

extension=redis.soerror: externally-managed-environment

  This environment is externally managed
> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

That resulted in service-runner not starting with this error:

Aug 23 20:24:59 emontest systemd[1]: service-runner.service: Scheduled restart job, restart counter is at 8.
Aug 23 20:24:59 emontest systemd[1]: Stopped service-runner.service - Emoncms service-runner Input Script.
Aug 23 20:24:59 emontest systemd[1]: Started service-runner.service - Emoncms service-runner Input Script.
Aug 23 20:24:59 emontest service-runner[20745]: Traceback (most recent call last):
Aug 23 20:24:59 emontest service-runner[20745]:   File "/var/www/emoncms/scripts/services/service-runner/service-runner.py", line 13, in <module>
Aug 23 20:24:59 emontest service-runner[20745]:     import redis
Aug 23 20:24:59 emontest service-runner[20745]: ModuleNotFoundError: No module named 'redis'
Aug 23 20:24:59 emontest systemd[1]: service-runner.service: Main process exited, code=exited, status=1/FAILURE
Aug 23 20:24:59 emontest systemd[1]: service-runner.service: Failed with result 'exit-code'.

Per PEP 668 we should not be doing sudo pip3 install. There are other ways around using pip/pipx. However in this occasion, running

systemctl stop service-runner.service
apt install python3-redis
systemctl start service-runner.service

Fixed this issue.

The user specified in config.ini is not applied to the demandshaper module resulting in this error

Aug 23 20:25:04 emontest systemd[1]: demandshaper.service: Scheduled restart job, restart counter is at 4.
Aug 23 20:25:04 emontest systemd[1]: Stopped demandshaper.service - Emoncms Demandshaper service.
Aug 23 20:25:04 emontest systemd[1]: Started demandshaper.service - Emoncms Demandshaper service.
Aug 23 20:25:04 emontest systemd[20746]: demandshaper.service: Failed to determine user credentials: No such process
Aug 23 20:25:04 emontest systemd[20746]: demandshaper.service: Failed at step USER spawning /usr/bin/php: No such process
Aug 23 20:25:04 emontest systemd[1]: demandshaper.service: Main process exited, code=exited, status=217/USER
Aug 23 20:25:04 emontest systemd[1]: demandshaper.service: Failed with result 'exit-code'.

It appears lines 93-108 of emoncms_core.sh fixes this for some other services:

    # Install service-runner drop-in if system user is different
    if [ "$user" != "pi" ]; then
        echo "installing service-runner drop-in User=$user"
        if [ ! -d /lib/systemd/system/service-runner.service.d ]; then
            sudo mkdir /lib/systemd/system/service-runner.service.d
        fi
        echo $'[Service]\nUser='$user > service-runner.conf
        sudo mv service-runner.conf /lib/systemd/system/service-runner.service.d/service-runner.conf

        echo "installing feedwriter drop-in User=$user"
        if [ ! -d /lib/systemd/system/feedwriter.service.d ]; then
            sudo mkdir /lib/systemd/system/feedwriter.service.d
        fi
        echo $'[Service]\nEnvironment="USER='$user'"' > feedwriter.conf
        sudo mv feedwriter.conf /lib/systemd/system/feedwriter.service.d/feedwriter.conf
    fi

However it appears the ‘correct’ way to do this is to place an override in /etc/systemd/system/servicename.service.d/override.conf rather than in /lib/systemd

I ran

mkdir /etc/systemd/system/demandshaper.service.d/
touch /etc/systemd/system/demandshaper.service.d/override.conf
systemctl edit demandshaper.service

And added

[Service]
User=emoncms

Between the correct lines and then restarted the service to get it to work.

I can also see /usr/bin/php /var/www/emoncms/scripts/services/emoncms_mqtt/emoncms_mqtt.php is running as root which probably isn’t ideal from a security standpoint! However I’ve not looked into this yet.

borpin · 24 August 2023 06:19

The Ubuntu instructions say ‘create a user Pi’. RTFM.

borpin · 24 August 2023 06:48

I banged on long and hard about this and thought we had eliminated all of the sudo pip commands in the scripts

Yes we did. @TrystanLea this is needed for DemandShaper as well.

Man page - systemd.unit

You can argue long and hard about which location is technically correct but the one in use isn’t wrong.

Looking at UFS Unix filesystem - Wikipedia I’m not sure why we used /lib as opposed to /usr/lib. Perhaps something has changed. I note /lib in symlinked to /usr/lib in Ubuntu.

Interestingly, Ubuntu doesn’t seem to have a /usr/local/lib/systemd folder by default.

Mainly, ain’t broke (other than Demandshaper override), don’t fix

You don’t need the first 2 commands as the final command creates the folder & file if it isn’t there (you need to be root or sudo to run it).

This is of course a specific user name to you (for anyone following on).

BTW, I have noticed in the past Demandshaper is a bit of a resource hog, so if you don’t use it, don’t install it (I don’t hence I did not find this issue).

lamixer · 24 August 2023 07:45

Hi Brian. Sorry if I have wasted your time.

I appreciate that you (and perhaps others) have put much time and effort into this script which does a lot of work to make this easy for lower-skilled linux users such as myself. Thank you. Perhaps we can help avoid this coming up again with a little discussion about the instructions.

I referred to instructions here: https://docs.openenergymonitor.org/emonsd/install.html and then just now double-checked the instructions here https://github.com/openenergymonitor/EmonScripts/blob/master/docs/install.md and unless my eyes deceive me (which does happen!) I don’t see an instruction to create a user Pi on the Ubuntu system.

Under the section “Building Your Own - Base OS Preparation” there are three sections, Pi, Ubuntu, and DO. The Rpi instructions say to create user pi. the DO one says to use OEM. The Ubuntu one says (implies?) you can use your own user when you modify the config.ini:

user=[YOUR_UBUNTU_USERNAME]
emonSD_pi_env=0

Since I was not going to install on a Raspberri Pi, I did not look at that page and just followed the instructions for Ubuntu, picked a username and put that in the config.ini before running the script.

And actually … for the main emoncms part, I think the script worked fine with a different chosen username.

Mostly I am running into trouble when trying to get the emonhub component working, maybe that requires a user pi. If there are instructions for the emonhub.sh installer script I have missed those. Or if I have missed some other manual I should have read, then I apologise.

I’m going to try again with the Ubuntu CT with user pi shortly and will report back.

Bill.Thomson · 24 August 2023 08:12

Recommendation is to not change the username.

More info in this thread:

borpin · 24 August 2023 10:31

Ah, yes. Sorry. I think that might have been the old advice, but the changes we made meant it should have worked (but TBH I’d not tested it with emonhub or Demandshaper).

I’ll see if I can find the time to fire up an LXC and test the other bits.

borpin · 24 August 2023 10:33

That thread is specifically about the pre-built image I think. That might be worth adding to the docs, though.

fruit-offbeat-dastar · 24 August 2023 11:07

This isn’t mentioned here EmonCMS Install Scripts — OpenEnergyMonitor 0.0.1 documentation so did I RTWM?

fruit-offbeat-dastar · 24 August 2023 11:12

I found I received an error when saving the file after using systemctl edit if the folder didn’t exist first (tbh I didn’t test creating just the folder, I just solved it by making the folder and file).

To my mind if the native tools (eg systemctl edit) don’t work to change configurable settings (eg system-specific variables) then it is broken? But I accept I haven’t looked into how other distros are using systemd. I know by convention that configuration information usually lives in /etc

From the manual you linked:

There are two methods of overriding vendor settings in unit files: copying the unit file from/usr/lib/systemd/system to /etc/systemd/system and modifying the chosen settings. Alternatively, one can create a directory named unit.d/ within /etc/systemd/systemand place a drop-in file name.conf there that only changes the specific settings one is interested in. Note that multiple such drop-in files are read if present, processed in lexicographic order of their filename.

The advantage of the first method is that one easily overrides the complete unit, the vendor unit is not parsed at all anymore. It has the disadvantage that improvements to the unit file by the vendor are not automatically incorporated on updates.

The advantage of the second method is that one only overrides the settings one specifically wants, where updates to the unit by the vendor automatically apply. This has the disadvantage that some future updates by the vendor might be incompatible with the local changes.

The idea is that if you ever modify the unit file in a future update any changes I’ve made aren’t nuked

I don’t need it but I didn’t see disabling it from installing in the config.ini but admittedly I didn’t look too hard! Is there a sensible way to remove it from an installation?

lamixer · 24 August 2023 11:53

Hello again.

I have had better luck with making user pi and installing from that account.

Again, basic emoncms install was error-free. I left the line install_emonhub=true just hoping that even if it does not get installed, the entries for it to appear in emoncms would be ready.

Installing emonhub using the script these errors occurred:

Defaulting to user installation because normal site-packages is not writeable
Collecting paho-mqtt
  Downloading paho-mqtt-1.6.1.tar.gz (99 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.4/99.4 KB 2.7 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting requests
  Downloading requests-2.31.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 KB 27.6 MB/s eta 0:00:00
Collecting pybluez
  Downloading PyBluez-0.23.tar.gz (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.6/97.6 KB 16.2 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      error in PyBluez setup command: use_2to3 is invalid.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Defaulting to user installation because normal site-packages is not writeable

I “solved” it (maybe) by apt install python3-requests python3-paho-mqtt

Then the systemd issue is solved by changing these:

Requires=rsyslog.service
After=network.target

plus the obligatory sudo systemctl daemon-reload.

Then emonhub will start! And, after adding user pi to the group dialout on the guest CT, and some fiddling with /dev/ttyUSB0 on Proxmox host ( chown 100000:100020 /dev/ttyUSB0 ), and modifying the CT config file on the Proxmox host to end with

lxc.cgroup2.devices.allow: c 188:00 rwm
lxc.mount.entry: /dev/ttyUSB0 dev/ttyUSB0 none bind,optional,create=file

and restarting it of course. YMMV, but this is working for my set up.

Notably, the entry for emonhub does NOT appear in emoncms webconfig settings. I’m not sure if that is expected or not. So I modified the emonhub config file commenting out things I don’t use and adding my emonTx4 with USB connection as directed:

[interfacers]
### Manually added for the emonTx4 unit
[[USB]]
    Type = EmonHubOEMInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyUSB0
        com_baud = 115200
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        nodename = emonTx4

when I tail the emonhub.log I see what looks like correct functionality:

2023-08-24 12:44:37,155 DEBUG    USB        Report 248  AC present V1 = 240.61 V2 = 0.00 V3 = 0.00 f=50.02
2023-08-24 12:44:37,357 DEBUG    USB        Ch 1 I=0.048 W=0 VA=11 Wh=0 pf=0.0087
2023-08-24 12:44:37,458 DEBUG    USB        Ch 2 I=0.045 W=0 VA=11 Wh=0 pf=-0.0658
2023-08-24 12:44:37,559 DEBUG    USB        Ch 3 I=0.023 W=0 VA=5 Wh=0 pf=-0.0418
2023-08-24 12:44:37,660 DEBUG    USB        Ch 4 I=0.022 W=0 VA=5 Wh=0 pf=-0.0647
2023-08-24 12:44:37,762 DEBUG    USB        Ch 5 I=0.022 W=0 VA=5 Wh=0 pf=-0.0074
2023-08-24 12:44:37,863 DEBUG    USB        Ch 6 I=0.009 W=0 VA=2 Wh=0 pf=-0.0083
2023-08-24 12:44:37,964 DEBUG    USB        Ch 7 I=0.009 W=0 VA=2 Wh=0 pf=-0.0757
2023-08-24 12:44:38,065 DEBUG    USB        Ch 8 I=0.009 W=0 VA=2 Wh=0 pf=-0.0303
2023-08-24 12:44:38,166 DEBUG    USB        Ch 9 I=0.010 W=0 VA=2 Wh=0 pf=0.0056
2023-08-24 12:44:38,268 DEBUG    USB        Ch 10 I=0.008 W=0 VA=2 Wh=0 pf=-0.0107
2023-08-24 12:44:38,369 DEBUG    USB        Ch 11 I=0.009 W=0 VA=2 Wh=0 pf=-0.0496
2023-08-24 12:44:38,471 DEBUG    USB        Ch 12 I=0.009 W=0 VA=2 Wh=0 pf=-0.0670
2023-08-24 12:44:46,312 DEBUG    USB        RF No Ack (1)
2023-08-24 12:44:46,815 DEBUG    USB        RF No Ack (2)
2023-08-24 12:44:46,917 DEBUG    USB        Report 249  AC present V1 = 240.75 V2 = 0.00 V3 = 0.00 f=49.91
2023-08-24 12:44:47,118 DEBUG    USB        Ch 1 I=0.047 W=0 VA=11 Wh=0 pf=-0.0042
2023-08-24 12:44:47,219 DEBUG    USB        Ch 2 I=0.045 W=0 VA=11 Wh=0 pf=-0.0708
2023-08-24 12:44:47,321 DEBUG    USB        Ch 3 I=0.022 W=0 VA=5 Wh=0 pf=-0.0344
2023-08-24 12:44:47,422 DEBUG    USB        Ch 4 I=0.022 W=0 VA=5 Wh=0 pf=-0.0690
2023-08-24 12:44:47,523 DEBUG    USB        Ch 5 I=0.022 W=0 VA=5 Wh=0 pf=0.0028
2023-08-24 12:44:47,624 DEBUG    USB        Ch 6 I=0.009 W=0 VA=2 Wh=0 pf=-0.0306
2023-08-24 12:44:47,725 DEBUG    USB        Ch 7 I=0.009 W=0 VA=2 Wh=0 pf=-0.0747
2023-08-24 12:44:47,827 DEBUG    USB        Ch 8 I=0.009 W=0 VA=2 Wh=0 pf=-0.0113
2023-08-24 12:44:47,928 DEBUG    USB        Ch 9 I=0.009 W=0 VA=2 Wh=0 pf=-0.0614
2023-08-24 12:44:48,029 DEBUG    USB        Ch 10 I=0.009 W=0 VA=2 Wh=0 pf=0.0096
2023-08-24 12:44:48,130 DEBUG    USB        Ch 11 I=0.008 W=0 VA=2 Wh=0 pf=-0.0672
2023-08-24 12:44:48,231 DEBUG    USB        Ch 12 I=0.009 W=0 VA=2 Wh=0 pf=-0.0681

(I haven’t connected the CT clamps to anything yet)

The docs say that once this is done I will see the sensors from the emonTx4 automatically populate in the INPUTS area of emoncms webconfig – but I don’t!

Maybe I should take that to a new thread. I did here.

When I look in emoncms ‘System Info’ I see the following services are green and happy:

emonhub
emoncms_mqtt
feedwriter
service-runner
redis-server
mosquitto
demandshaper

and emonPiLCD “not found or not installed”

Please let me know if you have any suggestions and thanks for bearing with me.