Emonhub will not start on Ubuntu install

stuart · 12 September 2023 15:38

Just for anyone following the above - its emonhub, not emomhub

Caught me out for a few moments

stuart · 14 September 2023 15:11

Hi all, I’m still struggling to get emonhub installed correctly.

I’ve used the automated installation scripts on Ubuntu 23.04.

They have run okay and I’ve installed as a usernamed “emoncms”

Below is the service definition, but I’m just getting a failed start on the service status=217/USER

If I run /usr/local/bin/emonhub/emonhub.py from the command line, it fires up and appears to work.

Help!

* emonhub.service - emonHub data multiplexer
     Loaded: loaded (/etc/systemd/system/emonhub.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/emonhub.service.d
             `-emonhub.conf
     Active: activating (auto-restart) (Result: exit-code) since Thu 2023-09-14 15:10:35 UTC; 593ms ago
    Process: 21446 ExecStartPre=/bin/mkdir -p ${LOG_PATH} (code=exited, status=217/USER)
        CPU: 1ms

This is the emonhub.service file…

[Unit]
Description=emonHub data multiplexer
# The config file lives in /etc/emonhub/emonhub.conf
# The log file lives in /var/log/emonhub/emonhub.log
Requires=-.mount
After=-.mount network.target

[Service]
Type=exec
ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
User=emoncms
Environment='USER=emoncms'
Environment='LOG_PATH=/var/log/emonhub'
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p ${LOG_PATH}
ExecStartPre=/bin/chown ${USER} ${LOG_PATH}

Restart=always
RestartSec=5

SyslogIdentifier=emonhub

[Install]
WantedBy=multi-user.target

Robert.Wall · 14 September 2023 16:02

Not that I’m an expert, but I believe emonCMS requires the user to be pi. It’s purely a guess that emonHub is going to be the same. (IIRC, it’s hard-coded somewhere )

borpin · 14 September 2023 16:16

That is asking for trouble

This isn’t correct, is it?

stuart:

User=emoncms
Environment='USER=emoncms'
Environment='LOG_PATH=/var/log/emonhub'
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p ${LOG_PATH}
ExecStartPre=/bin/chown ${USER} ${LOG_PATH}

Does the user emoncms have permissions on that folder to do the mkdir?

stuart · 14 September 2023 16:51

Yes, the var-log systemd unit doesn’t exist. Is this created by the script or an older version ofthe OS?

Permission is fine on the log folder.

If I run emonhib from command line as the emoncms user it works, just not as a service

borpin · 15 September 2023 06:15

~~But that directive is not correct syntax. Did you change it? It should be~~

Requires=var-log.mount
After=var-log.mount network.target

~~If you want to remove it then just delete/comment out the line.~~

mmm. Interesting.

It must be that the var-log mount point is something that happenes because we use log2ram.

On an emonpi

pi@emonpi:~ $ systemctl list-units --type=mount
  UNIT                          LOAD   ACTIVE SUB     DESCRIPTION
  -.mount                       loaded active mounted Root Mount
  boot.mount                    loaded active mounted /boot
  dev-mqueue.mount              loaded active mounted POSIX Message Queue File >
  run-rpc_pipefs.mount          loaded active mounted RPC Pipe File System
  run-user-1000.mount           loaded active mounted /run/user/1000
  sys-fs-fuse-connections.mount loaded active mounted FUSE Control File System
  sys-kernel-config.mount       loaded active mounted Kernel Configuration File>
  sys-kernel-debug.mount        loaded active mounted Kernel Debug File System
  sys-kernel-tracing.mount      loaded active mounted Kernel Trace File System
  tmp.mount                     loaded active mounted /tmp
  var-lib-php-sessions.mount    loaded active mounted /var/lib/php/sessions
  var-log.bak.mount             loaded active mounted /var/log.bak
  var-log.mount                 loaded active mounted var-log.mount
  var-opt-emoncms.mount         loaded active mounted /var/opt/emoncms
  var-tmp.mount                 loaded active mounted /var/tmp

On an ubuntu system

root@emoncms:~# systemctl list-units --type=mount
  UNIT                                LOAD   ACTIVE SUB     DESCRIPTION
  -.mount                             loaded active mounted Root Mount
  dev-.lxc-proc.mount                 loaded active mounted /dev/.lxc/proc
  dev-.lxc-sys.mount                  loaded active mounted /dev/.lxc/sys
  dev-full.mount                      loaded active mounted /dev/full
  dev-mqueue.mount                    loaded active mounted POSIX Message Queue File System
  dev-null.mount                      loaded active mounted /dev/null
  dev-ptmx.mount                      loaded active mounted /dev/ptmx
  dev-random.mount                    loaded active mounted /dev/random
  dev-tty.mount                       loaded active mounted /dev/tty
  dev-tty1.mount                      loaded active mounted /dev/tty1
  dev-tty2.mount                      loaded active mounted /dev/tty2
  dev-urandom.mount                   loaded active mounted /dev/urandom
  dev-zero.mount                      loaded active mounted /dev/zero
  proc-cpuinfo.mount                  loaded active mounted /proc/cpuinfo
  proc-diskstats.mount                loaded active mounted /proc/diskstats
  proc-loadavg.mount                  loaded active mounted /proc/loadavg
  proc-meminfo.mount                  loaded active mounted /proc/meminfo
  proc-stat.mount                     loaded active mounted /proc/stat
  proc-swaps.mount                    loaded active mounted /proc/swaps
  proc-sys-fs-binfmt_misc.mount       loaded active mounted Arbitrary Executable File Formats File System
  proc-sys-net.mount                  loaded active mounted /proc/sys/net
  proc-sysrq\x2dtrigger.mount         loaded active mounted /proc/sysrq-trigger
  proc-uptime.mount                   loaded active mounted /proc/uptime
  run-user-0.mount                    loaded active mounted /run/user/0
  sys-devices-system-cpu-online.mount loaded active mounted /sys/devices/system/cpu/online
  sys-devices-virtual-net.mount       loaded active mounted /sys/devices/virtual/net
  sys-fs-fuse-connections.mount       loaded active mounted FUSE Control File System

I’ve never used emonhub on an Ubuntu based system!

Can you try restarting the service in one terminal and monitor the syslog in another?

I’d take out the requires and after directive completely.

borpin · 15 September 2023 06:28

One other thing, did you use the install script? on my emonpi;

pi@emonpi:~ $ systemctl status emonhub.service
● emonhub.service - emonHub data multiplexer
     Loaded: loaded (/opt/openenergymonitor/emonhub/service/emonhub.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-09-01 11:17:49 BST; 1 weeks 6 days ago
    Process: 492 ExecStartPre=/bin/mkdir -p ${LOG_PATH} (code=exited, status=0/SUCCESS)
    Process: 508 ExecStartPre=/bin/chown ${USER} ${LOG_PATH} (code=exited, status=0/SUCCESS)
   Main PID: 514 (python3)
      Tasks: 4 (limit: 1596)
        CPU: 17h 7min 30.386s
     CGroup: /system.slice/emonhub.service
             └─514 python3 /usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log

stuart · 15 September 2023 10:12

Yes used the install script. This is running in lxc container - Ubuntu.

I’ve installed emoncms using the GITHUB script (EmonScripts/docs/install.md at master · openenergymonitor/EmonScripts · GitHub), and then ran /opt/openenergymonitor/EmonScripts/install/emonhub.sh

This has copied the emonhub files into /opt/openenergymonitor/emonhub and then ran /opt/openenergymonitor/emonhub/install.sh

The installation appears successful, but the systemd service won’t start.

Notice, that its actually failing before it even tried to run the python emonhub program - falls over at ExecStartPre

* emonhub.service - emonHub data multiplexer
     Loaded: loaded (/etc/systemd/system/emonhub.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/emonhub.service.d
             `-emonhub.conf
     Active: activating (auto-restart) (Result: exit-code) since Fri 2023-09-15 10:09:45 UTC; 1s ago
    Process: 41023 ExecStartPre=/bin/mkdir -p ${LOG_PATH} (code=exited, status=217/USER)
        CPU: 1ms

I commented out requires and after directive - no difference.
If I remove the ExecStartPre commands - it then fails at launching the emonhub python code.

As I mentioned, if I just run
/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
as user emoncms, it works fine.

10:11:47 INFO  emonhub:77 EmonHub v2.5.8
10:11:47 INFO  emonhub:78 Opening hub...
10:11:47 INFO  emonhub:79 Running as user: emoncms
10:11:47 ERROR EmonHubSerialInterfacer:57 Could not open serial port: /dev/ttyAMA0 @ 38400 bits/s (retry every 10s)
10:11:47 ERROR emonhub:246 Unable to create 'RFM2Pi' interfacer: 'bool' object has no attribute 'write'
10:11:47 WARNING EmonHubEmoncmsHTTPInterfacer:185 Setting emoncmsorg apikey: obscured

borpin · 15 September 2023 10:23

OK, so I do not understand why it seems to be installed in a different location as the script creats a symlink to the local repo file.

Might be a red herring and might be as a result of adding the dropin (the script does that).

Googling that error (status=217/USER) I find this Systemd Service Not Starting (status=217/USER) - ShellHacks - the error says it cannot find the user emoncms on the system.

Looks like there is still an issue with running from the command line.

It is a problem with the serial port. When you run it from the command line, are you running it as user emoncms and does that have access to the serial port?

stuart · 15 September 2023 10:28

Don’t worry about that one - this container doesn’t have that serial device, I’m yet to get that far in the configuration!

stuart · 15 September 2023 10:29

I found the same, but most of the errors on Google report issues with external users like LDAP services. I’m actually logged on as the emoncms user so it does exist!

Does it need root privs?

borpin · 15 September 2023 10:29

I’m also going to say, it is quite possible this part of the install script hasn’t been tested with a non pi user plus a non RaspberryOS install!

github.com

openenergymonitor/emonhub/blob/0627fe7f4cbeda4fe3124718d53427aee48317a0/install.sh#L122-L150


      
          # ---------------------------------------------------------
          # Install service
          # ---------------------------------------------------------
          if [ -d /lib/systemd/system ]; then
            if [ ! -f /lib/systemd/system/emonhub.service ]; then
              echo "Installing emonhub.service in /lib/systemd/system (creating symlink)"
              sudo ln -sf $script_dir/service/emonhub.service /lib/systemd/system
              sudo systemctl enable emonhub.service
              sudo systemctl restart emonhub.service
            else 
              echo "emonhub.service already installed"
            fi
          fi
          
          if [ "$user" != "pi" ]; then
              echo "installing emonhub drop-in User=$user"
              if [ ! -d /lib/systemd/system/emonhub.service.d ]; then
                  sudo mkdir /lib/systemd/system/emonhub.service.d
              fi
              echo $'[Service]\nUser='$user'\nEnvironment="USER='$user'"' > emonhub.service.conf

This file has been truncated. show original

I have certainly never tried it!

borpin · 15 September 2023 10:30

Did you add the user to sudo?

stuart · 15 September 2023 10:31

Must be down to this bit in the install script…

if [ "$user" != "pi" ]; then
    echo "installing emonhub drop-in User=$user"
    if [ ! -d /lib/systemd/system/emonhub.service.d ]; then
        sudo mkdir /lib/systemd/system/emonhub.service.d
    fi
    echo $'[Service]\nUser='$user'\nEnvironment="USER='$user'"' > emonhub.service.conf
    sudo mv emonhub.service.conf /lib/systemd/system/emonhub.service.d/emonhub.conf
fi

Should I scrap this install and create a “pi” user ?

borpin · 15 September 2023 10:31

That is certainly the advice

stuart · 15 September 2023 10:32

Yes. File /etc/sudoers.d/emoncms

emoncms ALL=(ALL) NOPASSWD: ALL

borpin · 15 September 2023 10:35

You could try adding multi-user.target as the requires directive.

[edit]
or the after directive. Can’t remember strictly which is correct.

borpin · 15 September 2023 10:39

I think it might need -

sudo usermod -a -G dialout $USER

stuart · 15 September 2023 10:42

Eureka!

* emonhub.service - emonHub data multiplexer
     Loaded: loaded (/etc/systemd/system/emonhub.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/emonhub.service.d
             `-emonhub.conf
     Active: active (running) since Fri 2023-09-15 10:40:06 UTC; 4s ago
    Process: 41798 ExecStartPre=/bin/mkdir -p ${LOG_PATH} (code=exited, status=0/SUCCESS)
    Process: 41799 ExecStartPre=/bin/chown ${USER} ${LOG_PATH} (code=exited, status=0/SUCCESS)
   Main PID: 41800 (python3)
      Tasks: 3 (limit: 8180)
     Memory: 23.1M
        CPU: 498ms
     CGroup: /system.slice/emonhub.service
             `-41800 python3 /usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log

Root cause:

If the user is not “pi” then this file is created /lib/systemd/system/emonhub.service.d/emonhub.conf

Which contains…

[Service]
User=emoncms\nEnvironment="USER=emoncms"

Note that the “\n” has not been escaped correctly by the install script. Editing the file and changing to…

[Service]
User=emoncms
Environment="USER=emoncms"

Allows the service to start up correctly - it was trying to run as user “emoncms\n”

Now the next problem… service-runner and demandshaper won’t launch!

Update:
service-runner was fixed by running sudo apt install python3-redis

demandshaper fixed by editing the “user” parameter in file /lib/systemd/system/demandshaper.service

borpin · 15 September 2023 11:34

Fixed by…

@glyn.hudson @TrystanLea