EmonHub Failing After Update

It’s for code too, eg

pi@emonpi(ro):emonhub$ sudo systemctl --now enable emonhub.service
systemctl: unrecognized option ‘–now’

would be

pi@emonpi(ro):emonhub$ sudo systemctl --now enable emonhub.service
systemctl: unrecognized option ‘–now’

not a whole lot different with short commands, but the lines do not wrap and certain chars are formatted

eg

<html tag>

using the "quote method comes out as

That’s normal,

That not so much!

Ok the long hand way of doing the same is

sudo systemctl enable emonhub.service
sudo systemctl start emonhub.service

the --now option was not available in earlier systemd versions, what version is your setup running?

systemd --version

[edit] apparently introduced in version 220 systemd - the --now switch of `systemctl` - Unix & Linux Stack Exchange

So I wasn’t going crazy! (on this occasion at least)

From the git-checkout man page

       git checkout -b|-B <new_branch> [<start point>]
           Specifying -b causes a new branch to be created as if git-branch(1) were called and then checked out. In this case you
           can use the --track or --no-track options, which will be passed to git branch. As a convenience, --track without -b
           implies branch creation; see the description of --track below.

           If -B is given, <new_branch> is created if it doesnât exist; otherwise, it is reset. This is the transactional
           equivalent of

               $ git branch -f <branch> [<start point>]
               $ git checkout <branch>

           that is to say, the branch is not reset/created unless "git checkout" is successful.

So I guess that command should have been

git checkout  -f -B emon-pi
> pi@emonpi(rw):service$ systemd --version
systemd 215
+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR

returns:

> pi@emonpi(rw):emonhub$ sudo systemctl enable emonhub.service
Failed to execute operation: No such file or directory

Should I go back and to a checkout with -B …?

So the version explains why the --now didn’t work, but why the enable command returns “No such file or directory” I know not!

I thought we had proved the service unit had changed and the links were there! If sudo systemctl cat emonhub.service and sudo systemctl status emonhub.service work, how can sudo systemctl enable emonhub.service be “No such file or directory”? How odd!

Can you run these commands (again) just to check all is still in place?

sudo systemctl cat emonhub.service
ls -la /{etc,lib}/systemd/system/emonhub.service
ls -la /usr/local/bin/emonhub
ls -la /usr/local/bin/emonhub/
cat /home/pi/emonhub/service/emonhub.service

top-tip, you can copy and paste the whole 5 lines to the command line of your SSH terminal and then just copy and paste the whole caboodle back again rather than doing them individually.

I had thought the repo was reset, perhaps do a

git status

before and after the “-B” command so we know whether it is up to date before and after running it.

First, many thanks for your help, I’d be lost without you!

> pi@emonpi(rw):~$ sudo systemctl cat emonhub.service
# /home/pi/emonhub/service/emonhub.service
[Unit]
Description=emonHub service description
DefaultDependencies=no
Before=shutdown.target
Conflicts=shutdown.target
Requires=local-fs.target
After=sysinit.target syslog.target local-fs.target

[Service]
User=emonhub
PIDFile=/var/run/emonhub.pid
ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhu
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/log/emonhub/
ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/
ExecStartPre=/bin/chmod 775 /var/log/emonhub/
Type=simple
Restart=always

[Install]
WantedBy=multi-user.target
lines 1-22/22 (END)pi@emonpi(rw):~$

2nd command: Ahh, error:

pi@emonpi(rw):emonhub$ ls -la /{etc,lib}/systemd/system/emonhub.service
ls: cannot access /etc/systemd/system/emonhub.service: No such file or directory
lrwxrwxrwx 1 root root 40 Jul 24 23:12 /lib/systemd/system/emonhub.service -> /home/pi/emonhub/service/emonhub.service

3rd command:

pi@emonpi(rw):emonhub$ ls -la /usr/local/bin/emonhub
lrwxrwxrwx 1 root staff 20 Jul 23 15:28 /usr/local/bin/emonhub -> /home/pi/emonhub/src

4th command:

pi@emonpi(rw):emonhub$ ls -la /usr/local/bin/emonhub/
total 80
drwxr-xr-x 4 pi   pi    4096 Jul 25 00:41 .
drwxr-xr-x 7 pi   pi    4096 Jul 24 23:12 ..
-rw-r--r-- 1 pi   pi    1142 Jul 24 23:12 Cargo.py
-rw-r--r-- 1 pi   pi    2834 Jul 24 23:12 emonhub_buffer.py
-rw-r--r-- 1 pi   pi    1148 Jan 26  2016 emonhub_coder.py
-rw-r--r-- 1 pi   pi   25255 Jul 24 23:12 emonhub_interfacer.py
-rwxr-xr-x 1 pi   pi   12341 Jul 24 23:12 emonhub.py
-rw-r--r-- 1 pi   pi    5219 Mar 29  2017 emonhub_setup.py
drwxr-xr-x 3 pi   pi    4096 Jul 24 23:12 interfacers
drwxr-xr-x 2 pi   pi    4096 Jul 13  2017 smalibrary
lrwxrwxrwx 1 root root    20 Jul 25 00:41 src -> /home/pi/emonhub/src

and 5th command:

pi@emonpi(rw):emonhub$ cat /home/pi/emonhub/service/emonhub.service
[Unit]
Description=emonHub service description
DefaultDependencies=no
Before=shutdown.target
Conflicts=shutdown.target
Requires=local-fs.target
After=sysinit.target syslog.target local-fs.target

[Service]
User=emonhub
PIDFile=/var/run/emonhub.pid
ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/log/emonhub/
ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/
ExecStartPre=/bin/chmod 775 /var/log/emonhub/
Type=simple
Restart=always

[Install]
WantedBy=multi-user.target

And the GIT status:

pi@emonpi(rw):emonhub$ git status
On branch emon-pi
Your branch is up-to-date with 'origin/emon-pi'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        src/src

nothing added to commit but untracked files present (use "git add" to track)

should have read your top-tip first, silly me!

Ahh, this time the ls commands differ. Back then I ran

s -la /usr/{local/bin,share}/emonhub
ls -la /etc/{init.d/,}emonhub

but this time it was:

ls -la /{etc,lib}/systemd/system/emonhub.service
ls -la /usr/local/bin/emonhub

i’m not entirely sure we’re not lost even with my input :smile:

No that’s a good error. If there was a service unit in /etc/systemd/system it would override anything we are doing to the correctly located unit in /lib/systemd/system.

So I’m at a loss as too why the path/file is found by systemctl for other commands but not for “enable” apparently! Everything looks as it should as far as I can tell (aside form that rogue “src” folder in the emonhub/src folder).

There is an option to force an “enable” but I don’t think that will over come a file perceived as absent.

sudo systemctl -f enable emonhub.service

Assuming no change, what does trying to start the service do?

sudo systemctl start emonhub.service

if there is no error, what is the status after that?

sudo systemctl status emonhub.service

Yeah, just slightly different to test different files and paths.

WAY less lost!

gets error:

Failed to execute operation: No such file or directory

gets no output, but the

produces the same as before:


pi@emonpi(ro):system$ sudo systemctl status emonhub.service
● emonhub.service - emonHub service description
   Loaded: loaded (/home/pi/emonhub/service/emonhub.service; disabled)
   Active: failed (Result: start-limit) since Thu 2019-07-25 01:24:51 NZST; 7s ago
  Process: 12425 ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log (code=exited, status =1/FAILURE)
  Process: 12422 ExecStartPre=/bin/chmod 775 /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 12419 ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 12416 ExecStartPre=/bin/mkdir -p /var/log/emonhub/ (code=exited, status=0/SUCCESS)
 Main PID: 12425 (code=exited, status=1/FAILURE)

Jul 25 01:24:51 emonpi systemd[1]: emonhub.service holdoff time over, sched...t.
Jul 25 01:24:51 emonpi systemd[1]: Stopping emonHub service description...
Jul 25 01:24:51 emonpi systemd[1]: Starting emonHub service description...
Jul 25 01:24:51 emonpi systemd[1]: emonhub.service start request repeated t...t.
Jul 25 01:24:51 emonpi systemd[1]: Failed to start emonHub service description.
Jul 25 01:24:51 emonpi systemd[1]: Unit emonhub.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.

Something quite different crossed my mind: Earlier this year, my backup.conf failed because of # comments no longer supported by PHP (I think). Could it be the same for emonhub.conf?

No, that was a change to php and how it reads ini files, emonhub is Python and uses configobj for it’s conf file formatting.

I really have to concentrate on something else for a while, sorry. You could try running a full update again now we know the repo is reset. As much as I hate suggesting it, you could also try a reboot as that might dislodge something. Otherwise i will need to comeback to this and go through a full list of checks to determine exactly what we do or don’t have right now. That also give me a chance to mull over the odd fact that systemctl can execute some commands but not all (eg enable) with the exact same file apparently “not found” in some instances.

Will do that and let you know.

FYI, I ran this again, this time without lines ellipsized:

pi@emonpi(ro):~$ sudo systemctl status emonhub.service -l
● emonhub.service - emonHub service description
   Loaded: loaded (/home/pi/emonhub/service/emonhub.service; disabled)
   Active: failed (Result: start-limit) since Thu 2019-07-25 01:24:51 NZST; 26min ago
  Process: 12425 ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log (code=exited, status=1/FAILURE)
  Process: 12422 ExecStartPre=/bin/chmod 775 /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 12419 ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 12416 ExecStartPre=/bin/mkdir -p /var/log/emonhub/ (code=exited, status=0/SUCCESS)
 Main PID: 12425 (code=exited, status=1/FAILURE)

Jul 25 01:24:51 emonpi systemd[1]: emonhub.service holdoff time over, scheduling restart.
Jul 25 01:24:51 emonpi systemd[1]: Stopping emonHub service description...
Jul 25 01:24:51 emonpi systemd[1]: Starting emonHub service description...
Jul 25 01:24:51 emonpi systemd[1]: emonhub.service start request repeated too quickly, refusing to start.
Jul 25 01:24:51 emonpi systemd[1]: Failed to start emonHub service description.
Jul 25 01:24:51 emonpi systemd[1]: Unit emonhub.service entered failed state.
pi@emonpi(ro):~$

It’s 2AM here now (New Zealand), we may have to plan a good time for you to continue this. Anytime is fine with me of course. Good night, and many thanks for your help.

Hi @pb66,
Success! The full update and reboot must have fixed whatever was blocking it. Emonhub started by itself, and all seems fine now. Thank you so much for your assistance with this rather weird issue. Very much appreciated.

BTW: I know my SD card (Nov16) is getting a bit old too. When should I upgrade? I believe there was talk about a major overhaul of the SD card (next year or so?)

Not entirely, looking at the last “status” you posted, that “loaded” line still says “disabled” at the end not “enabled” as it should.

The current emonSD image updates provide a entry in the rc.local that restarts everything after creating logfiles in tmpfs for those services that are unable to do so for themselves eg redis, mysql and apache. Although unnecessary, emonhub also appears on that list of aided services.

Therefore it doesn’t really even matter whether these services start automatically on boot or not, since they are then stopped and restarted all over again at the tail end of the boot up process. I have seen on some other users log info that the apache service isn’t “enabled” on some images, but because of the rc.local, everything seems to work fine. The concern here is that since the emonhub service doesn’t need to be in rc.local, it may get removed, at which time your emonhub will fail to start at boot time as it isn’t “enabled”.

You could try enabling it now with

sudo systemctl enable emonhub.service

hopefully what ever was blocking that from enabling before, has now cleared in the reboot?

If you are concerned about your physical sdcard condition, you could get a new card and install the Oct2018 image, transferring all your data and settings from the old card/image to the new card/image. There is a new “installer” image builder script in the pipeline, although I think it may be a while before that’s ready for full release.

You are right. While on my EMONCMS admin page it says emonhub is running, even after the reboot I still get the disabled status

pi@emonpi(ro):~$ sudo systemctl status emonhub.service
● emonhub.service - emonHub service description
   Loaded: loaded (/home/pi/emonhub/service/emonhub.service; disabled)
   Active: active (running) since Thu 2019-07-25 03:48:44 NZST; 1 day 6h ago
  Process: 2939 ExecStartPre=/bin/chmod 775 /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 2935 ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 2932 ExecStartPre=/bin/mkdir -p /var/log/emonhub/ (code=exited, status=0/SUCCESS)
 Main PID: 2946 (python)
   CGroup: /system.slice/emonhub.service
           └─2946 python /usr/local/bin/emonhub/emonhub.py --config-file=/etc...

So I followed your advice to enable it, but strangely it still cannot find that file even though the service is active and running:

pi@emonpi(rw):emonhub$ sudo systemctl enable emonhub.service
Failed to execute operation: No such file or directory

Isn’t that strange … the start and stop commands work fine, but not the enable … Is it looking in the wrong directory?

SD card: So far the SD cards shows no signs of problems, so I think I will wait until the new installer script is ready. No hurry.

That sounds very much like the issues we initially ran into with enabling systemd services on Raspbian jessie. … Looking…

Yep, that’s exactly the same…

See that link for further commentary on the problem (quite a discussion occurred :slight_smile:). But there is this from that link

There’s no “good” workaround, except to upgrade to a newer image.

Layman’s question: Seeing there is no good workaround and that the service starts up after booting (i.e. all is working again fine), would it be sensible to leave everything as is and wait for the new SD image to be released, then migrate the lot…? I know so little about the internals of Emonpi that I shy away from touching anything to keep support as simple as possible. A reasonable compromise?

Yep. Do that :slight_smile:

Just keep in mind pb66’s comment about rc.local:

So if it ever fails to start after a reboot, try and remember this discussion and make that the first thing to check!

[EDIT] Or do what I’m doing with my jessie based image… I’ve just stopped hitting the “Update” button until I get a new image on there. I’m not interested in any of the new features being added and I’m happy with the current operation of the device(s)

I concur with Greebo, it’s not a big issue it will work fine as it is.

Thanks @Greebo for confirming and providing the link, I have referred to memory of that discussion elsewhere on the forum but couldn’t find it or recall the exact detail. Although I hadn’t actually considered it being a factor with Juerg’s issue, it now makes a little more sense why the service won’t enable.

Quite a few months have passed since July 2019, and I have been a bit off-line to and therefore a bit uninformed: Is it still advisable to stick with my old SD image of Nov16, or has the mentioned major overhaul of the new SD card now been completed? Thanks for your insights in advance!

Yes it has been done and there is a method of transferring data and settings via USB. Just do a search here for the relevant posts.

Thank you Brian, very good to know. Cheers

1 Like