EmonHub Failing After Update

Hi,

I had same problem, but after /usr/local/bin/emonhub link fixed, then I also had problem with missing config file /etc/emonhub/emonhub.conf

So I created missing directory and link in it:

sudo mkdir /etc/emonhub
sudo ln -sf /home/pi/data/emonhub.conf /etc/emonhub/emonhub.conf

Not sure it is the right to do, or just edit emonhub.service in /home/pi/emonhub/service ?
But seems emonhub.service updated , so maybe those link is missing too.

 systemctl cat emonhub
# /home/pi/emonhub/service/emonhub.service
[Unit]
Description=emonHub service description
DefaultDependencies=no
Before=shutdown.target
Conflicts=shutdown.target
Requires=local-fs.target
After=sysinit.target syslog.target local-fs.target

[Service]
User=emonhub
PIDFile=/var/run/emonhub.pid
ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.l
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/log/emonhub/
ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/
ExecStartPre=/bin/chmod 775 /var/log/emonhub/
Type=simple
Restart=always

[Install]
WantedBy=multi-user.target

BR
Dimitar

Good to hear you sorted your problem out, it is better to create the directory and symlink than edit the service unit so that your setup continues to be in line with the stock OEM image. Had you edited the service unit, it would have most likely blocked future updates to emonhub via the updater as git will abort pulling in updates if it detects local changes in the files to be updated.

What you’ve done is exactly what should of happened in the update, just like Sam’s update, your’s has not completed succesfully either, in this case it appears to have skipped/missed these 2 parts

The symlink obviously can’t be created if the directory doesn’t exist.

It’s not clear to me why the updater appears to pick and choose which bits to run, but looking back at the update log snippet in the opening post, it seems there was no problem with the git pull and then it jumps straight to “emonhub.service already installed” and then “Running emonhub automatic node addition script”. These are the last 2 items in the emonhub update script and we should have seen “Creating emonhub logfile” and “Symlinking emonhub.conf to /etc/emonhub/emonhub.conf” echod to the log regardless of whether it actually performed those operations or not due to the fact they are located outside the conditional tests for executing those operations.

Thanks for letting us know Dimitar.

Hi,
just updated to the latest version and ran into what seems to be the same problem: Emonhub Failed.

I’m not very familiar with the technical stuff, and I don’t know where I need to enter the above two sudo commands. In SSH on the command line?

Many thanks for your help.

UPDATE:
I entered the quoted two sudo commands on SSH command line.
First sudo cmd finished, the second failed with:

:Job for emonhub.service failed. See systemctl status emonhub.service and journalctl -xn for details.

Status of emonhub.service:

● emonhub.service - LSB: Start/stop emonHub
   Loaded: loaded (/etc/init.d/emonhub)
   Active: failed (Result: exit-code) since Tue 2019-07-23 16:06:01 NZST; 10s ago
  Process: 8168 ExecStart=/etc/init.d/emonhub start (code=exited, status=1/FAILURE)

and: journalctl -xn:

No journal files were found.

When I tried to update EMONHUB for the second time, the log shows:

Update Log

Starting update via service-runner-update.sh (v2.0) >
Filesystem is unlocked - Write access
type ’ rpi-ro ’ to lock

  • emonSD version: emonSD-07Nov16
    emonSD base image check passed…continue update
    git pull /home/pi/emonpi
  • master
    wifiap
    On branch master
    Your branch is up-to-date with ‘origin/master’.
    Untracked files:
    (use “git add …” to include in what will be committed)

    1
    restart.sh

nothing added to commit but untracked files present (use “git add” to track)
Already up-to-date.

Main Update Script

Date: Tue 23 Jul 16:41:31 NZST 2019
EUID: 1000
usrdir: /home/pi
type: emonhub
firmware: emonpi
Checking environment:

  • pi user found

EUID: 1000
Hardware detected: EmonPi
Stopping emonPiLCD service
Display update message on LCD
I2C LCD DETECTED Ox3f
Start emonhub update script:

emonHub update started

  • usr directory: /home/pi
    git pull /home/pi/emonhub
    K0den-wibeee_interface
  • emon-pi
    hmm01i-syslogging
    On branch emon-pi
    Your branch is behind ‘origin/emon-pi’ by 147 commits, and can be fast-forwarded.
    (use “git pull” to update your local branch)
    Changes not staged for commit:
    (use “git add …” to update what will be committed)
    (use “git checkout – …” to discard changes in working directory)

    modified: src/emonhub.py

Untracked files:
(use “git add …” to include in what will be committed)

src/src

no changes added to commit (use “git add” and/or “git commit -a”)
error: Your local changes to the following files would be overwritten by merge:
src/emonhub.py
Please, commit your changes or stash them before you can merge.
Aborting
Updating 756a33d…ee1f537
Creating emonhub logfile
Symlinking emonhub.conf to /etc/emonhub/emonhub.conf
Job for emonhub.service failed. See ‘systemctl status emonhub.service’ and ‘journalctl -xn’ for details.

Running emonhub automatic node addition script
EUID: 1000
[[5]]
Node 5 already present
[[6]]
Node 6 already present
[[7]]
Node 7 already present
[[8]]
Node 8 already present
[[9]]
Node 9 already present
[[10]]
Node 10 already present
[[11]]
Node 11 already present
[[12]]
Node 12 already present
[[13]]
Node 13 already present
[[14]]
Node 14 already present
[[19]]
Node 19 already present
[[20]]
Node 20 already present
[[21]]
Node 21 already present
[[22]]
Node 22 already present
[[23]]
Node 23 already present
[[24]]
Node 24 already present
[[25]]
Node 25 already present
[[26]]
Node 26 already present

Starting emonPi LCD service…

Filesystem is locked - Read Only access
type ’ rpi-rw ’ to unlock


emonPi update done: Tue 23 Jul 16:41:41 NZST 2019


I also cannot make sense of the line:

error: Your local changes to the following files would be overwritten by merge:
src/emonhub.py

As far as I know there are no local changes, the Emonpi is stock standard, except for having to remove the # comment lines in the backup.conf file earlier this year after causing PHP errors for backups after updating to 10.0.2.

I am a bit lost now.
BTW, I’m on Nov16 SD card, but 10.1.2 emoncms. Before the update I was on 10.0.2.

I have yet to find git wrong when reporting local changes, so whilst i can’t say it’s not possible I do think it’s unlikely (at this stage). If you go to the emonhub folder you can ask git what the changes are.

cd /home/pi/emonhub
git diff src/emonhub.py

This may jog the memory or at least tell you what you might lose if we force an update.

It looks like it hasn’t updated for quite some time as there are 147 commits outstanding, I don’t know how frequently you run an update, but it looks like emonhub hasn’t succesfully updated since mid 2017’ish, this may just be the first time the failed (ie partial) update has caused it to stop running.

There is also a second local change reported in that there is apparently an untracked src folder inside the original src folder, couls you run

ls -la src/src

whilst you are still in the emonhub folder and post what you find.

I’m just heading out soon so I may not be around til later. If you can run these couple of commands, it will tell us the status of the symlinks to see if you have actually had a similar issue to the cases above. So far it just looks like the git pull failed due to the local changes, there’s nothing (yet) to suggest any of the symlinking has failed.

ls -la /usr/{local/bin,share}/emonhub
ls -la /etc/{init.d/,}emonhub
systemctl cat emonhub.service

If you are happy to lose the local changes to src/emonhub.py, you can force a update with (from within the emonhub folder again)

git checkout -f -b emon-pi

It may also be worth running these 2 commands for good measure

sudo systemctl daemon-reload
sudo systemctl --now enable emonhub.service

assuming there are no other errors, you could run the update again to confirm all is ticketty-boo. If the above command doesn’t work, there is a longer “hard reset” procedure we can try, but I don’t know those commands off hand, I tend to use the above without any issue.

Sorry for the info overload, but I’m trying to give you everything you might need to get it running today rather than wait for me, please post any errors or results etc, if you would rather wait and go through step bey step we can.

Thanks @pb66 for your help.

The git diff produces nothing that I would know or remember: (or is it a fix suggested by your team many years ago with some issue we had back then?)

pi@emonpi(ro):emonhub$ git diff src/emonhub.py
diff --git a/src/emonhub.py b/src/emonhub.py
index aee21c0…84c4271 100755
— a/src/emonhub.py
+++ b/src/emonhub.py
@@ -138,6 +138,7 @@ class EmonHub(object):
if not I.isAlive():
#I.start()
self._log.warning(I.name + " thread is dead") # had to be restarted")
self._exit = True
# Sleep until next iteration
time.sleep(0.2)

I have updated EMONCMS last in June 2019, to version 10.0.2, but not since then. Are you saying when I updated that emonhub was left behind with a much older version? Without reading every line of the update logs, I was not aware that anything ever failed. Perhaps my fault for not reading all the lines …?

When I run ls -la src/src I get:

pi@emonpi(ro):emonhub$ ls -la src/src
lrwxrwxrwx 1 root root 20 Jul 23 16:41 src/src → /home/pi/emonhub/src

Unfortunately that means nothing to me either.

As far as the remaining commands go that you gave me, where does the forced update start? When I tried the following next of your commands, it failed with

pi@emonpi(ro):emonhub$ ls -la /usr/{local/bin,share}/emonhub
ls: cannot access /usr/share/emonhub: No such file or directory
lrwxrwxrwx 1 root staff 20 Jul 23 15:28 /usr/local/bin/emonhub → /home/pi/emonhub/src

With my lack of knowledge I think I better stop here and await your advice. I really have no idea what I would be overwriting.

Looks like I should have started a new topic - the cause of this seems unrelated to the original issue. Apologies.

Done some searching, and found the ‘local’ change below was related to a fix in Jun 2017, due to Emonpi requiring periodic restarting. See EmonPi needs periodic restarting - #9 by TrystanLea for details.
I think this is no longer needed, right?

pi@emonpi(ro):emonhub$ git diff src/emonhub.py
diff --git a/src/emonhub.py b/src/emonhub.py
index aee21c0…84c4271 100755
— a/src/emonhub.py
+++ b/src/emonhub.py
@@ -138,6 +138,7 @@ class EmonHub(object):
if not I.isAlive():
#I.start()
self._log.warning(I.name + " thread is dead") # had to be restarted")
self._exit = True
`# Sleep until next iteration
time.sleep(0.2)

In fact, the watchdog for this fix is still in my crontab.

 * * * * * /home/pi/watchdog.sh >> /home/pi/data/watchdog.log 2>&1

Is it obsolete and safe to remove now? I asked that question in May 2019 but did not get replies:

Seeing that, I thought I can force an update for emonhub without losing anything important. I started with the two ls and the one cat commands:

pi@emonpi(rw):emonhub$ ls -la /usr/{local/bin,share}/emonhub
ls: cannot access /usr/share/emonhub: No such file or directory
lrwxrwxrwx 1 root staff 20 Jul 23 15:28 /usr/local/bin/emonhub -> /home/pi/emonhub/src
 
pi@emonpi(rw):emonhub$ ls -la /etc/{init.d/,}emonhub
lrwxrwxrwx 1 root root   32 Jan 26  2016 /etc/init.d/emonhub -> /home/pi/emonhub/service/emonhub
/etc/emonhub:
total 8
drwxr-xr-x   2 root root 4096 Jul 23 16:41 .
drwxr-xr-x 100 root root 4096 Jul 23 15:39 ..
lrwxrwxrwx   1 root root   26 Jul 23 16:41 emonhub.conf -> /home/pi/data/emonhub.conf

pi@emonpi(rw):emonhub$ systemctl cat emonhub.service
# /run/systemd/generator.late/emonhub.service
# Automatically generated by systemd-sysv-generator

[Unit]
SourcePath=/etc/init.d/emonhub
Description=LSB: Start/stop emonHub
Before=runlevel2.target runlevel3.target runlevel4.target runlevel5.target shutdown.target
After=remote-fs.target systemd-journald-dev-log.socket
Conflicts=shutdown.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SysVStartPriority=3
ExecStart=/etc/init.d/emonhub start
ExecStop=/etc/init.d/emonhub stop
pi@emonpi(rw):emonhub$

Then I tried the forced update of emonhub: git checkout -f -b emon-pi, but it fails with:

pi@emonpi(rw):emonhub$ git checkout -f -b emon-pi
fatal: A branch named 'emon-pi' already exists.

Hmm, what should I do next? Many thanks for your help.

Not a problem, it is related though. We do not have all the info on the previous 2 occurrences, but from your last reply it would seem the update failed to complete (aside from the failed git pull), possibly or even probably due to the git pull failing because of local changes. The update script needs to handle these situations and either complete or back out so that a working system is maintained, and more importantly it should clearly tell the user there is a issue. Perhaps a final status at the end of the update as a minimum, a user should not need to read the whole log line by line to know if the update was done successfully or not.

Correct, glad you worked out the history, local changes are usually there for a reason and blindly resetting isn’t always without consequence. But here you can indeed safely reset and get rid of the line in crontab.

The first command checked both the original and the new locations for the link to emonhub/src, the result tells us nothing was found at the original location and a link to the emonhub/src was found at the new location, so that part seems to have succeeded ok.

The second command tested for 3 things, whether the old original int.d service unit had been removed, whether the etc/emonhub folder exists and whether a link to the emonhub.conf on the writable data partition exists. The latter 2 passed. But the init.d service script seems to still exist, it should of been deleted.

This is further confirmed but the last command as the systemd service unit’s source path is /etc/init.d/emonhub.

My apologies, that “-b” shouldn’t be there, that creates a new branch, we were trying to reset the existing branch, hence the “already exists” error.

So from within the emonhub folder, try

git checkout -f emon-pi

At this point I would say try doing a full update again. I say this because the init.d script that wasn’t deleted, should have been deleted by a sub-script whilst symlinking reloading, restarting and enabling the new service unit. We do not know where it fell over so running it again may fill in any “missed” parts.

once completed run

sudo system status emonhub.service

to confirm emonhub is running ok, if there are any errors then run these and post results here along with the results of the previous command

sudo systemctl cat emonhub.service
ls -la /{etc,lib}/systemd/system/emonhub.service
ls -la /etc/init.d/emonhub

and we can sort it out manually.

I ran that, and got

pi@emonpi(rw):emonhub$ git checkout -f emon-pi
Already on ‘emon-pi’
Your branch is behind ‘origin/emon-pi’ by 147 commits, and can be fast-forwarded.
(use “git pull” to update your local branch)

Then I accidentally ran a ‘Update EMONHUB’, and when I noticed my mistake a full update from EMONCMS admin page. Emonhub still shows FAILED after that. Got a very long log, which I struggled a bit to include here in a readable format:

Starting update via service-runner-update.sh (v2.0) >
Filesystem is unlocked - Write access
type ’ rpi-ro ’ to lock
- emonSD version: emonSD-07Nov16 emonSD base image check passed...continue update git pull /home/pi/emonpi * master
wifiap
On branch master
Your branch is up-to-date with ‘origin/master’.
Untracked files:
(use “git add …” to include in what will be committed)
1 restart.sh
nothing added to commit but untracked files present (use “git add” to track)
Already up-to-date.
------------------------------------------------------------- Main Update Script -------------------------------------------------------------
Date: Wed 24 Jul 23:18:46 NZST 2019
EUID: 1000
usrdir: /home/pi
type: all
firmware: emonpi
Checking environment:
- pi user found
EUID: 1000
Hardware detected: EmonPi
Stopping emonPiLCD service
Display update message on LCD
I2C LCD DETECTED Ox3f
------------------------------------------------------------- Ign http://dl.bintray.com stable InRelease Get:1 http://dl.bintray.com stable Release.gpg [821 B] Hit http://ppa.launchpad.net xenial InRelease Get:2 http://dl.bintray.com stable Release [6,051 B] Get:3 http://dl.bintray.com stable/main armhf Packages [77.8 kB] Hit http://ppa.launchpad.net xenial/main Sources Hit http://ppa.launchpad.net xenial/main armhf Packages Hit http://ppa.launchpad.net xenial/main Translation-en Get:4 http://repo.mosquitto.org jessie InRelease [11.0 kB] Ign http://dl.bintray.com stable/main Translation-en_GB Hit http://mirrordirector.raspbian.org jessie InRelease Err http://repo.mosquitto.org jessie InRelease
Ign http://dl.bintray.com stable/main Translation-en
Hit http://mirrordirector.raspbian.org jessie/main armhf Packages
Hit http://mirrordirector.raspbian.org jessie/contrib armhf Packages
Hit http://mirrordirector.raspbian.org jessie/non-free armhf Packages
Hit http://mirrordirector.raspbian.org jessie/rpi armhf Packages
Hit http://archive.raspberrypi.org jessie InRelease
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en
Ign http://mirrordirector.raspbian.org jessie/main Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/main Translation-en
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en
Hit http://archive.raspberrypi.org jessie/main armhf Packages
Hit http://archive.raspberrypi.org jessie/ui armhf Packages
Ign http://archive.raspberrypi.org jessie/main Translation-en_GB
Ign http://archive.raspberrypi.org jessie/main Translation-en
Ign http://archive.raspberrypi.org jessie/ui Translation-en_GB
Ign http://archive.raspberrypi.org jessie/ui Translation-en
Fetched 84.6 kB in 51s (1,657 B/s)
Reading package lists…
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://repo.mosquitto.org jessie InRelease: The following signatures were invalid: KEYEXPIRED 1515017477 KEYEXPIRED 1515017477 KEYEXPIRED 1515017477
W: Failed to fetch http://repo.mosquitto.org/debian/dists/jessie/InRelease
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists…
Building dependency tree…
Reading state information…
rng-tools is already the newest version.
python-gpiozero is already the newest version.
python-rpi.gpio is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 211 not upgraded.
Requirement already up-to-date: paho-mqtt in /usr/local/lib/python2.7/dist-packages
You are using pip version 9.0.1, however version 19.1.1 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.
------------------------------------------------------------- git pull /home/pi/RFM2Pi * master
On branch master
Your branch is up-to-date with ‘origin/master’.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/usefulscripts * master
On branch master
Your branch is behind ‘origin/master’ by 45 commits, and can be fast-forwarded.
(use “git pull” to update your local branch)
Changes not staged for commit:
(use “git add …” to update what will be committed)
(use “git checkout – …” to discard changes in working directory)
modified: convertdata/phptimestore_to_phpfina.php modified: process/power_to_kwh.php modified: readme.md
Untracked files:
(use “git add …” to include in what will be committed)
update_emoncms.sh
no changes added to commit (use “git add” and/or “git commit -a”)
error: Your local changes to the following files would be overwritten by merge:
convertdata/phptimestore_to_phpfina.php
process/power_to_kwh.php
readme.md
Please, commit your changes or stash them before you can merge.
error: The following untracked working tree files would be overwritten by merge:
update_emoncms.sh
Please move or remove them before you can merge.
Aborting
Updating 6cf8e48…6d3942c
git pull /home/pi/huawei-hilink-status* master
On branch master
Your branch is up-to-date with ‘origin/master’.
nothing to commit, working directory clean
error: cannot open .git/FETCH_HEAD: Permission denied
git pull /home/pi/oem_openHab* master
On branch master
Your branch is up-to-date with ‘origin/master’.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/oem_node-red* master
On branch master
Your branch is up-to-date with ‘origin/master’.
nothing to commit, working directory clean
Already up-to-date.
-------------------------------------------------------------
EmonPi Firmware Update
------------------------------------------------------------- Start ATmega328 serial upload using avrdude with latest.hex Discrete Sampling avrdude -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 115200 -U flash:w:/home/pi/emonpi/firmware/compiled/latest.hex avrdude-original: Using autoreset DTR on GPIO Pin 7
avrdude-original: AVR device initialized and ready to accept instructions
Reading | ################################################## | 100% 0.00s
avrdude-original: Device signature = 0x1e950f
avrdude-original: NOTE: “flash” memory has been specified, an erase cycle will be performed
To disable this feature, specify the -D option.
avrdude-original: erasing chip
avrdude-original: reading input file “/home/pi/emonpi/firmware/compiled/latest.hex”
avrdude-original: input file /home/pi/emonpi/firmware/compiled/latest.hex auto detected as Intel Hex
avrdude-original: writing flash (17886 bytes):
Writing | ################################################## | 100% 2.45s
avrdude-original: 17886 bytes of flash written
avrdude-original: verifying flash memory against /home/pi/emonpi/firmware/compiled/latest.hex:
avrdude-original: load data flash data from input file /home/pi/emonpi/firmware/compiled/latest.hex:
avrdude-original: input file /home/pi/emonpi/firmware/compiled/latest.hex auto detected as Intel Hex
avrdude-original: input file /home/pi/emonpi/firmware/compiled/latest.hex contains 17886 bytes
avrdude-original: reading on-chip flash data:
Reading | ################################################## | 100% 1.83s
avrdude-original: verifying …
avrdude-original: 17886 bytes of flash verified
strace: |autoreset: Broken pipe
strace: |autoreset: Broken pipe
strace: |autoreset: Broken pipe
strace: |autoreset: Broken pipe
strace: |autoreset: Broken pipe
avrdude-original done. Thank you.
strace: |autoreset: Broken pipe
Start emonhub update script:
------------------------------------------------------------- emonHub update started -------------------------------------------------------------
- usr directory: /home/pi git pull /home/pi/emonhub K0den-wibeee_interface * emon-pi
hmm01i-syslogging
On branch emon-pi
Your branch is up-to-date with ‘origin/emon-pi’.
Untracked files:
(use “git add …” to include in what will be committed)
src/src
nothing added to commit but untracked files present (use “git add” to track)
Already up-to-date.
Creating emonhub logfile
Symlinking emonhub.conf to /etc/emonhub/emonhub.conf
emonhub.service already installed
Running emonhub automatic node addition script [[5]] Node 5 already present [[6]] Node 6 already present [[7]] Node 7 already present [[8]] Node 8 already present [[9]] Node 9 already present [[10]] Node 10 already present [[11]] Node 11 already present [[12]] Node 12 already present [[13]] Node 13 already present [[14]] Node 14 already present [[19]] Node 19 already present [[20]] Node 20 already present [[21]] Node 21 already present [[22]] Node 22 already present [[23]] Node 23 already present [[24]] Node 24 already present [[25]] Node 25 already present [[26]] Node 26 already present
Start emoncms update:
-------------------------------------------------------------
Emoncms update started
Emoncms update script V1.3 (26th March 2019)
------------------------------------------------------------- - usr directory: /home/pi
- emoncms directory: /var/www/emoncms
current settings.php md5: 0e4803a641b7784ed23edf87506ce761
Default settings.php md5: 0e4803a641b7784ed23edf87506ce761
settings.php has NOT been user modifed
Checking status of /var/www/emoncms git repository- git branch: stable
- no local changes - running: git pull origin stable
From https://github.com/emoncms/emoncms * branch stable → FETCH_HEAD
Already up-to-date.
NEW default settings.php md5: 0e4803a641b7784ed23edf87506ce761 settings.php not updated
------------------------------------------------------------- Updating Emoncms Modules -------------------------------------------------------------
------------------------------------------
Updating /var/www/emoncms/Modules/app module
------------------------------------------ - git branch: stable
- git tags: 1.2.0-108-g6e5016a - no local changes
- running: git pull origin stable
From GitHub - emoncms/app: Emoncms App module: application specific dashboards: includes myelectric, mysolarpv, myheatpump and solar + wind app.
* branch stable -> FETCH_HEAD Already up-to-date.
------------------------------------------ Updating /var/www/emoncms/Modules/config module ------------------------------------------
- git branch: stable - git tags: 1.0.0-63-g1d9238e
- no local changes - running: git pull origin stable
From https://github.com/emoncms/config * branch stable → FETCH_HEAD
Already up-to-date.
------------------------------------------
Updating /var/www/emoncms/Modules/dashboard module
------------------------------------------ - git branch: stable
- git tags: 1.3.1-134-g337df80 - no local changes
- running: git pull origin stable
From GitHub - emoncms/dashboard: dashboard module for emoncms
* branch stable -> FETCH_HEAD Already up-to-date.
------------------------------------------ Updating /var/www/emoncms/Modules/device module ------------------------------------------
- git branch: master - git tags: 1.1.1-50-g7dd5c26
- no local changes - running: git pull origin master
From https://github.com/emoncms/device * branch master → FETCH_HEAD
Already up-to-date.
------------------------------------------
Updating /var/www/emoncms/Modules/graph module
------------------------------------------ - git branch: stable
- git tags: 1.2.1-151-gc7f4ea9 - no local changes
- running: git pull origin stable
From GitHub - emoncms/graph: Versatile graphing module for emoncms
* branch stable -> FETCH_HEAD Already up-to-date.
------------------------------------------ Updating /var/www/emoncms/Modules/wifi module ------------------------------------------
- git branch: stable - git tags: 1.2.0-39-gb67783d
- no local changes - running: git pull origin stable
From https://github.com/emoncms/wifi * branch stable → FETCH_HEAD
Already up-to-date.
------------------------------------------
Updating postprocess module
------------------------------------------ - git branch: emonpi
- git tags: 2.1.0-3-g558173e - no local changes
- running: git pull origin emonpi
Already up-to-date.
Already on ‘emonpi’
Your branch is up-to-date with ‘remotes/origin/emonpi’.
------------------------------------------
Updating sync module ------------------------------------------
- git branch: master - git tags: 2.0.1-11-ge7dcf33
- no local changes - running: git pull origin master
Already up-to-date.
Already on 'master' Your branch is up-to-date with 'origin/master'.
------------------------------------------ Updating backup module ------------------------------------------
- git branch: stable - git tags: 2.0.0-12-g67a1f1d
- no local changes - running: git pull origin stable
Already up-to-date. Already on 'stable' Your branch is up-to-date with 'origin/stable'.
------------------------------------------ Auto Installation of Emoncms Modules ------------------------------------------
- graph module already installed - device module already installed
- postprocess module already installed - sync module already installed
- backup module already installed
www-data : www-data video systemd-journal
------------------------------------------
SERVICES
------------------------------------------ emoncms_mqtt.service already installed feedwriter.service already installed service-runner.service already installed ------------------------------------------
Reloading systemctl deamon
Update Emoncms database

Restarting Services…
- sudo systemctl restart feedwriter.service — ActiveState=active —
- sudo systemctl restart emoncms_mqtt.service — ActiveState=active —
- sudo systemctl restart emonhub.service — ActiveState=active —
- sudo /etc/init.d/openhab restart Restarting openhab (via systemctl): openhab.service. openhab.service - Starts and stops the openHAB Home Automation Bus Loaded: loaded (/usr/lib/systemd/system/openhab.service; enabled) Active: active (running) since Wed 2019-07-24 23:21:07 NZST; 88ms ago Docs: http://www.openhab.org Process: 28667 ExecStartPre=/usr/share/openhab/bin/setpermissions.sh (code=exited, status=0/SUCCESS) Main PID: 28715 (openhab.sh) CGroup: /system.slice/openhab.service ├─28715 /bin/sh /usr/share/openhab/bin/openhab.sh -o └─28737 /usr/bin/java -Dlogback.configurationFile=/etc/openhab/logback.xml -Dosgi.clean=true -Declipse.ignoreApp=true -Dosgi.noShutdown=true -Djetty.port=8080 -Dopenhab.configfile=/etc/openhab/configurations/openhab.cfg -Dopenhab.configdir=/etc/openhab/configurations -Dopenhab.logdir=/var/log/openhab -Dsmarthome.userdata=/var/lib/openhab -Djetty.home=/usr/share/openhab -Djetty.port.ssl=8443 -Djetty.config=/etc/openhab/jetty -Djetty.logs=/var/log/openhab -Djetty.rundir=/usr/share/openhab -Dfelix.fileinstall.dir=/usr/share/openhab/addons -Dfelix.fileinstall.filter=.*\.jar -Djava.library.path=/usr/share/openhab/lib -Djava.security.auth.login.config=/etc/openhab/login.conf -Dorg.quartz.properties=/etc/openhab/quartz.properties -Dequinox.ds.block_timeout=240000 -Dequinox.scr.waitTimeOnBlock=60000 -Dfelix.fileinstall.active.level=4 -Djava.awt.headless=true -jar /usr/share/openhab/server/plugins/org.eclipse.equinox.launcher_1.3.0.v20120522-1813.jar -configuration /var/lib/openhab/workspace -data /var/lib/openhab/workspace -console 5555
Jul 24 23:21:07 emonpi setpermissions.sh[28667]: setting owner openhab:openhab recursively for /var/lib/openhab
Jul 24 23:21:07 emonpi setpermissions.sh[28667]: setting permissions and owner openhab:openhab recursively for /var/log/openhab
Jul 24 23:21:07 emonpi setpermissions.sh[28667]: setting permissions and owner openhab:openhab recursively for /etc/openhab/configurations
Jul 24 23:21:07 emonpi setpermissions.sh[28667]: setting permissions and owner openhab:openhab for /usr/share/openhab/webapps/static
Jul 24 23:21:07 emonpi setpermissions.sh[28667]: setting permissions and owner openhab:openhab for /etc/openhab/jetty/etc/keystore
Jul 24 23:21:07 emonpi setpermissions.sh[28667]: setting permissions and owner openhab:openhab for /etc/openhab/configurations/users.cfg
Jul 24 23:21:07 emonpi systemd[1]: Started Starts and stops the openHAB Home Automation Bus.
Symlinking emonpiupdate.log
Installing emoncms logrotate…
Now setting up Logrotate... Backing up old logrotate configuration... Linked to new logrotate configuration... Backing up old logrotate cron job... mv: cannot stat ‘/etc/cron.daily/logrotate’: No such file or directory Linked to new logrotate cron job...
Completed

Completed
Running logrotate...
set log rotate config owner to root
------------------------------------------
Emoncms update script complete
------------------------------------------
Starting emonPi LCD service..
Filesystem is locked - Read Only access
type ’ rpi-rw ’ to unlock
-------------------------------------------------------------
emonPi update done: Wed 24 Jul 23:21:15 NZST 2019
------------------------------------------------------------- restarting service-runner Then I ran sudo systemctl status emonhub.service and got
pi@emonpi(ro):emonhub$ sudo systemctl status emonhub.service
emonhub.service - emonHub service description
Loaded: loaded (/home/pi/emonhub/service/emonhub.service; disabled)
Active: failed (Result: start-limit) since Wed 2019-07-24 23:13:04 NZST; 2min 50s ago
Process: 24955 ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log (code=exited, status=1/FAILURE)
Process: 24952 ExecStartPre=/bin/chmod 775 /var/log/emonhub/ (code=exited, status=0/SUCCESS)
Process: 24949 ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/ (code=exited, status=0/SUCCESS)
Process: 24946 ExecStartPre=/bin/mkdir -p /var/log/emonhub/ (code=exited, status=0/SUCCESS)
Main PID: 24955 (code=exited, status=1/FAILURE)
`
Jul 24 23:13:03 emonpi systemd[1]: Unit emonhub.service entered failed state.
Jul 24 23:13:04 emonpi systemd[1]: emonhub.service holdoff time over, scheduling restart.
Jul 24 23:13:04 emonpi systemd[1]: Stopping emonHub service description…
Jul 24 23:13:04 emonpi systemd[1]: Starting emonHub service description…
Jul 24 23:13:04 emonpi systemd[1]: emonhub.service start request repeated too quickly, refusing to start.
Jul 24 23:13:04 emonpi systemd[1]: Failed to start emonHub service description.
Jul 24 23:13:04 emonpi systemd[1]: Unit emonhub.service entered failed state.

Then I ran

pi@emonpi(ro):emonhub$ sudo systemctl cat emonhub.service
# /home/pi/emonhub/service/emonhub.service [Unit] Description=emonHub service description DefaultDependencies=no Before=shutdown.target Conflicts=shutdown.target Requires=local-fs.target After=sysinit.target syslog.target local-fs.target
[Service]
User=emonhub
PIDFile=/var/run/emonhub.pid
ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhu
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/log/emonhub/
ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/
ExecStartPre=/bin/chmod 775 /var/log/emonhub/
Type=simple
Restart=always
`
[Install]
WantedBy=multi-user.target

Then I ran:

pi@emonpi(ro):emonhub$ ls -la /{etc,lib}/systemd/system/emonhub.service
ls: cannot access /etc/systemd/system/emonhub.service: No such file or directory
lrwxrwxrwx 1 root root 40 Jul 24 23:12 /lib/systemd/system/emonhub.service → /home/pi/emonhub/service/emonhub.service

And last I ran:

pi@emonpi(ro):emonhub$ ls -la /etc/init.d/emonhub
ls: cannot access /etc/init.d/emonhub: No such file or directory

Sorry I’m a bit too tied up to look into this right now, but a brief look at the man page i’m now thinking that “-b” should have been “-B” but I will need to read up some more later. Many moons ago I used to switch branch and then switch back using something like

git checkout -f <someotherbranch>
git checkout <branch>

but I changed that habit when I found a way to reset the branch in one move, shame I can’t recall what that move was now!

Or I think the other reset command is something like

git reset --hard

but I haven’t much experience of using it myself, since there’s nothing you want to retain in the way of local changes, I don’t think it can hurt.

[edit - I was just about to post this when I noticed your edit], so the comments below relate to the updated content]

Looking at your logs above, it seems you now have the systemd symink as the service unit path has changed, in fact the whole service unit content has changed and that would only happen if the repo had updated, so perhaps the reset has worked at some point?

So the last 3 tests above ( from the second “then I ran” on wards) all return the correct info. But the previous “status” doesn’t look right. Firstly it says “disabled” at the end of the loaded line, it should be “enabled”.

Try

sudo systemctl daemon-reload
sudo systemctl --now enable emonhub.service

and then check the status again.

[edit 2] When you post code or logs try using 3x backticks (```) on the lines before and after your code/log, it formats much better. eg

```
code goes here
```

Excellent, thanks for the tip with copying logs. Next time …

First command ran fine, without any output though
Second command fails with

pi@emonpi(ro):emonhub$ sudo systemctl --now enable emonhub.service
systemctl: unrecognized option ‘–now’

It’s for code too, eg

pi@emonpi(ro):emonhub$ sudo systemctl --now enable emonhub.service
systemctl: unrecognized option ‘–now’

would be

pi@emonpi(ro):emonhub$ sudo systemctl --now enable emonhub.service
systemctl: unrecognized option ‘–now’

not a whole lot different with short commands, but the lines do not wrap and certain chars are formatted

eg

<html tag>

using the "quote method comes out as

That’s normal,

That not so much!

Ok the long hand way of doing the same is

sudo systemctl enable emonhub.service
sudo systemctl start emonhub.service

the --now option was not available in earlier systemd versions, what version is your setup running?

systemd --version

[edit] apparently introduced in version 220 systemd - the --now switch of `systemctl` - Unix & Linux Stack Exchange

So I wasn’t going crazy! (on this occasion at least)

From the git-checkout man page

       git checkout -b|-B <new_branch> [<start point>]
           Specifying -b causes a new branch to be created as if git-branch(1) were called and then checked out. In this case you
           can use the --track or --no-track options, which will be passed to git branch. As a convenience, --track without -b
           implies branch creation; see the description of --track below.

           If -B is given, <new_branch> is created if it doesnât exist; otherwise, it is reset. This is the transactional
           equivalent of

               $ git branch -f <branch> [<start point>]
               $ git checkout <branch>

           that is to say, the branch is not reset/created unless "git checkout" is successful.

So I guess that command should have been

git checkout  -f -B emon-pi
> pi@emonpi(rw):service$ systemd --version
systemd 215
+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR

returns:

> pi@emonpi(rw):emonhub$ sudo systemctl enable emonhub.service
Failed to execute operation: No such file or directory

Should I go back and to a checkout with -B …?

So the version explains why the --now didn’t work, but why the enable command returns “No such file or directory” I know not!

I thought we had proved the service unit had changed and the links were there! If sudo systemctl cat emonhub.service and sudo systemctl status emonhub.service work, how can sudo systemctl enable emonhub.service be “No such file or directory”? How odd!

Can you run these commands (again) just to check all is still in place?

sudo systemctl cat emonhub.service
ls -la /{etc,lib}/systemd/system/emonhub.service
ls -la /usr/local/bin/emonhub
ls -la /usr/local/bin/emonhub/
cat /home/pi/emonhub/service/emonhub.service

top-tip, you can copy and paste the whole 5 lines to the command line of your SSH terminal and then just copy and paste the whole caboodle back again rather than doing them individually.

I had thought the repo was reset, perhaps do a

git status

before and after the “-B” command so we know whether it is up to date before and after running it.

First, many thanks for your help, I’d be lost without you!

> pi@emonpi(rw):~$ sudo systemctl cat emonhub.service
# /home/pi/emonhub/service/emonhub.service
[Unit]
Description=emonHub service description
DefaultDependencies=no
Before=shutdown.target
Conflicts=shutdown.target
Requires=local-fs.target
After=sysinit.target syslog.target local-fs.target

[Service]
User=emonhub
PIDFile=/var/run/emonhub.pid
ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhu
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/log/emonhub/
ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/
ExecStartPre=/bin/chmod 775 /var/log/emonhub/
Type=simple
Restart=always

[Install]
WantedBy=multi-user.target
lines 1-22/22 (END)pi@emonpi(rw):~$

2nd command: Ahh, error:

pi@emonpi(rw):emonhub$ ls -la /{etc,lib}/systemd/system/emonhub.service
ls: cannot access /etc/systemd/system/emonhub.service: No such file or directory
lrwxrwxrwx 1 root root 40 Jul 24 23:12 /lib/systemd/system/emonhub.service -> /home/pi/emonhub/service/emonhub.service

3rd command:

pi@emonpi(rw):emonhub$ ls -la /usr/local/bin/emonhub
lrwxrwxrwx 1 root staff 20 Jul 23 15:28 /usr/local/bin/emonhub -> /home/pi/emonhub/src

4th command:

pi@emonpi(rw):emonhub$ ls -la /usr/local/bin/emonhub/
total 80
drwxr-xr-x 4 pi   pi    4096 Jul 25 00:41 .
drwxr-xr-x 7 pi   pi    4096 Jul 24 23:12 ..
-rw-r--r-- 1 pi   pi    1142 Jul 24 23:12 Cargo.py
-rw-r--r-- 1 pi   pi    2834 Jul 24 23:12 emonhub_buffer.py
-rw-r--r-- 1 pi   pi    1148 Jan 26  2016 emonhub_coder.py
-rw-r--r-- 1 pi   pi   25255 Jul 24 23:12 emonhub_interfacer.py
-rwxr-xr-x 1 pi   pi   12341 Jul 24 23:12 emonhub.py
-rw-r--r-- 1 pi   pi    5219 Mar 29  2017 emonhub_setup.py
drwxr-xr-x 3 pi   pi    4096 Jul 24 23:12 interfacers
drwxr-xr-x 2 pi   pi    4096 Jul 13  2017 smalibrary
lrwxrwxrwx 1 root root    20 Jul 25 00:41 src -> /home/pi/emonhub/src

and 5th command:

pi@emonpi(rw):emonhub$ cat /home/pi/emonhub/service/emonhub.service
[Unit]
Description=emonHub service description
DefaultDependencies=no
Before=shutdown.target
Conflicts=shutdown.target
Requires=local-fs.target
After=sysinit.target syslog.target local-fs.target

[Service]
User=emonhub
PIDFile=/var/run/emonhub.pid
ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/log/emonhub/
ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/
ExecStartPre=/bin/chmod 775 /var/log/emonhub/
Type=simple
Restart=always

[Install]
WantedBy=multi-user.target

And the GIT status:

pi@emonpi(rw):emonhub$ git status
On branch emon-pi
Your branch is up-to-date with 'origin/emon-pi'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        src/src

nothing added to commit but untracked files present (use "git add" to track)

should have read your top-tip first, silly me!

Ahh, this time the ls commands differ. Back then I ran

s -la /usr/{local/bin,share}/emonhub
ls -la /etc/{init.d/,}emonhub

but this time it was:

ls -la /{etc,lib}/systemd/system/emonhub.service
ls -la /usr/local/bin/emonhub

i’m not entirely sure we’re not lost even with my input :smile:

No that’s a good error. If there was a service unit in /etc/systemd/system it would override anything we are doing to the correctly located unit in /lib/systemd/system.

So I’m at a loss as too why the path/file is found by systemctl for other commands but not for “enable” apparently! Everything looks as it should as far as I can tell (aside form that rogue “src” folder in the emonhub/src folder).

There is an option to force an “enable” but I don’t think that will over come a file perceived as absent.

sudo systemctl -f enable emonhub.service

Assuming no change, what does trying to start the service do?

sudo systemctl start emonhub.service

if there is no error, what is the status after that?

sudo systemctl status emonhub.service

Yeah, just slightly different to test different files and paths.

WAY less lost!

gets error:

Failed to execute operation: No such file or directory

gets no output, but the

produces the same as before:


pi@emonpi(ro):system$ sudo systemctl status emonhub.service
● emonhub.service - emonHub service description
   Loaded: loaded (/home/pi/emonhub/service/emonhub.service; disabled)
   Active: failed (Result: start-limit) since Thu 2019-07-25 01:24:51 NZST; 7s ago
  Process: 12425 ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log (code=exited, status =1/FAILURE)
  Process: 12422 ExecStartPre=/bin/chmod 775 /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 12419 ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 12416 ExecStartPre=/bin/mkdir -p /var/log/emonhub/ (code=exited, status=0/SUCCESS)
 Main PID: 12425 (code=exited, status=1/FAILURE)

Jul 25 01:24:51 emonpi systemd[1]: emonhub.service holdoff time over, sched...t.
Jul 25 01:24:51 emonpi systemd[1]: Stopping emonHub service description...
Jul 25 01:24:51 emonpi systemd[1]: Starting emonHub service description...
Jul 25 01:24:51 emonpi systemd[1]: emonhub.service start request repeated t...t.
Jul 25 01:24:51 emonpi systemd[1]: Failed to start emonHub service description.
Jul 25 01:24:51 emonpi systemd[1]: Unit emonhub.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.

Something quite different crossed my mind: Earlier this year, my backup.conf failed because of # comments no longer supported by PHP (I think). Could it be the same for emonhub.conf?

No, that was a change to php and how it reads ini files, emonhub is Python and uses configobj for it’s conf file formatting.

I really have to concentrate on something else for a while, sorry. You could try running a full update again now we know the repo is reset. As much as I hate suggesting it, you could also try a reboot as that might dislodge something. Otherwise i will need to comeback to this and go through a full list of checks to determine exactly what we do or don’t have right now. That also give me a chance to mull over the odd fact that systemctl can execute some commands but not all (eg enable) with the exact same file apparently “not found” in some instances.

Will do that and let you know.

FYI, I ran this again, this time without lines ellipsized:

pi@emonpi(ro):~$ sudo systemctl status emonhub.service -l
● emonhub.service - emonHub service description
   Loaded: loaded (/home/pi/emonhub/service/emonhub.service; disabled)
   Active: failed (Result: start-limit) since Thu 2019-07-25 01:24:51 NZST; 26min ago
  Process: 12425 ExecStart=/usr/local/bin/emonhub/emonhub.py --config-file=/etc/emonhub/emonhub.conf --logfile=/var/log/emonhub/emonhub.log (code=exited, status=1/FAILURE)
  Process: 12422 ExecStartPre=/bin/chmod 775 /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 12419 ExecStartPre=/bin/chgrp -R emonhub /var/log/emonhub/ (code=exited, status=0/SUCCESS)
  Process: 12416 ExecStartPre=/bin/mkdir -p /var/log/emonhub/ (code=exited, status=0/SUCCESS)
 Main PID: 12425 (code=exited, status=1/FAILURE)

Jul 25 01:24:51 emonpi systemd[1]: emonhub.service holdoff time over, scheduling restart.
Jul 25 01:24:51 emonpi systemd[1]: Stopping emonHub service description...
Jul 25 01:24:51 emonpi systemd[1]: Starting emonHub service description...
Jul 25 01:24:51 emonpi systemd[1]: emonhub.service start request repeated too quickly, refusing to start.
Jul 25 01:24:51 emonpi systemd[1]: Failed to start emonHub service description.
Jul 25 01:24:51 emonpi systemd[1]: Unit emonhub.service entered failed state.
pi@emonpi(ro):~$

It’s 2AM here now (New Zealand), we may have to plan a good time for you to continue this. Anytime is fine with me of course. Good night, and many thanks for your help.