Problems after updating to emoncms 10.1.13

I just updated from 10.1.5 to 10.1.13 and now lots of things don’t work :frowning:
The Input screen still shows all the defined inputs but claims no data has been received. emonhub.log shows data being received succesfully. Here is the Admin screen:

Server Information

Server Information

Services

  • emonhub :- Active Exited
  • mqtt_input :- Activating Auto-restart
  • feedwriter :- Active ExitedService is not running0 feed points pending write
  • emonPiLCD :- Active Exited
  • redis-server :- Active Running
  • mosquitto :- Active Running

Emoncms

Server

  • OS :- Linux 4.4.26-v7+
  • Host :- Pi3 | Pi3 | (192.168.0.18)
  • Date :- 2019-12-11 23:00:26 UTC
  • Uptime :- 23:00:26 up 33 min, 1 user, load average: 0.13, 0.17, 0.18

Memory

  • RAM :- Used: 25.84%
    • Total :- 973.11 MB
    • Used :- 251.4 MB
    • Free :- 721.7 MB
      Write Load Period

Disk

  • / :- Used: 88.20%
    • Total :- 3.33 GB
    • Used :- 2.94 GB
    • Free :- 237.13 MB
    • Write Load :- 0 B/s (30 mins)
  • /boot :- Used: 34.76%
    • Total :- 59.95 MB
    • Used :- 20.84 MB
    • Free :- 39.11 MB
    • Write Load :- 0 B/s (30 mins)
  • /home/pi/data :- Used: 77.09%
    • Total :- 193.66 MB
    • Used :- 149.3 MB
    • Free :- 34.37 MB
    • Write Load :- 2.82 B/s (30 mins)

HTTP

  • Server :- Apache/2.4.10 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 5.5.52-0+deb8u1
  • Host :- localhost:6379 (127.0.0.1)
  • Date :- 2019-12-11 23:00:26 (UTC 00:00‌​)
  • Stats :- Uptime: 1967 Threads: 2 Questions: 787 Slow queries: 0 Opens: 60 Flush tables: 1 Open tables: 53 Queries per second avg: 0.400

Redis

  • Version :-
    • Redis Server :- 2.8.17
    • PHP Redis :- 2.2.7
  • Host :- localhost:6379
  • Size :- 173 keys (480.35K)
  • Uptime :- 0 days

MQTT Server

  • Version :- Mosquitto 1.4.10
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 5.6.27-0+deb8u1 (Zend Version 2.6.0)
  • Modules :- apache2handler | bcmath | bz2 | calendar | Core v5.6.27-0+deb8u1 | ctype | curl | date v5.6.27-0+deb8u1 | dba | dio v0.0.4RC4 | dom v20031129 | ereg | exif v1.4 $Id: 8b0e34c10dc8a04b8e81d9d79985b2566141b03d $ | fileinfo v1.0.5 | filter v0.11.0 | ftp | gettext | hash v1.0 | iconv | json v1.3.6 | libxml | mbstring | mcrypt | mhash | mosquitto v0.3.0 | mysql v1.0 | mysqli v0.1 | openssl | pcre | PDO v1.0.4dev | pdo_mysql v1.0.2 | Phar v2.0.2 | posix | readline v5.6.27-0+deb8u1 | redis v2.2.7 | Reflection v$Id: 5f15287237d5f78d75b19c26915aa7bd83dee8b8 $ | session | shmop | SimpleXML v0.1 | soap | sockets | SPL v0.2 | standard v5.6.27-0+deb8u1 | sysvmsg | sysvsem | sysvshm | tokenizer v0.1 | wddx | xml | xmlreader v0.1 | xmlwriter v0.1 | Zend OPcache v7.0.6-devFE | zip v1.12.5 | zlib v2.0

Pi

  • Model :- Raspberry Pi 3 Model B Rev 1.2 - 1GB (Sony UK)

  • Serial num. :- 8574B2A6

  • Temperature :- 51.54°C - 51.5°C

  • emonpiRelease :- emonSD-07Nov16

  • File-system :- read-only

Client Information

Client Information

HTTP

  • Browser :- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0
  • Language :- en-GB,en;q=0.5

Window

  • Size :- 1263 x 597

Screen

  • Resolution :- 1280 x 720

I am using an Pi 3 emonbase +rfm69pi and logging locally on this unit.

I only updated as I was getting the error " Received data not in correct format" in the graphs module when trying to display data, as reported here and it seemed updating to 10.1.13 was going to help.

I accessed the Admin page, and chose to only update emoncms (if it ain’t broke don’t fix it), although initially the page advised that /var/log/emoncms/emoncms.log was missing so I created it, went out and back in to the Admin page, and that error went away. I then clicked to update only emoncms, and watched as message filled the log window.

I was alarmed to see the log window show messages related to updating emonpi firmware, especially as this is running on an emonbase with rfm69pi and not an emonpi (and as I was not updating everything I had not bothered to select rfm69pi in the drop down box).
The update process ran for a while, and eventually stopped with a message that service runner had been restarted, and did nothing else. I was SSH’d into the pi and ran top which seemed to show nothing was happening. Back in the admin page, under Services, only redis-server and mosquitto were running, the rest showed Active Exited.
I checked service-runner.log and one of the last messages was something like

mount / device busy.

I assumed something had got stuck, and so thought I would try a reboot, which I triggered from the emoncms admin page. But after the reboot, nothing changed - most services are still not started. And, the /var/log/emoncms directory has dissapeared, along with the log file, so I can’t show you what really happened.

Incidentally, yesterday I had to create /var/log/emoncms directory and postprocess.log as it was also missing, which prevented me from being able to use the post-process module, but once I created it, that module appeared to work. Obviously that has gone as the directory has gone.

I am running emonSD-07Nov16 which I know is a bit old, but the emoncms update log file did show it had checked and was happy with this version.

Any suggestions?

I’ve managed to get feed-writer working by restarting it manually, and searching the forum I found this this post that showed the same error regarding mqtt_input Mqtt_input service down: 'activating (auto-restart)' state
I have the same error in /var/log/daemon.log:

mqtt_input[16236]: PHP Warning: require(Lib/EmonLogger.php): failed to open stream: No such file or directory in /var/www/emoncms/scripts/phpmqtt_input.php on line 47
but the file exists:

ls -l /var/www/emoncms/Lib/EmonLogger.php
-rw-r–r-- 1 pi pi 2371 Dec 11 22:11 /var/www/emoncms/Lib/EmonLogger.php

Any help appreciated

How long has it been running?

Can you post the update log please.

No feed will appear as Feedwriter service is not running and neither is service_runner

can you post the output of this command please

sudo systemctl status {emonhub.service,feedwriter.service,mqtt_input.service,service_runner.service}

Hi @borpin, thanks for your reply.
This system has been running this version for a couple of years.
I have now found the update.log in ~/data/emonpiupdate.log and so update the post to add the log below.
I did get feedwriter running in my 2nd post, and service-runner process was always running.

Output from the requested command:

pi@Pi3(ro):log$ sudo systemctl status {emonhub.service,feedwriter.service,mqtt_input.service,service_runner.service}
● emonhub.service - LSB: Start/stop emonHub
   Loaded: loaded (/etc/init.d/emonhub)
   Active: active (exited) since Wed 2019-12-11 22:27:43 GMT; 24h ago

Dec 11 22:27:42 Pi3 systemd[1]: Starting LSB: Start/stop emonHub...
Dec 11 22:27:43 Pi3 sudo[3214]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/mkdir -p /var/log/emonhub
Dec 11 22:27:43 Pi3 sudo[3214]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 11 22:27:43 Pi3 sudo[3214]: pam_unix(sudo:session): session closed for user root
Dec 11 22:27:43 Pi3 sudo[3224]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/touch /var/log/emonhub/emonhub.log
Dec 11 22:27:43 Pi3 sudo[3224]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 11 22:27:43 Pi3 sudo[3224]: pam_unix(sudo:session): session closed for user root
Dec 11 22:27:43 Pi3 sudo[3237]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/chown -R emonhub /var/log/emonhub
Dec 11 22:27:43 Pi3 sudo[3237]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 11 22:27:43 Pi3 sudo[3237]: pam_unix(sudo:session): session closed for user root
Dec 11 22:27:43 Pi3 sudo[3249]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/chown -R emonhub /var/log/emonhub
Dec 11 22:27:43 Pi3 sudo[3249]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 11 22:27:43 Pi3 sudo[3249]: pam_unix(sudo:session): session closed for user root
Dec 11 22:27:43 Pi3 emonhub[3210]: Starting OpenEnergyMonitor emonHub: emonhub has been started ok.
Dec 11 22:27:43 Pi3 systemd[1]: Started LSB: Start/stop emonHub.
Dec 11 23:15:33 Pi3 systemd[1]: Started LSB: Start/stop emonHub.

● feedwriter.service - LSB: feedwriter script daemon
   Loaded: loaded (/etc/init.d/feedwriter)
   Active: active (running) since Wed 2019-12-11 23:34:39 GMT; 23h ago
  Process: 11141 ExecStop=/etc/init.d/feedwriter stop (code=exited, status=0/SUCCESS)
  Process: 11159 ExecStart=/etc/init.d/feedwriter start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/feedwriter.service
           └─11293 /usr/bin/php -f /var/www/emoncms/scripts/feedwriter.php

Dec 11 23:34:32 Pi3 feedwriter[11159]: Log is turned off
Dec 11 23:34:32 Pi3 feedwriter[11159]: Starting RPI
Dec 11 23:34:39 Pi3 systemd[1]: Started LSB: feedwriter script daemon.

● mqtt_input.service - Emoncms MQTT Input Script
   Loaded: loaded (/etc/systemd/system/mqtt_input.service; enabled)
   Active: activating (auto-restart) (Result: exit-code) since Thu 2019-12-12 23:12:56 GMT; 40s ago
     Docs: https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
  Process: 4943 ExecStart=/usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php (code=exited, status=255)
 Main PID: 4943 (code=exited, status=255)

Dec 12 23:12:56 Pi3 systemd[1]: mqtt_input.service: main process exited, code=exited, status=255/n/a
Dec 12 23:12:56 Pi3 systemd[1]: Unit mqtt_input.service entered failed state.

● service_runner.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

As service_runner was not found, but ps-ef shows there is a service-runner, I tried the command on that:

pi@Pi3(ro):log$ ps -ef |grep -i service
pi        7223  7219  0 Dec11 ?        00:00:00 /bin/sh -c /home/pi/emonpi/service-runner >> /var/log/service-runner.log 2>&1
pi        7224  7223  2 Dec11 ?        00:31:03 /bin/bash /home/pi/emonpi/service-runner
pi       10247   341  0 23:17 pts/0    00:00:00 grep --color=auto -i service
pi@Pi3(ro):log$ sudo systemctl status {service-runner.service}
● \x7bservice-runner.service\x7d.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

emonpiupdate.log - which shows more errors than appeared in the log window at the time of the update.

Starting emonPi Update >
via service-runner-update.sh
Service Runner update script V1.1.1
EUID: 1000
Argument:
Wed 11 Dec 22:28:23 GMT 2019
#############################################################

emonSD version: emonSD-07Nov16

emonSD base image check passed...continue update

#############################################################
I2C LCD NOT DETECTED on either 0x['27', '3f'] ...exiting LCD script
Filesystem is unlocked - Write access
type ' rpi-ro ' to lock
git pull /home/pi/emonpi
* master
  wifiap
On branch master
Your branch is behind 'origin/master' by 270 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   rc.local_jessieminimal

no changes added to commit (use "git add" and/or "git commit -a")
error: Your local changes to the following files would be overwritten by merge:
        rc.local_jessieminimal
Please, commit your changes or stash them before you can merge.
Aborting
Updating d2be4f4..0cbdb5a
git pull /home/pi/RFM2Pi
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/emonhub
  K0den-wibeee_interface
* emon-pi
  hmm01i-syslogging
On branch emon-pi
Your branch is up-to-date with 'origin/emon-pi'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/oem_openHab
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
Already up-to-date.
git pull /home/pi/usefulscripts
* master
On branch master
Your branch is behind 'origin/master' by 40 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   update_emoncms.sh

no changes added to commit (use "git add" and/or "git commit -a")
error: Your local changes to the following files would be overwritten by merge:
        update_emoncms.sh
Please, commit your changes or stash them before you can merge.
Aborting
Updating 7f1ac2a..ea00787
git pull /home/pi/huawei-hilink-status
* master
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
error: cannot open .git/FETCH_HEAD: Permission denied


Start emonPi Atmega328 firmware update:

=================================
EmonPi update started
=================================

EUID: 1000

Requirement already up-to-date: paho-mqtt in /usr/local/lib/python2.7/dist-packages
You are using pip version 8.1.2, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Stopping OpenEnergyMonitor emonHub: emonhub has been stopped ok.
Start ATmega328 serial upload using avrdude with latest.hex
Discrete Sampling
avrdude -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 115200 -U flash:w:/home/pi/emonpi/firmware/compiled/latest.hex
avrdude-original: Using autoreset DTR on GPIO Pin 7
avrdude-original: stk500_getsync() attempt 1 of 10: not in sync: resp=0xfc
avrdude-original: stk500_getsync() attempt 2 of 10: not in sync: resp=0xe0
avrdude-original: stk500_getsync() attempt 3 of 10: not in sync: resp=0x8e
avrdude-original: stk500_getsync() attempt 4 of 10: not in sync: resp=0x1c
avrdude-original: stk500_getsync() attempt 5 of 10: not in sync: resp=0x8e
avrdude-original: stk500_getsync() attempt 6 of 10: not in sync: resp=0xe0
avrdude-original: stk500_getsync() attempt 7 of 10: not in sync: resp=0x0e
avrdude-original: stk500_getsync() attempt 8 of 10: not in sync: resp=0xf8
avrdude-original: stk500_getsync() attempt 9 of 10: not in sync: resp=0xe0
avrdude-original: stk500_getsync() attempt 10 of 10: not in sync: resp=0xf0

avrdude-original done.  Thank you.

strace: |autoreset: Broken pipe
Starting OpenEnergyMonitor emonHub: emonhub has been started ok.
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

+ [email protected]
updated 1 package and audited 1031 packages in 21.286s
found 2 vulnerabilities (1 moderate, 1 high)
  run `npm audit fix` to fix them, or `npm audit` for details
npm WARN checkPermissions Missing write access to /home/pi/data/node-red/node_modules/node-red-node-emoncms
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

npm ERR! path /home/pi/data/node-red/node_modules/node-red-node-emoncms
npm ERR! code EACCES
npm ERR! errno -13
npm ERR! syscall access
npm ERR! Error: EACCES: permission denied, access '/home/pi/data/node-red/node_modules/node-red-node-emoncms'
npm ERR!  { Error: EACCES: permission denied, access '/home/pi/data/node-red/node_modules/node-red-node-emoncms'
npm ERR!   stack: 'Error: EACCES: permission denied, access \'/home/pi/data/node-red/node_modules/node-red-node-emoncms\'',
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'access',
npm ERR!   path: '/home/pi/data/node-red/node_modules/node-red-node-emoncms' }
npm ERR!
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR!
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator (though this is not recommended).

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/pi/.npm/_logs/2019-12-11T22_29_16_210Z-debug.log


Start emonhub update script:

=================================
EmonPi update started
=================================
Running emonhub automatic node addition script
EUID: 1000
emonhub location does not exist

Start emoncms update:

=================================
Emoncms update started
Emoncms update script V1.2.0

Wed 11 Dec 22:29:16 GMT 2019

#############################################################

emonSD version: emonSD-07Nov16

emonSD base image check pass...continue update

#############################################################
EUID: 1000
Checking cron tab for service runner entry...
service runner crontab entry already installed

current settings.php md5: 59e80f731935867f5f93753d49c0fe86
Default settings.php md5: 59e80f731935867f5f93753d49c0fe86

git pull /var/www/emoncms
  dev-mosquitto-php
  master
* stable
  symlinked_modules
On branch stable
Your branch is up-to-date with 'origin/stable'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        settings.php_7-11-2016

nothing added to commit but untracked files present (use "git add" to track)
Already up-to-date.

NEW default settings.php md5: 59e80f731935867f5f93753d49c0fe86
settings.php has NOT been user modifed
settings.php not updated


git pull /var/www/emoncms/Modules/app
  9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Already on 'stable'
Your branch is up-to-date with 'origin/stable'.

git pull /var/www/emoncms/Modules/config
  9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Already on 'stable'
Your branch is up-to-date with 'origin/stable'.

git pull /var/www/emoncms/Modules/wifi
  9.0
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Already on 'stable'
Your branch is up-to-date with 'origin/stable'.
git pull /var/www/emoncms/Modules/dashboard
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Already on 'stable'
Your branch is up-to-date with 'origin/stable'.

git pull /var/www/emoncms/Modules/graph
  master
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Already on 'stable'
Your branch is up-to-date with 'origin/stable'.

git pull /home/pi/postprocess
Already on 'emonpi'
Your branch is up-to-date with 'remotes/origin/emonpi'.
Already up-to-date.
git pull /home/pi/sync
Already on 'master'
Your branch is up-to-date with 'origin/master'.
Already up-to-date.
git pull /home/pi/backup
  master
* stable
On branch stable
Your branch is up-to-date with 'origin/stable'.
nothing to commit, working directory clean
Already up-to-date.
Already on 'stable'
Your branch is up-to-date with 'origin/stable'.

update mqtt_input systemd unit file

Update Emoncms database
[]

Restarting Services...
Restarting OpenEnergyMonitor emonHub: emonhub has been restarted ok.
Log is turned off
Restarting feedwriter

Restarting openhab (via systemctl): openhab.service.


set log rotate config owner to root
Restarting Services...


Starting emonPi LCD service..

Filesystem is locked - Read Only access
type ' rpi-rw ' to unlock
Wed 11 Dec 22:30:04 GMT 2019


...................
emonPi update done
restarting service-runner\n

I managed to get emonhub started. Although using the ‘service’ command would display emonhub as active/exited, using ps -ef, I could see emonhub was running.I tried using /etc/init.d/emonhub stop to stop emonhub, as well as sudo service emonhub stop, and then started it just using sudo service emonhub start and this time it worked, and the admin page showed emonhub working.

That left mqtt, and I tried the fix mentioned in Mqtt_input service down: ‘activating (auto-restart)’ of updating line 46, and thenwaiting for the next auto-restart, and then mqtt was active.
I still didn’t have any data appearing on the inputs page, so I stopped and restarted emonhub service, and then the inputs re-appeared, as well as the feeds.

It’s quite possible this installation is getting a bit creaky, but it’s odd/interesting that the fix sugested by @fcattaneo seems to have solved the problem for me as well.

For future reference, when posting code or bash output, put in 3 ‘backticks’ (found at the top left of the keyboard normally) on a line on their own, then the code, then 3 more backticks on a line following the code.

    ```
   put your code here
    ```

If it is something like php you can add a language identifier that after the first 3 backticks so ```php

Here is the first problem. This means your update script has not been updated.

I would suggest taking a backup first.

Fix the rc.local_jessieminimal issue and update.

However, this is a Jessie image, 2 releases behind the current Buster and out of support.

I strongly suggest you take a backup, get a new SD card, and import the backup.

There are system changes that mean Jessie based systems are not really supportable.

Hi @borpin thanks for your response. I did see the pop up message about backticks, but to be honest I didn’t understand what it was trying to tell me. I guess this is a feature of the forum that I haven’t seen before. I’ve now updated my post to use back ticks, just for the practice and maybe to help others that might read my long post.

Regarding the issue, and once I found the update log, (which I thought had been lost from the update, but turned out I was looking in the wrong place) it’s clear there are a number of issues. A few months ago I set out to test the upgrade to the newer SD card image on my Pi3, before applying the upgrade to my Pi Model B which is my ‘production’ system. However this work has been side-tracked by the prep and installation of the Powervault 3, so i’ll have to limp on a bit longer. Thanks for your assistance.

Because of the differences in the architecture this is not good practice. I think Raspbian does some clever things on first boot to make the distribution common across different boards and architectures (from Pi A to Pi 4 including Pi Zero).

My suggestion for a Pi B, would be to use the EmonScript process installing Raspbian and running the scripts actually on the Pi B.

Nothing complicated, just some text entries in config.txt
(maybe something OEM should be using?)

Ref: Raspberry Pi Documentation - Configuration

Conditional filters in config.txt

When a single SD card (or card image) is being used with one Pi and one monitor, it is easy to set config.txt as required for that specific combination and keep it that way, amending it only when something changes.

However, if one Pi is swapped between different monitors, or if the SD card (or card image) is being swapped between multiple Pis, a single set of settings may no longer be sufficient. Conditional filters allow you to define certain sections of the config file to be used only in specific cases, allowing a single config.txt to create different configurations when read by different hardware.

e.g.:
image

1 Like

FWIW, on my system:

$ systemctl status service_runner.service
Unit service_runner.service could not be found.

But the system is running normally and according to http://192.168.1.24/emoncms/admin/view

service-runner **Active** Running

and

$ ls -l /etc/systemd/system/service-runner.service 
lrwxrwxrwx 1 root root 71 Oct  3  2018 /etc/systemd/system/service-runner.service -> /var/www/emoncms/scripts/services/service-runner/service-runner.service

So dunno what the issue is. My system is low-write 9.9.8