Community
OpenEnergyMonitor

Community

Another EmonCMS graphs stopped working question

Tags: #<Tag:0x00007fa343c35530>

Hi chaps,
I’ve got another instance of the graphs stopped working days ago issue.
This time it’s not browser cache related (first thing I tried) or lack of disk space (plenty free).
The feedwriter service is running fine and all the data is being stored locally on the Pi.

The only thing I’ve done to this Pi recently is I ran an “apt update/upgrade” on it to pull the OS up to date.
Everything seems to be running OK, the main dash is up to date, the inputs are arriving and look like they are getting logged into the feeds, but the charts all stop four days ago.

As a last resort, I’ve rebooted the Pi, but it doesn’t seem to have made any difference.
Any ideas?

Screenshot%20from%202019-07-29%2009-11-12

Was that around the time the data appears to have stopped?

Are there any error messages logged by emoncms? Can you check the admin page are all the services running? Can you post the “server info” ?

Did you do a “update & upgrade” or a “update & dist-upgrade” ? (any other commands used eg rpi-update)

Are any of the inputs duplicated on the inputs page?

Hi Paul,

I can’t say 100%, but they were both some time that evening. :frowning:

The full EmonCMS log is -

2019-07-29 07:46:05.066|WARN|phpmqtt_input.php|Not connected, retrying connection
2019-07-29 07:46:05.117|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0
2019-07-29 07:46:06.877|WARN|phpmqtt_input.php|Not connected, retrying connection
2019-07-29 07:46:06.937|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0
2019-07-29 07:47:05.502|ERROR|input_controller.php|{“success”: false, “message”: “Format error: csv value is not numeric”} for User: 1
2019-07-29 07:47:05.511|ERROR|input_controller.php|{“success”: false, “message”: “Format error: csv value is not numeric”} for User: 1

Which is all this morning before I found the problem.
Yes, everything looked like it was running OK before I rebooted.
Server Info below (yes, it’s an old version, I’m still playing with the new version on a different Pi)

Server Information
Emoncms Version low-write 9.8.30 : 2018.05.08
Modules Administration : App v1.0.0 : Backup v1.1.2 : Config v1.0.0 : Dashboard v1.1.1 : Device v1.0.0 : EventProcesses : Feed : Graph v1.2.0 : Input : CoreProcess : Schedule : sync : Time : User : Visualisation : wifi
Buffer loading…
Writer Daemon is running with sleep 600s
Server OS Linux 4.9.35-v7+
Host emonpi emonpi (10.0.0.4)
Date 2019-07-29 11:56:30 BST
Uptime 11:56:30 up 3:10, 1 user, load average: 0.47, 0.47, 0.46
HTTP Server Apache/2.4.10 (Raspbian) HTTP/1.1 CGI/1.1 80
MySQL Version 5.5.62-0+deb8u1
Host localhost (127.0.0.1)
Date 2019-07-29 11:56:30 (UTC 01:00‌​)
Stats Uptime: 11444 Threads: 3 Questions: 39095 Slow queries: 0 Opens: 60 Flush tables: 1 Open tables: 53 Queries per second avg: 3.416
Redis Version 2.8.17
Host localhost:6379 (127.0.0.1)
Size 632 keys (675.05K)
Uptime 0 days
MQTT Version
Host localhost:1883 (127.0.0.1)
Pi CPU Temp 42.24°C
Release emonSD-03May16
File-system Set root file-system temporarily to read-write, (default read-only)
Memory RAM Used: 30.54% Total: 970.93 MB Used: 296.52 MB Free: 674.41 MB
Disk Mount Stats
/ Used: 68.92% Total: 3.33 GB Used: 2.3 GB Free: 894.66 MB
/boot Used: 36.27% Total: 59.95 MB Used: 21.74 MB Free: 38.2 MB
/home/pi/data Used: 55.04% Total: 4 GB Used: 2.2 GB Free: 1.59 GB
/media/usb Used: 60.93% Total: 3.61 GB Used: 2.2 GB Free: 1.21 GB
PHP Version 5.6.40-0+deb8u4 (Zend Version 2.6.0)
Modules apache2handler : bcmath : bz2 : calendar : Core v5.6.40-0+deb8u4 : ctype : curl : date v5.6.40-0+deb8u4 : dba : dio v0.0.4RC4 : dom v20031129 : ereg : exif v1.4 : fileinfo v1.0.5 : filter v0.11.0 : ftp : gettext : hash v1.0 : iconv : json v1.3.6 : libxml : mbstring : mcrypt : mhash : mosquitto v0.3.0 : mysql v1.0 : mysqli v0.1 : openssl : pcre : PDO v1.0.4dev : pdo_mysql v1.0.2 : Phar v2.0.2 : posix : readline v5.6.40-0+deb8u4 : redis v4.0.2 : Reflection : session : shmop : SimpleXML v0.1 : soap : sockets : SPL v0.2 : standard v5.6.40-0+deb8u4 : sysvmsg : sysvsem : sysvshm : tokenizer v0.1 : wddx : xml : xmlreader v0.1 : xmlwriter v0.1 : Zend OPcache v7.0.6-devFE : zip v1.12.5 : zlib v2.0 :
Client Information
HTTP Browser Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
Screen Resolution 1600 x 900
Window Size 1543 x 801

No, it was just an update & upgrade. I’ve had issues with dist-upgrades, hence building a new version.
No, no other updates done. It’s odd, I’ve done this several times to this Pi/EmonCMS & not had issues.
No, none of the inputs were duplicated. I’ve had that before too.

Rubbish!
Just checked the graphs again, and it looks like data is being stored again, but I’ve lost 3.5 days of data.
:sob:

These lines might point to something, can you check emonhub.log?

can you also confirm that the mosquitto broker is running ok, I notice there is no “mqtt version” in your server info.

Ahh! something else to check! Mosquitto used to be installed direct from the mosquitto repo’s (not the debian repos) to get the latest version. This has changed in more recent emonSD images. The most recent Mosquitto versions (1.5.7’ish) have breaking changes. When you used apt-get to upgrade it will have pulled in a “broken” version.

Please check the Mosquitto version, if it’s not 1.4.10’ish then I think you need to rollback by removing the current mosquitto broker and then editing the source.list to remove the mosquitto repo url and then re-install from the Debian repos.

[edit - just seen your latest post]

Ok, maybe your Mosquitto wasn’t updated? Still worth checking I think.

[edit2] - also, unrelated but, your sleep setting for the feedwriter is 10 mins (600s), emonpi’s are normally set to 60s. 0mins of data is a lot to be buffering, this can delay data being displayed in graphs etc and/or if there’s a “hiccup” it’s better to lose up to 1 min of data rather than 10mins. This can be set in emoncms/settings.php

Hi Paul,

That’s a good idea, unfortunately they’re been log rotated and the mornings logs are gone.
There’s no repeat of the error in the three logs that currently exist.
I’ll check again tomorrow at 7:47 to see it it reoccurs.

Yes, I noticed that about Mosquitto too. It was running before the reboot, and it’s still running now and has been since the reboot.

[email protected]:/var/log/emonhub $ sudo service mosquitto status
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
Loaded: loaded (/etc/init.d/mosquitto)
Active: active (running) since Mon 2019-07-29 08:45:47 BST; 13h ago
Process: 1569 ExecStop=/etc/init.d/mosquitto stop (code=exited, status=0/SUCCESS)
Process: 1585 ExecStart=/etc/init.d/mosquitto start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/mosquitto.service
└─1592 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

Jul 29 08:45:47 emonpi systemd[1]: Starting LSB: mosquitto MQTT v3.1 message broker…
Jul 29 08:45:47 emonpi mosquitto[1585]: Starting network daemon:: mosquitto.
Jul 29 08:45:47 emonpi systemd[1]: Started LSB: mosquitto MQTT v3.1 message broker.

But this is where it gets sticky…

[email protected]:~ $ dpkg -s mosquitto
Package: mosquitto
Status: install ok installed
Priority: optional
Section: net
Installed-Size: 419
Maintainer: Roger A. Light [email protected]
Architecture: armhf
Multi-Arch: foreign
Version: 1.6.3-0mosquitto1~jessie1
Depends: adduser (>= 3.10), libuuid1, lsb-base (>= 4.1+Debian3), libc6 (>= 2.4), libssl1.0.0 (>= 1.0.0), libwebsockets3 (>= 1.2), libwrap0 (>= 7.6-4~)
Suggests: apparmor
Conffiles:
/etc/init.d/mosquitto 7c1c057c3c625deaac042ba97032ec69
/etc/logrotate.d/mosquitto d02c7d232915df80a983e120d96b3b08
/etc/mosquitto/ca_certificates/README c1c6ae67f2def06c6a483be09b49d4de
/etc/mosquitto/certs/README 4d8a70d4cefab07d4dabc5be1f786c1f
/etc/mosquitto/conf.d/README b4ac621550824082a735732bfb42b51d
/etc/mosquitto/mosquitto.conf 379a6d6e30d19bfdedf8099a9f3f8770
/etc/init/mosquitto.conf 97d9bc16841e4ba66a68ef1ee280fd9f obsolete
Description: MQTT version 3.1/3.1.1 compatible message broker
This is a message broker that supports version 3.1 and 3.1.1 of the MQTT
protocol.
.
MQTT provides a method of carrying out messaging using a publish/subscribe
model. It is lightweight, both in terms of bandwidth usage and ease of
implementation. This makes it particularly useful at the edge of the network
where a sensor or other simple device may be implemented using an arduino for
example.
Homepage: http://mosquitto.org/

So I appear to be running version 1.6.3.

APT says I can have these versions -

1.6.3-0mosquitto1~jessie1 -
1.6.2-0mosquitto1~jessie1 -
1.6.1-0mosquitto1~jessie1 -
1.5.8-0mosquitto1~jessie1 -
1.5.6-0mosquitto1~jessie1 -
1.5.5-0mosquitto1~jessie1 -
1.5.4-0mosquitto2~jessie1 -
1.5.4-0mosquitto1 -
1.5.4-0mosquitto1~jessie1 -
1.5.3-0mosquitto1~jessie1 -
1.5-0mosquitto2~jessie1 -
1.4.15-0mosquitto4~jessie1 -
1.3.4-2+deb8u3 -

So I guess aim for 1.4.15 ?
Do you know what are the problems were with later releases? As I’m pretty sure I upgraded at the beginning of this years sometime, so I’m guess I’d have been running a newer version of Mosquitto for a while?

I don’t know the specific reasons behind the issues, I’m just aware there have been issues. I just looked up a couple of threads and realised my memory wasn’t 100% regards the version numbers. 1.5.7 is actually the latest stable version from the Debian Buster repos and is ok (with Buster at least), Stretch is 1.4.10 and Jessie is still trailing behind at 1.3.4 so whilst you are still running Jessie I guess you will need to run a mosquitto source rather than debian. The problem versions are from 1.6’ish

I can’t answer why your’s is ok with 1.6.2, nor tell you why others have had less luck. I can only say that the known good versions are 1.4.10 and 1.5.7. 1.4.15 rings a bell but I seem to recall it wasn’t for a good reason and a quick search turned up this Mosquitto issues and Stretch (DietPi).

Without doing some research I can’t even tell you why you have such a short shortlist, (versions available) unless there was good reason to do otherwise, I would try to go for 1.4.10 on Jessie as it is tried and tested. I wish I could be of more help but I’m not sure what to suggest.

Ah, sounds bad. I think I should revert then.

The list comes from the “apt-cache showpkg mosquitto” command that shows what versions of a package are available in the APT repository. This presents a slight problem in that 1.4.10 and 1.5.7 are not there.

I see them on the list on the Mosquitto website that you pointed to, but those downloads don’t seem to be tailored to the RPi and must be compiled from source. The website jst points us back to the repo.

I think I need to test the compiling on a different Pi, as while I’m messing about MQTT won’t be running and data will be getting lost. Unless anyone else can point me in the right direction?