All inputs stopped logging this morning?!

Hardware emonPi
1

Hi

Seemingly random failure of all my inputs this morning just after 1am. All showing as NULL, n/a in the feed list. The emonPi is alive and well, I can login etc. I tried multiple restarts of services to no avail (although if relevant, the feedwriter is showing ‘Active Running - sleep 300s 0 feed points pending write’ - expected?
I had a look in previous threads about looking at the log but it seems to only contain data since the last reboot, so not much use to debug (unless there is a longer term one stored somewhere).
EmonCMS is low-write 11.2.3

Many thanks,
Tom

2

Hello @tommyt it sounds like there’s not data coming through from emonhub to emoncms on the emonPi after the reboot. What does the emonhub log show?

3

Is there data coming through on the log after the reboot?

4

Yes there is, here are the last few entries:

2022-11-07 10:11:14.-51|WARN|emoncms_mqtt.php|Not connected, retrying connection

2022-11-07 10:11:14.-16|WARN|emoncms_mqtt.php|Connecting to MQTT server: Connection Accepted.: code: 0

2022-11-07 10:16:43.-83|WARN|emoncms_mqtt.php|Not connected, retrying connection

2022-11-07 10:16:44.023|WARN|emoncms_mqtt.php|Connecting to MQTT server: Connection Accepted.: code: 0

2022-11-07 10:20:15.-430|ERROR|index.php|Not Authenticated|feed/view

5

Can you check the emonhub log. That is the emoncms log (click on emonhub in the left menu).

6

Sure here it is from when the issue happened to now:

2022-11-07 01:04:07,122 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1667783047 (296 bytes of data, 6 frames, uncompressed)
2022-11-07 01:04:09,993 INFO     MQTT       Publishing 'node' formatted msg
2022-11-07 01:04:15,034 INFO     MQTT       Publishing 'node' formatted msg
2022-11-07 01:04:20,175 INFO     MQTT       Publishing 'node' formatted msg
2022-11-07 01:04:25,018 INFO     MQTT       Publishing 'node' formatted msg
2022-11-07 01:04:30,170 INFO     MQTT       Publishing 'node' formatted msg
2022-11-07 01:04:37,591 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?sentat=1667783077 (243 bytes of data, 5 frames, uncompressed)
2022-11-07 07:29:50,036 INFO     MainThread Exiting hub...
2022-11-07 07:29:50,256 INFO     MainThread Exit completed
2022-11-07 07:30:20,688 INFO     MainThread EmonHub v2.4.2
2022-11-07 07:30:20,689 INFO     MainThread Opening hub...
2022-11-07 07:30:20,690 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2022-11-07 07:30:22,696 WARNING  MainThread Device communication error - check settings
2022-11-07 07:30:22,697 INFO     MainThread Setting RFM2Pi baseid: 5 (5i)
2022-11-07 07:30:23,700 INFO     MainThread Setting RFM2Pi frequency: 433 (4b)
2022-11-07 07:30:24,702 INFO     MainThread Setting RFM2Pi group: 210 (210g)
2022-11-07 07:30:25,704 INFO     MainThread Setting RFM2Pi quiet: 1 (1q)
2022-11-07 07:30:26,706 INFO     MainThread Setting RFM2Pi calibration: 230V (1p)
2022-11-07 07:30:27,710 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
2022-11-07 07:30:27,714 INFO     MainThread Setting MQTT node_format_enable: 1
2022-11-07 07:30:27,714 INFO     MainThread Setting MQTT nodevar_format_enable: 1
2022-11-07 07:30:27,715 INFO     MainThread Setting MQTT nodevar_format_basetopic: emon/
2022-11-07 07:30:27,716 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2022-11-07 07:30:27,719 INFO     MainThread Setting emoncmsorg apikey: set
2022-11-07 07:30:27,719 INFO     MainThread Setting emoncmsorg url: https://emoncms.org
2022-11-07 07:30:27,720 INFO     MainThread Setting emoncmsorg senddata: 1
2022-11-07 07:30:27,721 INFO     MainThread Setting emoncmsorg sendstatus: 1
2022-11-07 07:30:27,721 INFO     MainThread Setting emoncmsorg sendnames: 0
2022-11-07 07:30:27,722 INFO     MainThread Setting emoncmsorg compress: 0
2022-11-07 10:10:58,892 INFO     MainThread Exiting hub...
2022-11-07 10:10:59,172 INFO     MainThread Exit completed
2022-11-07 10:11:02,266 INFO     MainThread EmonHub v2.4.2
2022-11-07 10:11:02,267 INFO     MainThread Opening hub...
2022-11-07 10:11:02,268 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2022-11-07 10:11:04,273 WARNING  MainThread Device communication error - check settings
2022-11-07 10:11:04,275 INFO     MainThread Setting RFM2Pi baseid: 5 (5i)
2022-11-07 10:11:05,276 INFO     MainThread Setting RFM2Pi frequency: 433 (4b)
2022-11-07 10:11:06,278 INFO     MainThread Setting RFM2Pi group: 210 (210g)
2022-11-07 10:11:07,281 INFO     MainThread Setting RFM2Pi quiet: 1 (1q)
2022-11-07 10:11:08,283 INFO     MainThread Setting RFM2Pi calibration: 230V (1p)
2022-11-07 10:11:09,287 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
2022-11-07 10:11:09,290 INFO     MainThread Setting MQTT node_format_enable: 1
2022-11-07 10:11:09,291 INFO     MainThread Setting MQTT nodevar_format_enable: 1
2022-11-07 10:11:09,291 INFO     MainThread Setting MQTT nodevar_format_basetopic: emon/
2022-11-07 10:11:09,293 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2022-11-07 10:11:09,295 INFO     MainThread Setting emoncmsorg apikey: set
2022-11-07 10:11:09,296 INFO     MainThread Setting emoncmsorg url: https://emoncms.org
2022-11-07 10:11:09,297 INFO     MainThread Setting emoncmsorg senddata: 1
2022-11-07 10:11:09,297 INFO     MainThread Setting emoncmsorg sendstatus: 1
2022-11-07 10:11:09,298 INFO     MainThread Setting emoncmsorg sendnames: 0
2022-11-07 10:11:09,298 INFO     MainThread Setting emoncmsorg compress: 0
2022-11-07 10:11:14,933 INFO     MainThread Exiting hub...
2022-11-07 10:11:15,035 INFO     MainThread Exit completed
2022-11-07 10:11:18,177 INFO     MainThread EmonHub v2.4.2
2022-11-07 10:11:18,179 INFO     MainThread Opening hub...
2022-11-07 10:11:18,180 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2022-11-07 10:11:20,185 WARNING  MainThread Device communication error - check settings
2022-11-07 10:11:20,186 INFO     MainThread Setting RFM2Pi baseid: 5 (5i)
2022-11-07 10:11:21,188 INFO     MainThread Setting RFM2Pi frequency: 433 (4b)
2022-11-07 10:11:22,191 INFO     MainThread Setting RFM2Pi group: 210 (210g)
2022-11-07 10:11:23,193 INFO     MainThread Setting RFM2Pi quiet: 1 (1q)
2022-11-07 10:11:24,195 INFO     MainThread Setting RFM2Pi calibration: 230V (1p)
2022-11-07 10:11:25,199 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
2022-11-07 10:11:25,202 INFO     MainThread Setting MQTT node_format_enable: 1
2022-11-07 10:11:25,203 INFO     MainThread Setting MQTT nodevar_format_enable: 1
2022-11-07 10:11:25,204 INFO     MainThread Setting MQTT nodevar_format_basetopic: emon/
2022-11-07 10:11:25,205 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2022-11-07 10:11:25,208 INFO     MainThread Setting emoncmsorg apikey: set
2022-11-07 10:11:25,209 INFO     MainThread Setting emoncmsorg url: https://emoncms.org
2022-11-07 10:11:25,209 INFO     MainThread Setting emoncmsorg senddata: 1
2022-11-07 10:11:25,210 INFO     MainThread Setting emoncmsorg sendstatus: 1
2022-11-07 10:11:25,210 INFO     MainThread Setting emoncmsorg sendnames: 0
2022-11-07 10:11:25,211 INFO     MainThread Setting emoncmsorg compress: 0
2022-11-07 10:16:16,928 INFO     MainThread Exiting hub...
2022-11-07 10:16:17,126 INFO     MainThread Exit completed
2022-11-07 10:16:45,660 INFO     MainThread EmonHub v2.4.2
2022-11-07 10:16:45,661 INFO     MainThread Opening hub...
2022-11-07 10:16:45,662 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2022-11-07 10:16:47,668 WARNING  MainThread Device communication error - check settings
2022-11-07 10:16:47,670 INFO     MainThread Setting RFM2Pi baseid: 5 (5i)
2022-11-07 10:16:48,672 INFO     MainThread Setting RFM2Pi frequency: 433 (4b)
2022-11-07 10:16:49,674 INFO     MainThread Setting RFM2Pi group: 210 (210g)
2022-11-07 10:16:50,677 INFO     MainThread Setting RFM2Pi quiet: 1 (1q)
2022-11-07 10:16:51,678 INFO     MainThread Setting RFM2Pi calibration: 230V (1p)
2022-11-07 10:16:52,682 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
2022-11-07 10:16:52,685 INFO     MainThread Setting MQTT node_format_enable: 1
2022-11-07 10:16:52,686 INFO     MainThread Setting MQTT nodevar_format_enable: 1
2022-11-07 10:16:52,686 INFO     MainThread Setting MQTT nodevar_format_basetopic: emon/
2022-11-07 10:16:52,688 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2022-11-07 10:16:52,691 INFO     MainThread Setting emoncmsorg apikey: set
2022-11-07 10:16:52,691 INFO     MainThread Setting emoncmsorg url: https://emoncms.org
2022-11-07 10:16:52,692 INFO     MainThread Setting emoncmsorg senddata: 1
2022-11-07 10:16:52,692 INFO     MainThread Setting emoncmsorg sendstatus: 1
2022-11-07 10:16:52,693 INFO     MainThread Setting emoncmsorg sendnames: 0
2022-11-07 10:16:52,694 INFO     MainThread Setting emoncmsorg compress: 0
7 
2022-11-07 07:30:27,716 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2022-11-07 07:30:27,719 INFO     MainThread Setting emoncmsorg apikey: set
2022-11-07 07:30:27,719 INFO     MainThread Setting emoncmsorg url: https://emoncms.org
2022-11-07 07:30:27,720 INFO     MainThread Setting emoncmsorg senddata: 1
2022-11-07 07:30:27,721 INFO     MainThread Setting emoncmsorg sendstatus: 1
2022-11-07 07:30:27,721 INFO     MainThread Setting emoncmsorg sendnames: 0
2022-11-07 07:30:27,722 INFO     MainThread Setting emoncmsorg compress: 0
2022-11-07 10:10:58,892 INFO     MainThread Exiting hub...
2022-11-07 10:10:59,172 INFO     MainThread Exit completed
2022-11-07 10:11:02,266 INFO     MainThread EmonHub v2.4.2
2022-11-07 10:11:02,267 INFO     MainThread Opening hub...

@TrystanLea looks like an issue with the HTTP interfacer.

@tommyt - are you able to SSH into the Pi?

8

I’m not local to it so doing everything via Dataplicity at the moment, and I can’t raise a terminal via that (instant disconnect when I attempt reconnect), so no SSH.

9

Ok.

Can you set the log level in emonhub to DEBUG?

[edit]
Have you upgraded recently?

[edit2]

Can you post the System Info, please (click on copy as markdown from admin menu and just paste)
image

10

Already at debug level. No updates recently (I did do a full check anyway this morning after the problem started and no out of date components were reported)

11
Server Information

Server Information

Services

  • emonhub :- Active Running

  • emoncms_mqtt :- Active Running

  • feedwriter :- Active Running - sleep 300s 0 feed points pending write

  • service-runner :- Active Running

  • emonPiLCD :- Active Running

  • redis-server :- Active Running

  • mosquitto :- Active Running

  • demandshaper :- Active Running

Emoncms

  • Version :- low-write 11.2.3
  • Git :-
  • Components :- Emoncms Core v11.2.3 | App v2.6.7 | EmonHub Config v2.1.4 | Dashboard v2.3.3 | Device v2.2.0 | Graph v2.2.3 | Network Setup v1.0.2 | WiFi v2.1.1 | Backup v2.3.2 | DemandShaper v2.2.2 | Postprocess v2.2.5 | Sync v2.1.4 | Usefulscripts v2.3.9 | EmonScripts v1.5.3 | RFM2Pi v1.4.1 | Emonhub v2.4.2 | EmonPi v2.9.5

Server

  • CPU :- 1 Threads(s) | 4 Core(s) | 1 Sockets(s) | Cortex-A53 | 51.20MIPS |
  • OS :- Linux 5.4.51-v7+
  • Host :- emonpi | emonpi | (127.0.0.1)
  • Date :- 2022-11-07 12:02:53 UTC
  • Uptime :- 12:02:53 up 1:46, 0 users, load average: 0.02, 0.05, 0.00

Memory

  • RAM :- Used: 20.68%
    • Total :- 925.86 MB
    • Used :- 191.44 MB
    • Free :- 734.42 MB
  • Swap :- Used: 0.00%
    • Total :- 100 MB
    • Used :- 0 B
    • Free :- 100 MB

Disk

  • **** :- - / :- Used: 50.87%
    • Total :- 4.06 GB
    • Used :- 2.07 GB
    • Free :- 1.8 GB
    • Read Load :- 2.46 KB/s
    • Write Load :- 140.9 B/s
    • Load Time :- 1 hours 22 mins
  • /var/opt/emoncms :- Used: 1.52%
    • Total :- 9.84 GB
    • Used :- 153.06 MB
    • Free :- 9.19 GB
    • Read Load :- 0 B/s
    • Write Load :- 0 B/s
    • Load Time :- 1 hours 22 mins
  • /boot :- Used: 21.15%
    • Total :- 252.05 MB
    • Used :- 53.32 MB
    • Free :- 198.73 MB
    • Read Load :- 0 B/s
    • Write Load :- 0 B/s
    • Load Time :- 1 hours 22 mins
  • /var/log :- Used: 6.91%
    • Total :- 50 MB
    • Used :- 3.46 MB
    • Free :- 46.54 MB
    • Read Load :- n/a
    • Write Load :- n/a
    • Load Time :- n/a

HTTP

  • Server :- Apache/2.4.38 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 5.5.5-10.3.23-MariaDB-0+deb10u1
  • Host :- 127.0.0.1 (127.0.0.1)
  • Date :- 2022-11-07 12:02:52 (UTC 00:00‌​)
  • Stats :- Uptime: 6374 Threads: 11 Questions: 4069 Slow queries: 0 Opens: 46 Flush tables: 1 Open tables: 40 Queries per second avg: 0.638

Redis

  • Version :-
    • Redis Server :- 5.0.3
    • PHP Redis :- 5.3.1
  • Host :- localhost:6379
  • Size :- 434 keys (864.20K)
  • Uptime :- 0 days

MQTT Server

  • Version :- Mosquitto 1.5.7
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 7.3.19-1~deb10u1 (Zend Version 3.3.19)
  • Run user :- User: www-data Group: www-data video Script Owner: pi
  • Modules :- apache2handlercalendar Core ctype curl date dom v20031129exif fileinfo filter ftp gd gettext hash iconv json v1.7.0libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $openssl pcre PDO pdo_mysql Phar posix readline redis v5.3.1Reflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer wddx xml xmlreader xmlwriter xsl Zend OPcache zlib

Pi

  • Model :- Raspberry Pi 3 Model B Rev 1.2 - 1GB (Sony UK)

  • Serial num. :- 23210675

  • CPU Temperature :- 32.71°C

  • GPU Temperature :- 32.2°C

  • emonpiRelease :- emonSD-24Jul20

  • File-system :- read-write

Client Information

Client Information

HTTP

  • Browser :- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
  • Language :- en-US,en;q=0.9

Window

  • Size :- 1519 x 882

Screen

  • Resolution :- 1536 x 1024
12

It looks like emonhub is crashing. The only way to get the error message is to SSH in and look at the system log.

journalctl -n 50 -u emonhub -f

You could try and comment out the HTTP section from the emonhub config.

It may need a reboot (from system admin menu).

If you only have the EmonPi, then you could comment out the RFM section as well (at a different time - only change one thing at a time :slight_smile:).

13

‘You could try and comment out the HTTP section from the emonhub config.’ ← do you mean the subsection starting [[emoncmsorg]]?
RFM subsection is already commented out.

14

2 things in general,

  1. if replying to a specific message, use the reply at the bottom of the message rather than the reply at the very bottom (makes it easier to understand what you are replying to as it creates a link)
  2. if you highlight some text, you get a ‘Quote’ box appear - click it.
  3. Please user the preformatted button if posting SSH output, text, logs etc.
    (sorry 3!)

Can you post your config down to the ‘nodes’ section please (using the preformatted button).

15

Looks to me like an issue affecting the emonPi hardware side of things… emonhub doesnt look like it’s reading any data from the emonPi…

Might be worth trying a full power cycle rather than just a reboot.

16

OK I’ll do that tonight. When it says ‘check settings’, that seems a bit general - which settings in particular should be checked (bear in mind I haven’t changed anything!)

17

@TrystanLea Did a nice long power cycle to no avail
@borpin I seem to have ‘misplaced’ my ssh password (assuming I did change it, I tried the default as well and that didn’t work so I assume I did). Is it possible to reset to default without losing the reset of the emonpi config?

18

Scrub that, I managed to get in. This is running as instructed

 **pi@emonpi**:**~ $** journalctl -n 50 -u emonhub -f

-- Logs begin at Mon 2022-11-07 17:17:36 GMT. --

Nov 07 17:33:44 emonpi systemd[1]: Starting emonHub data multiplexer...

Nov 07 17:33:44 emonpi systemd[1]: Started emonHub data multiplexer.
19

Please @tommyt

20

Is the emonhub log level definitely set to DEBUG in emonhub.conf?

[hub]

### loglevel must be one of DEBUG, INFO, WARNING, ERROR, and CRITICAL

loglevel = DEBUG

Full config as Brian suggests would be useful. Blank out the apikey if you have it set