Emonhub fails to start after power outage today

Hardware emonPi
1

After a power outage today, the emonpi rebooted, but EmonHub gives me the following error:

2019-12-20 12:39:07,679 CRITICAL MainThread Error parsing config file "/etc/emonhub/emonhub.conf": Parsing failed with several errors.
First error at line 243.

The emonhub.conf line 243-252 are as follows:

[[5]]
    nodename = emonPi
    firmware = emonPi_RFM69CW_RF12Demo_DiscreteSampling.ino
    hardware = emonpi
    [[[rx]]]
        names = power1,power2,power1_plus_power2,Vrms,T1,T2,T3,T4,T5,T6,pulseCount
        datacodes = h, h, h, h, h, h, h, h, h, h, L
        scales = 1,1,1,0.01,0.1,0.1,0.1,0.1,0.1,0.1,1
        units = W,W,W,V,C,C,C,C,C,C,p

All feeds from the emonpi and the two IotaWatts (which are both working fine) are either delayed in time or marked as inactive in the feed list. These are the last two entries in the emoncms log:

2019-12-20 11:35:49.772|WARN|PHPFina.php|post_bulk_prepare() timestamp=1546892260 older than feed starttime=1574030360 feedid=120
2019-12-20 11:35:49.772|WARN|PHPFina.php|post_bulk_prepare() timestamp=1546892270 older tha

I have pasted the server info below. The SD card is only a few days old, and I created an image from the Oct19 version. Rebooting did not help, I have not tried cycling power yet.

Any suggestions?

Server Information
-----------------------

Services
	emonhub :	 Failed Failed
	emoncms_mqtt :	 Active Running
	feedwriter :	 Active Running - sleep 300s 63566 feed points pending write
	service-runner :	 Active Running
	emonPiLCD :	 Active Running
	redis-server :	 Active Running
	mosquitto :	 Active Running
	demandshaper :	 Active Running

Emoncms
	Version :	 low-write 10.1.13
	Modules :	 Administration | App v2.0.9 | Backup v2.1.7 | EmonHub Config v2.0.4 | Dashboard v2.0.5 | DemandShaper v1.1.0 | Device v2.0.3 | EventProcesses | Feed | Graph v2.0.6 | Input | Postprocess v2.1.2 | CoreProcess | Schedule | Network Setup v1.0.0 | sync | Time | User | Visualisation | WiFi v2.0.2
	Git :	 
		URL :	 https://github.com/emoncms/emoncms.git
		Branch :	 * stable
		Describe :	 10.1.11-7-g90f39158

Server
	OS :	 Linux 4.19.75-v7+
	Host :	 emonpi | emonpi | (192.168.1.132)
	Date :	 2019-12-20 12:52:24 UTC
	Uptime :	 12:52:24 up 13 min,  0 users,  load average: 0.75, 0.78, 0.54

Memory
	RAM :	 Used: 18.65%
		Total :	 975.62 MB
		Used :	 181.99 MB
		Free :	 793.63 MB
	Swap :	 Used: 0.00%
		Total :	 100 MB
		Used :	 0 B
		Free :	 100 MB
Write Load Period
Disk
	/ :	 Used: 48.48%
		Total :	 3.92 GB
		Used :	 1.9 GB
		Free :	 1.83 GB
		Write Load :	 117.33 B/s (6 mins)
	/var/opt/emoncms :	 Used: 0.88%
		Total :	 9.98 GB
		Used :	 90.29 MB
		Free :	 9.38 GB
		Write Load :	 0 B/s (6 mins)
	/boot :	 Used: 20.55%
		Total :	 252.05 MB
		Used :	 51.79 MB
		Free :	 200.25 MB
		Write Load :	 0 B/s (6 mins)
	/var/log :	 Used: 100.00%
		Total :	 50 MB
		Used :	 50 MB
		Free :	 0 B
		Write Load :	 n/a

HTTP
	Server :	 Apache/2.4.38 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL
	Version :	 5.5.5-10.3.17-MariaDB-0+deb10u1
	Host :	 localhost:6379 (127.0.0.1)
	Date :	 2019-12-20 12:52:23 (UTC 00:00ā€Œā€‹)
	Stats :	 Uptime: 831  Threads: 13  Questions: 5329  Slow queries: 0  Opens: 49  Flush tables: 1  Open tables: 43  Queries per second avg: 6.412

Redis
	Version :	 
		Redis Server :	 5.0.3
		PHP Redis :	 5.0.2
	Host :	 localhost:6379
	Size :	 407 keys (6.70M)
	Uptime :	 0 days
MQTT Server
	Version :	 Mosquitto 1.5.7
	Host :	 localhost:1883 (127.0.0.1)

PHP
	Version :	 7.3.9-1~deb10u1 (Zend Version 3.3.9)
	Modules :	 apache2handler | calendar v7.3.9-1~deb10u1 | Core v7.3.9-1~deb10u1 | ctype v7.3.9-1~deb10u1 | curl v7.3.9-1~deb10u1 | date v7.3.9-1~deb10u1 | dom v20031129 | exif v7.3.9-1~deb10u1 | fileinfo v7.3.9-1~deb10u1 | filter v7.3.9-1~deb10u1 | ftp v7.3.9-1~deb10u1 | gd v7.3.9-1~deb10u1 | gettext v7.3.9-1~deb10u1 | hash v7.3.9-1~deb10u1 | iconv v7.3.9-1~deb10u1 | json v1.7.0 | libxml v7.3.9-1~deb10u1 | mbstring v7.3.9-1~deb10u1 | mosquitto v0.4.0 | mysqli v7.3.9-1~deb10u1 | mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $ | openssl v7.3.9-1~deb10u1 | pcre v7.3.9-1~deb10u1 | PDO v7.3.9-1~deb10u1 | pdo_mysql v7.3.9-1~deb10u1 | Phar v7.3.9-1~deb10u1 | posix v7.3.9-1~deb10u1 | readline v7.3.9-1~deb10u1 | redis v5.0.2 | Reflection v7.3.9-1~deb10u1 | session v7.3.9-1~deb10u1 | shmop v7.3.9-1~deb10u1 | SimpleXML v7.3.9-1~deb10u1 | sockets v7.3.9-1~deb10u1 | sodium v7.3.9-1~deb10u1 | SPL v7.3.9-1~deb10u1 | standard v7.3.9-1~deb10u1 | sysvmsg v7.3.9-1~deb10u1 | sysvsem v7.3.9-1~deb10u1 | sysvshm v7.3.9-1~deb10u1 | tokenizer v7.3.9-1~deb10u1 | wddx v7.3.9-1~deb10u1 | xml v7.3.9-1~deb10u1 | xmlreader v7.3.9-1~deb10u1 | xmlwriter v7.3.9-1~deb10u1 | xsl v7.3.9-1~deb10u1 | Zend OPcache v7.3.9-1~deb10u1 | zlib v7.3.9-1~deb10u1

Pi
	Model :	 Raspberry Pi 3 Model B+ Rev 1.3 - 1GB (Sony UK)
	Serial num. :	 56D9027C
	Temperature :	 59.07Ā°C - 59.1Ā°C
	emonpiRelease :	 emonSD-17Oct19
	File-system :	 read-write

Client Information
-----------------------

HTTP
	Browser :	 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36
	Language :	 nb-NO,nb;q=0.9,no;q=0.8,nn;q=0.7,en-US;q=0.6,en;q=0.5,da;q=0.4
2

Shutting down (using the emonpi top button), and cycling power did not help.
Temperature is about 10Ā°C higher than normal, up to 60Ā°C.

3

Thanks for the system info but it is better to post your system info by clicking on the button Copy as Markdown next to Server Information on the Admin page and pasting (no further formatting required).

Is the power supply to the EmonPi on a spike protection plug? As Trystan and Glyn found, spikes (often accompany power loss) can cause havoc.

Looking at the default emonhub.conf, the entries should finish at 241 after node 26.

Try commenting out those lines and see what happens.

4

Worth taking a backup of the system as well.

5

To add to @borpinā€™s reply, could you copy and paste here the result of trying to restart emonhub via SSH?

sudo systemctl restart emonhub
sudo systemctl status emonhub

This is the thread @borpin was referring too that both Glyn and I had suffered a potentially similar issue after a power cut: Backup module usb-import.sh script (beta) - #9 by TrystanLea

6

1st cousin of was is? :wink: :grin:

1 Like
7

gunner2

1 Like
8

The outage was in the entire town area where we live, possibly due to a cable failure, so there may very well have been a power surge. Our fuse cabinet has a surge protection for our entire house grid, but the emonpi is just connected to an ordinary power outlet, together with two iotawatts and thus three reference voltage adaptors.

9

Iā€™ll see if I can play with this tonight or tomorrow. Thanks for quick responses.

10

You like those little smilies, dontcha? highfive
Iā€™m glad to see someoneā€™s gettinā€™ some milage out of them! thumbsup

1 Like
11

It sounds like you have multiple node [[5]] entries. The reason it is failing to restart is that once running if an error is made to the config emonhub will log the error but continue to run on the previous settings, when starting from afresh it doesnā€™t have that option. You possibly edited emonhub.conf sometime in the past and not noticed that the new settings were not loaded.

Either commenting out that section (as suggested above) or simply changing one of the [[5]] entries to an unused number should prove if this is the case or not.

1 Like
12

From day 1 for this emonpi I added two emonTH units, then after a few months I added two IotaWatts. This configuration was running until last week, when I replaced the microSD with a brand new, faster one (and larger). During the card replacement, I first took a backup then flashed the new card with emonSD Oct19, and uploaded the backup. I have never knowingly edited the emonhub.conf file, and the new setup ran just fine for a few days until the outage yesterday.

Anyway, Iā€™ll try commenting the section and see what happens.

EDIT: Ok, so I think you are into something here. There was an incident where I messed up the feed setup in the emonpi gui after using the restore backup process on the new card. And I thought I fixed it then. But looking into emonhub.conf now I see that multiple nodes have two entries, starting with [[5]] on line 243. Nodes 5 to 14 suddenly has appeared twice, and with less info in the first entry, example from 2019-12-21:

[[5]]
    nodename = emonpi
    [[[rx]]]
        names = power1,power2,power1pluspower2,vrms,t1,t2,t3,t4,t5,t6,pulsecount
        datacodes = h, h, h, h, h, h, h, h, h, h, L
        scales = 1,1,1,0.01,0.1,0.1,0.1,0.1,0.1,0.1,1
        units = W,W,W,V,C,C,C,C,C,C,p
		
		
[[5]]
    nodename = emonPi
    firmware = emonPi_RFM69CW_RF12Demo_DiscreteSampling.ino
    hardware = emonpi
    [[[rx]]]
        names = power1,power2,power1_plus_power2,Vrms,T1,T2,T3,T4,T5,T6,pulseCount
        datacodes = h, h, h, h, h, h, h, h, h, h, L
        scales = 1,1,1,0.01,0.1,0.1,0.1,0.1,0.1,0.1,1
        units = W,W,W,V,C,C,C,C,C,C,p

Looking at my backup from 2019-12-16 (from the old SD card), this was not the case.

Conclusion now is that I need to tidy up the conf file. Which one of the [[5]] entries above look correct?

13

By the way, I never noticed that bright red LED inside the emonpi case before. That may be because I have never looked, or because some warning LED was switched on. I think it is on the main RPI board, and I can see it through the small gaps around the USBs. Do you know if this red LED is normal?

14

I removed all duplicate entries in the emonhub.conf file, and now it is up and running again - sort of.

The nodes defined in the conf file seem ok, but the two iotawatt nodes are not. One of them are still Inactive, the other one has status ā€œn/aā€ on the 18 inputs channels, but 18 new channels (withouth defined feeds) have appeared. And I donā€™t see the iotawatts appear in the emonhub.conf file at all, not in the old (where everything worked) nor in the new one.

Any suggestions, please?

15

[Content deleted - not relevant to the topic.]

16

Since the iotawatt sends directly to emoncms, emonhub plays no part.

Unless of course you are running something non-standard.

1 Like
17

Delete the new Inputs without the processes. This happens sometimes. No one has been able to work out why.

As Paul said, IoTaWatt does not use emonhub but uses the HTTP API to send the data. Are their configurations OK after the power outage?

[edit] IIRC there was something else recently about IoTaWatt and data input not being restored correctly - have a search on the forum.

18

Thatā€™s what I thought, too. It has worked very well on my previous emoncms setup on a self-configured RPI. Now I have had another issue with the feeds on this emonpi (see this thread and this), so I ended up with deleting all the iotawatt feeds altogether and have now rebuilt them. Since the iotawatts hold all historic data, that should work just fine. The feeds will take some hours to rebuild, but I see that some data are starting to appear now, so things seem to work out.

2 Likes
19

This just happened to me. We was a brief outage, I realised when I saw all of my graphing was down, when I logged into my terminal server in my garage the uptime was only a few minutes. I was able to edit the emonhub config to remove the duplicate nodes and restart the service which got everything running again. Very weird. Could anyone recommend some sort of mini UPS to power the emonpi for less than a minute to prevent this some happening again? I expect this is rare as my electricity is usually reliable but it would be useful to have it. I will sort out a surge protection myself.

Can the VMS plug connected to the emonpi run properly behind a surge protection plug?

20

Raspberry pi are pretty bad when it comes to power failures. They can fail to come back at all, or come back with some services running and others not etc etc. Theyā€™re not like ā€˜properā€™ computers in this regard.

My solution has been to buy an Adafruit PowerBoost unit and a battery. Iā€™m still testing and they appear to work but I havenā€™t installed them live yet. Be warned though that I destroyed one Pi 3B+ just by connecting a wire from the LBO terminal to a GPIO pin. You MUST put a diode in the line. I have a simple python script that watches the LBO line and shuts down the pi safely when the line indicates that the battery is running out of power. That part seems to work OK.