Emonhub failing to start

Emoncms
1

Hi,
My issue seems similar to this:

but not exactly?
After a power outage a few days ago, my emonPi system has not come alive properly.
It looked like /var/log was pretty full, so I truncated /var/log/logrotate/logrotate.log (it was he main culprit).
Rebooted. Still get the following (emonhub fails to start):
Nov 13 08:59:33 emonpi systemd[1]: emonhub.service: Service RestartSec=5s expired, scheduling restart.
Nov 13 08:59:33 emonpi systemd[1]: emonhub.service: Scheduled restart job, restart counter is at 221.
Nov 13 08:59:33 emonpi systemd[1]: Stopped emonHub data multiplexer.
Nov 13 08:59:33 emonpi systemd[1]: Starting emonHub data multiplexer…
Nov 13 08:59:33 emonpi systemd[1]: Started emonHub data multiplexer.
Nov 13 08:59:34 emonpi emonhub.py[7122]: Unable to load configuration file: /etc/emonhub/emonhub.conf
Nov 13 08:59:35 emonpi systemd[1]: emonhub.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 08:59:35 emonpi systemd[1]: emonhub.service: Failed with result ‘exit-code’.

Server Information

Server Information

Services

  • emonhub :- Activating Auto-restart
  • emoncms_mqtt :- Active Running
  • feedwriter :- Active Running - sleep 300s 592 feed points pending write
  • service-runner :- Active Running
  • emonPiLCD :- Inactive Dead
  • redis-server :- Active Running
  • mosquitto :- Active Running
  • demandshaper :- Active Running

Emoncms

Server

  • OS :- Linux 4.19.75-v7+
  • Host :- emonpi | emonpi | (172.16.0.13)
  • Date :- 2020-11-13 09:09:11 UTC
  • Uptime :- 09:09:11 up 40 min, 1 user, load average: 0.42, 0.47, 0.66

Memory

  • RAM :- Used: 25.63%
    • Total :- 975.62 MB
    • Used :- 250.05 MB
    • Free :- 725.57 MB
  • Swap :- Used: 0.00%
    • Total :- 100 MB
    • Used :- 0 B
    • Free :- 100 MB
      Write Load Period

Disk

  • / :- Used: 58.18%
    • Total :- 3.92 GB
    • Used :- 2.28 GB
    • Free :- 1.45 GB
    • Write Load :- 158.72 KB/s (38 mins)
  • /var/opt/emoncms :- Used: 25.12%
    • Total :- 9.98 GB
    • Used :- 2.51 GB
    • Free :- 6.97 GB
    • Write Load :- 163.73 B/s (38 mins)
  • /boot :- Used: 20.55%
    • Total :- 252.05 MB
    • Used :- 51.79 MB
    • Free :- 200.26 MB
    • Write Load :- 6.42 B/s (38 mins)
  • /var/log :- Used: 13.43%
    • Total :- 50 MB
    • Used :- 6.71 MB
    • Free :- 43.29 MB
    • Write Load :- n/a

HTTP

  • Server :- Apache/2.4.38 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 5.5.5-10.3.17-MariaDB-0+deb10u1
  • Host :- localhost:6379 (127.0.0.1)
  • Date :- 2020-11-13 09:09:10 (UTC 00:00‌​)
  • Stats :- Uptime: 2420 Threads: 17 Questions: 39035 Slow queries: 0 Opens: 47 Flush tables: 1 Open tables: 41 Queries per second avg: 16.130

Redis

  • Version :-
    • Redis Server :- 5.0.3
    • PHP Redis :- 5.0.2
  • Host :- localhost:6379
  • Size :- 476 keys (950.13K)
  • Uptime :- 0 days

MQTT Server

  • Version :- Mosquitto 1.5.7
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 7.3.9-1~deb10u1 (Zend Version 3.3.9)
  • Modules :- apache2handlercalendar Core ctype curl date dom v20031129exif fileinfo filter ftp gd gettext hash iconv json v1.7.0libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $openssl pcre PDO pdo_mysql Phar posix readline redis v5.0.2Reflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer wddx xml xmlreader xmlwriter xsl Zend OPcache zlib

Pi

  • Model :- Raspberry Pi 3 Model B Rev 1.2 - 1GB (Sony UK)

  • Serial num. :- 5D7A6220

  • CPU Temperature :- 53.69°C

  • GPU Temperature :- 54.2°C

  • emonpiRelease :- emonSD-17Oct19

  • File-system :- read-write

Client Information

Client Information

HTTP

  • Browser :- Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0
  • Language :- en-GB,en;q=0.5

Window

  • Size :- 1846 x 1411

Screen

  • Resolution :- 2560 x 1600

So I can’t see any more logs with helpful hints on what the issue might be. Is my only option a backup/new sd image/data restore?

2

The most likely answer is that the SD card is damaged so you will need a new one.

What does

ls -la  /etc/emonhub/

give and does

cat /etc/emonhub/emonhub.conf

list the file?

3

Hi Brian,
emonhub.conf does appear to contain a valid config file.
I kept a backup at some stage, so I might see if I can diif the files just to make 100% sure.

Just checked, my emonhub.conf backup I kept was pre-SD image update in 2018, so not the same file unfortunately. I have other backups from the emonpi but they are only the data, or does the gui backup also back up the configs… I guess it must. Perhaps I’ll look in there.

OK, so visually comparing a backup, and it looks like the conf file on the pi is corrupt. Looks like it has a partial duplicate concatentated on the end, e.g. duplicate node entries. This will likely cause issues. Odd there is not an error message that the file is invaild. I guess it parses OK, just not valid. I’ll attempt to edit it back to a sane state and see how that goes.

Wondering, what’s the file system used on the pi… maybe a file system check will straighten out some fs issues… mmm mounted filesystem, so no e2fsck… Not sure if trying to force a check on reboot is going to end in tears.

4

OK, so I’ll attach the offending? emonhub.conf file.
I deleted everything after line 239, and things seemed to start normally.
It looks like some duplicate node entries have been added to the file, but they are not exact duplicates, e.g. node [[5]] (the emonpi) at line 97 is not exactly the same as the duplicate at line 258, so I’m not really sure where these extra entries came from. I’m presuming they were not there before the power outage, as the system was logging data OK up till then, and they are not in the backups I have.

emonhub.conf.txt (15.0 KB)

Anyone know what may have happened here? Is it a corrupted file system on the SD card (due to the power outage), or something else?

5

We occasionally saw these duplicated entries after an update, but thought that had been cured.

All fixed?

6

Seems better after the edits, and nothing strange in the logs.
Will take regular backups and if it goes bad again I’ll start fresh with a clean SD image and data restore!

7

Hia. Seems like this is resolved now but out of interest did you do a manual emoncms/emonhub update recently from the admin page?

8

I tried a manual update from the admin page, but only after the problem started. Can’t say for sure what the contents of the conf file were at that point, but syslog was showing that emonhub was failing to start (before I tried an update), and after editing the file it was OK. So it seems unlikely that the update actually caused the problem.

9

I’ve just seen the same issue following an upgrade from Version: low-write 10.2.2 to 10.2.6 using the Admin Full Update button. Looks like all my node entries were duplicated. I restored emonhub.conf using the backup I took before the upgrade and its running ok now. Corrupt version with duplicates attached in case its useful. emonhub.conf.corrupt.txt (14.8 KB)

10

@TrystanLea