emonPi hangs and stops logging

2018-03-20 19:17:39,943 WARNING  MainThread Setting emoncmsorg apikey: obscured

That line tells me you haven’t added your emoncmsorg write apikey to emonhub.conf.

If you add that you should see data at emoncms.org, from there you can see what data is present or missing.

You could try restarting the mqtt_input service as sometimes that needs restarting when new nodes are added (old fault but service may not have been restarted since update)

sudo systemctl restart mqtt_input

In fact, if you haven’t already, try a reboot to reset everything after the updates.

Thank you for your help, I am making progress.

I ran the updaters on the Admin page on the emonPi and let them complete, they both appeared to make no changes.

/$ sudo systemctl status emonhub
● emonhub.service - LSB: Start/stop emonHub
   Loaded: loaded (/etc/init.d/emonhub)
   Active: active (exited) since Tue 2018-03-20 21:40:24 UTC; 8h ago

Mar 20 21:40:24 emonpi sudo[6436]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/mkdir -p /var/log/emonhub
Mar 20 21:40:24 emonpi sudo[6436]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 20 21:40:24 emonpi sudo[6436]: pam_unix(sudo:session): session closed for user root
Mar 20 21:40:24 emonpi sudo[6444]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/touch /var/log/emonhub/emonhub.log
Mar 20 21:40:24 emonpi sudo[6444]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 20 21:40:24 emonpi sudo[6444]: pam_unix(sudo:session): session closed for user root
Mar 20 21:40:24 emonpi sudo[6453]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/chown -R emonhub /var/log/emonhub
Mar 20 21:40:24 emonpi sudo[6453]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 20 21:40:24 emonpi sudo[6453]: pam_unix(sudo:session): session closed for user root
Mar 20 21:40:24 emonpi sudo[6462]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/chown -R emonhub /var/log/emonhub
Mar 20 21:40:24 emonpi sudo[6462]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 20 21:40:24 emonpi sudo[6462]: pam_unix(sudo:session): session closed for user root
Mar 20 21:40:24 emonpi emonhub[6434]: Starting OpenEnergyMonitor emonHub: emonhub has been started ok.
Mar 20 21:40:24 emonpi systemd[1]: Started LSB: Start/stop emonHub.
/$ sudo systemctl status feedwriter
● feedwriter.service - LSB: feedwriter script daemon
   Loaded: loaded (/etc/init.d/feedwriter)
   Active: active (exited) since Tue 2018-03-20 21:28:47 UTC; 9h ago

Mar 20 21:17:22 emonpi systemd[1]: Starting LSB: feedwriter script daemon...
Mar 20 21:17:22 emonpi feedwriter[1660]: Log is turned off
Mar 20 21:17:22 emonpi feedwriter[1660]: Starting RPI
Mar 20 21:28:47 emonpi systemd[1]: Started LSB: feedwriter script daemon.
/$ sudo systemctl status mqtt_input
● mqtt_input.service - Emoncms MQTT Input Script
   Loaded: loaded (/etc/systemd/system/mqtt_input.service; enabled)
   Active: active (running) since Tue 2018-03-20 21:40:59 UTC; 8h ago
     Docs: https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
 Main PID: 7045 (php)
   CGroup: /system.slice/mqtt_input.service
           └─7045 /usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php

Mar 20 21:40:59 emonpi systemd[1]: Started Emoncms MQTT Input Script.

API corrected and data is now flowing from emonPi to emonCMS. You can see it working here - Emoncms - dashboard view

However, all feeds still showing as “inactive” on emonPi but all feeds now present and correct at emonCMS. Are there any other settings in emonhub.conf or anywhere else that I should look at? I had assumed they would come across with the backup file.

Running the command “sudo systemctl restart mqtt_input” seems to do nothing, the window just hangs.

Also the Admin option in the Setup drop down keeps disappearing, if I log out and log back in it reappears for a while and then disappears again. When the Admin option is not available, if I go to http://emonpi.local/emoncms/admin/view then I get the message “Admin re-authentication required”. Log out and back in again, and Admin becomes available again.

I have rebooted, no change.

I have to go to work now, I may not be home for 36 hours but I will be back, thank you again for all your help!

I’m not sure what “the backup file” is as I do not know what you’ve done to migrate to the new image. If you have exported and imported the mysql tables then yes i would expect the same.

When you look at the inputs page is there any processing in the inputs or are they blank?

Since you have feeds on the feeds page but they are not updating, I suspect you have migrated the tables.

That doesn’t sound good.

That sounds correct, the user can remain logged in (using remember me) but the admin level session times out as a security feature (see Admin session expiry and logout).

There is another discussion going on about high memory usage by the mqtt_input service (see Mqtt_input service high memory usage) that may (or may not) be related, perhaps we might know more by the time you return.

I used the backup facility on the EmonPi to back it up, then I cleared the SD card and installed the new OS onto it, then I used the facility on the same page in the EmonPi to import the backup files.

When I look at the Inputs page on the local EmonPi I see their current values, when I look at the Feeds page it says each is “inactive”.

When I run the “sudo systemctl status emonhub” or “sudo systemctl status feedwriter” or “sudo systemctl status mqtt_input” I get a response in the terminal window, stuff happens. When I run “sudo systemctl restart mqtt_input” nothing happens. I’ll read the high memory usage by the mqtt_input service thread now.

I’ll ignore the loss of Admin level access if that is a feature. Sorry for the red herring.

I am back home at some point tomorrow and will try out any suggestions then.

Thank you again!

On the inputs page are there any blue blocks next to any of the inputs? These are the input processes that connect the feeds to the inputs.

When the inputs update the processes are run and that updates the feed values in redis which gets persisted every 60s by the feedwriter.

So

  1. Are the correct inputs being updated, ie are there any duplicated inputs where some are being upadted and some not?
  2. Is the input processing present and correct? ie are those blue blocks there on the inputs page and no red error blocks?
  3. is the feedwriter running? You can check that on the admin page and/or using “sudo systemctl status feedwriter”
  4. Are the permissions for the data files ok? Check to see if there are any errors in the emoncms,log from the admin page.
  5. Is there room for the data? did you expand the emonSD image to fit the SD card? Use the “df -h” command to check the disk usage.

Ok then can you try these 3 commands in turn and post the complete output

sudo systemctl status mqtt_input
sudo systemctl stop mqtt_input
sudo systemctl start mqtt_input

Although I’m starting to think the issue is less likely to be the mqtt_input service if you can see the input values changing on the emoncms inputs page. (and therefore the other thread is unlikely to be related)

Both strike me as odd as I’d expect Active: active (running) in the status but if data is getting to emoncms.org both must be running. Confused!

Did you go through the setup process? Although some of the config will come across, I’d expect some of the system configurations to need updating.

Strictly you do not get any output to the terminal (which in this case, no news is good news). if you then do a ‘status’ command you will see it has been running for a few seconds (i.e. it has just been restarted).

Good point Brian :slight_smile: I was still focused on the previous comment

@stabuck can you confirm if the command prompt is returning almost imediately or if you are waiting indefinitely for that to happen? and as Brian says, you can check the running time has restarted by checking the status.

At http://emonpi.local/emoncms/input/view
I see no blue or red blocks. All the Inputs are present and correct, I see none that are not updating, I see no weird values, I see no duplications.

/var/log/emoncms.log

2018-03-21 07:51:32.957|ERROR|phpmqtt_input.php|exception 'Mosquitto\Exception' in /var/www/emoncms/scripts/phpmqtt_input.php:125
Stack trace:
#0 /var/www/emoncms/scripts/phpmqtt_input.php(125): Mosquitto\Client->connect('localhost', 1883, 5)
#1 {main}
2018-03-21 07:51:32.976|WARN|phpmqtt_input.php|Not connected, retrying connection
2018-03-21 07:51:38.002|WARN|phpmqtt_input.php|Not connected, retrying connection
2018-03-21 07:51:38.004|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0
2018-03-21 07:51:46.866|WARN|phpmqtt_input.php|Not connected, retrying connection
2018-03-21 07:51:46.889|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0
2018-03-22 12:10:35.390|WARN|phpmqtt_input.php|Not connected, retrying connection
2018-03-22 12:10:35.415|WARN|phpmqtt_input.php|Connecting to MQTT server: Connection Accepted.: code: 0

/$ sudo systemctl status feedwriter

/$ sudo systemctl status feedwriter
feedwriter.service - LSB: feedwriter script daemon
Loaded: loaded (/etc/init.d/feedwriter)
Active: active (running) since Wed 2018-03-21 07:51:33 UTC; 1 day 4h ago
Process: 1663 ExecStart=/etc/init.d/feedwriter start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/feedwriter.service
           └─1851 /usr/bin/php -f /var/www/emoncms/scripts/feedwriter.php

Mar 21 07:17:22 emonpi feedwriter[1663]: Log is turned off
Mar 21 07:17:22 emonpi feedwriter[1663]: Starting RPI
Mar 21 07:51:33 emonpi systemd[1]: Started LSB: feedwriter script daemon.

/$ df -h

/$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.4G  2.0G  1.2G  63% /
devtmpfs        481M     0  481M   0% /dev
tmpfs           486M     0  486M   0% /dev/shm
tmpfs           486M   19M  467M   4% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           486M     0  486M   0% /sys/fs/cgroup
tmpfs            40M  6.6M   34M  17% /var/lib/openhab
tmpfs           1.0M  4.0K 1020K   1% /var/lib/dhcpcd5
tmpfs           1.0M  4.0K 1020K   1% /var/lib/dhcp
tmpfs            50M  5.3M   45M  11% /var/log
tmpfs            30M   36K   30M   1% /tmp
/dev/mmcblk0p1   60M   22M   39M  37% /boot
/dev/mmcblk0p3  969M   97M  823M  11% /home/pi/data

/$ sudo systemctl status mqtt_input

/$ sudo systemctl status mqtt_input
mqtt_input.service - Emoncms MQTT Input Script
Loaded: loaded (/etc/systemd/system/mqtt_input.service; enabled)
Active: active (running) since Wed 2018-03-21 07:51:44 UTC; 1 day 4h ago
Docs: https://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
Main PID: 1993 (php)
CGroup: /system.slice/mqtt_input.service
           └─1993 /usr/bin/php /var/www/emoncms/scripts/phpmqtt_input.php

Mar 21 07:51:44 emonpi systemd[1]: Started Emoncms MQTT Input Script.
/$ sudo systemctl stop mqtt_input
/$ sudo systemctl start mqtt_input

Did you go through the setup process? Although some of the config will come across, I’d expect some of the system configurations to need updating.

I am going to send this message and then check to see…

OK, the EmonPi keeps crashing / being completely unresponsive without any obvious cause, I am not doing anything to it, one munte it is alive and the next it is dead and only a hard reset brings it back to life. I am at the limit of my understanding here and I’m getting very frustrated with it.

I am wondering what happens if I reflash the SD card with a virgin OS, I then don’t import the old data and start the setup process from scratch as though I’d just bought a new EmonPi. Will it be easy for me to reconnected the “new” EmonPi to the existing EmonCMS account?

by “existing EmonCMS account” are you referring to a emoncms.org hosted account? If so yes, if you are referring to a local emoncms, that;s in your backups and can only be accessed if restored.

I think it might be a good idea to start over and take smaller steps.

Step one - start with a sdcard freshly flashed with the emonSD image, give the emonpi a wired Ethernet connection (don’t worry about any sensors at this point if you need to move it to a wired network point). With the card inserted and the Ethernet connected, power it up and walk away, leave it for an hour to be sure it completes it’s initial upgrades and updates.

Only when that is completely finished and you can confirm all is well by ssh’ing in and doing some checks (share the emonpiupdate.log with us), should you start configuring wifi etc.

At this point you can check the emonhub.logs and check the services are running but do not set up local emoncms yet.

Then you can return it to the sensors, connect them all before powering the emonpi up, then add the emoncms.org apikey to resume posting data to the online “existing” emoncms (if that’s what you have).

At this point you can confirm the network connection is still good and check everything is working as expected and make a decision on whether to import your old local emoncms data or start a fresh.

There is no question, you should not be having these issues and I cannot rule out any software or hardware at this point either, but moving from a new downloaded image through to a completely configured, updated, imported old data and deployed system in one move is only good if it works first time, the additional checks will help get it right, debug any issues and possibly save you from landing in the same position a second time and none the wiser.

Thank you. That is very helpful.

By “EmonCMS” I do mean my emoncms-dot-org account, yes. (I can’t post links!) I am not very worried about losing local data, but it would be nice to keep what is at emoncms-dot-org and connect the “new” EmonPi to it. I will backup the data (if it will let me) before I start with a fresh OS and config it from new.
I suspect the issue is an incompetent operator (me) who broke things just over 12 months ago, but yes I appreciate it could be a hardware fault, or indeed something else.

I will let you know how I get on.

Thank you again.

You can now, I bumped you up a couple of user levels. :slight_smile:

And I’ve “Unspammed” your posts!

And somehow in the process the previously valid post (6 posts back/up) has been determined as spam by “the system” retrospectively!

Unhidden now. I wonder when “System” will get the message?

Actually looking closer at those posts you’ve “unspammed” they are all existing old posts, there was nothing new there. “The system” has gone crazy!

That’s what I’d decided. As they’d been moved by a moderator, they should thereafter be immune from “System” intervention.

Perhaps @starbuck isn’t content with just breaking his emonpi, he’s havin’ a go at breaking the forum too :grin:

1 Like

… ably assisted by @system

1 Like

I think I have it working correctly… I’ll let you know if it stops logging again but so far so good.

1 Like