Community
OpenEnergyMonitor

Community

Emonhub fails to start

(John) #1

– Logs begin at Wed 2019-04-24 10:17:01 EDT, end at Wed 2019-04-24 11:48:18 EDT. – Apr 24 11:48:02 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart. Apr 24 11:48:02 emonpi systemd[1]: Stopped emonHub service description. Apr 24 11:48:02 emonpi systemd[1]: Started emonHub service description. Apr 24 11:48:02 emonpi systemd[1]: emonhub.service: Main process exited, code=killed, status=11/SEGV Apr 24 11:48:02 emonpi systemd[1]: emonhub.service: Unit entered failed state. Apr 24 11:48:02 emonpi systemd[1]: emonhub.service: Failed with result ‘signal’. Apr 24 11:48:02 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart. Apr 24 11:48:02 emonpi systemd[1]: Stopped emonHub service description. Apr 24 11:48:02 emonpi systemd[1]: Started emonHub service description. Apr 24 11:48:02 emonpi systemd[1]: emonhub.service: Main process exited, code=killed, status=11/SEGV Apr 24 11:48:02 emonpi systemd[1]: emonhub.service: Unit entered failed state. Apr 24 11:48:02 emonpi systemd[1]: emonhub.service: Failed with result ‘signal’. Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart. Apr 24 11:48:03 emonpi systemd[1]: Stopped emonHub service description. Apr 24 11:48:03 emonpi systemd[1]: Started emonHub service description. Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Main process exited, code=killed, status=11/SEGV Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Unit entered failed state. Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Failed with result ‘signal’. Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart. Apr 24 11:48:03 emonpi systemd[1]: Stopped emonHub service description. Apr 24 11:48:03 emonpi systemd[1]: Started emonHub service description. Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Main process exited, code=killed, status=11/SEGV Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Unit entered failed state. Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Failed with result ‘signal’. Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart. Apr 24 11:48:03 emonpi systemd[1]: Stopped emonHub service description. Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Start request repeated too quickly. Apr 24 11:48:03 emonpi systemd[1]: Failed to start emonHub service description. Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Unit entered failed state. Apr 24 11:48:03 emonpi systemd[1]: emonhub.service: Failed with result ‘signal’.

Did soft and hard reboots and pi/data is only used to 9% capacity

(Brian Orpin) #2

Hi John,

Can you try and post the log again as it is impossible to read.

If you use 3 backticks (top left of keyboard usually) on a line before and a line after, it should format correctly.

[Edit]What command did you use to get the log?

(John) #3

Services
emonhub Failed Failed
emoncms_mqtt Active Running
feedwriter Active Running - sleep 60s 0 feed points pending write
service-runner Activating Auto-restart
emonPiLCD Active Exited
redis-server Active Running
mosquitto Active Running
Emoncms Version low-write 9.9.8
Modules Administration App v1.2.1 Backup v1.1.6 EmonHub Config v1.1.0 Dashboard v1.3.3 Device v1.2.1 EventProcesses Feed Graph v1.2.3 Input Postprocess v1.0.0 CoreProcess Schedule Network Setup v1.0.0 sync Time User Visualisation WiFi v1.3.1
Git URL: https://github.com/emoncms/emoncms.git Branch: * stable Describe: 9.9.4-247-gd0db7a57
Server OS Linux 4.14.71-v7+
Host emonpi emonpi (192.168.1.71)
Date 2019-04-24 21:06:20 UTC
Uptime 17:06:20 up 6:27, 1 user, load average: 0.00, 0.01, 0.00
HTTP Server Apache/2.4.25 (Raspbian) HTTP/1.1 CGI/1.1 80
MySQL Version 5.5.5-10.1.23-MariaDB-9+deb9u1
Host 127.0.0.1 (127.0.0.1)
Date 2019-04-24 21:06:19 (UTC 00:00‌​)
Stats Uptime: 24546 Threads: 3 Questions: 67546 Slow queries: 0 Opens: 23 Flush tables: 1 Open tables: 17 Queries per second avg: 2.751
Redis Version 3.2.6
Host localhost:6379 (127.0.0.1)
Size 145 keys (729.26K)

Uptime 0 days
MQTT Server Version Mosquitto 1.4.10
Host localhost:1883 (127.0.0.1)
Pi Model Raspberry Pi 3 Model B+ Rev 1.3 - 1 GB (Sony UK)
SoC Broadcom BCM2835
Serial num. 92A84C41
Temperature CPU: 44.01°C - GPU: 44.0’C
Release emonSD-30Oct18
Memory RAM Used: 13.71%

Total: 976.74 MB Used: 133.96 MB Free: 842.78 MB
Swap Used: 0.00%

Total: 100 MB Used: 0 B Free: 100 MB
Disk Mount Stats
/ Used: 39.94%

Total: 3.81 GB Used: 1.52 GB Free: 2.11 GB
/boot Used: 51.69%

Total: 42.52 MB Used: 21.98 MB Free: 20.54 MB
/home/pi/data Used: 1.89%

Total: 10.32 GB Used: 200.04 MB Free: 9.6 GB

(John) #4
-- Logs begin at Wed 2019-04-24 10:17:01 EDT, end at Wed 2019-04-24 17:08:01 EDT. --
Apr 24 17:03:16 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart.
Apr 24 17:03:16 emonpi systemd[1]: Stopped emonHub service description.
Apr 24 17:03:16 emonpi systemd[1]: Started emonHub service description.
Apr 24 17:03:16 emonpi systemd[1]: emonhub.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:03:16 emonpi systemd[1]: emonhub.service: Unit entered failed state.
Apr 24 17:03:16 emonpi systemd[1]: emonhub.service: Failed with result 'signal'.
Apr 24 17:03:17 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart.
Apr 24 17:03:17 emonpi systemd[1]: Stopped emonHub service description.
Apr 24 17:03:17 emonpi systemd[1]: Started emonHub service description.
Apr 24 17:03:17 emonpi systemd[1]: emonhub.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:03:17 emonpi systemd[1]: emonhub.service: Unit entered failed state.
Apr 24 17:03:17 emonpi systemd[1]: emonhub.service: Failed with result 'signal'.
Apr 24 17:03:17 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart.
Apr 24 17:03:17 emonpi systemd[1]: Stopped emonHub service description.
Apr 24 17:03:17 emonpi systemd[1]: Started emonHub service description.
Apr 24 17:03:17 emonpi systemd[1]: emonhub.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:03:17 emonpi systemd[1]: emonhub.service: Unit entered failed state.
(Brian Orpin) #6

Can you tell me what your setup is please (EmonPi, EmonBase) and what has changed from the working setup. How long has it been in use?

I presume you have rebooted?

(John) #7

We have several EmonTXs communicating with a single EmonPi

Everything was working fine and nothing was changed when unit stopped reporting or gathering feeds from TXs

Has been down alittle more than a day now

Tried hard reboots of all
The devices, soft reboots. Ssh emonhub service starts, all fail with same info as per logs

This setup has been run Running since March 8th of this year

(Brian Orpin) #8

@pb66 @TrystanLea any ideas?

(Brian Orpin) #9

In the web interface, is there anything in the emonhub log window (under settings)?

(Paul) #10

It has zero uptime and the versions are the latest, could it be this was triggered by an update?

The faults relate to a segmentation error, Is the log partition full?

df -h

Is there a emonhub.log?

ls -la /var/log/emonhub

in fact lets see what is happening with the logs in general

ls -la /var/log

Ahhh, ok that would most likely (not-confirmed) be the latest revisions anyway, so maybe not updated?

Not seen this error before, if this is a stock set up and the log partition isn’t maxing out (or even if it is for that matter) I would look at the recent changes to emonhub systemd and logging first.

(John) #11

there is actually no file named emonhub.log at all in var/log

i will dump all log files here in an hour as im on a mobile device right now

perhaps remote ssh access would be better?? To whom would I send the credentials to?

(John) #12

I messaged brian with credentials for ssh

Regards,

(Brian Orpin) #13

Hi @mickeymouse690, this is a community forum rather than a formal support desk. We are not really set up for remote access help.

The error is odd as it is a segmentation error. I’m not finding anything else like it here.

Have you done any OS upgrades or installed anything else on the Pi?

I also note service-runner is not starting correctly Could you do

sudo journalctl -u service-runner
(John) #14

I did an update maybe 2-3 weeks ago and everything was running the same after until a day ago. Other than that it wasn’t touched.

I understand this is a community forum, was just hoping someone could login and take a look as this would

I guess I should have went for a Commercial solution :frowning:

I’ll provide what was asked when u get in front of a desktop compter. This downtime is costing me serious money

(John) #15
-- Logs begin at Thu 2016-11-03 13:16:43 EDT, end at Thu 2019-04-25 21:53:30 EDT
Apr 24 17:54:20 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:54:25 emonpi systemd[1]: service-runner.service: Main process exited,
Apr 24 17:54:25 emonpi systemd[1]: service-runner.service: Unit entered failed s
Apr 24 17:54:25 emonpi systemd[1]: service-runner.service: Failed with result 's
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Service hold-off time
Apr 24 17:55:06 emonpi systemd[1]: Stopped Emoncms service-runner Input Script.
Apr 24 17:55:06 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Main process exited,
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Unit entered failed s
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Failed with result 's
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Service hold-off time
Apr 24 17:55:36 emonpi systemd[1]: Stopped Emoncms service-runner Input Script.
Apr 24 17:55:36 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Main process exited,
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Unit entered failed s
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Failed with result 's
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Service hold-off time
Apr 24 17:56:06 emonpi systemd[1]: Stopped Emoncms service-runner Input Script.
Apr 24 17:56:06 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Main process exited,
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Unit entered failed s
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Failed with result 's
lines 1-23
-- Logs begin at Thu 2016-11-03 13:16:43 EDT, end at Thu 2019-04-25 21:53:30 EDT. --
Apr 24 17:54:20 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:54:25 emonpi systemd[1]: service-runner.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:54:25 emonpi systemd[1]: service-runner.service: Unit entered failed state.
Apr 24 17:54:25 emonpi systemd[1]: service-runner.service: Failed with result 'signal'.
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Service hold-off time over, scheduling restart.
Apr 24 17:55:06 emonpi systemd[1]: Stopped Emoncms service-runner Input Script.
Apr 24 17:55:06 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Unit entered failed state.
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Failed with result 'signal'.
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Service hold-off time over, scheduling restart.
Apr 24 17:55:36 emonpi systemd[1]: Stopped Emoncms service-runner Input Script.
Apr 24 17:55:36 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Unit entered failed state.
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Failed with result 'signal'.
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Service hold-off time over, scheduling restart.
Apr 24 17:56:06 emonpi systemd[1]: Stopped Emoncms service-runner Input Script.
Apr 24 17:56:06 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Unit entered failed state.
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Failed with result 'signal'.
lines 1-23...skipping...
-- Logs begin at Thu 2016-11-03 13:16:43 EDT, end at Thu 2019-04-25 21:53:30 EDT. --
Apr 24 17:54:20 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:54:25 emonpi systemd[1]: service-runner.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:54:25 emonpi systemd[1]: service-runner.service: Unit entered failed state.
Apr 24 17:54:25 emonpi systemd[1]: service-runner.service: Failed with result 'signal'.
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Service hold-off time over, scheduling restart.
Apr 24 17:55:06 emonpi systemd[1]: Stopped Emoncms service-runner Input Script.
Apr 24 17:55:06 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Unit entered failed state.
Apr 24 17:55:06 emonpi systemd[1]: service-runner.service: Failed with result 'signal'.
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Service hold-off time over, scheduling restart.
Apr 24 17:55:36 emonpi systemd[1]: Stopped Emoncms service-runner Input Script.
Apr 24 17:55:36 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Unit entered failed state.
Apr 24 17:55:36 emonpi systemd[1]: service-runner.service: Failed with result 'signal'.
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Service hold-off time over, scheduling restart.
Apr 24 17:56:06 emonpi systemd[1]: Stopped Emoncms service-runner Input Script.
Apr 24 17:56:06 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Unit entered failed state.
Apr 24 17:56:06 emonpi systemd[1]: service-runner.service: Failed with result 'signal'.
Apr 24 17:56:36 emonpi systemd[1]: service-runner.service: Service hold-off time over, scheduling restart.
Apr 24 17:56:36 emonpi systemd[1]: Stopped Emoncms service-runner Input Script.
Apr 24 17:56:36 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:56:36 emonpi systemd[1]: service-runner.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:56:36 emonpi systemd[1]: service-runner.service: Unit entered failed state.
Apr 24 17:56:36 emonpi systemd[1]: service-runner.service: Failed with result 'signal'.
Apr 24 17:57:07 emonpi systemd[1]: service-runner.service: Service hold-off time over, scheduling restart.
Apr 24 17:57:07 emonpi systemd[1]: Stopped Emoncms service-runner Input Script.
Apr 24 17:57:07 emonpi systemd[1]: Started Emoncms service-runner Input Script.
Apr 24 17:57:07 emonpi systemd[1]: service-runner.service: Main process exited, code=killed, status=11/SEGV
Apr 24 17:57:07 emonpi systemd[1]: service-runner.service: Unit entered failed state.
Apr 24 17:57:07 emonpi systemd[1]: service-runner.service: Failed with result 'signal'.
Apr 24 17:57:37 emonpi systemd[1]: service-runner.service: Service hold-off time over, scheduling restart.
(John) #16

I exported the emonpi config via ssh
performed a restore to factory default via ssh

did NOT import config just to see if emonhub is now running… Emonhub stil fails to start with exact same failure messages as before…

any input?
Regards

(Brian Orpin) #17

As both of these are python scripts I think that might be the culprit. You could try and reinstall it.

However, I suggest getting a brand new image on a new SD card https://github.com/openenergymonitor/emonpi/wiki/emonSD-pre-built-SD-card-Download-&-Change-Log and try that.

emonSD next steps
(Robin Davies) #18

My EmonPi stopped reporting readings a couple of days ago (the 27th or 28th April)…but I only noticed yesterday. It had been running perfectly for some weeks.I restarted my EmonPi, tried updating all the software…but to no avail, - I just can’t get readings from the EmonPi. The error that’s reported is the same as @mickeymouse690 - EmonHub fails to start

-- Logs begin at Thu 2016-11-03 17:16:43 UTC, end at Thu 2019-05-02 08:28:05 UTC. -- May 01 21:41:32 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart.
May 01 21:41:32 emonpi systemd[1]: Stopped emonHub service description.
May 01 21:41:32 emonpi systemd[1]: Started emonHub service description.
May 01 21:41:32 emonpi systemd[1]: emonhub.service: Main process exited, code=killed, status=11/SEGV May 01 21:41:32 emonpi systemd[1]: emonhub.service: Unit entered failed state.
May 01 21:41:32 emonpi systemd[1]: emonhub.service: Failed with result 'signal'.
May 01 21:41:32 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart.
May 01 21:41:33 emonpi systemd[1]: Stopped emonHub service description.
May 01 21:41:33 emonpi systemd[1]: Started emonHub service description.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Main process exited, code=killed, status=11/SEGV May 01 21:41:33 emonpi systemd[1]: emonhub.service: Unit entered failed state.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Failed with result 'signal'.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart.
May 01 21:41:33 emonpi systemd[1]: Stopped emonHub service description.
May 01 21:41:33 emonpi systemd[1]: Started emonHub service description.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Main process exited, code=killed, status=11/SEGV May 01 21:41:33 emonpi systemd[1]: emonhub.service: Unit entered failed state.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Failed with result 'signal'.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart.
May 01 21:41:33 emonpi systemd[1]: Stopped emonHub service description.
May 01 21:41:33 emonpi systemd[1]: Started emonHub service description.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Main process exited, code=killed, status=11/SEGV May 01 21:41:33 emonpi systemd[1]: emonhub.service: Unit entered failed state.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Failed with result 'signal'.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Service hold-off time over, scheduling restart.
May 01 21:41:33 emonpi systemd[1]: Stopped emonHub service description.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Start request repeated too quickly.
May 01 21:41:33 emonpi systemd[1]: Failed to start emonHub service description.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Unit entered failed state.
May 01 21:41:33 emonpi systemd[1]: emonhub.service: Failed with result 'signal'.

(interestingly, the log viewer has the error messages, but when I try and download the log it tells me that /var/log/emonhub/emonhub.log does not exist!)

In case this is a more systemic issue, rather than just a coincidence…is there any information that i can post that might help someone debug the issue?

Thanks

(Brian Orpin) #20

Did the emonhub not restart at this point? Did you wait and see?

Did you just update emoncms or Raspbian as well?

(Robin Davies) #21

I waited…and waited…but no restart. I then tried to restart EmonHub separately through the admin interface as well, but again to no avail!

I only updated the EmonCMS software via the admin interface buttons…I’ve done nothing to update Raspbian (which I assume i would need to ssh into the RaspPi and run ‘apt-get update’. I could look at doing that if recommended.

I have just tried ssh’ing into the EmonPi (from my phone over a VPN…isn’t technology great!) - and I have access to try some commands now.

Thanks

(Brian Orpin) #22

Hi Robin, certainly do not update via apt. It is breaking the database setup.

This error is really strange.

How old is the setup?