Emoncms not receiving data

bgrattan · 19 March 2018 14:42

Hello,

It seems my house power monitoring system has been working perfectly for the past couple of months and, for some unknown reason, stopped posting to emoncms last night. I have the following system:

House power monitoring system (USA) with the following:

RPI 3 + RFM69Pi_433 loaded with flashed OEM software on 32 GB SD (latest updates)
emonTXV3 + AC/AC power, (purchased July 2017)
2 x SCT023R (200A:50MA) on 220 mains (North America 2 x 110)
emonTH V2 (purchased Oct 2017)

weewx weather station (VP2) on Node 0

The log indicates that data is being published but the only thing showing up in emoncms (graphs, feeds, inputs) is the weather station data which never seemed to stop.

+++++++++++++++++++++++++++++++++++
Sample log:

MQTT       Publishing: emon/emonth5/rssi -69
2018-03-19 10:30:03,729 INFO     MQTT       Publishing: emonhub/rx/23/values 12.4,0,52,3,1,-69
2018-03-19 10:30:07,340 DEBUG    RFM2Pi     1082 NEW FRAME : OK 8 182 2 184 2 134 0 75 0 192 42 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-22)
2018-03-19 10:30:07,343 DEBUG    RFM2Pi     1082 Timestamp : 1521469807.34
2018-03-19 10:30:07,343 DEBUG    RFM2Pi     1082 From Node : 8
2018-03-19 10:30:07,344 DEBUG    RFM2Pi     1082    Values : [1483.4944, 1487.7695999999999, 134, 75, 120.54159360000001, 300, 300, 300, 300, 300, 300, 0]
2018-03-19 10:30:07,345 DEBUG    RFM2Pi     1082      RSSI : -22
2018-03-19 10:30:07,345 DEBUG    RFM2Pi     1082 Sent to channel(start)' : ToEmonCMS
2018-03-19 10:30:07,346 DEBUG    RFM2Pi     1082 Sent to channel(end)' : ToEmonCMS
2018-03-19 10:30:07,591 DEBUG    MQTT       Publishing: emon/emontx3/power1 1483.4944
2018-03-19 10:30:07,593 DEBUG    MQTT       Publishing: emon/emontx3/power2 1487.7696
2018-03-19 10:30:07,595 DEBUG    MQTT       Publishing: emon/emontx3/power3 134
2018-03-19 10:30:07,597 DEBUG    MQTT       Publishing: emon/emontx3/power4 75
2018-03-19 10:30:07,598 DEBUG    MQTT       Publishing: emon/emontx3/vrms 120.5415936
2018-03-19 10:30:07,600 DEBUG    MQTT       Publishing: emon/emontx3/temp1 300
2018-03-19 10:30:07,601 DEBUG    MQTT       Publishing: emon/emontx3/temp2 300
2018-03-19 10:30:07,603 DEBUG    MQTT       Publishing: emon/emontx3/temp3 300
2018-03-19 10:30:07,604 DEBUG    MQTT       Publishing: emon/emontx3/temp4 300
2018-03-19 10:30:07,606 DEBUG    MQTT       Publishing: emon/emontx3/temp5 300
2018-03-19 10:30:07,607 DEBUG    MQTT       Publishing: emon/emontx3/temp6 300
2018-03-19 10:30:07,609 DEBUG    MQTT       Publishing: emon/emontx3/pulse 0
2018-03-19 10:30:07,611 DEBUG    MQTT       Publishing: emon/emontx3/rssi -22
2018-03-19 10:30:07,612 INFO     MQTT       Publishing: emonhub/rx/8/values 1483.4944,1487.7696,134,75,120.5415936,300,300,300,300,300,300,0,-22
2018-03-19 10:30:17,321 DEBUG    RFM2Pi     1083 NEW FRAME : OK 8 179 2 184 2 144 0 74 0 200 42 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-30)

+++++++++++++++++++++++++++++++++++++
Any suggestions on where to look would be appreciated. Thanks
Bob

borpin · 19 March 2018 21:28

How is the weather data being sent to you EmonCMS install?

Can you get the output from the 3 command below (possible linked issue though not an EmonPi)

pb66 · 19 March 2018 22:07

It looks like data is pasing through emonhub ok, next thing to check as Brian suggests it the mqtt_input service

sudo systemctl status mqtt_input

if it’s not running try

sudo systemctl restart mqtt_input
sudo systemctl status mqtt_input

bgrattan · 19 March 2018 22:19

Brian,

Many thanks for answering my cry for help. I’m not much of a programmer so I’m easily confused when something goes wrong.

To answer your first question. My weewx software has the following info in weewx.conf:

#   This section is for uploading data to Internet sites
[StdRESTful]
 [[EmonCMS]]
        url = http://192.168.1.115/emoncms/input/post.json
        token = [token from emon]

As for the emoncms, I have the following:

● emonhub.service - LSB: Start/stop emonHub
   Loaded: loaded (/etc/init.d/emonhub)
   Active: active (exited) since Mon 2018-03-19 14:19:28 EDT; 3h 29min ago

● feedwriter.service - LSB: feedwriter script daemon
   Loaded: loaded (/etc/init.d/feedwriter)
   Active: active (running) since Mon 2018-03-19 16:37:51 EDT; 1h 11min ago

● mqtt_input.service - Emoncms MQTT Input Script
   Loaded: loaded (/etc/systemd/system/mqtt_input.service; enabled)
   Active: active (running) since Mon 2018-03-19 14:19:51 EDT; 3h 29min ago

I restarted the emonhub service. Earlier, I got the same “active (exited)” message and restarted it. However,
this didn’t solve the problem.
My “inputs” do show activity but the “feeds” show 30 hours since updated.
As I mentioned, the weather station, on the same local network (Ethernet) has been running and updating all along.
Yesterday I did an apt update/upgrade and an rpi-update on the firmware. I also did an emonpi update which I do several times a week and repeated this morning.

Much appreciated for the help so far.
Bob

pb66 · 19 March 2018 22:30

I think what Brian was trying to determine when asking about your weatherstation was how the data was routed to emoncms to help debug the issue you have, which it looks like the weatherstation is posting direct to emoncms via http, so that narrows the field a little.

can you provide some emonhub.logs, preferably leading up to when it fails, if you don’t have those you mat need to wait til it occurs again.

Is that “30hrs” reducing? If there was data in the emonhub buffers when you restarted emonhub it will have beeb lost and you will have a hole in your data. But it should resume logging if your inputs are updating.

bgrattan · 19 March 2018 22:39

Paul,
emonCMS has been down for about 30 hours and the number is advancing. An app on my iPad that usually shows the day’s KWHs now shows error, cannot connect to emonCMS.

end of latest emonhub.log:

2018-03-19 18:37:29,268 INFO     MQTT       Publishing: emonhub/rx/8/values 256.512,331.328,136,81,121.98448,300,300,300,300,300,300,0,-32
2018-03-19 18:37:33,090 DEBUG    RFM2Pi     340 NEW FRAME : OK 23 142 0 0 0 4 2 30 0 1 0 0 0 (-68)
2018-03-19 18:37:33,092 DEBUG    RFM2Pi     340 Timestamp : 1521499053.09
2018-03-19 18:37:33,093 DEBUG    RFM2Pi     340 From Node : 23
2018-03-19 18:37:33,094 DEBUG    RFM2Pi     340    Values : [14.200000000000001, 0, 51.6, 3, 1]
2018-03-19 18:37:33,095 DEBUG    RFM2Pi     340      RSSI : -68
2018-03-19 18:37:33,095 DEBUG    RFM2Pi     340 Sent to channel(start)' : ToEmonCMS
2018-03-19 18:37:33,096 DEBUG    RFM2Pi     340 Sent to channel(end)' : ToEmonCMS
2018-03-19 18:37:33,308 DEBUG    MQTT       Publishing: emon/emonth5/temperature 14.2
2018-03-19 18:37:33,310 DEBUG    MQTT       Publishing: emon/emonth5/external temperature 0
2018-03-19 18:37:33,312 DEBUG    MQTT       Publishing: emon/emonth5/humidity 51.6
2018-03-19 18:37:33,314 DEBUG    MQTT       Publishing: emon/emonth5/battery 3
2018-03-19 18:37:33,316 DEBUG    MQTT       Publishing: emon/emonth5/pulsecount 1
2018-03-19 18:37:33,318 DEBUG    MQTT       Publishing: emon/emonth5/rssi -68
2018-03-19 18:37:33,320 INFO     MQTT       Publishing: emonhub/rx/23/values 14.2,0,51.6,3,1,-68
2018-03-19 18:37:39,066 DEBUG    RFM2Pi     341 NEW FRAME : OK 8 121 0 152 0 141 0 81 0 68 43 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-20)
2018-03-19 18:37:39,069 DEBUG    RFM2Pi     341 Timestamp : 1521499059.07
2018-03-19 18:37:39,070 DEBUG    RFM2Pi     341 From Node : 8
2018-03-19 18:37:39,070 DEBUG    RFM2Pi     341    Values : [258.6496, 324.91519999999997, 141, 81, 121.99549440000001, 300, 300, 300, 300, 300, 300, 0]
2018-03-19 18:37:39,071 DEBUG    RFM2Pi     341      RSSI : -20
2018-03-19 18:37:39,072 DEBUG    RFM2Pi     341 Sent to channel(start)' : ToEmonCMS
2018-03-19 18:37:39,072 DEBUG    RFM2Pi     341 Sent to channel(end)' : ToEmonCMS
2018-03-19 18:37:39,364 DEBUG    MQTT       Publishing: emon/emontx3/power1 258.6496
2018-03-19 18:37:39,367 DEBUG    MQTT       Publishing: emon/emontx3/power2 324.9152
2018-03-19 18:37:39,369 DEBUG    MQTT       Publishing: emon/emontx3/power3 141
2018-03-19 18:37:39,371 DEBUG    MQTT       Publishing: emon/emontx3/power4 81
2018-03-19 18:37:39,374 DEBUG    MQTT       Publishing: emon/emontx3/vrms 121.9954944
2018-03-19 18:37:39,376 DEBUG    MQTT       Publishing: emon/emontx3/temp1 300
2018-03-19 18:37:39,379 DEBUG    MQTT       Publishing: emon/emontx3/temp2 300
2018-03-19 18:37:39,381 DEBUG    MQTT       Publishing: emon/emontx3/temp3 300
2018-03-19 18:37:39,383 DEBUG    MQTT       Publishing: emon/emontx3/temp4 300
2018-03-19 18:37:39,386 DEBUG    MQTT       Publishing: emon/emontx3/temp5 300
2018-03-19 18:37:39,388 DEBUG    MQTT       Publishing: emon/emontx3/temp6 300
2018-03-19 18:37:39,390 DEBUG    MQTT       Publishing: emon/emontx3/pulse 0
2018-03-19 18:37:39,392 DEBUG    MQTT       Publishing: emon/emontx3/rssi -20
2018-03-19 18:37:39,395 INFO     MQTT       Publishing: emonhub/rx/8/values 258.6496,324.9152,141,81,121.9954944,300,300,300,300,300,300,0,-20
2018-03-19 18:37:49,055 DEBUG    RFM2Pi     342 NEW FRAME : OK 8 121 0 156 0 136 0 79 0 89 43 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-25)
2018-03-19 18:37:49,058 DEBUG    RFM2Pi     342 Timestamp : 1521499069.05
2018-03-19 18:37:49,059 DEBUG    RFM2Pi     342 From Node : 8
2018-03-19 18:37:49,060 DEBUG    RFM2Pi     342    Values : [258.6496, 333.4656, 136, 79, 122.2267968, 300, 300, 300, 300, 300, 300, 0]
2018-03-19 18:37:49,061 DEBUG    RFM2Pi     342      RSSI : -25
2018-03-19 18:37:49,062 DEBUG    RFM2Pi     342 Sent to channel(start)' : ToEmonCMS
2018-03-19 18:37:49,062 DEBUG    RFM2Pi     342 Sent to channel(end)' : ToEmonCMS
2018-03-19 18:37:49,160 DEBUG    MQTT       Publishing: emon/emontx3/power1 258.6496
2018-03-19 18:37:49,162 DEBUG    MQTT       Publishing: emon/emontx3/power2 333.4656
2018-03-19 18:37:49,163 DEBUG    MQTT       Publishing: emon/emontx3/power3 136
2018-03-19 18:37:49,166 DEBUG    MQTT       Publishing: emon/emontx3/power4 79
2018-03-19 18:37:49,168 DEBUG    MQTT       Publishing: emon/emontx3/vrms 122.2267968
2018-03-19 18:37:49,169 DEBUG    MQTT       Publishing: emon/emontx3/temp1 300
2018-03-19 18:37:49,171 DEBUG    MQTT       Publishing: emon/emontx3/temp2 300
2018-03-19 18:37:49,173 DEBUG    MQTT       Publishing: emon/emontx3/temp3 300
2018-03-19 18:37:49,175 DEBUG    MQTT       Publishing: emon/emontx3/temp4 300
2018-03-19 18:37:49,177 DEBUG    MQTT       Publishing: emon/emontx3/temp5 300
2018-03-19 18:37:49,178 DEBUG    MQTT       Publishing: emon/emontx3/temp6 300
2018-03-19 18:37:49,180 DEBUG    MQTT       Publishing: emon/emontx3/pulse 0
2018-03-19 18:37:49,182 DEBUG    MQTT       Publishing: emon/emontx3/rssi -25
2018-03-19 18:37:49,184 INFO     MQTT       Publishing: emonhub/rx/8/values 258.6496,333.4656,136,79,122.2267968,300,300,300,300,300,300,0,-25

Thanks.
Bob

pb66 · 19 March 2018 22:48

Are you able to log into emoncms? You have said

“My “inputs” do show activity” and “emonCMS has been down for about 30 hours” which conflict.

Assuming you have not changed any input processing, if the inputs are updating the feeds should be logging, if they are not it could be the feed writer

sudo systemctl status feedwriter

but that appeared fine in your earlier info.

bgrattan · 19 March 2018 22:53

● feedwriter.service - LSB: feedwriter script daemon
   Loaded: loaded (/etc/init.d/feedwriter)
   Active: active (running) since Mon 2018-03-19 16:37:51 EDT; 2h 12min ago
  Process: 19726 ExecStop=/etc/init.d/feedwriter stop (code=exited, status=0/SUCCESS)
  Process: 19741 ExecStart=/etc/init.d/feedwriter start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/feedwriter.service
           └─19758 /usr/bin/php -f /var/www/emoncms/scripts/feedwriter.php

Mar 19 16:37:44 emonpi feedwriter[19741]: Log is turned off
Mar 19 16:37:44 emonpi feedwriter[19741]: Starting RPI
Mar 19 16:37:51 emonpi systemd[1]: Started LSB: feedwriter script daemon.

I can log into emonCMS and the “inputs” screen shows they are getting data on schedule. The “feeds” screen shows 30 hours since anything was received (except for the Node 0 which is the weather station and it shows data every 15 minutes which is correct).

pb66 · 19 March 2018 23:04

Is there space for data?

df -h

Have the permissions/ownership of the data files been altered? There would normally be some errors in the emoncms.log if this was the case (viewed via admin page).

but you could try this to check them

ls -la /home/pi/data/php*

If the data files are the correct path, ownership and permissions, there is room for the data and the feedwriter is running ok, then it would suggest an issue in the input processing. Have you made any changes?

None of the above explains why your tablet won’t connect when your weatherstation can. Have you tried a browser refresh or tried navigating to another emoncms page?

bgrattan · 19 March 2018 23:31

ls -la /home/pi/data/php*
/home/pi/data/phpfina:
total 122349
drwxr-xr-x  2 www-data root        1024 Nov 24 14:26 .
drwxrwxrwx 10 pi       pi          1024 Nov 20 07:58 ..
-rw-r--r--  1 www-data www-data 7252312 Mar 18 12:38 14.dat
-rw-r--r--  1 www-data www-data      16 Aug 20  2017 14.meta

…etc

/home/pi/data/phptimeseries:
total 2
drwxr-xr-x  2 www-data root 1024 Nov  7  2016 .
drwxrwxrwx 10 pi       pi   1024 Nov 20 07:58 ..

root@emonpi(ro):emonhub# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.4G  2.2G  1.1G  69% /
devtmpfs        484M     0  484M   0% /dev
tmpfs           489M     0  489M   0% /dev/shm
tmpfs           489M   13M  476M   3% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           489M     0  489M   0% /sys/fs/cgroup
tmpfs            40M  6.1M   34M  16% /var/lib/openhab
tmpfs           1.0M  8.0K 1016K   1% /var/lib/dhcpcd5
tmpfs            50M  5.5M   45M  11% /var/log
tmpfs            30M  144K   30M   1% /tmp
tmpfs           1.0M  4.0K 1020K   1% /var/lib/dhcp
/dev/mmcblk0p1   60M   23M   38M  38% /boot
/dev/mmcblk0p3   25G  159M   24G   1% /home/pi/data
tmpfs            98M     0   98M   0% /run/user/1000

I haven’t changed anything except for the updates. Nothing in the feeds, configurations, etc…
After the rpi updates, everything ran fine for about 8 hours.
I can log into emonCMS but it just isn’t getting any data in the feeds. Refresh does nothing new.

The system was working very well and I was getting data withing several KWHs of the power company over two months. All of a sudden it stops sending data to the established feeds…

Thanks,
Bob

PS Don’t you guys ever sleep…?_

pb66 · 19 March 2018 23:46

Of course we do, I’m actually asleep right now! (Well not far off)

I’m afraid I’m running a bit short on idea’s right now. If you have run updates and restarted services I don’t think you will have any data buffered anywhere, it might be worth trying a reboot and keeping an eye on it if it appears ok. It’s not a fix, but it might help get you restarted quicker, and it will tell us if this is a permanent fault or not (if it persists a reboot).

bgrattan · 19 March 2018 23:56

I’ll reboot and get back with you tomorrow. Thanks again for the help.
Bob

bgrattan · 20 March 2018 00:04

After reboot. Feeds not updating. Not sure why there are two sets. Let me know if the page print doesn’t come through and I’ll resend it. Thanks
Bob

pb66 · 20 March 2018 00:19

I can see what’s happened but I can’t say why at this point. But if you delete all the second set of inputs, essentially delete all the green updating inputs, but be aware as you do the red ones will turn green so be careful not to delete too many. Start at the 2nd “power1” (first green input) and work down the list as each is deleted it will move up a position.

What’s happened is there has been a “glitch” that has resulted in a duplicate set of inputs and as they are the newest, they are getting updated whilst the original ones with your processing are getting stale.

Once deleted all will be well, but you should keep an eye on it in case it reoccurs.

bgrattan · 20 March 2018 15:58

I followed your directions and seem to have everything back to normal. After deleting the duplicate inputs, I had to reconfigure several of the feeds which were more complicated than a simple “log to feed.” Do you have any idea what might have caused the problem? As I mentioned before, the only changes I made were the updates to the rpi firmware/software and to the emon software and things ran fine for about 8 hours.
After adjustments to the AC voltage and CTs (Fluke meter), my measurements were coming within several KWHs of what the power company charges me for (average 1200/month).
Much appreciated for the help.
Bob

pb66 · 20 March 2018 16:29

That was what my rather wordy explanations were trying to head off.

There was no need to delete any inputs that had processing, the duplicate inputs were void of any processing so it sounds like you got a little keen with the deleting.

In short, no I do not know why it occurs, but I am pleased to say we see it less often these days. I think there is something causing the originals not to be found at the right time, eg redis not populating at the first input, maybe it clashes with something else, I have no idea. But it does seem rare these days. If it does happen again, please try and recall what might be the trigger, now you know what to look for you might notice it nearer the event. But it may never happen again.

borpin · 20 March 2018 17:23

@TrystanLea on an EmonPi is there a requirement to check the databases or does that update button do it for the user?

pb66 · 20 March 2018 17:27

It’s included in the emonpi update routine.

borpin · 20 March 2018 17:44

@pb66 one other general thought, I’ve seen myself when posting multiple items to MQTT from a microprocessor, it is not always as quick as you’d think, which can cause problems. As part of the emonhub development perhaps using a JSON format that posts the lot as a single MQTT feed might help.

pb66 · 20 March 2018 23:16

I totally agree, Have a read of the EmonHub Development thread. Trystan is looking into making the mqtt inputs in emoncms work as per the http bulk upload and changing emoncms so that both indexed (csv) and named (key:value) inputs can at last be used interchangeably rather than having to use one or the other exclusively (per input), emonhub will hopefully be able to post complete payloads via mqtt and in batches too if buffering is needed (eg mqtt across machines or networks).