Emoncms updates stop until local emonhub restart

Emoncms
1

Hi,

I’ve been advised to post on here following a Twitter post regarding a problem I have experienced with emoncms updates failing after running out of credit.

Since approx 5th July I have had four separate instances where the upload to emoncms just stops and so the usage registered is frozen on the last upload value and repeated thereby causing incorrect values to be registered. During this time until I realise the upload has ‘crashed’ the local system continues to log correctly.

Remote emoncms with corrupt data due to failed upload:

Screenshot_20200725-005439
Screenshot_20200725-0054391080×2160 376 KB

Local emoncms with accurate data:

Screenshot_20200725-013814
Screenshot_20200725-0138141080×2160 501 KB

If I restart emonhub then the updates start to work again for a random amount of time before crashing again, but I lose all of the accurate feed data on emoncms.

At first I thought this was related to running out of credit on emoncms, which I wasn’t aware of as no reminder was sent, but I’ve been told this wouldn’t have been the cause.

I’m not able to upload the emonhub log but the system details are as follows:

Server Information

Server Information

Services

  • emonhub :- Active Running
  • emoncms_mqtt :- Active Running
  • feedwriter :- Active Running - sleep 300s 530 feed points pending write
  • service-runner :- Active Running
  • emonPiLCD :- Active Running
  • redis-server :- Active Running
  • mosquitto :- Active Running
  • demandshaper :- Active Running

Emoncms

Server

  • OS :- Linux 4.19.75-v7+
  • Host :- emonpi | emonpi | (192.168.1.130)
  • Date :- 2020-07-25 01:04:38 BST
  • Uptime :- 01:04:38 up 34 days, 10:45, 0 users, load average: 0.23, 0.20, 0.23

Memory

  • RAM :- Used: 22.09%
    • Total :- 975.63 MB
    • Used :- 215.53 MB
    • Free :- 760.1 MB
  • Swap :- Used: 58.37%
    • Total :- 100 MB
    • Used :- 58.37 MB
    • Free :- 41.63 MB
      Write Load Period

Disk

  • / :- Used: 52.11%
    • Total :- 3.92 GB
    • Used :- 2.04 GB
    • Free :- 1.69 GB
    • Write Load :- 1.33 KB/s (19 days 2 hours 50 mins)
  • /boot :- Used: 20.55%
    • Total :- 252.05 MB
    • Used :- 51.79 MB
    • Free :- 200.26 MB
    • Write Load :- 0.01 B/s (19 days 2 hours 50 mins)
  • /var/opt/emoncms :- Used: 7.04%
    • Total :- 9.98 GB
    • Used :- 719.65 MB
    • Free :- 8.77 GB
    • Write Load :- 522.27 B/s (19 days 2 hours 50 mins)
  • /var/log :- Used: 74.77%
    • Total :- 50 MB
    • Used :- 37.39 MB
    • Free :- 12.61 MB
    • Write Load :- n/a

HTTP

  • Server :- Apache/2.4.38 (Raspbian) HTTP/1.1 CGI/1.1 80

MySQL

  • Version :- 5.5.5-10.3.17-MariaDB-0+deb10u1
  • Host :- localhost:6379 (127.0.0.1)
  • Date :- 2020-07-25 01:04:37 (UTC 01:00‌​)
  • Stats :- Uptime: 2976406 Threads: 10 Questions: 86940 Slow queries: 0 Opens: 60 Flush tables: 1 Open tables: 54 Queries per second avg: 0.029

Redis

  • Version :-
    • Redis Server :- 5.0.3
    • PHP Redis :- 5.0.2
  • Host :- localhost:6379
  • Size :- 600 keys (996.12K)
  • Uptime :- 34 days

MQTT Server

  • Version :- Mosquitto 1.5.7
  • Host :- localhost:1883 (127.0.0.1)

PHP

  • Version :- 7.3.9-1~deb10u1 (Zend Version 3.3.9)
  • Modules :- apache2handlercalendar Core ctype curl date dom v20031129exif fileinfo filter ftp gd gettext hash iconv json v1.7.0libxml mbstring mosquitto v0.4.0mysqli mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: 7cc7cc96e675f6d72e5cf0f267f48e167c2abb23 $openssl pcre PDO pdo_mysql Phar posix readline redis v5.0.2Reflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer wddx xml xmlreader xmlwriter xsl Zend OPcache zlib

Pi

  • Model :- Raspberry Pi 2 Model B Rev 1.1 - 1GB (Sony UK)

  • Serial num. :- D80D88EC

  • CPU Temperature :- 39.01°C

  • GPU Temperature :- 39.0°C

  • emonpiRelease :- emonSD-17Oct19

  • File-system :- read-write

Client Information # Client Information ## HTTP
  • Browser :- Mozilla/5.0 (Linux; Android 10; ONEPLUS A5010) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36
  • Language :- en-GB,en-US;q=0.9,en;q=0.8

Window

  • Size :- 412 x 783

Screen

  • Resolution :- 412 x 823

I have attempted to run a system update but it looks to already be up to date and the problem has reoccurred since then anyway.

Any troubleshooting tips to assist in diagnosing the problem would be appreciated.

Also, how do I go about re-uploading feed data from local to remote to replace the corrupt data?

Thanks

2

IIRC there is currently or recently an issue here emonhub stops sending data to emoncms.org due to the emonhub http interfacer crashing/dying. I do not know if that has been fixed off hand, @TrystanLea. @borpin or @bwduncan may be able to assist here.

To confirm if this is the case, can you please check or post your emonhub.log for a period when emoncms.org is not updaing or better still, the time when it stops posting, although the latter can be somewhat harder to achieve.

This assume there is no local network issue like a broken WAN or DNS issue etc as that would also stop just the emoncms.org data.

The local emoncms will continue to update via MQTT and isn’t effected by the above.

Well actually the absent data is just missing whilst it isn’t able to post to emoncms.org. What you see is the last good value, before the “crash” and the first good value after the crash linked together as the graphing just goes from point to point so it only appears stationary at the last good value, in fact it’s empty.

Having said that, when the first new value arrives, it will assume the new power value to be an average of the period since the previous value, so all your kWh feeds derived from the missing power inputs will be updated based on the post-crash input rather than the pre-crash input, so still wrong, but it won’t match the graph appearance. ie if the is 0w power at first post-crash post then 0w x duration of crash = 0kWh so the kWh feeds won’t increase but the graph will show a straight line from the last pre-crash input value to 0 across the crash period, indicating a steady drop to 0 over that period, which is at odds with the 0kWh recorded.

I guess the system may have stopped you as you are a new user, I have up’d user user level so you should be able to post it now.

I do not think there is any easy way to do that, but that is something that can looked at once it’s fixed and no more holes are likely to appear.

3

Thank you very much for your reply @pb66 and the explanation of the averaging of the feeds while the uploads aren’t taking place.

On seeing your reply I have checked emoncms and the uploads stopped again at approximately 13:15 today, I have attached the emonhub log but it appears to only store half hour or so, the last time I checked emoncms was this morning so unless there is an archive of the log I guess that data has been overwritten?

I don’t believe there to be an issue locally, the internet connection is stable and other devices (myEnergi, Powerwall, FireTV, etc.) have remained connected and operational during today.

I have attached the log files from last night/this morning when it has crashed and I’d restarted emonhub (emonhub.log.txt) and one from the last half hour after realising the feed upload had stopped again (emonhub (1).log.txt).

Thanks,
Colin

emonhub.log.txt (458.9 KB) emonhub (1).log.txt (288.8 KB)

4

Just noticed that the upload stopped again about 23:44 after only an hour of successful operation this time round, log file attached.

Thanks,
Colin
emonhub (2) (1).log.txt (66.8 KB)

5

The upload stopped again but this time the error is logged in the emonhub log…

2020-07-26 01:04:51,096 WARNING emoncmsorg emoncmsorg couldn’t send to server: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))
2020-07-26 01:04:51,097 WARNING emoncmsorg send failure: wanted ‘ok’ but got ‘None’

Looks to have failed around 00:58 and then every attempt since then timesout after about 50 seconds with the ‘remote end closed connection error’.

Is this enough to identify what is causing the problem and is it at the hosted emonCMS end as the message suggests?

Thanks,
Colin

emonhub (captured error 2).log.txt (1.4 MB)

6

Thanks for all the logs, I will take a look at them later hopefully, but yes on the face of it, it does look like it’s the emoncms.org end at fault. But whether that is in (poor) response to a malformed request or a recent change in the emonhub code is still a possibility at this stage.

Apparently the logs are now rotated at just 0.5M so yes you would need to be very quick to catch the error messages in the live log. The logs are rotated out of RAM to disk so there should be an archive of several recent logs. I do not run the OEM images so I do not know offhand where or how much etc try running the following command in an ssh session ls -la /var/log* to see if there is a /var/log_old directory, if there is, you should find some old logs in there, possibly in a emonhub sub folder.

I will post here if/when I find anything useful in your logs and in the meantime I’ll nudge @TrystanLea again as this may affect emoncms.org regardless of which end is at fault, he maybe able to see whats happening at that end.

7

I think it should be fixed by this that has been merged to stable

Can you ping emoncms.org? This error is raised when emonhub cannot reach the server. Do you see that error repeated? If so, this is usually a problem with your local network.

8

That is correct. emonhub does that itself rather than logrotate being involved.

9

Hi,

I can ping the domain and when the problem occurs I have other devices on the network that continue to work (powerwall, myEnergi devices, SolarEdge app, etc) and today when it has failed I’ve been at home connected to VPN which is very sensitive to network loss or glitches and I haven’t noticed any.

I’ve ran an update on the emonPi again and rebooted but it has still failed at about 18:00 today. I only realised a little before 21:00 so the attached log doesn’t show the actual failure entry like before. But after the emonhub restart it does show the upload attempts to emoncms every minute or so, and this isn’t present before the restart, almost like the system is failing once (for whatever reason) and then never attempts to upload again until the emonhub restart.

Even if there was a problem with the internal network (DNS, internet connection, a.n.other) I would have expected the system to fail gracefully and attempt the connection request at the next polling interval, or is it coded to just stop and wait for manual intervention?

Thanks,
Colin

emonhub_20200727_1.log.txt (1.2 MB)

10

Ah, you’ve got a TX as well as a TH and an emonPi - correct?

11

This may well not make sense as I thought this was the emonTH issue another user is having!!

That is (probably) irrelevant to the TH issues as the comms is via RFM not Wi-Fi. responding to wrong issue - sorry However,

So looking at your log, are you sending data to emoncms.org? It looks like it, but equally looks like this is failing (as there are no log entries before the restart) and that could be related.

So to get a bigger log, you need to pipe the tail to a file. Point it to the data partition so you don’t run out of space. Start the tail, then restart emonhub and wait for it to fail. You may need to zip the resulting file to post it.

tail -f /var/log/emonhub/emonhub.log >> /var/opt/emoncms/backup/emonhubtail.log

We always welcome code submissions to make this software better.

[edit]
Might be worth running a log against syslog to catch any python exceptions

journalctl -f -u emonhub

You can pipe that as well if desired.

12

I’ve just gone back over this log, and I strongly suggest it is an internal network issue. Whilst you may not see it on other machines, that does not exclude your router ‘not liking’ the Pi. I guess it is connected via Wi-Fi and that is a fickle beast. To exclude Wi-Fi as the issue, you need to either connect to a different AP (preferably one with just a 2.4MHz SSID available) or use an ethernet cable.

Note the software does recover where it can; the failures are interspersed with sending the buffered data.

2020-07-26 00:56:31,799 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1595721365.2993395,5,39,-3,36,241.77,0,0,0,0,0,0,0],[1595721370.2700872,5,41,0,41,242.70000000000002,0,0,0,0,0,0,0],[1595721375.322579,5,38,8,46,239.65,0,0,0,0,0,0,0],[1595721375.4420671,8,0,1,229,0,252.66,300,300,300,300,300,300,0,-29],[1595721380.3053825,5,41,6,47,242.47,0,0,0,0,0,0,0],[1595721385.2897246,5,14,1,15,242.67000000000002,0,0,0,0,0,0,0],[1595721385.4076974,8,0,1,256,0,252.81,300,300,300,300,300,300,0,-28],[1595721390.2920363,5,17,-8,9,242.93,0,0,0,0,0,0,0]]&sentat=1595721391
2020-07-26 00:57:49,201 WARNING  emoncmsorg emoncmsorg couldn't send to server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
2020-07-26 00:57:49,202 WARNING  emoncmsorg send failure: wanted 'ok' but got 'None'
2020-07-26 00:58:19,290 DEBUG    emoncmsorg Buffer size: 40
2020-07-26 00:58:19,295 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1595721365.2993395,5,39,-3,36,241.77,0,0,0,0,0,0,0],[1595721370.2700872,5,41,0,41,242.70000000000002,0,0,0,0,0,0,0],[1595721375.322579,5,38,8,46,239.65,0,0,0,0,0,0,0],[1595721375.4420671,8,0,1,229,0,252.66,300,300,300,300,300,300,0,-29],[1595721380.3053825,5,41,6,47,242.47,0,0,0,0,0,0,0],[1595721385.2897246,5,14,1,15,242.67000000000002,0,0,0,0,0,0,0],[1595721385.4076974,8,0,1,256,0,252.81,300,300,300,300,300,300,0,-28],[1595721390.2920363,5,17,-8,9,242.93,0,0,0,0,0,0,0],[1595721395.285541,5,34,7,41,242.9,0,0,0,0,0,0,0],[1595721395.409052,8,0,1,264,0,252.92000000000002,300,300,300,300,300,300,0,-28],[1595721400.373789,5,-4,16,12,242.43,0,0,0,0,0,0,0],[1595721404.8403149,8,0,1,253,0,252.84,300,300,300,300,300,300,0,-28],[1595721405.4609249,5,16,2,18,242.39000000000001,0,0,0,0,0,0,0],[1595721410.3332882,5,25,-3,22,242.18,0,0,0,0,0,0,0],[1595721414.7877738,8,0,0,266,0,252.95000000000002,300,300,300,300,300,300,0,-28],[1595721415.412424,5,-6,12,6,242.22,0,0,0,0,0,0,0],[1595721420.3501878,5,29,-4,25,243.21,0,0,0,0,0,0,0],[1595721424.6179383,8,0,1,233,0,253.06,300,300,300,300,300,300,0,-28],[1595721425.4440935,5,37,10,47,241.94,0,0,0,0,0,0,0],[1595721430.406607,5,23,2,25,242.91,0,0,0,0,0,0,0],[1595721434.5781276,8,0,1,227,0,253.11,300,300,300,300,300,300,0,-28],[1595721435.3994215,5,48,-4,44,242.09,0,0,0,0,0,0,0],[1595721440.3642502,5,28,0,28,241.68,0,0,0,0,0,0,0],[1595721444.5209508,8,0,0,246,0,253,300,300,300,300,300,300,0,-28],[1595721445.3485134,5,28,2,30,242.86,0,0,0,0,0,0,0],[1595721450.395642,5,8,-1,7,242.98000000000002,0,0,0,0,0,0,0],[1595721454.3391504,8,0,1,248,0,253.04,300,300,300,300,300,300,0,-28],[1595721455.3619828,5,8,9,17,242.15,0,0,0,0,0,0,0],[1595721460.4250638,5,66,-5,61,242.11,0,0,0,0,0,0,0],[1595721464.2810664,8,1,1,213,0,252.96,300,300,300,300,300,300,0,-28],[1595721465.4058502,5,0,-9,-9,241.63,0,0,0,0,0,0,0],[1595721470.359067,5,26,-6,20,242.67000000000002,0,0,0,0,0,0,0],[1595721474.2093008,8,0,0,245,0,253.01000000000002,300,300,300,300,300,300,0,-28],[1595721475.432744,5,52,-1,51,241.32,0,0,0,0,0,0,0],[1595721480.3935678,5,13,4,17,242.36,0,0,0,0,0,0,0],[1595721484.0546203,8,0,1,271,0,253.04,300,300,300,300,300,300,0,-29],[1595721485.477837,5,71,-2,69,242.84,0,0,0,0,0,0,0],[1595721490.5415988,5,10,-9,1,243.13,0,0,0,0,0,0,0],[1595721493.9912837,8,1,2,253,0,252.72,300,300,300,300,300,300,0,-32],[1595721495.530174,5,35,6,41,241.99,0,0,0,0,0,0,0]]&sentat=1595721499
2020-07-26 00:58:19,627 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org
2020-07-26 00:58:49,716 DEBUG    emoncmsorg Buffer size: 9
2020-07-26 00:58:49,718 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1595721500.4966364,5,76,12,88,242.04,0,0,0,0,0,0,0],[1595721503.9340851,8,0,0,285,0,252.92000000000002,300,300,300,300,300,300,0,-29],[1595721505.5649207,5,32,3,35,242.76,0,0,0,0,0,0,0],[1595721510.6261597,5,50,2,52,242.37,0,0,0,0,0,0,0],[1595721513.7884817,8,0,1,304,0,253,300,300,300,300,300,300,0,-28],[1595721515.5335834,5,6,2,8,242.73000000000002,0,0,0,0,0,0,0],[1595721520.5960112,5,52,6,58,243.04,0,0,0,0,0,0,0],[1595721523.7556305,8,1,1,239,0,253.13,300,300,300,300,300,300,0,-29],[1595721525.5933404,5,41,0,41,242.91,0,0,0,0,0,0,0]]&sentat=1595721529
2020-07-26 00:59:46,927 WARNING  emoncmsorg emoncmsorg couldn't send to server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
2020-07-26 00:59:46,928 WARNING  emoncmsorg send failure: wanted 'ok' but got 'None'
2020-07-26 01:00:17,007 DEBUG    emoncmsorg Buffer size: 36
2020-07-26 01:00:17,014 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1595721500.4966364,5,76,12,88,242.04,0,0,0,0,0,0,0],[1595721503.9340851,8,0,0,285,0,252.92000000000002,300,300,300,300,300,300,0,-29],[1595721505.5649207,5,32,3,35,242.76,0,0,0,0,0,0,0],[1595721510.6261597,5,50,2,52,242.37,0,0,0,0,0,0,0],[1595721513.7884817,8,0,1,304,0,253,300,300,300,300,300,300,0,-28],[1595721515.5335834,5,6,2,8,242.73000000000002,0,0,0,0,0,0,0],[1595721520.5960112,5,52,6,58,243.04,0,0,0,0,0,0,0],[1595721523.7556305,8,1,1,239,0,253.13,300,300,300,300,300,300,0,-29],[1595721525.5933404,5,41,0,41,242.91,0,0,0,0,0,0,0],[1595721530.564131,5,11,10,21,241.91,0,0,0,0,0,0,0],[1595721533.6248238,8,0,1,278,0,252.88,300,300,300,300,300,300,0,-28],[1595721535.551502,5,9,0,9,242.39000000000001,0,0,0,0,0,0,0],[1595721540.6006122,5,26,-6,20,242.28,0,0,0,0,0,0,0],[1595721543.541161,8,0,1,263,0,253.11,300,300,300,300,300,300,0,-28],[1595721545.572008,5,36,8,44,242.79,0,0,0,0,0,0,0],[1595721550.543105,5,28,-2,26,242.52,0,0,0,0,0,0,0],[1595721553.3978488,8,0,0,286,0,252.62,300,300,300,300,300,300,0,-28],[1595721555.6400719,5,32,3,35,241.81,0,0,0,0,0,0,0],[1595721560.5924678,5,16,11,27,242.72,0,0,0,0,0,0,0],[1595721563.335027,8,0,1,285,0,252.93,300,300,300,300,300,300,0,-28],[1595721565.5725543,5,33,-2,31,242.72,0,0,0,0,0,0,0],[1595721570.6284306,5,21,-5,16,242.73000000000002,0,0,0,0,0,0,0],[1595721573.2747355,8,0,1,290,0,252.84,300,300,300,300,300,300,0,-29],[1595721575.719619,5,3,-17,-14,242.39000000000001,0,0,0,0,0,0,0],[1595721580.6719935,5,66,8,74,242.85,0,0,0,0,0,0,0],[1595721583.1107914,8,0,2,250,0,253.08,300,300,300,300,300,300,0,-29],[1595721585.6476479,5,5,12,17,242.62,0,0,0,0,0,0,0],[1595721590.6154983,5,32,-1,31,241.77,0,0,0,0,0,0,0],[1595721593.0571353,8,0,1,240,0,252.9,300,300,300,300,300,300,0,-28],[1595721595.588118,5,46,6,52,242.34,0,0,0,0,0,0,0],[1595721600.6626167,5,46,7,53,242.97,0,0,0,0,0,0,0],[1595721603.0053282,8,0,0,308,0,252.88,300,300,300,300,300,300,0,-28],[1595721605.668775,5,30,11,41,242.74,0,0,0,0,0,0,0],[1595721610.6431515,5,37,-2,35,242.59,0,0,0,0,0,0,0],[1595721612.8798814,8,0,1,229,0,252.8,300,300,300,300,300,300,0,-29],[1595721615.7176247,5,-4,-14,-18,241.5,0,0,0,0,0,0,0]]&sentat=1595721617
2020-07-26 01:00:32,376 WARNING  emoncmsorg emoncmsorg couldn't send to server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
2020-07-26 01:00:32,378 WARNING  emoncmsorg send failure: wanted 'ok' but got 'None'
2020-07-26 01:01:02,464 DEBUG    emoncmsorg Buffer size: 49
2020-07-26 01:01:02,471 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1595721500.4966364,5,76,12,88,242.04,0,0,0,0,0,0,0],[1595721503.9340851,8,0,0,285,0,252.92000000000002,300,300,300,300,300,300,0,-29],[1595721505.5649207,5,32,3,35,242.76,0,0,0,0,0,0,0],[1595721510.6261597,5,50,2,52,242.37,0,0,0,0,0,0,0],[1595721513.7884817,8,0,1,304,0,253,300,300,300,300,300,300,0,-28],[1595721515.5335834,5,6,2,8,242.73000000000002,0,0,0,0,0,0,0],[1595721520.5960112,5,52,6,58,243.04,0,0,0,0,0,0,0],[1595721523.7556305,8,1,1,239,0,253.13,300,300,300,300,300,300,0,-29],[1595721525.5933404,5,41,0,41,242.91,0,0,0,0,0,0,0],[1595721530.564131,5,11,10,21,241.91,0,0,0,0,0,0,0],[1595721533.6248238,8,0,1,278,0,252.88,300,300,300,300,300,300,0,-28],[1595721535.551502,5,9,0,9,242.39000000000001,0,0,0,0,0,0,0],[1595721540.6006122,5,26,-6,20,242.28,0,0,0,0,0,0,0],[1595721543.541161,8,0,1,263,0,253.11,300,300,300,300,300,300,0,-28],[1595721545.572008,5,36,8,44,242.79,0,0,0,0,0,0,0],[1595721550.543105,5,28,-2,26,242.52,0,0,0,0,0,0,0],[1595721553.3978488,8,0,0,286,0,252.62,300,300,300,300,300,300,0,-28],[1595721555.6400719,5,32,3,35,241.81,0,0,0,0,0,0,0],[1595721560.5924678,5,16,11,27,242.72,0,0,0,0,0,0,0],[1595721563.335027,8,0,1,285,0,252.93,300,300,300,300,300,300,0,-28],[1595721565.5725543,5,33,-2,31,242.72,0,0,0,0,0,0,0],[1595721570.6284306,5,21,-5,16,242.73000000000002,0,0,0,0,0,0,0],[1595721573.2747355,8,0,1,290,0,252.84,300,300,300,300,300,300,0,-29],[1595721575.719619,5,3,-17,-14,242.39000000000001,0,0,0,0,0,0,0],[1595721580.6719935,5,66,8,74,242.85,0,0,0,0,0,0,0],[1595721583.1107914,8,0,2,250,0,253.08,300,300,300,300,300,300,0,-29],[1595721585.6476479,5,5,12,17,242.62,0,0,0,0,0,0,0],[1595721590.6154983,5,32,-1,31,241.77,0,0,0,0,0,0,0],[1595721593.0571353,8,0,1,240,0,252.9,300,300,300,300,300,300,0,-28],[1595721595.588118,5,46,6,52,242.34,0,0,0,0,0,0,0],[1595721600.6626167,5,46,7,53,242.97,0,0,0,0,0,0,0],[1595721603.0053282,8,0,0,308,0,252.88,300,300,300,300,300,300,0,-28],[1595721605.668775,5,30,11,41,242.74,0,0,0,0,0,0,0],[1595721610.6431515,5,37,-2,35,242.59,0,0,0,0,0,0,0],[1595721612.8798814,8,0,1,229,0,252.8,300,300,300,300,300,300,0,-29],[1595721615.7176247,5,-4,-14,-18,241.5,0,0,0,0,0,0,0],[1595721620.678896,5,37,0,37,242.18,0,0,0,0,0,0,0],[1595721622.8181,8,0,0,242,0,252.99,300,300,300,300,300,300,0,-29],[1595721625.6719697,5,19,3,22,242.65,0,0,0,0,0,0,0],[1595721630.6494544,5,52,3,55,242.67000000000002,0,0,0,0,0,0,0],[1595721632.6971965,8,0,0,230,0,252.68,300,300,300,300,300,300,0,-28],[1595721635.6549573,5,10,0,10,242.17000000000002,0,0,0,0,0,0,0],[1595721640.6230807,5,40,2,42,242.85,0,0,0,0,0,0,0],[1595721642.5685434,8,0,0,237,0,253.1,300,300,300,300,300,300,0,-28],[1595721645.621127,5,13,9,22,242.9,0,0,0,0,0,0,0],[1595721650.6847603,5,10,1,11,242.61,0,0,0,0,0,0,0],[1595721652.519857,8,0,1,242,0,252.66,300,300,300,300,300,300,0,-29],[1595721655.6785293,5,55,12,67,241.67000000000002,0,0,0,0,0,0,0],[1595721660.7612135,5,53,4,57,242.8,0,0,0,0,0,0,0]]&sentat=1595721662
2020-07-26 01:01:54,935 WARNING  emoncmsorg emoncmsorg couldn't send to server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
2020-07-26 01:01:54,936 WARNING  emoncmsorg send failure: wanted 'ok' but got 'None'
2020-07-26 01:02:25,021 DEBUG    emoncmsorg Buffer size: 74
2020-07-26 01:02:25,030 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1595721500.4966364,5,76,12,88,242.04,0,0,0,0,0,0,0],[1595721503.9340851,8,0,0,285,0,252.92000000000002,300,300,300,300,300,300,0,-29],[1595721505.5649207,5,32,3,35,242.76,0,0,0,0,0,0,0],[1595721510.6261597,5,50,2,52,242.37,0,0,0,0,0,0,0],[1595721513.7884817,8,0,1,304,0,253,300,300,300,300,300,300,0,-28],[1595721515.5335834,5,6,2,8,242.73000000000002,0,0,0,0,0,0,0],[1595721520.5960112,5,52,6,58,243.04,0,0,0,0,0,0,0],[1595721523.7556305,8,1,1,239,0,253.13,300,300,300,300,300,300,0,-29],[1595721525.5933404,5,41,0,41,242.91,0,0,0,0,0,0,0],[1595721530.564131,5,11,10,21,241.91,0,0,0,0,0,0,0],[1595721533.6248238,8,0,1,278,0,252.88,300,300,300,300,300,300,0,-28],[1595721535.551502,5,9,0,9,242.39000000000001,0,0,0,0,0,0,0],[1595721540.6006122,5,26,-6,20,242.28,0,0,0,0,0,0,0],[1595721543.541161,8,0,1,263,0,253.11,300,300,300,300,300,300,0,-28],[1595721545.572008,5,36,8,44,242.79,0,0,0,0,0,0,0],[1595721550.543105,5,28,-2,26,242.52,0,0,0,0,0,0,0],[1595721553.3978488,8,0,0,286,0,252.62,300,300,300,300,300,300,0,-28],[1595721555.6400719,5,32,3,35,241.81,0,0,0,0,0,0,0],[1595721560.5924678,5,16,11,27,242.72,0,0,0,0,0,0,0],[1595721563.335027,8,0,1,285,0,252.93,300,300,300,300,300,300,0,-28],[1595721565.5725543,5,33,-2,31,242.72,0,0,0,0,0,0,0],[1595721570.6284306,5,21,-5,16,242.73000000000002,0,0,0,0,0,0,0],[1595721573.2747355,8,0,1,290,0,252.84,300,300,300,300,300,300,0,-29],[1595721575.719619,5,3,-17,-14,242.39000000000001,0,0,0,0,0,0,0],[1595721580.6719935,5,66,8,74,242.85,0,0,0,0,0,0,0],[1595721583.1107914,8,0,2,250,0,253.08,300,300,300,300,300,300,0,-29],[1595721585.6476479,5,5,12,17,242.62,0,0,0,0,0,0,0],[1595721590.6154983,5,32,-1,31,241.77,0,0,0,0,0,0,0],[1595721593.0571353,8,0,1,240,0,252.9,300,300,300,300,300,300,0,-28],[1595721595.588118,5,46,6,52,242.34,0,0,0,0,0,0,0],[1595721600.6626167,5,46,7,53,242.97,0,0,0,0,0,0,0],[1595721603.0053282,8,0,0,308,0,252.88,300,300,300,300,300,300,0,-28],[1595721605.668775,5,30,11,41,242.74,0,0,0,0,0,0,0],[1595721610.6431515,5,37,-2,35,242.59,0,0,0,0,0,0,0],[1595721612.8798814,8,0,1,229,0,252.8,300,300,300,300,300,300,0,-29],[1595721615.7176247,5,-4,-14,-18,241.5,0,0,0,0,0,0,0],[1595721620.678896,5,37,0,37,242.18,0,0,0,0,0,0,0],[1595721622.8181,8,0,0,242,0,252.99,300,300,300,300,300,300,0,-29],[1595721625.6719697,5,19,3,22,242.65,0,0,0,0,0,0,0],[1595721630.6494544,5,52,3,55,242.67000000000002,0,0,0,0,0,0,0],[1595721632.6971965,8,0,0,230,0,252.68,300,300,300,300,300,300,0,-28],[1595721635.6549573,5,10,0,10,242.17000000000002,0,0,0,0,0,0,0],[1595721640.6230807,5,40,2,42,242.85,0,0,0,0,0,0,0],[1595721642.5685434,8,0,0,237,0,253.1,300,300,300,300,300,300,0,-28],[1595721645.621127,5,13,9,22,242.9,0,0,0,0,0,0,0],[1595721650.6847603,5,10,1,11,242.61,0,0,0,0,0,0,0],[1595721652.519857,8,0,1,242,0,252.66,300,300,300,300,300,300,0,-29],[1595721655.6785293,5,55,12,67,241.67000000000002,0,0,0,0,0,0,0],[1595721660.7612135,5,53,4,57,242.8,0,0,0,0,0,0,0],[1595721662.397204,8,0,1,261,0,252.72,300,300,300,300,300,300,0,-32],[1595721665.7427611,5,14,10,24,242.47,0,0,0,0,0,0,0],[1595721670.7069654,5,52,24,76,241.76,0,0,0,0,0,0,0],[1595721672.3522832,8,0,1,253,0,252.58,300,300,300,300,300,300,0,-28],[1595721675.709659,5,11,-12,-1,242.24,0,0,0,0,0,0,0],[1595721680.6707008,5,39,-9,30,242.55,0,0,0,0,0,0,0],[1595721682.1971116,8,0,1,230,0,252.3,300,300,300,300,300,300,0,-28],[1595721685.7427049,5,28,4,32,240.75,0,0,0,0,0,0,0],[1595721690.7162955,5,73,13,86,241.70000000000002,0,0,0,0,0,0,0],[1595721692.1463468,8,0,1,261,0,252.27,300,300,300,300,300,300,0,-27],[1595721695.7059405,5,54,0,54,241.23000000000002,0,0,0,0,0,0,0],[1595721700.7218435,5,9,1,10,242.11,0,0,0,0,0,0,0],[1595721702.0700736,8,0,1,262,0,252.42000000000002,300,300,300,300,300,300,0,-29],[1595721705.731052,5,8,-3,5,242.27,0,0,0,0,0,0,0],[1595721710.6990197,5,14,-3,11,242.22,0,0,0,0,0,0,0],[1595721711.9357183,8,0,0,254,0,252.39000000000001,300,300,300,300,300,300,0,-29],[1595721715.6870236,5,29,-8,21,242.48000000000002,0,0,0,0,0,0,0],[1595721720.7699866,5,12,-17,-5,242.61,0,0,0,0,0,0,0],[1595721721.8058774,8,0,2,257,0,252.70000000000002,300,300,300,300,300,300,0,-28],[1595721725.7497613,5,0,3,3,242.51,0,0,0,0,0,0,0],[1595721730.7118385,5,31,7,38,242.25,0,0,0,0,0,0,0],[1595721731.7424705,8,0,1,258,0,252.46,300,300,300,300,300,300,0,-28],[1595721735.7077775,5,-1,4,3,241.5,0,0,0,0,0,0,0],[1595721740.7000027,5,18,7,25,242.52,0,0,0,0,0,0,0],[1595721741.6381173,8,0,1,276,0,252.77,300,300,300,300,300,300,0,-28]]&sentat=1595721745
2020-07-26 01:02:25,376 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org
2020-07-26 01:02:55,464 DEBUG    emoncmsorg Buffer size: 9
2020-07-26 01:02:55,467 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1595721745.696244,5,51,2,53,242.39000000000001,0,0,0,0,0,0,0],[1595721750.672602,5,43,-1,42,241.63,0,0,0,0,0,0,0],[1595721751.6044698,8,0,1,234,0,252.61,300,300,300,300,300,300,0,-28],[1595721755.7444847,5,13,11,24,242.24,0,0,0,0,0,0,0],[1595721760.7227302,5,13,-9,4,242.3,0,0,0,0,0,0,0],[1595721761.4483879,8,1,2,261,0,252.55,300,300,300,300,300,300,0,-28],[1595721765.707056,5,31,6,37,242.65,0,0,0,0,0,0,0],[1595721770.7791088,5,29,1,30,242.48000000000002,0,0,0,0,0,0,0],[1595721771.3100579,8,0,1,257,0,252.5,300,300,300,300,300,300,0,-28]]&sentat=1595721775
2020-07-26 01:02:55,792 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org