Wh accumulator - strange behavior

Emoncms
1

I use the Wh accumulator for a accumulated pulse feed as mentioned in the guide: + Add Optical Pulse Sensor - Guide | OpenEnergyMonitor

I noticed that the new feed value sometimes is lower than the old one, which should IMHO not happen with a accumulated feed. This strange behavior happens to the exact same time, if the graph tool is showing a missing value (it’s an emonTh in my case, feed: PHPFINA, interval 60s). After that it is returning to the correct value:

05
05888×533 40.1 KB

07
071077×138 16.7 KB

What could be the reason for that?

2

Unless the power value was negative, and that was added to the feed

3

True, but that isn’t the “strangeness”. The strangeness is it jumped negatively as if it had received a large negative value, but then came back with an equal sized but positive value.

4

One way to get negative numbers is an incorrect handling of signed and unsigned numbers somewhere along the way. If an unsigned number is copied into a signed number it works fine for the first half of the number range, then flips to negative once the upper bit becomes set.

Is the timing of the point of error aligned with higher than average load?

5

Can you zoom in closer and also check the csv via the graphs page?

You are right that shouldn’t happen with the whaccumulator, this is the code for that process.

Knowing why or how those datapoints are missing from the other feed might also shed some light on the events at the time, can you see if there are any clues in emonhub.log and if still none the wiser, add another feed in front of these 2 on the same input and make it a phptimeseries feed, you can delete it again in a few days or when you get to the bottom of the issue. PHPtimeseries will faithfully record all data points with the timestamp, so you can see exactly what data points arrived and when to get a clearer picture than looking at the fixed interval feeds.

[EDIT}
When I count pulses I tend to use a “Total pulse count to pulse increment” followed by an “whAccumulator”, it uses more disk space but it is IMO more dependable, more accurate and far easier to debug pulse counting because it is simpler code and the first process retains the incoming data to a feed (not the increment) and the second one records the new value (last value + increment passed from first process).

You could try setting this method up to compare.

6

Hi @pb66,

first of all thank you very much for your very detailed answer with a lot ideas how to debug that!

Here is a closer zoom:

46
46998×507 19.8 KB

and that’s the corresponding CSV-output:

2017-10-21 10:38:20, 1591.0, 3478.0
2017-10-21 10:38:25, 1591.0, 3478.0
2017-10-21 10:38:30, 1591.0, 3478.0
2017-10-21 10:38:35, 1591.0, 3478.0
2017-10-21 10:38:40, 1591.0, 3478.0
2017-10-21 10:38:45, 1591.0, 3478.0
2017-10-21 10:38:50, 1591.0, 3478.0
2017-10-21 10:38:55, 1591.0, 3478.0
2017-10-21 10:39:00, 1591.0, 3478.0
2017-10-21 10:39:05, 1591.0, 3478.0
2017-10-21 10:39:10, 1591.0, 3478.0
2017-10-21 10:39:15, 1591.0, 3478.0
2017-10-21 10:39:20, 1591.0, 3478.0
2017-10-21 10:39:25, 1591.0, 3478.0
2017-10-21 10:39:30, null, 3441.0
2017-10-21 10:39:35, null, 3441.0
2017-10-21 10:39:40, null, 3441.0
2017-10-21 10:39:45, null, 3441.0
2017-10-21 10:39:50, null, 3441.0
2017-10-21 10:39:55, null, 3441.0
2017-10-21 10:40:00, null, 3441.0
2017-10-21 10:40:05, null, 3441.0
2017-10-21 10:40:10, null, 3441.0
2017-10-21 10:40:15, null, 3441.0
2017-10-21 10:40:20, null, 3441.0
2017-10-21 10:40:25, null, 3441.0
2017-10-21 10:40:30, 1591.0, 3478.0
2017-10-21 10:40:35, 1591.0, 3478.0
2017-10-21 10:40:40, 1591.0, 3478.0
2017-10-21 10:40:45, 1591.0, 3478.0
2017-10-21 10:40:50, 1591.0, 3478.0
2017-10-21 10:40:55, 1591.0, 3478.0
2017-10-21 10:41:00, 1591.0, 3478.0
2017-10-21 10:41:05, 1591.0, 3478.0

Here is another one:

30
30987×554 25.6 KB

This one looks actually fine, but looking at the CSV …

2017-10-21 10:59:30, 1592.0, 3479.0
2017-10-21 10:59:35, 1592.0, 3479.0
2017-10-21 10:59:40, 1592.0, 3479.0
2017-10-21 10:59:45, 1592.0, 3479.0
2017-10-21 10:59:50, 1592.0, 3479.0
2017-10-21 10:59:55, 1592.0, 3479.0
2017-10-21 11:00:00, 1592.0, 3479.0
2017-10-21 11:00:05, 1592.0, 3479.0
2017-10-21 11:00:10, 1592.0, 3479.0
2017-10-21 11:00:15, 1592.0, 3479.0
2017-10-21 11:00:20, 1592.0, 3479.0
2017-10-21 11:00:25, 1592.0, 3479.0
2017-10-21 11:00:30, null, 3478.5
2017-10-21 11:00:35, null, 3478.5
2017-10-21 11:00:40, null, 3478.5
2017-10-21 11:00:45, null, 3478.5
2017-10-21 11:00:50, null, 3478.5
2017-10-21 11:00:55, null, 3478.5
2017-10-21 11:01:00, null, 3478.5
2017-10-21 11:01:05, null, 3478.5
2017-10-21 11:01:10, null, 3478.5
2017-10-21 11:01:15, null, 3478.5
2017-10-21 11:01:20, null, 3478.5
2017-10-21 11:01:25, null, 3478.5
2017-10-21 11:01:30, 1592.0, 3479.0
2017-10-21 11:01:35, 1592.0, 3479.0
2017-10-21 11:01:40, 1592.0, 3479.0
2017-10-21 11:01:45, 1592.0, 3479.0
2017-10-21 11:01:50, 1592.0, 3479.0
2017-10-21 11:01:55, 1592.0, 3479.0
2017-10-21 11:02:00, 1592.0, 3479.0
2017-10-21 11:02:05, 1592.0, 3479.0
2017-10-21 11:02:10, 1592.0, 3479.0
2017-10-21 11:02:15, 1592.0, 3479.0
2017-10-21 11:02:20, 1592.0, 3479.0
2017-10-21 11:02:25, 1592.0, 3479.0
2017-10-21 11:02:30, 1594.0, 3481.0
2017-10-21 11:02:35, 1594.0, 3481.0
2017-10-21 11:02:40, 1594.0, 3481.0
2017-10-21 11:02:45, 1594.0, 3481.0
2017-10-21 11:02:50, 1594.0, 3481.0
2017-10-21 11:02:55, 1594.0, 3481.0
2017-10-21 11:03:00, 1594.0, 3481.0

… it shows, also a slight change in value (from 3479.0 to 3478.5).

What I already did, was adding a accumulator and it behaved similar to the wh_accumulator:

21
21925×498 27.1 KB

Very good idea, added it and will report as soon as I have the data.

Thanks for the hint, I added this as well to compare.

7

Just a couple of observations

  1. In your opening post you say the “(it’s an emonTh in my case, feed: PHPFINA, interval 60s)” but the csv is at 5sec intervals. Can you confirm the data source/update interval and the feed types/intervals?

  2. is it just coincidence that both the csv examples fault for 12 x 5sec datapoints? Can you check another occurrence or 2 to see what the duration of the error is?

At first glance I would of chanced a guess that this might have been related to the difference in intervals if you are sending data on 60s intervals and saving it on 5sec intervals, however that would only be visible in the graphs until you zoom in, so that’s unlikely. The whaccumulator csv also confirms the data is being saved every 5secs, but the pulse count seems to contradict that by recording null values ie it suggests the data was backfilled with null values once an input value was received.

  1. I also find it odd that the erroneous value returns to it’s previous value, it doesn’t drop back and continue to accumulate from the lower value, nor does it stay at the lower value despite no new increase, it simply corrects itself.

Is it possible you have rebooted the server or flushed redis so the last value was lost somehow? the nulls suggest a reset of some sort but I imagine a reset of the pulse count source would mean a return to zero, which doesn’t appear to happen.

Hopefully the phptimeseries will give some insight.

8

The feed-API: http://192.168.11.76/emoncms/feed/getmeta.json?id=3 returns: {"interval":60,"start_time":1504647240,"npoints":69201}
45

and for the pulscount (feed id 2): {"interval":60,"start_time":1504647120,"npoints":69215}
I don’t understand why the graph/csv is showing values every 5 second.

I don’t think it’s coincidence. The emonth is sending every minute (=12 x 5 sec) and the fault in the accumulated graph is only as long as the missing data gap.

No, it’s up and running since 20 days:
52

9

Ok! So I just did some tests and discovered, if the Graph module doesn’t find the datapoints needed to fulfil your query, it manufactures them… It seems that you can zoom into a 60sec feed and get 5sec resolution, it seems to replicate exisiting datapoints so that explains why both error examples are 12 datapoints long, it is in fact only 1 erroneous datapoint, repeated 12 times. (it still reports a “quality” of 100%, that should in fact be 1200%)

That data is a little misleading, that reports the uptime of the “physical server” ie the emonPi not the uptime of the actual “Apache server”, you could take your emoncms server offline for a week by shutting down the apache2 service and it would not impact the reported uptime. An example is updating the emonPi, each time you press the “Update emonPi” button on the admin page the apache server is taken offline whilst the update runs regardless of whether there is any updates to be had. Also if you were to use the “flush” button in the redis section it wouldn’t effect the uptime.

Although we now know (strongly suspect?) there is only one missing 60s pulsecount per occurrence that results in a both a null on the pulsecount feed and a single erroneous/reduced value on the whaccumulator feed, the latter still shouldn’t happen. We will probably have to wait and see what the phptimeseries feeds record if/when it occurs again.

In the meantime keep a record of the times of any emonPi updates, reboots, power cuts or any other events that might impact the performance just to eliminate. Plus if you are really keen you could hunt out the emonhub log for the time(s) it missed a count to try and determine why it was missing. Not that knowing will change the fact emoncms should be able cope, but if you can reduce the occurrences until it’s fixed or invoke an occurrence whilst debugging, it might help.

10

I have now a day of values with at least 5 missing packets.

Like @pb66 suggested I added a timeseries feed to store the accumulated values. This feed is not showing erroneous/reduced values. Actually it shows nothing instead, because it does not log any values if no new packets arrive.

I also added a “total pulse to pulse increment” and a “accumulator”. And now it’s getting interesting. The accumulator is showing the same behavior and reduces the values by the exact same number at the same time. For example:

At 2017-10-25 13:27:35 (UTC)
phpfina wh_accumulator reduces from 3984 to 3964.5 = 19.5
phpfina accumulator from 364 to 344.5 = 19.5

At 2017-10-25 03:47:30 (UTC)
from 3871 to 3866 = 5
from 251 to 246 = 5

So it seems that the actual feed values are not relevant.

11

Can you post the phptimeseries csv and if it looks like data points are missing can you post emonhub.log for the same period too.

12

That’s the CSV output of the phptimeseries log (left = log to feed, right = wh_accumulator):

2017-10-25 23:54:59, 2227.0, 494.0
2017-10-25 23:55:57, 2227.0, 494.0
2017-10-25 23:56:56, 2227.0, 494.0
2017-10-25 23:57:54, 2227.0, 494.0
2017-10-25 23:59:51, 2227.0, 494.0
2017-10-26 00:00:49, 2227.0, 494.0
2017-10-26 00:01:47, 2227.0, 494.0
2017-10-26 00:02:46, 2227.0, 494.0
2017-10-26 00:03:44, 2227.0, 494.0

One data point is missing between 2017-10-25 23:57:54 and 2017-10-25 23:59:51. There is no corresponding data in the emonhub-log. I think the packet is lost during transmissing:

Emonhub.log:

2017-10-25 21:56:56,142 DEBUG    RFM2Pi     133462 NEW FRAME : OK 20 206 0 0 0 183 2 29 0 179 8 0 0 (-67)
2017-10-25 21:56:56,144 DEBUG    RFM2Pi     133462 Timestamp : 1508968616.14
2017-10-25 21:56:56,145 DEBUG    RFM2Pi     133462 From Node : 20
2017-10-25 21:56:56,146 DEBUG    RFM2Pi     133462    Values : [20.6, 0, 69.5, 2.9000000000000004, 2227]
2017-10-25 21:56:56,146 DEBUG    RFM2Pi     133462      RSSI : -67
2017-10-25 21:56:56,147 DEBUG    RFM2Pi     133462 Sent to channel(start)' : ToEmonCMS
2017-10-25 21:56:56,147 DEBUG    RFM2Pi     133462 Sent to channel(end)' : ToEmonCMS
2017-10-25 21:56:56,278 DEBUG    MQTT       Publishing: emon/emonth2/temperature 20.6
2017-10-25 21:56:56,280 DEBUG    MQTT       Publishing: emon/emonth2/external temperature 0
2017-10-25 21:56:56,282 DEBUG    MQTT       Publishing: emon/emonth2/humidity 69.5
2017-10-25 21:56:56,283 DEBUG    MQTT       Publishing: emon/emonth2/battery 2.9
2017-10-25 21:56:56,285 DEBUG    MQTT       Publishing: emon/emonth2/pulsecount 2227
2017-10-25 21:56:56,287 INFO     MQTT       Publishing: emon/emonth2/rssi -67
2017-10-25 21:56:56,288 INFO     MQTT       Publishing: emonhub/rx/20/values 20.6,0,69.5,2.9,2227
2017-10-25 21:56:56,290 INFO     MQTT       Publishing: emonhub/rx/20/rssi -67


2017-10-25 21:57:54,424 DEBUG    RFM2Pi     133475 NEW FRAME : OK 20 206 0 0 0 183 2 29 0 179 8 0 0 (-67)
2017-10-25 21:57:54,425 DEBUG    RFM2Pi     133475 Timestamp : 1508968674.42
2017-10-25 21:57:54,426 DEBUG    RFM2Pi     133475 From Node : 20
2017-10-25 21:57:54,426 DEBUG    RFM2Pi     133475    Values : [20.6, 0, 69.5, 2.9000000000000004, 2227]
2017-10-25 21:57:54,426 DEBUG    RFM2Pi     133475      RSSI : -67
2017-10-25 21:57:54,427 DEBUG    RFM2Pi     133475 Sent to channel(start)' : ToEmonCMS
2017-10-25 21:57:54,427 DEBUG    RFM2Pi     133475 Sent to channel(end)' : ToEmonCMS
2017-10-25 21:57:54,485 DEBUG    MQTT       Publishing: emon/emonth2/temperature 20.6
2017-10-25 21:57:54,486 DEBUG    MQTT       Publishing: emon/emonth2/external temperature 0
2017-10-25 21:57:54,486 DEBUG    MQTT       Publishing: emon/emonth2/humidity 69.5
2017-10-25 21:57:54,487 DEBUG    MQTT       Publishing: emon/emonth2/battery 2.9
2017-10-25 21:57:54,488 DEBUG    MQTT       Publishing: emon/emonth2/pulsecount 2227
2017-10-25 21:57:54,489 INFO     MQTT       Publishing: emon/emonth2/rssi -67
2017-10-25 21:57:54,489 INFO     MQTT       Publishing: emonhub/rx/20/values 20.6,0,69.5,2.9,2227
2017-10-25 21:57:54,490 INFO     MQTT       Publishing: emonhub/rx/20/rssi -67


2017-10-25 21:59:51,084 DEBUG    RFM2Pi     133499 NEW FRAME : OK 20 206 0 0 0 183 2 30 0 179 8 0 0 (-67)
2017-10-25 21:59:51,085 DEBUG    RFM2Pi     133499 Timestamp : 1508968791.08
2017-10-25 21:59:51,085 DEBUG    RFM2Pi     133499 From Node : 20
2017-10-25 21:59:51,086 DEBUG    RFM2Pi     133499    Values : [20.6, 0, 69.5, 3, 2227]
2017-10-25 21:59:51,086 DEBUG    RFM2Pi     133499      RSSI : -67
2017-10-25 21:59:51,086 DEBUG    RFM2Pi     133499 Sent to channel(start)' : ToEmonCMS
2017-10-25 21:59:51,086 DEBUG    RFM2Pi     133499 Sent to channel(end)' : ToEmonCMS
2017-10-25 21:59:51,188 DEBUG    MQTT       Publishing: emon/emonth2/temperature 20.6
2017-10-25 21:59:51,189 DEBUG    MQTT       Publishing: emon/emonth2/external temperature 0
2017-10-25 21:59:51,190 DEBUG    MQTT       Publishing: emon/emonth2/humidity 69.5
2017-10-25 21:59:51,191 DEBUG    MQTT       Publishing: emon/emonth2/battery 3
2017-10-25 21:59:51,191 DEBUG    MQTT       Publishing: emon/emonth2/pulsecount 2227
2017-10-25 21:59:51,192 INFO     MQTT       Publishing: emon/emonth2/rssi -67
2017-10-25 21:59:51,193 INFO     MQTT       Publishing: emonhub/rx/20/values 20.6,0,69.5,3,2227
2017-10-25 21:59:51,194 INFO     MQTT       Publishing: emonhub/rx/20/rssi -67

Btw.: I configured the emonPi yesterday to send the values also to a self hosted emoncms instance. No reduce values are shown there. Not sure what’s the difference to the emonPi (OEM shop bought and not modified :wink:

13

Do you have quiet = false set in emonhub.conf [[RFM2Pi]] [[[runtimesettings]]]? it would be good to see if it is recieved but corrupt or too weak etc.

Regarding the csv and the logs, you have given us csv for 23:55 to 00:03 and emonhub.log for 21:56 to 22:00 so there is no cross referencing, there is also no errors visible. Can you confirm if any TZ adjustment is included in the csv data? I was expecting raw unix timestamps. Be wary of passing csv of multiple feeds together this way as the graph module is a little quirky and only uses the timestamp of the first dataseries correctly. since you are comparing data produced in the same input processlist and stored in feeds of the same type you should be ok but when using different feed types or different intervals or results from different input processlists always run them separately and merge them yourself outside emoncms when debugging stuff like this.

Also have you edited the emonhub.log? it doesn’t normally look like that and if you are sending to another instance that is not shown either.

It is important to see the raw data (emonhub.log and phptimeseries csv with unix timestamps) and suspect processed data (whaccumulator) for the same period that demonstrates a “glitch”. Try setting quiet = true and we’ll try again tomorrow if it happens again.

14

Yes, sorry for that I use timezone +2, so emonhub log 21:00 (UTC) is graph module 23:00 (UTC+2). I will copy the raw UNIX-timestamps this time.

I have a emonTx, sending every 5 seconds which blows up the log. I only deleted these lines. I will add the whole log this time.

New data, new luck :slight_smile: This one is interesting, because it has two missing data packets behind each other which results in two different erroneous PHPFINA wh_accumulator values.

PHPtimeseries:

47
471150×510 61.8 KB

CSV:

1509210269, 1052.0, 2785.0
1509210327, 1052.0, 2785.0
1509210386, 1052.0, 2785.0
1509210444, 1052.0, 2785.0
1509210502, 1052.0, 2785.0
1509210561, 1053.0, 2786.0
1509210619, 1055.0, 2788.0
1509210678, 1057.0, 2790.0
1509210736, 1057.0, 2790.0
1509210795, 1057.0, 2790.0
1509210853, 1057.0, 2790.0
1509210911, 1057.0, 2790.0
1509210970, 1057.0, 2790.0
1509211028, 1057.0, 2790.0
1509211086, 1057.0, 2790.0
1509211145, 1057.0, 2790.0
1509211203, 1057.0, 2790.0
1509211261, 1057.0, 2790.0
1509211319, 1057.0, 2790.0
1509211378, 1057.0, 2790.0
1509211436, 1057.0, 2790.0
1509211495, 1057.0, 2790.0
1509211553, 1058.0, 2791.0
1509211611, 1058.0, 2791.0
1509211670, 1058.0, 2791.0
1509211728, 1058.0, 2791.0
1509211786, 1058.0, 2791.0
1509211845, 1058.0, 2791.0
1509211903, 1058.0, 2791.0
1509211961, 1058.0, 2791.0
1509212020, 1058.0, 2791.0
1509212078, 1059.0, 2792.0
1509212137, 1059.0, 2792.0
1509212195, 1059.0, 2792.0
1509212253, 1059.0, 2792.0
1509212311, 1059.0, 2792.0
1509212370, 1059.0, 2792.0
1509212428, 1059.0, 2792.0
1509212486, 1059.0, 2792.0
1509212545, 1059.0, 2792.0
1509212603, 1059.0, 2792.0
1509212662, 1059.0, 2792.0
1509212720, 1060.0, 2793.0
1509212778, 1060.0, 2793.0
1509212837, 1060.0, 2793.0
1509212895, 1061.0, 2794.0
1509212953, 1061.0, 2794.0
1509213012, 1061.0, 2794.0
1509213070, 1061.0, 2794.0
1509213129, 1061.0, 2794.0
1509213187, 1061.0, 2794.0
1509213245, 1061.0, 2794.0
1509213303, 1061.0, 2794.0
1509213362, 1061.0, 2794.0
1509213420, 1061.0, 2794.0
1509213478, 1061.0, 2794.0
1509213537, 1061.0, 2794.0
1509213595, 1061.0, 2794.0
1509213654, 1061.0, 2794.0
1509213712, 1061.0, 2794.0
1509213770, 1061.0, 2794.0
1509213828, 1061.0, 2794.0
1509213887, 1061.0, 2794.0
1509213945, 1061.0, 2794.0
1509214004, 1062.0, 2795.0
1509214062, 1062.0, 2795.0
1509214120, 1062.0, 2795.0
1509214179, 1062.0, 2795.0
1509214237, 1062.0, 2795.0
1509214295, 1062.0, 2795.0
1509214354, 1062.0, 2795.0
1509214412, 1062.0, 2795.0
1509214470, 1062.0, 2795.0
1509214529, 1062.0, 2795.0
1509214587, 1062.0, 2795.0
1509214645, 1062.0, 2795.0
1509214704, 1062.0, 2795.0
1509214762, 1063.0, 2796.0
1509214937, 1063.0, 2796.0
1509214996, 1063.0, 2796.0
1509215054, 1063.0, 2796.0
1509215112, 1063.0, 2796.0
1509215171, 1063.0, 2796.0
1509215229, 1063.0, 2796.0
1509215287, 1063.0, 2796.0
1509215346, 1063.0, 2796.0
1509215404, 1063.0, 2796.0
1509215462, 1063.0, 2796.0
1509215521, 1063.0, 2796.0
1509215579, 1063.0, 2796.0
1509215637, 1063.0, 2796.0
1509215696, 1063.0, 2796.0
1509215754, 1063.0, 2796.0
1509215812, 1064.0, 2797.0
1509215871, 1064.0, 2797.0
1509215929, 1064.0, 2797.0
1509215988, 1064.0, 2797.0
1509216046, 1064.0, 2797.0
1509216104, 1064.0, 2797.0
1509216162, 1064.0, 2797.0
1509216221, 1064.0, 2797.0
1509216279, 1064.0, 2797.0
1509216337, 1064.0, 2797.0
1509216396, 1064.0, 2797.0
1509216454, 1064.0, 2797.0
1509216512, 1064.0, 2797.0
1509216571, 1064.0, 2797.0
1509216629, 1064.0, 2797.0
1509216688, 1064.0, 2797.0
1509216746, 1064.0, 2797.0
1509216804, 1064.0, 2797.0
1509216863, 1064.0, 2797.0
1509216921, 1065.0, 2798.0
1509216979, 1065.0, 2798.0
1509217038, 1065.0, 2798.0
1509217096, 1065.0, 2798.0
1509217154, 1065.0, 2798.0
1509217213, 1065.0, 2798.0
1509217271, 1065.0, 2798.0
1509217329, 1065.0, 2798.0
1509217388, 1065.0, 2798.0
1509217446, 1065.0, 2798.0
1509217504, 1065.0, 2798.0
1509217563, 1065.0, 2798.0
1509217621, 1066.0, 2799.0
1509217679, 1066.0, 2799.0
1509217738, 1066.0, 2799.0
1509217796, 1066.0, 2799.0
1509217854, 1066.0, 2799.0
1509217913, 1066.0, 2799.0
1509217971, 1066.0, 2799.0
1509218029, 1066.0, 2799.0
1509218088, 1066.0, 2799.0
1509218204, 1066.0, 2799.0
1509218263, 1066.0, 2799.0
1509218321, 1066.0, 2799.0
1509218380, 1066.0, 2799.0
1509218438, 1066.0, 2799.0
1509218496, 1066.0, 2799.0
1509218554, 1066.0, 2799.0
1509218613, 1066.0, 2799.0
1509218671, 1066.0, 2799.0
1509218730, 1067.0, 2800.0
1509218788, 1067.0, 2800.0
1509218846, 1067.0, 2800.0
1509218905, 1067.0, 2800.0
1509218963, 1067.0, 2800.0
1509219021, 1067.0, 2800.0
1509219080, 1067.0, 2800.0
1509219138, 1067.0, 2800.0
1509219196, 1067.0, 2800.0
1509219255, 1067.0, 2800.0

PHPFINA:

26
261143×474 63.3 KB

CSV:

1509210310, 4672.0, 2785.0
1509210320, 4672.0, 2785.0
1509210330, 4672.0, 2785.0
1509210340, 4672.0, 2785.0
1509210350, 4672.0, 2785.0
1509210360, 4672.0, 2785.0
1509210370, 4672.0, 2785.0
1509210380, 4672.0, 2785.0
1509210390, 4672.0, 2785.0
1509210400, 4672.0, 2785.0
1509210410, 4672.0, 2785.0
1509210420, 4672.0, 2785.0
1509210430, 4672.0, 2785.0
1509210440, 4672.0, 2785.0
1509210450, 4672.0, 2785.0
1509210460, 4672.0, 2785.0
1509210470, 4672.0, 2785.0
1509210480, 4672.0, 2785.0
1509210490, 4672.0, 2785.0
1509210500, 4672.0, 2785.0
1509210510, 4673.0, 2786.0
1509210520, 4673.0, 2786.0
1509210530, 4673.0, 2786.0
1509210540, 4673.0, 2786.0
1509210550, 4673.0, 2786.0
1509210560, 4673.0, 2786.0
1509210570, 4675.0, 2788.0
1509210580, 4675.0, 2788.0
1509210590, 4675.0, 2788.0
1509210600, 4675.0, 2788.0
1509210610, 4675.0, 2788.0
1509210620, 4675.0, 2788.0
1509210630, 4677.0, 2790.0
1509210640, 4677.0, 2790.0
1509210650, 4677.0, 2790.0
1509210660, 4677.0, 2790.0
1509210670, 4677.0, 2790.0
1509210680, 4677.0, 2790.0
1509210690, 4677.0, 2790.0
1509210700, 4677.0, 2790.0
1509210710, 4677.0, 2790.0
1509210720, 4677.0, 2790.0
1509210730, 4677.0, 2790.0
1509210740, 4677.0, 2790.0
1509210750, 4677.0, 2790.0
1509210760, 4677.0, 2790.0
1509210770, 4677.0, 2790.0
1509210780, 4677.0, 2790.0
1509210790, 4677.0, 2790.0
1509210800, 4677.0, 2790.0
1509210810, 4677.0, 2790.0
1509210820, 4677.0, 2790.0
1509210830, 4677.0, 2790.0
1509210840, 4677.0, 2790.0
1509210850, 4677.0, 2790.0
1509210860, 4677.0, 2790.0
1509210870, 4677.0, 2790.0
1509210880, 4677.0, 2790.0
1509210890, 4677.0, 2790.0
1509210900, 4677.0, 2790.0
1509210910, 4677.0, 2790.0
1509210920, 4677.0, 2790.0
1509210930, 4677.0, 2790.0
1509210940, 4677.0, 2790.0
1509210950, 4677.0, 2790.0
1509210960, 4677.0, 2790.0
1509210970, 4677.0, 2790.0
1509210980, 4677.0, 2790.0
1509210990, 4677.0, 2790.0
1509211000, 4677.0, 2790.0
1509211010, 4677.0, 2790.0
1509211020, 4677.0, 2790.0
1509211030, 4677.0, 2790.0
1509211040, 4677.0, 2790.0
1509211050, 4677.0, 2790.0
1509211060, 4677.0, 2790.0
1509211070, 4677.0, 2790.0
1509211080, 4677.0, 2790.0
1509211090, 4677.0, 2790.0
1509211100, 4677.0, 2790.0
1509211110, 4677.0, 2790.0
1509211120, 4677.0, 2790.0
1509211130, 4677.0, 2790.0
1509211140, 4677.0, 2790.0
1509211150, 4677.0, 2790.0
1509211160, 4677.0, 2790.0
1509211170, 4677.0, 2790.0
1509211180, 4677.0, 2790.0
1509211190, 4677.0, 2790.0
1509211200, 4677.0, 2790.0
1509211210, 4677.0, 2790.0
1509211220, 4677.0, 2790.0
1509211230, 4677.0, 2790.0
1509211240, 4677.0, 2790.0
1509211250, 4677.0, 2790.0
1509211260, 4677.0, 2790.0
1509211270, 4677.0, 2790.0
1509211280, 4677.0, 2790.0
1509211290, 4677.0, 2790.0
1509211300, 4677.0, 2790.0
1509211310, 4677.0, 2790.0
1509211320, 4677.0, 2790.0
1509211330, 4677.0, 2790.0
1509211340, 4677.0, 2790.0
1509211350, 4677.0, 2790.0
1509211360, 4677.0, 2790.0
1509211370, 4677.0, 2790.0
1509211380, 4677.0, 2790.0
1509211390, 4677.0, 2790.0
1509211400, 4677.0, 2790.0
1509211410, 4677.0, 2790.0
1509211420, 4677.0, 2790.0
1509211430, 4677.0, 2790.0
1509211440, 4677.0, 2790.0
1509211450, 4677.0, 2790.0
1509211460, 4677.0, 2790.0
1509211470, 4678.0, 2791.0
1509211480, 4678.0, 2791.0
1509211490, 4678.0, 2791.0
1509211500, 4678.0, 2791.0
1509211510, 4678.0, 2791.0
1509211520, 4678.0, 2791.0
1509211530, 4678.0, 2791.0
1509211540, 4678.0, 2791.0
1509211550, 4678.0, 2791.0
1509211560, 4678.0, 2791.0
1509211570, 4678.0, 2791.0
1509211580, 4678.0, 2791.0
1509211590, 4678.0, 2791.0
1509211600, 4678.0, 2791.0
1509211610, 4678.0, 2791.0
1509211620, 4678.0, 2791.0
1509211630, 4678.0, 2791.0
1509211640, 4678.0, 2791.0
1509211650, 4678.0, 2791.0
1509211660, 4678.0, 2791.0
1509211670, 4678.0, 2791.0
1509211680, 4678.0, 2791.0
1509211690, 4678.0, 2791.0
1509211700, 4678.0, 2791.0
1509211710, 4678.0, 2791.0
1509211720, 4678.0, 2791.0
1509211730, 4678.0, 2791.0
1509211740, 4678.0, 2791.0
1509211750, 4678.0, 2791.0
1509211760, 4678.0, 2791.0
1509211770, 4678.0, 2791.0
1509211780, 4678.0, 2791.0
1509211790, 4678.0, 2791.0
1509211800, 4678.0, 2791.0
1509211810, 4678.0, 2791.0
1509211820, 4678.0, 2791.0
1509211830, 4678.0, 2791.0
1509211840, 4678.0, 2791.0
1509211850, 4678.0, 2791.0
1509211860, 4678.0, 2791.0
1509211870, 4678.0, 2791.0
1509211880, 4678.0, 2791.0
1509211890, 4678.0, 2791.0
1509211900, 4678.0, 2791.0
1509211910, 4678.0, 2791.0
1509211920, 4678.0, 2791.0
1509211930, 4678.0, 2791.0
1509211940, 4678.0, 2791.0
1509211950, 4678.0, 2791.0
1509211960, 4678.0, 2791.0
1509211970, 4678.0, 2791.0
1509211980, 4678.0, 2791.0
1509211990, 4678.0, 2791.0
1509212000, 4678.0, 2791.0
1509212010, 4679.0, 2792.0
1509212020, 4679.0, 2792.0
1509212030, 4679.0, 2792.0
1509212040, 4679.0, 2792.0
1509212050, 4679.0, 2792.0
1509212060, 4679.0, 2792.0
1509212070, 4679.0, 2792.0
1509212080, 4679.0, 2792.0
1509212090, 4679.0, 2792.0
1509212100, 4679.0, 2792.0
1509212110, 4679.0, 2792.0
1509212120, 4679.0, 2792.0
1509212130, 4679.0, 2792.0
1509212140, 4679.0, 2792.0
1509212150, 4679.0, 2792.0
1509212160, 4679.0, 2792.0
1509212170, 4679.0, 2792.0
1509212180, 4679.0, 2792.0
1509212190, 4679.0, 2792.0
1509212200, 4679.0, 2792.0
1509212210, 4679.0, 2792.0
1509212220, 4679.0, 2792.0
1509212230, 4679.0, 2792.0
1509212240, 4679.0, 2792.0
1509212250, 4679.0, 2792.0
1509212260, 4679.0, 2792.0
1509212270, 4679.0, 2792.0
1509212280, 4679.0, 2792.0
1509212290, 4679.0, 2792.0
1509212300, 4679.0, 2792.0
1509212310, 4679.0, 2792.0
1509212320, 4679.0, 2792.0
1509212330, 4679.0, 2792.0
1509212340, 4679.0, 2792.0
1509212350, 4679.0, 2792.0
1509212360, 4679.0, 2792.0
1509212370, 4679.0, 2792.0
1509212380, 4679.0, 2792.0
1509212390, 4679.0, 2792.0
1509212400, 4679.0, 2792.0
1509212410, 4679.0, 2792.0
1509212420, 4679.0, 2792.0
1509212430, 4679.0, 2792.0
1509212440, 4679.0, 2792.0
1509212450, 4679.0, 2792.0
1509212460, 4679.0, 2792.0
1509212470, 4679.0, 2792.0
1509212480, 4679.0, 2792.0
1509212490, 4679.0, 2792.0
1509212500, 4679.0, 2792.0
1509212510, 4679.0, 2792.0
1509212520, 4679.0, 2792.0
1509212530, 4679.0, 2792.0
1509212540, 4679.0, 2792.0
1509212550, 4679.0, 2792.0
1509212560, 4679.0, 2792.0
1509212570, 4679.0, 2792.0
1509212580, 4679.0, 2792.0
1509212590, 4679.0, 2792.0
1509212600, 4679.0, 2792.0
1509212610, 4679.0, 2792.0
1509212620, 4679.0, 2792.0
1509212630, 4679.0, 2792.0
1509212640, 4679.0, 2792.0
1509212650, 4679.0, 2792.0
1509212660, 4679.0, 2792.0
1509212670, 4680.0, 2793.0
1509212680, 4680.0, 2793.0
1509212690, 4680.0, 2793.0
1509212700, 4680.0, 2793.0
1509212710, 4680.0, 2793.0
1509212720, 4680.0, 2793.0
1509212730, 4680.0, 2793.0
1509212740, 4680.0, 2793.0
1509212750, 4680.0, 2793.0
1509212760, 4680.0, 2793.0
1509212770, 4680.0, 2793.0
1509212780, 4680.0, 2793.0
1509212790, 4680.0, 2793.0
1509212800, 4680.0, 2793.0
1509212810, 4680.0, 2793.0
1509212820, 4680.0, 2793.0
1509212830, 4680.0, 2793.0
1509212840, 4680.0, 2793.0
1509212850, 4681.0, 2794.0
1509212860, 4681.0, 2794.0
1509212870, 4681.0, 2794.0
1509212880, 4681.0, 2794.0
1509212890, 4681.0, 2794.0
1509212900, 4681.0, 2794.0
1509212910, 4681.0, 2794.0
1509212920, 4681.0, 2794.0
1509212930, 4681.0, 2794.0
1509212940, 4681.0, 2794.0
1509212950, 4681.0, 2794.0
1509212960, 4681.0, 2794.0
1509212970, 4681.0, 2794.0
1509212980, 4681.0, 2794.0
1509212990, 4681.0, 2794.0
1509213000, 4681.0, 2794.0
1509213010, 4681.0, 2794.0
1509213020, 4681.0, 2794.0
1509213030, 4681.0, 2794.0
1509213040, 4681.0, 2794.0
1509213050, 4681.0, 2794.0
1509213060, 4681.0, 2794.0
1509213070, 4681.0, 2794.0
1509213080, 4681.0, 2794.0
1509213090, 4681.0, 2794.0
1509213100, 4681.0, 2794.0
1509213110, 4681.0, 2794.0
1509213120, 4681.0, 2794.0
1509213130, 4681.0, 2794.0
1509213140, 4681.0, 2794.0
1509213150, 4681.0, 2794.0
1509213160, 4681.0, 2794.0
1509213170, 4681.0, 2794.0
1509213180, 4681.0, 2794.0
1509213190, 4681.0, 2794.0
1509213200, 4681.0, 2794.0
1509213210, 4681.0, 2794.0
1509213220, 4681.0, 2794.0
1509213230, 4681.0, 2794.0
1509213240, 4681.0, 2794.0
1509213250, 4681.0, 2794.0
1509213260, 4681.0, 2794.0
1509213270, 4681.0, 2794.0
1509213280, 4681.0, 2794.0
1509213290, 4681.0, 2794.0
1509213300, 4681.0, 2794.0
1509213310, 4681.0, 2794.0
1509213320, 4681.0, 2794.0
1509213330, 4681.0, 2794.0
1509213340, 4681.0, 2794.0
1509213350, 4681.0, 2794.0
1509213360, 4681.0, 2794.0
1509213370, 4681.0, 2794.0
1509213380, 4681.0, 2794.0
1509213390, 4681.0, 2794.0
1509213400, 4681.0, 2794.0
1509213410, 4681.0, 2794.0
1509213420, 4681.0, 2794.0
1509213430, 4681.0, 2794.0
1509213440, 4681.0, 2794.0
1509213450, 4681.0, 2794.0
1509213460, 4681.0, 2794.0
1509213470, 4681.0, 2794.0
1509213480, 4681.0, 2794.0
1509213490, 4681.0, 2794.0
1509213500, 4681.0, 2794.0
1509213510, 4681.0, 2794.0
1509213520, 4681.0, 2794.0
1509213530, 4681.0, 2794.0
1509213540, 4681.0, 2794.0
1509213550, 4681.0, 2794.0
1509213560, 4681.0, 2794.0
1509213570, 4681.0, 2794.0
1509213580, 4681.0, 2794.0
1509213590, 4681.0, 2794.0
1509213600, 4681.0, 2794.0
1509213610, 4681.0, 2794.0
1509213620, 4681.0, 2794.0
1509213630, 4681.0, 2794.0
1509213640, 4681.0, 2794.0
1509213650, 4681.0, 2794.0
1509213660, 4681.0, 2794.0
1509213670, 4681.0, 2794.0
1509213680, 4681.0, 2794.0
1509213690, 4681.0, 2794.0
1509213700, 4681.0, 2794.0
1509213710, 4681.0, 2794.0
1509213720, 4681.0, 2794.0
1509213730, 4681.0, 2794.0
1509213740, 4681.0, 2794.0
1509213750, 4681.0, 2794.0
1509213760, 4681.0, 2794.0
1509213770, 4681.0, 2794.0
1509213780, 4681.0, 2794.0
1509213790, 4681.0, 2794.0
1509213800, 4681.0, 2794.0
1509213810, 4681.0, 2794.0
1509213820, 4681.0, 2794.0
1509213830, 4681.0, 2794.0
1509213840, 4681.0, 2794.0
1509213850, 4681.0, 2794.0
1509213860, 4681.0, 2794.0
1509213870, 4681.0, 2794.0
1509213880, 4681.0, 2794.0
1509213890, 4681.0, 2794.0
1509213900, 4681.0, 2794.0
1509213910, 4681.0, 2794.0
1509213920, 4681.0, 2794.0
1509213930, 4682.0, 2795.0
1509213940, 4682.0, 2795.0
1509213950, 4682.0, 2795.0
1509213960, 4682.0, 2795.0
1509213970, 4682.0, 2795.0
1509213980, 4682.0, 2795.0
1509213990, 4682.0, 2795.0
1509214000, 4682.0, 2795.0
1509214010, 4682.0, 2795.0
1509214020, 4682.0, 2795.0
1509214030, 4682.0, 2795.0
1509214040, 4682.0, 2795.0
1509214050, 4682.0, 2795.0
1509214060, 4682.0, 2795.0
1509214070, 4682.0, 2795.0
1509214080, 4682.0, 2795.0
1509214090, 4682.0, 2795.0
1509214100, 4682.0, 2795.0
1509214110, 4682.0, 2795.0
1509214120, 4682.0, 2795.0
1509214130, 4682.0, 2795.0
1509214140, 4682.0, 2795.0
1509214150, 4682.0, 2795.0
1509214160, 4682.0, 2795.0
1509214170, 4682.0, 2795.0
1509214180, 4682.0, 2795.0
1509214190, 4682.0, 2795.0
1509214200, 4682.0, 2795.0
1509214210, 4682.0, 2795.0
1509214220, 4682.0, 2795.0
1509214230, 4682.0, 2795.0
1509214240, 4682.0, 2795.0
1509214250, 4682.0, 2795.0
1509214260, 4682.0, 2795.0
1509214270, 4682.0, 2795.0
1509214280, 4682.0, 2795.0
1509214290, 4682.0, 2795.0
1509214300, 4682.0, 2795.0
1509214310, 4682.0, 2795.0
1509214320, 4682.0, 2795.0
1509214330, 4682.0, 2795.0
1509214340, 4682.0, 2795.0
1509214350, 4682.0, 2795.0
1509214360, 4682.0, 2795.0
1509214370, 4682.0, 2795.0
1509214380, 4682.0, 2795.0
1509214390, 4682.0, 2795.0
1509214400, 4682.0, 2795.0
1509214410, 4682.0, 2795.0
1509214420, 4682.0, 2795.0
1509214430, 4682.0, 2795.0
1509214440, 4682.0, 2795.0
1509214450, 4682.0, 2795.0
1509214460, 4682.0, 2795.0
1509214470, 4682.0, 2795.0
1509214480, 4682.0, 2795.0
1509214490, 4682.0, 2795.0
1509214500, 4682.0, 2795.0
1509214510, 4682.0, 2795.0
1509214520, 4682.0, 2795.0
1509214530, 4682.0, 2795.0
1509214540, 4682.0, 2795.0
1509214550, 4682.0, 2795.0
1509214560, 4682.0, 2795.0
1509214570, 4682.0, 2795.0
1509214580, 4682.0, 2795.0
1509214590, 4682.0, 2795.0
1509214600, 4682.0, 2795.0
1509214610, 4682.0, 2795.0
1509214620, 4682.0, 2795.0
1509214630, 4682.0, 2795.0
1509214640, 4682.0, 2795.0
1509214650, 4682.0, 2795.0
1509214660, 4682.0, 2795.0
1509214670, 4682.0, 2795.0
1509214680, 4682.0, 2795.0
1509214690, 4682.0, 2795.0
1509214700, 4682.0, 2795.0
1509214710, 4683.0, 2796.0
1509214720, 4683.0, 2796.0
1509214730, 4683.0, 2796.0
1509214740, 4683.0, 2796.0
1509214750, 4683.0, 2796.0
1509214760, 4683.0, 2796.0
1509214770, 4673.0, null
1509214780, 4673.0, null
1509214790, 4673.0, null
1509214800, 4673.0, null
1509214810, 4673.0, null
1509214820, 4673.0, null
1509214830, 4678.0, null
1509214840, 4678.0, null
1509214850, 4678.0, null
1509214860, 4678.0, null
1509214870, 4678.0, null
1509214880, 4678.0, null
1509214890, 4683.0, 2796.0
1509214900, 4683.0, 2796.0
1509214910, 4683.0, 2796.0
1509214920, 4683.0, 2796.0
1509214930, 4683.0, 2796.0
1509214940, 4683.0, 2796.0
1509214950, 4683.0, 2796.0
1509214960, 4683.0, 2796.0
1509214970, 4683.0, 2796.0
1509214980, 4683.0, 2796.0
1509214990, 4683.0, 2796.0
1509215000, 4683.0, 2796.0
1509215010, 4683.0, 2796.0
1509215020, 4683.0, 2796.0
1509215030, 4683.0, 2796.0
1509215040, 4683.0, 2796.0
1509215050, 4683.0, 2796.0
1509215060, 4683.0, 2796.0
1509215070, 4683.0, 2796.0
1509215080, 4683.0, 2796.0
1509215090, 4683.0, 2796.0
1509215100, 4683.0, 2796.0
1509215110, 4683.0, 2796.0
1509215120, 4683.0, 2796.0
1509215130, 4683.0, 2796.0
1509215140, 4683.0, 2796.0
1509215150, 4683.0, 2796.0
1509215160, 4683.0, 2796.0
1509215170, 4683.0, 2796.0
1509215180, 4683.0, 2796.0
1509215190, 4683.0, 2796.0
1509215200, 4683.0, 2796.0
1509215210, 4683.0, 2796.0
1509215220, 4683.0, 2796.0
1509215230, 4683.0, 2796.0
1509215240, 4683.0, 2796.0
1509215250, 4683.0, 2796.0
1509215260, 4683.0, 2796.0
1509215270, 4683.0, 2796.0
1509215280, 4683.0, 2796.0
1509215290, 4683.0, 2796.0
1509215300, 4683.0, 2796.0
1509215310, 4683.0, 2796.0
1509215320, 4683.0, 2796.0
1509215330, 4683.0, 2796.0
1509215340, 4683.0, 2796.0
1509215350, 4683.0, 2796.0
1509215360, 4683.0, 2796.0
1509215370, 4683.0, 2796.0
1509215380, 4683.0, 2796.0
1509215390, 4683.0, 2796.0
1509215400, 4683.0, 2796.0
1509215410, 4683.0, 2796.0
1509215420, 4683.0, 2796.0
1509215430, 4683.0, 2796.0
1509215440, 4683.0, 2796.0
1509215450, 4683.0, 2796.0
1509215460, 4683.0, 2796.0
1509215470, 4683.0, 2796.0
1509215480, 4683.0, 2796.0
1509215490, 4683.0, 2796.0
1509215500, 4683.0, 2796.0
1509215510, 4683.0, 2796.0
1509215520, 4683.0, 2796.0
1509215530, 4683.0, 2796.0
1509215540, 4683.0, 2796.0
1509215550, 4683.0, 2796.0
1509215560, 4683.0, 2796.0
1509215570, 4683.0, 2796.0
1509215580, 4683.0, 2796.0
1509215590, 4683.0, 2796.0
1509215600, 4683.0, 2796.0
1509215610, 4683.0, 2796.0
1509215620, 4683.0, 2796.0
1509215630, 4683.0, 2796.0
1509215640, 4683.0, 2796.0
1509215650, 4683.0, 2796.0
1509215660, 4683.0, 2796.0
1509215670, 4683.0, 2796.0
1509215680, 4683.0, 2796.0
1509215690, 4683.0, 2796.0
1509215700, 4683.0, 2796.0
1509215710, 4683.0, 2796.0
1509215720, 4683.0, 2796.0
1509215730, 4684.0, 2797.0
1509215740, 4684.0, 2797.0
1509215750, 4684.0, 2797.0
1509215760, 4684.0, 2797.0
1509215770, 4684.0, 2797.0
1509215780, 4684.0, 2797.0
1509215790, 4684.0, 2797.0
1509215800, 4684.0, 2797.0
1509215810, 4684.0, 2797.0
1509215820, 4684.0, 2797.0
1509215830, 4684.0, 2797.0
1509215840, 4684.0, 2797.0
1509215850, 4684.0, 2797.0
1509215860, 4684.0, 2797.0
1509215870, 4684.0, 2797.0
1509215880, 4684.0, 2797.0
1509215890, 4684.0, 2797.0
1509215900, 4684.0, 2797.0
1509215910, 4684.0, 2797.0
1509215920, 4684.0, 2797.0
1509215930, 4684.0, 2797.0
1509215940, 4684.0, 2797.0
1509215950, 4684.0, 2797.0
1509215960, 4684.0, 2797.0
1509215970, 4684.0, 2797.0
1509215980, 4684.0, 2797.0
1509215990, 4684.0, 2797.0
1509216000, 4684.0, 2797.0
1509216010, 4684.0, 2797.0
1509216020, 4684.0, 2797.0
1509216030, 4684.0, 2797.0
1509216040, 4684.0, 2797.0
1509216050, 4684.0, 2797.0
1509216060, 4684.0, 2797.0
1509216070, 4684.0, 2797.0
1509216080, 4684.0, 2797.0
1509216090, 4684.0, 2797.0
1509216100, 4684.0, 2797.0
1509216110, 4684.0, 2797.0
1509216120, 4684.0, 2797.0
1509216130, 4684.0, 2797.0
1509216140, 4684.0, 2797.0
1509216150, 4684.0, 2797.0
1509216160, 4684.0, 2797.0
1509216170, 4684.0, 2797.0
1509216180, 4684.0, 2797.0
1509216190, 4684.0, 2797.0
1509216200, 4684.0, 2797.0
1509216210, 4684.0, 2797.0
1509216220, 4684.0, 2797.0
1509216230, 4684.0, 2797.0
1509216240, 4684.0, 2797.0
1509216250, 4684.0, 2797.0
1509216260, 4684.0, 2797.0
1509216270, 4684.0, 2797.0
1509216280, 4684.0, 2797.0
1509216290, 4684.0, 2797.0
1509216300, 4684.0, 2797.0
1509216310, 4684.0, 2797.0
1509216320, 4684.0, 2797.0
1509216330, 4684.0, 2797.0
1509216340, 4684.0, 2797.0
1509216350, 4684.0, 2797.0
1509216360, 4684.0, 2797.0
1509216370, 4684.0, 2797.0
1509216380, 4684.0, 2797.0
1509216390, 4684.0, 2797.0
1509216400, 4684.0, 2797.0
1509216410, 4684.0, 2797.0
1509216420, 4684.0, 2797.0
1509216430, 4684.0, 2797.0
1509216440, 4684.0, 2797.0
1509216450, 4684.0, 2797.0
1509216460, 4684.0, 2797.0
1509216470, 4684.0, 2797.0
1509216480, 4684.0, 2797.0
1509216490, 4684.0, 2797.0
1509216500, 4684.0, 2797.0
1509216510, 4684.0, 2797.0
1509216520, 4684.0, 2797.0
1509216530, 4684.0, 2797.0
1509216540, 4684.0, 2797.0
1509216550, 4684.0, 2797.0
1509216560, 4684.0, 2797.0
1509216570, 4684.0, 2797.0
1509216580, 4684.0, 2797.0
1509216590, 4684.0, 2797.0
1509216600, 4684.0, 2797.0
1509216610, 4684.0, 2797.0
1509216620, 4684.0, 2797.0
1509216630, 4684.0, 2797.0
1509216640, 4684.0, 2797.0
1509216650, 4684.0, 2797.0
1509216660, 4684.0, 2797.0
1509216670, 4684.0, 2797.0
1509216680, 4684.0, 2797.0
1509216690, 4684.0, 2797.0
1509216700, 4684.0, 2797.0
1509216710, 4684.0, 2797.0
1509216720, 4684.0, 2797.0
1509216730, 4684.0, 2797.0
1509216740, 4684.0, 2797.0
1509216750, 4684.0, 2797.0
1509216760, 4684.0, 2797.0
1509216770, 4684.0, 2797.0
1509216780, 4684.0, 2797.0
1509216790, 4684.0, 2797.0
1509216800, 4684.0, 2797.0
1509216810, 4684.0, 2797.0
1509216820, 4684.0, 2797.0
1509216830, 4684.0, 2797.0
1509216840, 4684.0, 2797.0
1509216850, 4684.0, 2797.0
1509216860, 4684.0, 2797.0
1509216870, 4685.0, 2798.0
1509216880, 4685.0, 2798.0
1509216890, 4685.0, 2798.0
1509216900, 4685.0, 2798.0
1509216910, 4685.0, 2798.0
1509216920, 4685.0, 2798.0
1509216930, 4685.0, 2798.0
1509216940, 4685.0, 2798.0
1509216950, 4685.0, 2798.0
1509216960, 4685.0, 2798.0
1509216970, 4685.0, 2798.0
1509216980, 4685.0, 2798.0
1509216990, 4685.0, 2798.0
1509217000, 4685.0, 2798.0
1509217010, 4685.0, 2798.0
1509217020, 4685.0, 2798.0
1509217030, 4685.0, 2798.0
1509217040, 4685.0, 2798.0
1509217050, 4685.0, 2798.0
1509217060, 4685.0, 2798.0
1509217070, 4685.0, 2798.0
1509217080, 4685.0, 2798.0
1509217090, 4685.0, 2798.0
1509217100, 4685.0, 2798.0
1509217110, 4685.0, 2798.0
1509217120, 4685.0, 2798.0
1509217130, 4685.0, 2798.0
1509217140, 4685.0, 2798.0
1509217150, 4685.0, 2798.0
1509217160, 4685.0, 2798.0
1509217170, 4685.0, 2798.0
1509217180, 4685.0, 2798.0
1509217190, 4685.0, 2798.0
1509217200, 4685.0, 2798.0
1509217210, 4685.0, 2798.0
1509217220, 4685.0, 2798.0
1509217230, 4685.0, 2798.0
1509217240, 4685.0, 2798.0
1509217250, 4685.0, 2798.0
1509217260, 4685.0, 2798.0
1509217270, 4685.0, 2798.0
1509217280, 4685.0, 2798.0
1509217290, 4685.0, 2798.0
1509217300, 4685.0, 2798.0
1509217310, 4685.0, 2798.0
1509217320, 4685.0, 2798.0
1509217330, 4685.0, 2798.0
1509217340, 4685.0, 2798.0
1509217350, 4685.0, 2798.0
1509217360, 4685.0, 2798.0
1509217370, 4685.0, 2798.0
1509217380, 4685.0, 2798.0
1509217390, 4685.0, 2798.0
1509217400, 4685.0, 2798.0
1509217410, 4685.0, 2798.0
1509217420, 4685.0, 2798.0
1509217430, 4685.0, 2798.0
1509217440, 4685.0, 2798.0
1509217450, 4685.0, 2798.0
1509217460, 4685.0, 2798.0
1509217470, 4685.0, 2798.0
1509217480, 4685.0, 2798.0
1509217490, 4685.0, 2798.0
1509217500, 4685.0, 2798.0
1509217510, 4685.0, 2798.0
1509217520, 4685.0, 2798.0
1509217530, 4685.0, 2798.0
1509217540, 4685.0, 2798.0
1509217550, 4685.0, 2798.0
1509217560, 4685.0, 2798.0
1509217570, 4685.0, 2798.0
1509217580, 4685.0, 2798.0
1509217590, 4686.0, 2799.0
1509217600, 4686.0, 2799.0
1509217610, 4686.0, 2799.0
1509217620, 4686.0, 2799.0
1509217630, 4686.0, 2799.0
1509217640, 4686.0, 2799.0
1509217650, 4686.0, 2799.0
1509217660, 4686.0, 2799.0
1509217670, 4686.0, 2799.0
1509217680, 4686.0, 2799.0
1509217690, 4686.0, 2799.0
1509217700, 4686.0, 2799.0
1509217710, 4686.0, 2799.0
1509217720, 4686.0, 2799.0
1509217730, 4686.0, 2799.0
1509217740, 4686.0, 2799.0
1509217750, 4686.0, 2799.0
1509217760, 4686.0, 2799.0
1509217770, 4686.0, 2799.0
1509217780, 4686.0, 2799.0
1509217790, 4686.0, 2799.0
1509217800, 4686.0, 2799.0
1509217810, 4686.0, 2799.0
1509217820, 4686.0, 2799.0
1509217830, 4686.0, 2799.0
1509217840, 4686.0, 2799.0
1509217850, 4686.0, 2799.0
1509217860, 4686.0, 2799.0
1509217870, 4686.0, 2799.0
1509217880, 4686.0, 2799.0
1509217890, 4686.0, 2799.0
1509217900, 4686.0, 2799.0
1509217910, 4686.0, 2799.0
1509217920, 4686.0, 2799.0
1509217930, 4686.0, 2799.0
1509217940, 4686.0, 2799.0
1509217950, 4686.0, 2799.0
1509217960, 4686.0, 2799.0
1509217970, 4686.0, 2799.0
1509217980, 4686.0, 2799.0
1509217990, 4686.0, 2799.0
1509218000, 4686.0, 2799.0
1509218010, 4686.0, 2799.0
1509218020, 4686.0, 2799.0
1509218030, 4686.0, 2799.0
1509218040, 4686.0, 2799.0
1509218050, 4686.0, 2799.0
1509218060, 4686.0, 2799.0
1509218070, 4684.5, null
1509218080, 4684.5, null
1509218090, 4684.5, null
1509218100, 4684.5, null
1509218110, 4684.5, null
1509218120, 4684.5, null
1509218130, 4686.0, 2799.0
1509218140, 4686.0, 2799.0
1509218150, 4686.0, 2799.0
1509218160, 4686.0, 2799.0
1509218170, 4686.0, 2799.0
1509218180, 4686.0, 2799.0
1509218190, 4686.0, 2799.0
1509218200, 4686.0, 2799.0
1509218210, 4686.0, 2799.0
1509218220, 4686.0, 2799.0
1509218230, 4686.0, 2799.0
1509218240, 4686.0, 2799.0
1509218250, 4686.0, 2799.0
1509218260, 4686.0, 2799.0
1509218270, 4686.0, 2799.0
1509218280, 4686.0, 2799.0
1509218290, 4686.0, 2799.0
1509218300, 4686.0, 2799.0
1509218310, 4686.0, 2799.0
1509218320, 4686.0, 2799.0
1509218330, 4686.0, 2799.0
1509218340, 4686.0, 2799.0
1509218350, 4686.0, 2799.0
1509218360, 4686.0, 2799.0
1509218370, 4686.0, 2799.0
1509218380, 4686.0, 2799.0
1509218390, 4686.0, 2799.0
1509218400, 4686.0, 2799.0
1509218410, 4686.0, 2799.0
1509218420, 4686.0, 2799.0
1509218430, 4686.0, 2799.0
1509218440, 4686.0, 2799.0
1509218450, 4686.0, 2799.0
1509218460, 4686.0, 2799.0
1509218470, 4686.0, 2799.0
1509218480, 4686.0, 2799.0
1509218490, 4686.0, 2799.0
1509218500, 4686.0, 2799.0
1509218510, 4686.0, 2799.0
1509218520, 4686.0, 2799.0
1509218530, 4686.0, 2799.0
1509218540, 4686.0, 2799.0
1509218550, 4686.0, 2799.0
1509218560, 4686.0, 2799.0
1509218570, 4686.0, 2799.0
1509218580, 4686.0, 2799.0
1509218590, 4686.0, 2799.0
1509218600, 4686.0, 2799.0
1509218610, 4686.0, 2799.0
1509218620, 4686.0, 2799.0
1509218630, 4686.0, 2799.0
1509218640, 4686.0, 2799.0
1509218650, 4686.0, 2799.0
1509218660, 4686.0, 2799.0
1509218670, 4687.0, 2800.0
1509218680, 4687.0, 2800.0
1509218690, 4687.0, 2800.0
1509218700, 4687.0, 2800.0
1509218710, 4687.0, 2800.0
1509218720, 4687.0, 2800.0
1509218730, 4687.0, 2800.0
1509218740, 4687.0, 2800.0
1509218750, 4687.0, 2800.0
1509218760, 4687.0, 2800.0
1509218770, 4687.0, 2800.0
1509218780, 4687.0, 2800.0
1509218790, 4687.0, 2800.0
1509218800, 4687.0, 2800.0
1509218810, 4687.0, 2800.0
1509218820, 4687.0, 2800.0
1509218830, 4687.0, 2800.0
1509218840, 4687.0, 2800.0
1509218850, 4687.0, 2800.0
1509218860, 4687.0, 2800.0
1509218870, 4687.0, 2800.0
1509218880, 4687.0, 2800.0
1509218890, 4687.0, 2800.0
1509218900, 4687.0, 2800.0
1509218910, 4687.0, 2800.0
1509218920, 4687.0, 2800.0
1509218930, 4687.0, 2800.0
1509218940, 4687.0, 2800.0
1509218950, 4687.0, 2800.0
1509218960, 4687.0, 2800.0
1509218970, 4687.0, 2800.0
1509218980, 4687.0, 2800.0
1509218990, 4687.0, 2800.0
1509219000, 4687.0, 2800.0
1509219010, 4687.0, 2800.0
1509219020, 4687.0, 2800.0
1509219030, 4687.0, 2800.0
1509219040, 4687.0, 2800.0
1509219050, 4687.0, 2800.0
1509219060, 4687.0, 2800.0
1509219070, 4687.0, 2800.0
1509219080, 4687.0, 2800.0
1509219090, 4687.0, 2800.0
1509219100, 4687.0, 2800.0
1509219110, 4687.0, 2800.0
1509219120, 4687.0, 2800.0
1509219130, 4687.0, 2800.0
1509219140, 4687.0, 2800.0
1509219150, 4687.0, 2800.0
1509219160, 4687.0, 2800.0
1509219170, 4687.0, 2800.0
1509219180, 4687.0, 2800.0
1509219190, 4687.0, 2800.0
1509219200, 4687.0, 2800.0
1509219210, 4687.0, 2800.0
1509219220, 4687.0, 2800.0
1509219230, 4687.0, 2800.0
1509219240, 4687.0, 2800.0
1509219250, 4687.0, 2800.0
1509219260, 4687.0, 2800.0
1509219270, 4687.0, 2800.0
1509219280, 4687.0, 2800.0

emonhub.log

emonhub.log.txt (111.4 KB)

15

How bizarre!

1509214762, 1063.0, 2796.0            1509214760, 4683.0, 2796.0
                                      1509214820, 4673.0, null
                                      1509214880, 4678.0, null
1509214937, 1063.0, 2796.0            1509214940, 4683.0, 2796.0

Well you can see the anomaly clearer when you compare the 2 phptimeseries feeds (left) with the phpfina (right), Every thing seem to be in order aside from the whaccumulator quite literally fudging the figures.

You can see when the input was updated and that the regular phpfina backfilled with nulls, but the whaccumulator seems to have backfilled with something else, it dropped 10 off the first infill and 5 off the second infill. Later in the data the next occurrence only drops 1.5 from 4686.0 to 4684.5 so the amount isn’t consistent either.

Perhaps @TrystanLea can take a look when he gets back from his Highland honeymoon, I will take a closer look at the whaccumulator code to work out how it is supposed to work, particularly when backfilling if I get a chance but right now I’m fried so it’s pointless trying.

[EDIT]

I couldn’t find the 2nd missing packet in the emonhub.log but here’s the 2 successful packets, with the first failed packet

2017-10-28 18:19:22,501 DEBUG    RFM2Pi     34799 NEW FRAME : OK 20 208 0 0 0 161 2 29 0 236 10 0 0 (-73)
2017-10-28 18:19:22,504 DEBUG    RFM2Pi     34799 Timestamp : 1509214762.5
2017-10-28 18:19:22,504 DEBUG    RFM2Pi     34799 From Node : 20
2017-10-28 18:19:22,505 DEBUG    RFM2Pi     34799    Values : [20.8, 0, 67.3, 2.9000000000000004, 2796]
2017-10-28 18:19:22,505 DEBUG    RFM2Pi     34799      RSSI : -73
2017-10-28 18:19:22,506 DEBUG    RFM2Pi     34799 Sent to channel(start)' : ToEmonCMS
2017-10-28 18:19:22,506 DEBUG    RFM2Pi     34799 Sent to channel(end)' : ToEmonCMS
2017-10-28 18:19:22,714 DEBUG    MQTT       Publishing: emon/emonth2/temperature 20.8
2017-10-28 18:19:22,718 DEBUG    MQTT       Publishing: emon/emonth2/external temperature 0
2017-10-28 18:19:22,722 DEBUG    MQTT       Publishing: emon/emonth2/humidity 67.3
2017-10-28 18:19:22,723 DEBUG    MQTT       Publishing: emon/emonth2/battery 2.9
2017-10-28 18:19:22,725 DEBUG    MQTT       Publishing: emon/emonth2/pulsecount 2796
2017-10-28 18:19:22,726 INFO     MQTT       Publishing: emon/emonth2/rssi -73
2017-10-28 18:19:22,728 INFO     MQTT       Publishing: emonhub/rx/20/values 20.8,0,67.3,2.9,2796
2017-10-28 18:19:22,729 INFO     MQTT       Publishing: emonhub/rx/20/rssi -73
...
2017-10-28 18:20:20,862 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 20 209 0 0 0 161 2 29 0 236 10 0 0 0 32 0 0 0 0 0 32 (-81)
...

...
2017-10-28 18:22:17,539 DEBUG    RFM2Pi     34836 NEW FRAME : OK 20 209 0 0 0 160 2 29 0 236 10 0 0 (-81)
2017-10-28 18:22:17,541 DEBUG    RFM2Pi     34836 Timestamp : 1509214937.54
2017-10-28 18:22:17,541 DEBUG    RFM2Pi     34836 From Node : 20
2017-10-28 18:22:17,541 DEBUG    RFM2Pi     34836    Values : [20.900000000000002, 0, 67.2, 2.9000000000000004, 2796]
2017-10-28 18:22:17,541 DEBUG    RFM2Pi     34836      RSSI : -81
2017-10-28 18:22:17,542 DEBUG    RFM2Pi     34836 Sent to channel(start)' : ToEmonCMS
2017-10-28 18:22:17,542 DEBUG    RFM2Pi     34836 Sent to channel(end)' : ToEmonCMS
2017-10-28 18:22:17,588 DEBUG    MQTT       Publishing: emon/emonth2/temperature 20.9
2017-10-28 18:22:17,589 DEBUG    MQTT       Publishing: emon/emonth2/external temperature 0
2017-10-28 18:22:17,589 DEBUG    MQTT       Publishing: emon/emonth2/humidity 67.2
2017-10-28 18:22:17,590 DEBUG    MQTT       Publishing: emon/emonth2/battery 2.9
2017-10-28 18:22:17,591 DEBUG    MQTT       Publishing: emon/emonth2/pulsecount 2796
2017-10-28 18:22:17,592 INFO     MQTT       Publishing: emon/emonth2/rssi -81
2017-10-28 18:22:17,592 INFO     MQTT       Publishing: emonhub/rx/20/values 20.9,0,67.2,2.9,2796
2017-10-28 18:22:17,593 INFO     MQTT       Publishing: emonhub/rx/20/rssi -81
16

That’s definitely best practice and should also be in the official guide. My emonpi was accidentally without power for some days, which is not a big problem as the emonTh continues to increment the pulses. Once the Pi was back online, the wh_accumulator did a bad job:

16
161263×546 67.7 KB

For any reason (probably because the wh_accumulator needs also the elapsed time for the calculation), it did’t count the 1971 pulses and it just shows a horizontal line. However “Total pulse count to pulse increment” worked as expected:

40
401001×499 41.9 KB