Emoncms graphs having missing data points

I am posting data to a third-party site, OpenEVSE, which is “Powered by openenergymonitor.org | 9.5 | 2016.04.19”

I have posted a ticket on the OpenEVSE site, but this may be an emoncms code problem so I am cross-posting here.

— begin included text —

#2074 emoncms incomplete display

Chuck Harrison
, reported 6 days ago
I have a simple dashboard at Emoncms - dashboard view (Emoncms - dashboard view)
which displays data from two current sensing channels through an unmodified OpenEVSE ESP8266 module. The dashboard contains one multigraph and
two individual rawdata graphs.
Initially everything behaved as I expected, then after about 5 days I observed something unexpected. Data on the multigraph after May 29 did not appear
unless the time scale was zoomed in to show 6 hrs or less span. At that time I added the two raw data graphs, which behaved normally. This situation
remained consistent for several days, after which I only checked in occasionally and then left the site unvisited for several weeks.
I looked at the site today and discovered that the multigraph has returned to displaying the most recent data at all time scale settings, but has a gap of
about three weeks with the anomalous behavior. Also, the raw data graphs are now behaving in a related (but not identical) way regarding display at longer
time spans.
This behavior occurs identically on Firefox (Ubuntu 14.04) and Chrome (both Ubuntu 14.04 and Windows 7).
Can you replicate this behavior? Any ideas?

Chuck Harrison
, said 3 minutes ago
I have gotten this far in debugging:
When drawing the graphs, the web page makes http requests to data.openevse.com/emoncms/feed/data.json
in order to get the data. These calls are handled by the function
get_data($name,$start,$end,$interval,$skipmissing,$limitinterval) in emoncms/Modules/feed/engine/PHPFina.php . These calls return less data than one
would expect.
Considering the raw data plot “L1 amps”, a typical request might be
data.openevse.com/emoncms/feed/data.json?
id=1792&start=1466803276323&end=1466889676323&interval=108&skipmissing=1&limitinterval=1&apikey=5cca0467b0ece4c5af430ddc809a813a
which returns a 320 ­data­point array. The expected length is 800 data points. By submitting a similar request with skipmissing=0, one obtains an array with
the missing points present, with null values. The total array length is 802 (index interval computation round­off?) and the missing points are scattered
throughout the data set.
data.openevse.com/emoncms/feed/data.json?
id=1792&start=1466803276323&end=1466889676323&interval=108&skipmissing=0&limitinterval=1&apikey=5cca0467b0ece4c5af430ddc809a813a

The situation is slightly different with the multigraph. A skipmissing=0 request like
data.openevse.com/emoncms/feed/data.json?
id=1793&start=1464298200000&end=1466890200000&interval=1800&skipmissing=0&limitinterval=1&apikey=5cca0467b0ece4c5af430ddc809a813a
returns a data array containing a large block of null data in the middle. This null data corresponds to the “gap” referred to in the original ticket.
Close observation of the json requests reveals that the multigraph generates “round number” time range and interval, while the raw data graph does not
seem to do any rounding.
My conjecture is that the problem lies somewhere in the PHPFina.php get_data() code, although I do not see an obvious error. My best guess is that $pos,
which is the floating­point result of round(), produces an incorrect argument to fseek when $pos*4 is truncated to integer. IANA PHP programmer.

Cheers,
Chuck

Problem is now understood.

The input source is publishing every 30 seconds, but the PHPfina feed is recording every 10 seconds. Therefore 2/3 of the data consists of “null” entries.

When extracting data with get_data(), the data file is sampled, and the distribution of null vs useful data values depends on the phasing of the reconstruction time intervals with respect to the original data timing. This can be a very slow “beat frequency” (multigraph intervals) or a less obvious one (raw data graph intervals).

It appears that to resolve this I will need to create a new feed with 30 second interval. The old 10-second feed will have to be retired. I can’t see any way to splice the old data to new data into a continuous historical record.

Chuck

I came across a similar problem. I wish we could just change the time intervals, instead of re-creating the feeds.