Sending historic.csv data to Emoncms.org

Emoncms Emoncms.org
1

Hi,

I have been using emoncms.org for recording live data from a number of community solar projects for nearly 12 months and it has been working perfectly. We use our own hardware for the monitoring and it is first sent to our linode server and then pushed to the emoncms inputs using the API key. Thank you to everyone here who helps make it possible.

However we now need to log data from a different sort of meter which records half hourly kWh data for 24 hours and then sends a single .csv file to us via ftp.

We would like the (timestamped) half hourly data to be logged in emoncms.org which would be updated daily (at slightly different times each day).

We have tried to restructure the data to send to emoncms using the code below. It works to the extent that new inputs are being created but the data does not seem to actually be recorded by emoncms. We have tried using both PHPFina and PHP timeseries for logging the feeds.

I have read through this thread but there doesn’t seem to be a solution for emoncms.org: Uploading historic data - #12 by thundersun

The is the code we are using:

curl -g ‘https://www.emoncms.org/input/bulk.json?data=[[0,18,0.0]]&time=1508279400&apikey=74b7a699ba984a9a20c0ba3a7e2e4b31’
curl -g ‘https://www.emoncms.org/input/bulk.json?data=[[0,18,0.0]]&time=1508281200&apikey=74b7a699ba984a9a20c0ba3a7e2e4b31’
curl -g ‘https://www.emoncms.org/input/bulk.json?data=[[0,18,0.0]]&time=1508283000&apikey=74b7a699ba984a9a20c0ba3a7e2e4b31’
curl -g ‘https://www.emoncms.org/input/bulk.json?data=[[0,18,0.0]]&time=1508284800&apikey=74b7a699ba984a9a20c0ba3a7e2e4b31’

from a linux command line

[0,18,0.0] 0 is the time offset, 18 is the node and 0.0 is the value
time=1508284800 is the linux time associated with the data point.

there are 18*48 of these lines sent up for 18 meters and 48 timestamps

I can send the raw .csv if that is helpful.

Thank you in advance,

Felix

2

When and how did you create the feeds?

When you first post data to a feed that is when it is created and thus, it’s the start time for the feed, if you later try and post earlier data, it will not accept data timestamped any earlier than that feed start time.

If you were (for example) to create the inputs, processing and feeds, then post data that was a year old, you could then at any time post further data that was more recent and as long as you always move forward in time you should be ok. You cannot “backfill” data to feeds.

Try replacing the feeds with new ones and use only the oldest data.

Also try using absolute time stamps within the data and &time=0 as anything else might skew the timestamps (see Best way to guarantee times when using BULK load with PHPTIMESERIES? - #2 by pb66)

3

Thanks Paul.

I created the feeds by adding the ‘log to feed’ process to each input node.

The input nodes had been created by the data being sent from our server to the Emoncms API.

This was after the timestamp of the first dataset, and subsequent data has been consecutive (i.e. no backfilling) so not sure if this is the problem.

However will try using absolute time stamps and time=0.

Thanks very much for your help!

Felix