Community
OpenEnergyMonitor

OpenEnergyMonitor Community

Problems with bulk upload

I’m having immense trouble with uploading a solar production forecast into emonCMS using the input/bulk. json command. The command works fine as long as I upload dataseries up to the present, in other words using a time= timestamp that is near the present and data before that time. Below is the code, it gives no error but also no data is found at the upload location. What could be the problem? Thanks for help.

https://emoncms.org/input/bulk.json?data=[[-8640,55,642],[-8340,55,656],[-8040,55,669],[-7740,55,683],[-7440,55,696],[-7140,55,710],[-6840,55,723],[-6540,55,737],[-6240,55,750],[-5940,55,764],[-5640,55,777],[-5340,55,791],[-5040,55,804],[-4740,55,810],[-4440,55,816],[-4140,55,822],[-3840,55,828],[-3540,55,834],[-3240,55,840],[-2940,55,846],[-2640,55,852],[-2340,55,858],[-2040,55,864],[-1740,55,870],[-1440,55,876],[-1140,55,877],[-840,55,878],[-540,55,879],[-240,55,880]]&time=1521458700&apikey=xxxxxxxxx

That timestamp is for 19th March 11.25 which is4.5 days ahead. you can only post data that is up to 48hrs in the future to some feed types eg phpfina.

There is a line you can edit in emoncms/Modules/feed/engine/PHPFina.php to change that

(If you’re interested, take a look at the Importing NREL Solar Models in to EmonCMS thread for a similar “forecasting” type project, for which I had to change that same line)

thank you Paul, that is useful to know even though I find 2 days a bit too restrictive. 2 weeks might be better as default as this is the range that medium range weather forecasts have these days

However, I am still having problems. When I upload the data for only one day ahead, I can see the input on emoncms.org. Subsequently I create a log to feed, run the code again, but nothing shows on the feed (it gives me NaN and inactive).


However, the input on emoncms.org correctly shows me the last value uploaded (in this case 880)

What could be the reason for the input not being logged to the feed that I created? is there also a way create the log to feed using python code?

My bad, I totally missed the fact you were using emoncms.org, so editing that restriction is not an option.

I can’t see any specific reason for it not working from your description. The key elements are to always initialize the feed with the oldest data as you can only move forward in time, so if you had run some tests and managed to post a frame 48hrs in the future you would no longer be able to “back-fill” the data between now and then.

The 48hrs is from the last datapoint entry unless it is the very first entry, in which case it is from the feeds “start time” in the meta file. The feed “start time” is set at initialization, you could check the start time of the feed using https://emoncms.org/feed/getmeta.json?id=220663 (add apikey if not logged in).

I have never used the bulk input format you are using so I cannot be sure it works, I have only ever sent full timestamps with each data frame and then used the “sentat=” or “time=0” parameters so it might be worth trying another format.

I do not know the code off hand to create a log to feed via Python but the request used in Python shouldn’t affect the code used to send it, it should be the same as any other, just the actual request details change.

All the feed and input api’s are documented from the link at the top right of the feed and input pages.

Hi Paul, thanks for your help. I have been trying all available formats for a number of days now and desparation is beginning to kick in…

In my experience the problem is less the format that is used but it has more to do with the interaction between the input and the feed. Why is the data showing in the input but not in the feed? Or how can I create both input and feed before I polulate the feed through the input? Because I have feeling that it is there it goes wrong.

Or is there any better way to upload multiple datapoints for one single feed in one go?

Possibly due to the feed start time? I gave you the url to check that.

Using the create feed api (docs links on feed page) and sending the earliest data first, the feed will then be initialized using that earliest datastamp.