Problems with bulk upload

franknfurther · 14 March 2018 21:43

I’m having immense trouble with uploading a solar production forecast into emonCMS using the input/bulk. json command. The command works fine as long as I upload dataseries up to the present, in other words using a time= timestamp that is near the present and data before that time. Below is the code, it gives no error but also no data is found at the upload location. What could be the problem? Thanks for help.

https://emoncms.org/input/bulk.json?data=[[-8640,55,642],[-8340,55,656],[-8040,55,669],[-7740,55,683],[-7440,55,696],[-7140,55,710],[-6840,55,723],[-6540,55,737],[-6240,55,750],[-5940,55,764],[-5640,55,777],[-5340,55,791],[-5040,55,804],[-4740,55,810],[-4440,55,816],[-4140,55,822],[-3840,55,828],[-3540,55,834],[-3240,55,840],[-2940,55,846],[-2640,55,852],[-2340,55,858],[-2040,55,864],[-1740,55,870],[-1440,55,876],[-1140,55,877],[-840,55,878],[-540,55,879],[-240,55,880]]&time=1521458700&apikey=xxxxxxxxx

pb66 · 14 March 2018 22:24

That timestamp is for 19th March 11.25 which is4.5 days ahead. you can only post data that is up to 48hrs in the future to some feed types eg phpfina.

There is a line you can edit in emoncms/Modules/feed/engine/PHPFina.php to change that

github.com

emoncms/emoncms/blob/master/Modules/feed/engine/PHPFina.php#L152


      
              }
          
              $meta->end_time = $meta->start_time + ($meta->interval * ($meta->npoints-1));
              if ($meta->end_time<0) $meta->end_time = 0;
          
              return $meta;
          }
          
          /**
           * Returns engine occupied size in bytes
           *
           * @param integer $id The id of the feed to be created
          */
          public function get_feed_size($id)
          {
              if (!$meta = $this->get_meta($id)) return false;
              return (16 + filesize($this->dir.$id.".dat"));
          }
          
          /**
           * Adds a data point to the feed

(If you’re interested, take a look at the Importing NREL Solar Models in to EmonCMS - #19 by pb66 thread for a similar “forecasting” type project, for which I had to change that same line)

franknfurther · 15 March 2018 10:26

thank you Paul, that is useful to know even though I find 2 days a bit too restrictive. 2 weeks might be better as default as this is the range that medium range weather forecasts have these days

However, I am still having problems. When I upload the data for only one day ahead, I can see the input on emoncms.org. Subsequently I create a log to feed, run the code again, but nothing shows on the feed (it gives me NaN and inactive).

However, the input on emoncms.org correctly shows me the last value uploaded (in this case 880)

What could be the reason for the input not being logged to the feed that I created? is there also a way create the log to feed using python code?

pb66 · 15 March 2018 11:21

My bad, I totally missed the fact you were using emoncms.org, so editing that restriction is not an option.

I can’t see any specific reason for it not working from your description. The key elements are to always initialize the feed with the oldest data as you can only move forward in time, so if you had run some tests and managed to post a frame 48hrs in the future you would no longer be able to “back-fill” the data between now and then.

The 48hrs is from the last datapoint entry unless it is the very first entry, in which case it is from the feeds “start time” in the meta file. The feed “start time” is set at initialization, you could check the start time of the feed using https://emoncms.org/feed/getmeta.json?id=220663 (add apikey if not logged in).

I have never used the bulk input format you are using so I cannot be sure it works, I have only ever sent full timestamps with each data frame and then used the “sentat=” or “time=0” parameters so it might be worth trying another format.

I do not know the code off hand to create a log to feed via Python but the request used in Python shouldn’t affect the code used to send it, it should be the same as any other, just the actual request details change.

All the feed and input api’s are documented from the link at the top right of the feed and input pages.

franknfurther · 15 March 2018 12:58

Hi Paul, thanks for your help. I have been trying all available formats for a number of days now and desparation is beginning to kick in…

In my experience the problem is less the format that is used but it has more to do with the interaction between the input and the feed. Why is the data showing in the input but not in the feed? Or how can I create both input and feed before I polulate the feed through the input? Because I have feeling that it is there it goes wrong.

Or is there any better way to upload multiple datapoints for one single feed in one go?

pb66 · 15 March 2018 13:03

Possibly due to the feed start time? I gave you the url to check that.

Using the create feed api (docs links on feed page) and sending the earliest data first, the feed will then be initialized using that earliest datastamp.