Uploading data to emoncms.org

Emoncms
1

Hi all,

I have searched through the forums but I can’t find a clear answer to this question: as of today what options are available to upload historical feed data to emoncms.org?
I understand that the sync module is not enabled on emoncms.org yet?
The only related option I found was to configure emonhub to post input data to emoncms.org, but that requires me to manually keep two feed configurations in sync doesn’t it?

Thanks!
Franck

2

This could be one for @TrystanLea

3

Hello @franck102

I needed to do the same this week in order to upload historic uk solar data used by the emailreports module on emoncms.org and so I wrote the following uploader script, which you could repurpose for your requirements.

It uses the emoncms/bulk/input.json api. There are usage examples on the input api helper page here Emoncms - site api

<?php
// ---------------------------------------------------------------------------------------------
// Emoncms data upload example
//
// 1. Create an input to receive the data upload
// https://emoncms.org/input/post?node=0&csv=100
// 
// 2. Setup a feed connected to the newly created input, set the interval to a suitable interval
// e.g data from sheffield solar below has an interval of 30 minutes.
// 
// 3. Run this script 
// ---------------------------------------------------------------------------------------------
$host = "https://emoncms.org";
$apikey = "WRITE APIKEY";
$nodeid = 0;

// Break upload into blocks of around 500-1000 datapoints
$start = time()-24*3600*365;
$time_block = 24*3600*14; // 672 datapoints at 30min interval
$end = $start + $time_block;

while (true) 
{
    $start = floor($start/3600)*3600;
    $end = floor($end/3600)*3600;
    
    // 1. Fetch data from source
    $startstr = date("Y-m-d",$start)."T".date("H:i:s",$start);
    $endstr = date("Y-m-d",$end)."T".date("H:i:s",$end);
    
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL,"https://api0.solar.sheffield.ac.uk/pvlive/v1?start=$startstr&end=$endstr");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $output = curl_exec ($ch);
    curl_close ($ch);
    
    print "start:".$startstr." end:".$endstr." strlen:".strlen($output)."\n";

    $data = array();
    $pv = json_decode($output);
    foreach ($pv->data as $row) {
      $time = strtotime($row[1]);
      $value = $row[2]*1;
      if ($value==null || $value==NAN) $value = 0;
      // 2. build bulk data format [time,nodeid,value,...]
      $data[] = array($time,$nodeid,$value);
    }
    
    // 3. send data to emoncms
    print "Result: ".http_request("POST","$host/input/bulk.json",array("time"=>0,"data"=>json_encode($data),"apikey"=>$apikey))."\n";
    
    // move forward to next block
    $start += $time_block;
    $end += $time_block;
    
    // please take your time uploading data
    // there is a 1s request limiter on emoncms.org
    sleep(1.0);
    
    // exit the script
    if ($end>time()) {
        print "complete\n";
        die;
    }
}

function http_request($method,$url,$data) {

    $options = array();
    $urlencoded = http_build_query($data);
    
    if ($method=="GET") { 
        $url = "$url?$urlencoded";
    } else if ($method=="POST") {
        $options[CURLOPT_POST] = 1;
        $options[CURLOPT_POSTFIELDS] = $data;
    }
    
    $options[CURLOPT_URL] = $url;
    $options[CURLOPT_RETURNTRANSFER] = 1;
    $options[CURLOPT_CONNECTTIMEOUT] = 2;
    $options[CURLOPT_TIMEOUT] = 5;

    $curl = curl_init();
    curl_setopt_array($curl,$options);
    $resp = curl_exec($curl);
    curl_close($curl);
    return $resp;
}
1 Like
4

Thanks Trystan, I like the idea of using REST te obtain the source data.

For me though the approach would involve feeding feed data into an input… so a completely different set of feeds on emoncms.org (also much simpler I guess, so this may ok).

Do you have any ETA for having sync enabled for feeds on emoncms.org?

Thanks!
Franck

1 Like
5

No ETA yet on enabling sync upload on emoncms.org, its a little more complicate because of the multi-server environment and all the additional testing and load management required. There are a number of other steps to do first, but it is my intention to enable it.

6

Thanks Trystan, keep us posted :slight_smile:

Franck

7

Hi Trystan and Team, Is it still the case that we cannot use the web gui to sync inputs, feeds and dashboards to emoncms.org? I spent some time trying to figure out why it wasnt working but if it’s not enable yet then I will leave it for now because i’d prefer not to delve into php and risk my install for something that I can view over VPN anyway!

8

Hello @drc yes still not available, I’m working on some fairly substantial changes to the emoncms back-end and will as part of this be looking at enabling this feature on emoncms.org. Probably at least a couple of months away.