Uploading data to emoncms.org

Hi all,

I have searched through the forums but I can’t find a clear answer to this question: as of today what options are available to upload historical feed data to emoncms.org?
I understand that the sync module is not enabled on emoncms.org yet?
The only related option I found was to configure emonhub to post input data to emoncms.org, but that requires me to manually keep two feed configurations in sync doesn’t it?

Thanks!
Franck

This could be one for @TrystanLea

Hello @franck102

I needed to do the same this week in order to upload historic uk solar data used by the emailreports module on emoncms.org and so I wrote the following uploader script, which you could repurpose for your requirements.

It uses the emoncms/bulk/input.json api. There are usage examples on the input api helper page here Emoncms - site api

<?php
// ---------------------------------------------------------------------------------------------
// Emoncms data upload example
//
// 1. Create an input to receive the data upload
// https://emoncms.org/input/post?node=0&csv=100
// 
// 2. Setup a feed connected to the newly created input, set the interval to a suitable interval
// e.g data from sheffield solar below has an interval of 30 minutes.
// 
// 3. Run this script 
// ---------------------------------------------------------------------------------------------
$host = "https://emoncms.org";
$apikey = "WRITE APIKEY";
$nodeid = 0;

// Break upload into blocks of around 500-1000 datapoints
$start = time()-24*3600*365;
$time_block = 24*3600*14; // 672 datapoints at 30min interval
$end = $start + $time_block;

while (true) 
{
    $start = floor($start/3600)*3600;
    $end = floor($end/3600)*3600;
    
    // 1. Fetch data from source
    $startstr = date("Y-m-d",$start)."T".date("H:i:s",$start);
    $endstr = date("Y-m-d",$end)."T".date("H:i:s",$end);
    
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL,"https://api0.solar.sheffield.ac.uk/pvlive/v1?start=$startstr&end=$endstr");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $output = curl_exec ($ch);
    curl_close ($ch);
    
    print "start:".$startstr." end:".$endstr." strlen:".strlen($output)."\n";

    $data = array();
    $pv = json_decode($output);
    foreach ($pv->data as $row) {
      $time = strtotime($row[1]);
      $value = $row[2]*1;
      if ($value==null || $value==NAN) $value = 0;
      // 2. build bulk data format [time,nodeid,value,...]
      $data[] = array($time,$nodeid,$value);
    }
    
    // 3. send data to emoncms
    print "Result: ".http_request("POST","$host/input/bulk.json",array("time"=>0,"data"=>json_encode($data),"apikey"=>$apikey))."\n";
    
    // move forward to next block
    $start += $time_block;
    $end += $time_block;
    
    // please take your time uploading data
    // there is a 1s request limiter on emoncms.org
    sleep(1.0);
    
    // exit the script
    if ($end>time()) {
        print "complete\n";
        die;
    }
}

function http_request($method,$url,$data) {

    $options = array();
    $urlencoded = http_build_query($data);
    
    if ($method=="GET") { 
        $url = "$url?$urlencoded";
    } else if ($method=="POST") {
        $options[CURLOPT_POST] = 1;
        $options[CURLOPT_POSTFIELDS] = $data;
    }
    
    $options[CURLOPT_URL] = $url;
    $options[CURLOPT_RETURNTRANSFER] = 1;
    $options[CURLOPT_CONNECTTIMEOUT] = 2;
    $options[CURLOPT_TIMEOUT] = 5;

    $curl = curl_init();
    curl_setopt_array($curl,$options);
    $resp = curl_exec($curl);
    curl_close($curl);
    return $resp;
}

Thanks Trystan, I like the idea of using REST te obtain the source data.

For me though the approach would involve feeding feed data into an input… so a completely different set of feeds on emoncms.org (also much simpler I guess, so this may ok).

Do you have any ETA for having sync enabled for feeds on emoncms.org?

Thanks!
Franck

1 Like

No ETA yet on enabling sync upload on emoncms.org, its a little more complicate because of the multi-server environment and all the additional testing and load management required. There are a number of other steps to do first, but it is my intention to enable it.

Thanks Trystan, keep us posted :slight_smile:

Franck

Hi Trystan and Team, Is it still the case that we cannot use the web gui to sync inputs, feeds and dashboards to emoncms.org? I spent some time trying to figure out why it wasnt working but if it’s not enable yet then I will leave it for now because i’d prefer not to delve into php and risk my install for something that I can view over VPN anyway!

Hello @drc yes still not available, I’m working on some fairly substantial changes to the emoncms back-end and will as part of this be looking at enabling this feature on emoncms.org. Probably at least a couple of months away.