Could emonESP also support CSV and post to a local emonHub?

Hardware
10

And I am going to do it again :wink: I think my original statement was correct; there is no difference between the order of parsing a CSV set of values and a ‘JSON’ set of values as in PHP an array is just a map with numbers as the ‘name’ and in both cases the map is ordered, see this post for more info.

But just to be sure walk with me through the input processing in /input/post.json (and I am talking about the single input here not the bulk input) …

                // code below processes input regardless of json or csv type
                if (isset($_GET['json'])) $datain = get('json');
                else if (isset($_GET['csv'])) $datain = get('csv');
                else if (isset($_GET['data'])) $datain = get('data');
                else if (isset($_POST['data'])) $datain = post('data');

This bit pulls the input CSV/JSON/whatever in to $datain

                if ($datain!="")
                {
                    $json = preg_replace('/[^\p{N}\p{L}_\s-.:,]/u','',$datain);

If we actually have any data strip out the anything that is not (^) a number (\p{N}), letter (\p{L}), _, whitespace (\s), -, ., :, or ,, ie for JSON input this will strip put all the JSON notation and leave you with just a name:value comma separated list.

                    $datapairs = explode(',', $json);

Split the string into an array ($datapairs) on the , char.

                    $data = array();
                    $csvi = 0;
                    for ($i=0; $i<count($datapairs); $i++)

loop through each item

                    {
                        $keyvalue = explode(':', $datapairs[$i]);

split into an array ($keyval) on the : char

                        if (isset($keyvalue[1])) {

If there is a second value in the $keyval array, ie name:value, then

                            if ($keyvalue[0]=='') {$valid = false; $error = "Format error, json key missing or invalid character"; }
                            if (!is_numeric($keyvalue[1])) {$valid = false; $error = "Format error, json value is not numeric"; }

do some validation, $keyvalue[0] (the ‘name’) must not be empty and $keyvalue[1] (the value) must be a number.

                            $data[$keyvalue[0]] = (float) $keyvalue[1];

store in the $data array

                        } else {

else, ie just a number

                            if (!is_numeric($keyvalue[0])) {$valid = false; $error = "Format error: csv value is not numeric"; }

make sure $keyvalue[0] (the value) is a number.

                            $data[$csvi+1] = (float) $keyvalue[0];
                            $csvi ++;

store in the $data array with the name of $csvi + 1, which as a side note for the input 10,20,30 will give you;

$data[1] = 10
$data[2] = 20
$data[3] = 30

will not be a problem, but worth noting

                        }
                    }

So now we have all our data parsed in to $data in the order added so for {p1:10,p2:20,p3:30} it will always be;

$data["p1"] = 10
$data["p2"] = 20
$data["p3"] = 30

and for 10,20,30 you will have

$data[1] = 10
$data[2] = 20
$data[3] = 30

                    $tmp = array();
                    foreach ($data as $name => $value)

enumerate the $data array;

                    {
                        if (!isset($dbinputs[$nodeid][$name])) {

If the input node does not exist then;

                            $inputid = $input->create_input($userid, $nodeid, $name);
                            $dbinputs[$nodeid][$name] = true;
                            $dbinputs[$nodeid][$name] = array('id'=>$inputid, 'processList'=>'');
                            $input->set_timevalue($dbinputs[$nodeid][$name]['id'],$time,$value);

Create the new input node and set the value, no need for processing as we have only just created it

                        } else {
                            $input->set_timevalue($dbinputs[$nodeid][$name]['id'],$time,$value);
                            if ($dbinputs[$nodeid][$name]['processList']) $tmp[] = array('value'=>$value,'processList'=>$dbinputs[$nodeid][$name]['processList'],'opt'=>array('sourcetype' => "INPUT",'sourceid'=>$dbinputs[$nodeid][$name]['id']));
                        }

else set the new value and if there is a process list add to the list of inputs to process ($tmp)

                    }

So at this point we have set all the values in the respective inputs and $tmp has a list of inputs that need processing but nothing has been actually processed.

                    foreach ($tmp as $i) $process->input($time,$i['value'],$i['processList'],$i['opt']);

The final step here is to actually go and process the inputs. Now the interesting thing here is that even though the inputs are kept in order it actually does not matter when summing or subtracting inputs as all the new values have already been updated.

I would also suggest modifying the input API documentation to change the examples to all use data=… instead of csv=… and json=… I think this may help show that it does not matter which format you use, and as a secondary point you can use the POST HTTP method which is a bit more RESTful :wink:

2 Likes