Flat file as input

TGC · 2018-10-04T20:12:33.196Z

Hi,
Totally new to to this, and trying to find a solution to collect and display data from my solar system.
I have input data (battery volts, harvested power, instant mains power), as a file on a linux server on my LAN, updated every 5s. How can I use that file as an input?
Alternatively, the server could easily push the data (http, ssh…).
Not sure if that is feasible at all?

Thanks for the help!

Bill.Thomson · 2018-10-04T20:23:29.864Z

We need some more info.

Are you speaking of using your flat file data as an input to emonCMS?
If so, are you speaking of a local instance of emonCMS or emoncms.org?

What is the format of your flat file data? Is it text? JSON? another format, perhaps?

TGC · 2018-10-04T22:19:31.598Z

Thanks Bill. Yes, input into emonCMS.
Local instance (remote Australia so … connectivity is at a premium here :). I don’t have it installed yet, I want to check if it is feasible before doing the install.
File is text, fixed width fields. Example of the harvested power one:
05-10-2018 07:41:23 8577.2250826167113

pb66 · 2018-10-05T08:25:04.080Z

There is nothing really “off the shelf” to do what you want, how complex it would be to implement something depends on what you have and what you want to achieve exactly.

The format of your data is still unclear

TGC:

I have input data (battery volts, harvested power, instant mains power), as a file

suggests a single file containing multiple metrics where as

TGC:

Example of the harvested power one:
05-10-2018 07:41:23 8577.2250826167113

suggests a separate file for each metric, assuming “harvested power” is a reference to the file and not just the entry in the file.

Do you have control over the format, creation and update of these files or is that dictated by a non-configurable device/software?

If the files are per metric you will possibly need to read and parse the data back into an array for each timestamp to post to emoncms in a way that allows more input processing, where as if you are just mirroring the data to emoncms with little or no input processing you can possibly do one file/metric at a time. eg if you want to combine the harvested power and instant mains power to calculate and create import/export or charge/discharge feeds, those 2 metrics must be posted simultaneously for each timestamp not as one batch and then the other as processing is generally done at input time.

Bramco · 2018-10-05T13:34:31.870Z

I think the guys are making it sound more difficult than it is.

Assuming that as you drop each reading into the file every 5 seconds you can also send all 3 as a package to your local instance using the http interface. You obviously need the IP address, the emoncms key and a node id which can be anything in the right range, say 20. that would give you 3 inputs in your local emoncms every 5 seconds on node 20 which you can then use to create feeds and then create a dashboard.

You might want to reduce the accuracy of the harvested power reading a tad, it’s remarkably accurate in your example.

I do this from an ESP 8266 based heat bank controller without any problems.

I’ve never tried to get historical data into emonCMS but I think there are some tutorials on how to do that, but it would essentially be the same as above but with the timestamp.

Simon

TGC · 2018-10-05T23:03:15.777Z

Great. Thanks to all.
Yes, I do have total control over the files, as I produce them from clamps and other sensors on a rPi (given the less-than-open-and-friendly nature of Schneider…), so I can push them into the http interface, yes. And probably less frequently
Good, all green to proceed with install ofenmonCMS.
Again, thanks.

Bill.Thomson · 2018-10-06T01:08:54.857Z

Here’s an example using BASH

In post number two of this thread:

Read the SDM220 modbus with the minimalmodbus python module Hardware

Thanks Bill!
···
On mar., 15 ago. 2017 16:14 Bill Thomson < noreply@community.openenergymonitor.org> wrote: Bill.Thomson <a href="https://community.openenergymonitor.org/u/bill.thomson">https://community.openenergymonitor.org/u/bill.thomson</a> Bill Thomson <a href="https://community.openenergymonitor.org/u/bill.thomson">https://community.openenergymonitor.org/u/bill.thomson</a> August 15 Hi Joaquin, I use this <a href="http://www.e…

is a script that reads data from a text file and uses curl to send the data emonCMS.