Uploading historic data

Tassie-Paul · 2016-12-20T10:07:08.414Z

Hi, I made a data logger for some Bee Hives and due to the location no internet access is available. The data is in CSV format. I can format it any way to make use of the Emoncms API.

The road block I am hitting is how to get the data into Emoncms, I did see some talk on the forum regarding Curl, Python and PHP.

I am on windows which may limit me somewhat. I can put it in to a text file with each API call in succession, but was after some direction as to how to automate the process somewhat.

Thanks Paul

Tassie-Paul · 2016-12-28T10:19:12.837Z

Ok, So a bit of an update.

After struggling with Curl, I found python to be better and after a crash course I have come up with a py script which will post the contents of a text file, did a test with 5 api calls.

My main question now is what are the restrictions on posting data/ being kind to the server and others.

i.e. Max posts per hour, I do have a fair bit of data but can leave it running in the background.

Cheers Paul

jimmy · 2016-12-28T16:18:29.939Z

Hi,

I’m still playing with this kind of thing at the moment.

With it being a self hosted install, its kind of hardware dependent.

At the moment, I’m importing data points in 5000 at a time, and its taking 2 hours to do a month. However, this is doing a full process list as well as if the data was just coming in.

If its just a straight log, it may be easier to call the phpfina post object in php direct. I will be trying this later on once this has finished importing my mysql from 2013 to 3 days ago.

Tassie-Paul · 2016-12-28T19:29:56.560Z

Thanks Jimmy.

I’m posting to emoncms.org and am only doing api calls like https://emncms.org/input.json?&nodeid=10 with UNIX time and csv values. Python was as complicated as my abilities would extend to.

I would be interested in uploading by php if I could get my head around the code and if it made everything easier.

Regards Paul

Tassie-Paul · 2017-01-01T11:41:40.350Z

Hi Jimmy, just wondering fi you would be prepared to share your method/ script for uploading 5000 points. My python script is working but as I have no error handling (a bit beyond me ) at random intervals it seems to be refused a connection from emoncms.org. Then I have to start it from where it stalled

Thanks

jimmy · 2017-01-02T06:30:57.713Z

Hi Paul,

Not a problem, for the most part this worked, looks as though the time went a bit funky on my solar feed for some reason. So it’s not imported from April 2016 onwards, but that’s nothing to do with the script.

I’m away from my scripts at the moment, but around 9ish I’ll be there so will upload then for you.

I see your running on windows, you may run into a execution time limit, but ill cover all that in my next post when I’ve got the meaningful script for you.

jimmy · 2017-01-02T08:55:03.389Z

Hi Again,

Right so I am around my scripts now. Mine all used mysql to pull the old data out, and then push it back in, however this concept will work with a csv with a few alterations.

What information is it you have that you are wanting to input and what is its current format?

Looking at what you are posting above it is set to node 10.

If you could give me a sample of around 10 lines of data with the headings I can get this script up and running for you as a php script.

jimmy · 2017-01-04T18:50:28.205Z

Hi,

I have written a quick script, attached emoncmsCSVImporter.php.txt (2.7 KB)
, at the moment it has the provision to take in the three parameters “Unix time”, “Temperature”, and “Humidity”.

The script is fully documented, so you can change it as you need to.

Generally speaking I prefer to run PHP scripts like this on the command line, as within Linux the timeouts are hard coded to not kill after a max execution time. Windows is not always the case.

Hope that it will help

Jimmy

Tassie-Paul · 2017-01-06T03:55:42.955Z

Hi jimmy.

Thanks heaps. I did have some luck with the bulk api, but after reading you php script and pretty well understanding it including changing it to suit my needs I think it will fit my needs perfectly.

I appreciate your help and sharing your code, the older I get the more I seem to learn.

I do have a machine running Linux at home so it may be easier to get it running on that, do I also need to install curl as well ??

Cheers Paul

jimmy · 2017-01-06T07:58:57.144Z

Hi,

Yes you will need curl installed, I cheat on Linux when installing it as I install php5-curl so it pulls all of the dependencies at once.

If you have emoncms running on a windows machine and Linux on another, you could just have the script and your CSV files on the Linux one posting to the windows one.

I do find it easier to learn coding when I have a need, I’ve tried books etc but nothing ever sunk in until I had an application for it and wrote what I needed

thundersun · 2017-01-14T00:23:54.302Z

This was failing for me with PHP Notice: Undefined offset. For some reason beyond my limited PHP comprehension, the script was reading a blank final line even though my CSV didn’t have one. Also, it wasn’t executing the final sendChunk.

Here’s my modified version of the script with those issues fixed.
emoncmsCSVImporter.php.txt (2.9 KB)

thundersun · 2017-01-14T17:18:22.535Z

Say, has anyone had any success using this to upload to emonCMS.org? It works fine to my local server, but it fails silently (with a response of OK, no less!) when I point it to my emoncms.org account.

[This question is asked again here: Historic uploads failing silently on emoncms.org, not on local server - Moderator (RW)]

thundersun · 2017-02-09T17:48:31.012Z

Followup: it appears that none of this works with emonCMS.org – it is silently failing at present on all API requests that include a time= or an offset= argument.

Here’s a more more extensive bulk upload script – I’ve included a function to repeatedly upload a random 1-10 value at a specified interval, perhaps useful for testing new feeds.

GitHub

GitHub - dsjstc/emoncsv

Contribute to dsjstc/emoncsv development by creating an account on GitHub.

pb66 · 2017-05-27T20:34:15.830Z

19 posts were split to a new topic: Uploading MATE data from an OutbackCC to emoncms