Hi, I made a data logger for some Bee Hives and due to the location no internet access is available. The data is in CSV format. I can format it any way to make use of the Emoncms API.
The road block I am hitting is how to get the data into Emoncms, I did see some talk on the forum regarding Curl, Python and PHP.
I am on windows which may limit me somewhat. I can put it in to a text file with each API call in succession, but was after some direction as to how to automate the process somewhat.
After struggling with Curl, I found python to be better and after a crash course I have come up with a py script which will post the contents of a text file, did a test with 5 api calls.
My main question now is what are the restrictions on posting data/ being kind to the server and others.
i.e. Max posts per hour, I do have a fair bit of data but can leave it running in the background.
I’m still playing with this kind of thing at the moment.
With it being a self hosted install, its kind of hardware dependent.
At the moment, I’m importing data points in 5000 at a time, and its taking 2 hours to do a month. However, this is doing a full process list as well as if the data was just coming in.
If its just a straight log, it may be easier to call the phpfina post object in php direct. I will be trying this later on once this has finished importing my mysql from 2013 to 3 days ago.
Hi Jimmy, just wondering fi you would be prepared to share your method/ script for uploading 5000 points. My python script is working but as I have no error handling (a bit beyond me ) at random intervals it seems to be refused a connection from emoncms.org. Then I have to start it from where it stalled
Not a problem, for the most part this worked, looks as though the time went a bit funky on my solar feed for some reason. So it’s not imported from April 2016 onwards, but that’s nothing to do with the script.
I’m away from my scripts at the moment, but around 9ish I’ll be there so will upload then for you.
I see your running on windows, you may run into a execution time limit, but ill cover all that in my next post when I’ve got the meaningful script for you.
Right so I am around my scripts now. Mine all used mysql to pull the old data out, and then push it back in, however this concept will work with a csv with a few alterations.
What information is it you have that you are wanting to input and what is its current format?
Looking at what you are posting above it is set to node 10.
If you could give me a sample of around 10 lines of data with the headings I can get this script up and running for you as a php script.
I have written a quick script, attached emoncmsCSVImporter.php.txt (2.7 KB)
, at the moment it has the provision to take in the three parameters “Unix time”, “Temperature”, and “Humidity”.
The script is fully documented, so you can change it as you need to.
Generally speaking I prefer to run PHP scripts like this on the command line, as within Linux the timeouts are hard coded to not kill after a max execution time. Windows is not always the case.
Thanks heaps. I did have some luck with the bulk api, but after reading you php script and pretty well understanding it including changing it to suit my needs I think it will fit my needs perfectly.
I appreciate your help and sharing your code, the older I get the more I seem to learn.
I do have a machine running Linux at home so it may be easier to get it running on that, do I also need to install curl as well ??
Yes you will need curl installed, I cheat on Linux when installing it as I install php5-curl so it pulls all of the dependencies at once.
If you have emoncms running on a windows machine and Linux on another, you could just have the script and your CSV files on the Linux one posting to the windows one.
I do find it easier to learn coding when I have a need, I’ve tried books etc but nothing ever sunk in until I had an application for it and wrote what I needed
This was failing for me with PHP Notice: Undefined offset. For some reason beyond my limited PHP comprehension, the script was reading a blank final line even though my CSV didn’t have one. Also, it wasn’t executing the final sendChunk.
Say, has anyone had any success using this to upload to emonCMS.org? It works fine to my local server, but it fails silently (with a response of OK, no less!) when I point it to my emoncms.org account.
Followup: it appears that none of this works with emonCMS.org – it is silently failing at present on all API requests that include a time= or an offset= argument.
Here’s a more more extensive bulk upload script – I’ve included a function to repeatedly upload a random 1-10 value at a specified interval, perhaps useful for testing new feeds.
) at random intervals it seems to be refused a connection from 