Background to this is my graphing challenges here. If data is written to the API at a sample rate less then the PHPTIMESERIES setting in emon you get a lot of null values in the database. When creating a feed the default is 10 second, I did this default for many feeds with data written every 30 or 60 seconds.
The script below can be run to extract the original data from the feed, via the API, between two timeranges into a Pandas Dataframe. It is then saved locally in a pickle file. Once in Pandas the whole array of data management/resampling functions are available.
It uses some functions emon_api_functions_utc.py I shared earlier (but are included in the zip file again)
Data can then be written back to a new feed (with a different interval) via the API. The script uses the emon bulk write function, writing hours (or days) of data at a time. There is a limit on th
e length of the URL you can call. The script checks for this - there is an option to simulate the writing process to check the length. if it fails just select a shorter time “chunk” for the writeback.
The write url length is dependent on the data e.g. a days worth of temperatures to 4 decimal places is longerstring than a days to 2 decimal places. The script limits numbers to 2 places.
I am currently connected to the emon remotely over a SSH tunnel on a rural broadbank link extracting 1 year of 10 second data takes 2-3 minutes. Running Pandas on a i5 Thinkpad I can resample it to 1 minute data in 12 seconds. The API writeback than takes another 4 minutes or so. It will probably be quicker when the laptop is on the same LAN as the emon.
emon_api_read_resample_rewrite.zip (4.3 KB)