The latest version of the Emoncms Sync module is now available Key improvements:
New emoncms_sync.service that handles automated upload of feed data at a user selected interval, e.g every 5, 10, 15, 30 minutes, hourly or daily. No need to setup a crontab entry.
Significantly improved upload efficiency by reducing the number of HTTP/s requests required. Instead of sending separate requests for each feed, data from multiple feeds is now consolidated into a single HTTP/s request. This reduces bandwidth consumption primarily by minimising the repeated transfer of HTTP/s headers.
These improvements make using the emoncms sync module a more viable option for uploading data from e.g a local emonPi2 up to emoncms.org in a regular automated way. This can replace using emonHub for data upload in many applications, saving the need to duplicate input processing on both local and remote emoncms instances and improving reliability of data upload when there are connectivity outages.
Other sync interface improvements include:
Option to authenticate with remote server using username and password login or write apikey - whichever is easiest to access.
Feeds grouped by node name
Quick select/deselect option to register feeds for upload or to stop uploading data for a feed.
Multi select to select/deselect multiple feeds.
emoncms_sync service status
To update:
Option 1: Update via Admin Update > Full Update
Option 2: Selectively update Emoncms Core and the Sync module via Admin > Components.
Reconfigure sync remote settings if you have an existing configuration in place.
If you added a manual crontab entry to run the older sync_upload.php periodically, please make sure to remove it.
Probably a dumb question, but do I need to change the senddata = 1 (below) to senddata = 0 in emonhub before enabling sync? I have not used sync previously.
Thanks,
Gordon
[[emoncmsorg]]
Type = EmonHubEmoncmsHTTPInterfacer
[[[init_settings]]]
[[[runtimesettings]]]
pubchannels = ToRFM12,
subchannels = ToEmonCMS,
url = http://emoncms.org
apikey = xxxxxxxxxxxxxxxxxxxxxxxxxx
senddata = 1 # Enable sending data to Emoncms.org
sendnames = 1 # Send full input names (compression will be automatically enabled)
interval = 30 # Bulk send interval to Emoncms.org in seconds
Apologies I have now fixed this issue, version is now 3.1.4. If you could both try this latest version that would be great, update either via full update or selective component update. If updating via git make sure to restart the emoncms_sync service.
On the local emonPi2, it appears to be working, with one minor hiccup below.
I initially clicked only the feeds I wanted to upload, and the “upload selected” button did not appear. I then clicked “Select all” and then unclicked the ones I did not want uploaded. The upload select button then appeared.
However, when I go into emoncms.org, it has not updated for the past 33 minutes (ever since I made the change).
I checked that the APIkeys were correct, and then started to wonder if I needed to sync the inputs as well.
So, back to the local emonPi2 to set sync for inputs, but no inputs are showing to select.
Hmm, I probably haven’t been clear about an important requirement that local and remote feeds have the precise same start times and intervals and I probably need to add better feedback in the interface about this…
I assume that you created one set of feeds locally and then a few moments later created a second set of feeds on the remote server? Using emonhub and manually setting up the same input processing?
If the completeness of your local feeds are better than the remote ones I would delete the remote feeds and then configure the sync module to sync up from the start… one way to test this safely would be to create a new emoncms.org account and sync to that fresh account just so that you can see what I mean and the process working?
You are correct in your assumptions. When we set up the emonPi2s, we do this in advance of our installations and then set emonCMS.org as we install. This could be days or weeks later.
To make sure I understand the process, please confirm my understanding below.
1 - configure the inputs and feeds on the emonPi2 and disable the emoncms interface in emonhub.
2 - establish sync to the emoncms.org instance
Deleting the feeds for our existing installations could be problematic as we are on metered cellular connections if I understand what you are suggesting correctly. By deleting the existing feeds in emoncms.org and then establishing sync, it would upload all the historical data first and then sync regularly from that point forward—correct? (in one location, this is over 100 MB)
It makes sense for our new deployments, but I have to think about how to update our existing sites.
Correct, if you started recording data on emoncms.org before the emonPi then you can also delete the local feeds and download the emoncms.org feeds, setup input processing locally and then sync upload from then on…
Yes something like that would work, one of the other things I want to add is a md5 or similar check to ensure both local and remote are identical and to offer re-upload if not. This would be needed if editing historic data in feeds or if feeds are recalculated using post processing…
Key improvements:
