Trystan, the sync works great! Thanks for setting this up.
I decided to bite the bullet and delete the feeds on emoncms.org for one of my systems. Then, I set up sync on the local Pi2. All the history is loaded onto emoncms.org, and it is working quite well.
I have one question about feed size vs. upload size. On the local machine, my feeds are around 5.2MB, but when the first sync upload occurred, it was under 1MB, with each subsequent sync being less than 1KB. All the data is showing in the emoncms.org cloud, so I am confused about the reported sizes. Is there some compression happening during the sync?
On the next site, I convert I will take some screen shots as I didn’t think to do this during the first conversion.
Hello @grod55, great! it may be that you need to click on ‘refresh feed size’ at the bottom of the feeds page on both local and remote to see the updated size.
Ok - tried one of my remote sites that uses a cellular connection and it does not appear to be syncing. I have tried deleting a feed in emoncms.org and starting over, but nothing is updating.
Not sure what do do next. Please advise.
Here are the feeds in sync on the Local emonPi2
And here is what is showing in emonCMS.org. This has been active for the past 45 minutes without any change.
The one feed that did a partial update looks like it only updated two months of data and then stopped. None of the other feeds have any data in emoncms.org
I am seeing a lot of timeout errors
Aug 24 17:19:45 emonpi emoncms_sync[24017]: - Upload size: 1048576
Aug 24 17:19:55 emonpi emoncms_sync[24017]: - timeout error
I had to stop the sync upload as it was consuming 1MB every 5 minutes and killing my data plan. I only have 1 feed that is fully sync’d. the other 4 are not due to timeout errors.
I tried only syncing just 1 of the 4 remaining, and I still see a timeout error.
entered the above and received.
Aug 24 19:20:29 emonpi emoncms_sync[1861]: 08/24/2024 07:20:29 pm
Aug 24 19:20:30 emonpi emoncms_sync[1861]: - Nothing to upload (sleep 270s)
Aug 24 19:23:26 emonpi systemd[1]: Stopping emoncms_sync service...
Aug 24 19:23:26 emonpi systemd[1]: emoncms_sync.service: Succeeded.
Aug 24 19:23:26 emonpi systemd[1]: Stopped emoncms_sync service.
Aug 24 19:23:26 emonpi systemd[1]: emoncms_sync.service: Consumed 1.464s CPU time.
Aug 24 19:23:26 emonpi systemd[1]: Started emoncms_sync service.
Aug 24 19:23:27 emonpi emoncms_sync[6104]: 08/24/2024 07:23:27 pm
Aug 24 19:23:28 emonpi emoncms_sync[6104]: - Nothing to upload (sleep 92s)
Sanity checking deleting a 45MB feed on emoncms.org and having sync re-upload it here, this is what I would expect to see in the log, a 1MB upload every 2-3 seconds, took 109 seconds in total to upload 45 MB after which it then returns to the standard longer sleep.
I’ve made another minor change so that it prints how long the request took in seconds when it gives the timeout error. If you want to pull in the latest again that would be great!
Manual process is:
cd /opt/emoncms/modules/sync
git pull origin master
sudo systemctl restart emoncms_sync
I am syncing the remaining feeds one at a time so that I can monitor the upload. It looks like it uploads a maximum of 1MB at a time.
All are now fully sync’d. I changed the sync interval from 5 minutes to hourly and noticed that it said “nothing to upload (sleep 2999s)” and the time was 10 minutes past the hour. I assume this means that hourly sync happens on the hour versus one hour since the last sync - correct?
thanks for your assistance on this. This is going to be very helpful going forward, I just need to stage our conversion of all sites as we have a monthly data pool that I do not want to exceed in a single month by updating all sites in the same month.
@TrystanLea I just started the update on another site and noticed that the Full Update contains sync release 3.15, not the 3.16 version we used yesterday, and I did not see where to get 3.16. Are you planning to include 3.16 into the Full Update? Or should I continue with
Manual process is:
cd /opt/emoncms/modules/sync
git pull origin master
sudo systemctl restart emoncms_sync
