How to automate the feed sync between local installation and emoncms

AlessandroVanzulli · 2023-11-07T22:22:15.801Z

Hi,

I’m using the EmonCms 11.3.22 and the Sync version 2.1.4 and I would like to automatically sync a feed between the local instance and the emoncms public instance. Right now, I’m using the Emoncms Sync functionality but I have to regularly press the upload button to sync the remote feed with the local one. The local one is getting created by a new python service getting data from melcloud and posting locally via the API.

I tried to found some documentation on how to sync … and I was not able to find any good article on how to setup such automation.
Happy to have a look into the code if required.

Thanks a lot for the help.

Cheers,

Ale

Dan_Nichols · 2023-11-12T19:53:04.409Z

AlessandroVanzulli:

The local one is getting created by a new python service getting data from melcloud and posting locally via the API.

Hi @AlessandroVanzulli, I have the same question. I was suprised that this does not appear to be documented somewhere.

@TrystanLea is there a feature in emoncms to automatically sync feeds between instances or do we need to write a script or otherwise schedule somehow?

Zarch · 2023-11-12T21:30:01.031Z

I just stuck this in cron on my local instance so I do daily uploads of the feeds i’ve preconfigured in sync feeds.

#01:01 am - sync local feeds to remote
1 1 * * * php /opt/emoncms/modules/sync/sync_upload.php 2>&1

TrystanLea · 2023-11-13T12:39:36.475Z

Dan_Nichols:

is there a feature in emoncms to automatically sync feeds between instances or do we need to write a script or otherwise schedule somehow?

As @Zarch mentioned. Yes this needs documenting!

Dan_Nichols · 2023-11-13T17:40:12.647Z

Zarch:

sync local feeds to remote

Thank you @Zarch I have now added a cron job to sync every 5mins - simple when you know how!

#Every five minutes sync local feeds to remote
*/5 * * * * php /opt/emoncms/modules/sync/sync_upload.php 2>&1

borpin · 2023-11-13T22:04:16.603Z

TrystanLea:

As @Zarch mentioned. Yes this needs documenting!

@TrystanLea - Perhaps, a systemd service could be added to the codebase to sync the data and that service could be enabled from the UI?

TrystanLea · 2023-11-14T10:31:36.856Z

Yes I think something like that would be nice.

The sync module was not really initially designed for use for regular data uploads. It was more designed as a tool for moving data from a local system to a remote system or vice versa or/and backups.

But it would be nice to make it more of a continuous replication option. The upload of the individual feed data is quite efficient as it’s done via binary, but the preparation for the upload and the way a request is made for each feed upload is less efficient, Ideally some way of combining all the requests in a single packaged upload would be good…

borpin · 2023-11-14T10:33:50.107Z

If adding flexibility to this, perhaps being able to define different targets with different feeds?

I’d not thought about using it to duplicate between systems. I have a partially finished migration stopped as the syncing of the data via rsync takes so long

TrystanLea · 2023-11-14T10:40:34.552Z

borpin:

as the syncing of the data via rsync takes so long

Yes that was part of the reason for the sync module.

One key downside with the current implementation is that it only uploads or downloads data that is new e.i after last end time of the feed that is behind local or remote. It does not upload edited datapoints if they are made before the end time.

borpin · 2023-11-14T10:43:30.049Z

Ah OK. If it is efficient, then a “just copy everything” option would be great.

If you have done a backup/restore (by any means), the inputs on the new system are running, then run a sync from the old system that should fill in all the gaps during migration.

TrystanLea · 2024-08-21T17:16:46.963Z

TrystanLea:

But it would be nice to make it more of a continuous replication option. The upload of the individual feed data is quite efficient as it’s done via binary, but the preparation for the upload and the way a request is made for each feed upload is less efficient, Ideally some way of combining all the requests in a single packaged upload would be good…

The new version of the emoncms sync module is now available that uploads data in a much more efficient way and also removes the need for the crontab entry discussed above. I would recommend updating to use this new version, please see release post here: Emoncms Sync module v3.x.x