Sync Module not Syncing

grod55 · 8 October 2024 00:18

I have one emonPi2 that is not syncing with emonCMS cloud for some reason. All my other emonPi2s are configured the same, but this one is lagging behind.

I am using the API key to authenticate and have the interval set to 5 minutes but keep seeing the message “No response from remote server”. I do have network connectivity as I have remotely logged into the emonPi2. I have also updated to the latest version of sync (v3.1.7) but continue to see Local ahead of Remote by xxxx points". the number of points is increasing by 30 every time the screen updates - which makes sense as my sync interval is every 5 minutes for 10 second data (6 points per minute).

Sync does show the message emoncms_sync service is running. Last upload xx seconds (or minutes) ago (1.0 MB)

Not sure what to do to correct this. Any suggestions?

thank you,

Gordon

TrystanLea · 8 October 2024 08:41

A post was split to a new topic: Home Assistant sync issue

TrystanLea · 8 October 2024 08:42

Hello @grod55 Can you PM me your emoncms.org account username for this system?

grod55 · 9 December 2024 19:09

@TrystanLea Is there a way to automatically restart sync or generate an alert if it starts to lag behind?

I have noticed that some sites tend to lag (likely with a weaker cellular connection) and I have to manually restart sync on the system information page to get it to catch up.

.

Today, one site was 37 hours behind and growing. It is almost as if once it lags to a certain level (unknown) it stops syncing and continually adds to the number of data points behind.

Thanks,

Gordon

TrystanLea · 18 December 2024 12:31

Apologies for the delay @grod55

I’m sorry that I still haven’t got to the bottom of why this is happening.

Yes probably the best interim approach is to add a cron entry to restart emoncms_sync e.g:

sudo crontab -e

and then add:

0 0 * * * systemctl restart emoncms_sync.service

borpin · 20 December 2024 07:28

Is the local clock correct?

@TrystanLea - is there a flag that could be created when the 2 sites get out of sync that could be raised as an error/warning?

grod55 · 20 December 2024 19:38

Brian, great question. When I log in to the emonPi and enter timedatectl, it shows

pi@emonpi:~ $ timedatectl
               Local time: Fri 2024-12-20 19:29:20 GMT
           Universal time: Fri 2024-12-20 19:29:20 UTC
                 RTC time: n/a
                Time zone: Europe/London (GMT, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

When I checked the account on the local machine, it, too, showed Europe/London, so I changed it to America/New_York.

In checking another of my installations, the local machine account showed correctly as America/New_York, but the emonPi shows:

pi@emonpi:~ $ timedatectl
               Local time: Fri 2024-12-20 19:29:20 GMT
           Universal time: Fri 2024-12-20 19:29:20 UTC
                 RTC time: n/a
                Time zone: Europe/London (GMT, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Should the Raspberry Pi always be in the Europe/London timezone, and should only the emoncms account be changed to the local time zone?

Thanks,

Gordon

TrystanLea · 21 December 2024 23:00

I don’t think this issue will be timezone related, it likely to do with the mechanism I’ve implemented to cache the position that the local machine is in, in terms of the sync upload. In order to save bandwidth it doesn’t poll the remote server to check how much data it needs to upload at every upload point and something is going wrong there I think. I need to add some better logging to pin it down.

@grod55 it’s best not to change the timezones on the emonpi’s. You only need to set the timezone in the Emoncms My Account page but that will be unrelated to this particular issue.

borpin · 23 December 2024 09:07

As Trystan said, it isn’t timezone related, it is the actual time - if the 2 machines were out by too much the sync might complain. Changing the timezone doesn’t change the actual time

Timezone should always be what your timezone actually is as it determines what you see as a translation of the underlying timestamp (which is stored as UTC).

grod55 · 5 January 2025 20:47

@TrystanLea

I am still seeing the lag issue sporadically in random sites and manually restarting sync resolves the issue. (I haven’t implemented your suggested cron entry yet).

One thing I did notice in my emonhub configuration is I still have the emoncmsorg settings in place (API key is x’d out in the screen grab below, but in emonhub it is the actual API key).

Could this be causing issues? I thought I had commented this out previously but it looks like I did not make the change or forgot to save it.

Thanks,

Gordon

 [[emoncmsorg]]
       # Type = EmonHubEmoncmsHTTPInterfacer
        [[[init_settings]]]
        [[[runtimesettings]]]
            pubchannels = ToRFM12,
            subchannels = ToEmonCMS,
            url = http://emoncms.org
            apikey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
            senddata = 1    # Enable sending data to Emoncms.org
            sendnames = 1    # Send full input names (compression will be automatically enabled)
            interval = 30    # Bulk send interval to Emoncms.org in seconds
    ```

TrystanLea · 7 January 2025 19:46

Hello @grod55 yes this could be a factor, if this is also writing to the feeds that could mess up the expected position in the sync. It’s probably not the only issue as Im sure I’ve seen this on my system and I dont have both running in parallel. Mine has been uploading now for sometime though without the issue.

grod55 · 7 January 2025 23:57

Thanks for the reply. I will comment this section out and let you know if it minimizes the issue.

Regards,

Gordon

grod55 · 9 January 2025 17:41

Hi @TrystanLea I had another site stop syncing today and I noticed a couple of things.

1 it was just over 15,000 datapoints behind (almost 2 days)
2 sync service had stopped - this is the first time that I have seen this message

3 for some reason the upload size changed from 100kb to 1MB
This is the site that was always having the sync issue until you made the change so that the file upload size was selectable. I changed it back to 100kb and all seems to work again. I restarted sync again to see if this caused the upload size to go to 1MB, but it remained at 100kb - so not sure what caused the change to 1MB

Lastly, in looking at the emoncms log, I do see a lot of format errors. I’m not sure it this is related or not.

Not sure about next steps.

Thanks,

Gordon

<details><summary>LAST ENTRIES ON THE LOG FILE</summary><br />
2025-01-07 18:58:49.320|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 18:58:59.418|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 18:59:09.-491|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 18:59:19.-387|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 18:59:29.-289|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 18:59:39.-197|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 18:59:49.-83|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:00:00.000|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:00:10.096|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:00:20.185|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:00:30.294|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:00:40.393|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:00:50.484|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:01:00.-420|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:01:10.-336|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:01:20.-235|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:01:30.-140|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:01:40.-41|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:01:51.041|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:02:01.140|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:02:11.257|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:02:31.460|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-07 19:02:41.-443|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-08 16:38:22.127|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1<br />
2025-01-08 16:38:32.240|ERROR|input_controller.php|{"success": false, "message": "Format error, json value is not numeric"} for User: 1</details><br />

TrystanLea · 14 January 2025 13:52

Thanks @grod55 noted re changing setting, not sure why that would have happened, lets see if that happens again.

The emoncms log errors loo unconnected as this is an input stage error. Do you have a device or script configured to post data to the input HTTP api? The standard emonHub route is typically via MQTT so these errors should be related to something else?

grod55 · 14 January 2025 14:39

@TrystanLea thanks for the response. Yes, I have a script that sends WiFi signal strength to emonCMS on the emonPi.
I need to look to see if this is causing my log errors.
Thanks,
Gordon

grod55 · 23 January 2025 14:42

@TrystanLea unfortunately, I am still encountering the issue of the local machine being ahead of emonCMS (see below).

So, I plan to add the cron job as you suggested a few weeks back to the site that experiences this issue most often and see if it addresses the issue.

TrystanLea:

grod55:

@TrystanLea Is there a way to automatically restart sync or generate an alert if it starts to lag behind?

Yes probably the best interim approach is to add a cron entry to restart emoncms_sync e.g:
sudo crontab -e
and then add:
0 0 * * * systemctl restart emoncms_sync.service

TrystanLea · 23 January 2025 16:22

Hello @grod55 I meant to get in contact. I’ve added an integrated feature to the sync module that will automatically restart the script if it fails to sync data after more than 6x the upload interval, so 30 mins if the upload interval is 5 mins.

It’s in the latest master branch, could you try updating and restarting the emoncms_sync service?

grod55 · 27 January 2025 23:00

Hi @TrystanLea, I just saw this message from you and had added the cron entry before seeing this.

Do I need to remove the cron entry and then do this update? I want to make sure I am not causing conflicting actions.

Thanks,

Gordon

TrystanLea · 28 January 2025 11:36

It’s not a requirement to remove the cron entry, but it may be harder to know if this change is effective with the cron entry in place…

grod55 · 5 February 2025 23:30

@TrystanLea

Hi Trystan,
I removed the cron entry and performed the sync update last week. Everything worked well for a few days, and then this morning, two sites ran behind again. The AVONFIT site is the one that most often gets out of sync.

I confirmed that both sites have applied the latest sync update.

The AVONFIT site uses wifi to connect the emonPi to a cellular router and is one of my slower sites. So I am wondering if this is contributing to the sync falling behind, but whenever I manually restart the sync module, it catches up.

The RSRP on the 4G router is not that great, nor is the RSRQ which could contribute to packet loss. I am going to try a high-gain antenna once the snow clears and we can get on the roof.

The emonPi wifi signal strength is OK

Now I am wondering if I did perform a restart of the sync module after updating it. I thought I did, but I can’t say for certain. If I didn’t perform that step, I think this means the old sync version was still running without the auto restart you added.

I will continue to monitor it for another week as I won’t be able to change out the antennas before then.

Thanks,

Gordon