Emoncms.org feeds not updating for some users today 30th September 3pm onwards

Emoncms Emoncms.org
1

It looks like there is a connectivity issue for some users today posting data to emoncms.org from I think mostly emonpi/emonbase systems starting 3pm onwards. It’s affecting about 10% of emoncms.org feeds.

This may be related to a now expired letsencrypt certificate Help thread for DST Root CA X3 expiration (September 2021) - Help - Let's Encrypt Community Support.

If you have an affected system and can post any emonhub logs that may show associated errors that would be really useful. I will hopefully get a clearer idea of the solution shortly.

It may require running: sudo apt-get update && sudo apt-get upgrade to pull in latest certificates.

2

Looks very likely to be this: Naked Security – Sophos News

And that’s just as well, because the DST Root CA X3 certificate that started it all will expire shortly after 3pm UK time on Thursday 30 September 2021 [2021-09-30T14:01:15Z].

3

I’ve been given some emonhub logs from an affected system, errors look like:

2021-09-30 19:30:32,497 WARNING emoncmsorg emoncmsorg couldn’t send to server, URLError: [Errno 113] No route to host

and

2021-09-30 19:33:56,452 WARNING emoncmsorg emoncmsorg couldn’t send to server, URLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

as expected.

The image the user is having problems with is emonSD-26Oct17, but this issue may well affect newer images.

The quick fix

The quick though **not secure** fix is to disable https in emonhub.conf:

[[emoncmsorg]]
   Type = EmonHubEmoncmsHTTPInterfacer
   [[[init_settings]]]
   [[[runtimesettings]]]
      pubchannels = ToRFM12,
      subchannels = ToEmonCMS,
      url = http://emoncms.org

and then at your earliest convenience upgrade to a new SD card running the latest image and importing any existing data using the emoncms import tool: Import / Backup / Restore / Update — OpenEnergyMonitor 0.0.1 documentation

Please note: Disabling https does mean your data including apikey will be passed to emoncms.org in plain text without encryption and this could compromise your emoncms.org account. We do recommend upgrading to to the latest image as soon as possible to keep your system secure.

Im still looking into other options for updating the cert in place.

4

Another user reporting this issue has emonSD-07Nov16.

5

You should be able to get things moving again with this line (run as root, of course):

sed -i 's/mozilla\/DST_Root_CA_X3.crt/!mozilla\/DST_Root_CA_X3.crt/g'  /etc/ca-certificates.conf && update-ca-certificates

It removes the now-expired root certificate from the CA store and allows openssl to use the newer trust chain. I believe this will work on jessie, but I can’t test it… Details here: Old Let’s Encrypt Root Certificate Expiration and OpenSSL 1.0.2 - OpenSSL Blog and the linked post on the LetsEncrypt discourse.

1 Like
6

Thanks @bwduncan @glyn.hudson is going to try that now, I got the impression that this needed an OpenSSL update as well and the old image I tried to fix for someone just now had difficulty with package lists and was out of LTS anyway. Would be great though if that works.

7

That’s possible. I believe the solutions are a bit less hacky with openssl >1.0.2k.

1 Like
8

Trystan

Error messages on my emonhub are:

2021-10-01 05:07:12,910 WARNING  emoncmsorg emoncmsorg couldn't send to server, URLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
2021-10-01 05:07:12,911 WARNING  emoncmsorg send failure: wanted 'ok' but got ''

I’ve tried your suggested fix to disable https but is still not updating.

I’ve got a newer image on SD card so I’ll swap it over when I can

9

I’ve just experienced a similar issue with one of the systems I maintain.

I found an alternative approach on the server side that’s given some breathing room, that was to change the ACME client to use a ZeroSSL certificate rather than Let’s Encrypt. ZeroSSL doesn’t rely on the same root certificates and so continues to work with older systems.

I realise that this is primarily a workaround for systems that should be updated, however it does give users time to get systems updated whilst everything continues to work. Also, I prefer Let’s Encrypt’s business model so will be returning to their certs as soon as possible.

2 Likes
10

Thanks, that’s very useful. I tried this on our emonSD-07Nov16 emonSD which is based on Debian 8 (Jessie). Unfortunately, it didn’t work. The image has got OpenSSL 1.0.1k 8 Jan 2015.

As @TrystanLea mentioned above, efforts to update packages are fraught with ‘unable to fetch packages’ error because the image is so old.

Give how old this image is, I think the sensible solution is to upgrade to the latest image.

11

That’s strange, did the emonhub log errors change by any chance after changing the url?

12

Trystan - I didn’t look - and I’ve already swapped over the emonSD to a new image now. Having issues with my EmonTXs with the new image - I’ve posted a question on this.

1 Like
13

Thanks @DelBert would you be able to share any info on the process that you took to do that. I assume using: GitHub - acmesh-official/acme.sh: A pure Unix shell script implementing ACME client protocol ?

14

Thanks for the prompt email Trystan. I’m in the strange position of having two systems running 2000km apart, both writing to emoncms.org. Unfortunately the remote one has been running low-write 9.8.0 2017.02.01 for over four years now with zero problems, but is now failing. It appears to have stopped about 10:00 UTC yesterday 30 Nov. A WeeWx weather station on the same site using the Emoncms add-on is working fine. With no physical access I may have to disable https as a workaround until I next get there!

15

Sorry forgot to add some lines from log file:

2021-10-01 13:30:32,257 WARNING emoncmsorg emoncmsorg couldn’t send to server, URLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
2021-10-01 13:31:02,210 WARNING emoncmsorg emoncmsorg couldn’t send to server, URLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

16

We’re using Windows based servers and the Win-Acme client, it was just a case of running that with a different server URL. I’m guessing that won’t help you much as it looks like your site is Ubuntu/Apache based.

There is a list of Acme clients compatible with ZeroSSL here: ACME Automation - ZeroSSL. acme.sh is one of those clients and it looks like they’ve been defaulting to ZeroSSL’s server on versions released since August 2021. For other Acme clients, once you know the setting, just point it at the server: https://acme.zerossl.com/v2/DV90

17

Thanks @derekb

I feel your pain with sorting these issues out at such a distance!

18

Thanks @DelBert just tried that one and it’s just timing out unfortunately, they must be busy with everyone trying to switch certs!! Will try again soon, testing on another server first to be sure I have the process right!

19

Thanks @bwduncan, this worked for me.

I’m not sure which emon image I originally installed but it’s running Debian 8 (Jessie) with openssl 1.0.1t

1 Like
20

I’d be surprised if their servers were overloaded but I guess it could have happened.

It might be worth forcing a re-issue of the Let’s Encrypt certificate with whatever tool you currently use to maintain the certificate. Since the root CA certificate expired, any new certs published aren’t including this path, which should mean the client will find the valid path instead, similar to @bwduncan solution, but from the server side instead.

You can see the current ‘dead’ path by expanding the ‘Certification Paths’ section here: SSL Server Test: emoncms.org (Powered by Qualys SSL Labs)