EmonTX missing data on CT4 input at regular intervals

Robert.Wall · 2019-04-29T12:54:41.973Z

I don’t think you’ve ever said which sketch you are using, so I assume it’s the standard default one. I also seem to remember that you’re using an ESP8266. But I do not know how the “quality” is calculated.

However, the standard sketch measures each active c.t. in order for 200 ms (in your case, I think that’s 1 & 4), parcels that up into a data packet that carries all 4 c.t. powers, voltage, 6 temperatures and the pulse count, and sends the whole lot to the ESP.

The output statements in your emonTx sketch look something like this:

  if (DEBUG==1) {
    Serial.print("ct1:"); Serial.print(emontx.power1);
    Serial.print(",ct2:"); Serial.print(emontx.power2);
    Serial.print(",ct3:"); Serial.print(emontx.power3);
    Serial.print(",ct4:"); Serial.print(emontx.power4);
    Serial.print(",vrms:"); Serial.print(emontx.Vrms);
    Serial.print(",pulse:"); Serial.print(emontx.pulseCount);
    if (DS18B20_STATUS==1){
      for(byte j=0;j<numSensors;j++){
        Serial.print(",t"); Serial.print(j); Serial.print(":");
        Serial.print(emontx.temp[j]);
      }
    }
    Serial.println();
    delay(50);
  }

My initial approach would be to take out all the unused variables and see what the effect is.

Some years ago, we had problems with the RFM modules when it was decided to send 6 temperature readings, because the long serial stream of 0 bits allowed the two clocks to drift out of sync and the checksum failed (and it’s resurfaced with 4 ‘long integer’ energy values and emonLibCM). There’s no checksum on the serial link between the emonTx and the ESP8266, but I wonder whether you have the same problem. If you have, taking out the unused variables should cure it. This might explain also your 12 kW spikes.

You might not need to amend your processing in emonCMS to cater for the change, if you’re sending the data labels (“ct1:”) as well as the data.

DickiePhitt · 2019-04-29T13:49:12.165Z

Thanks Robert,
I’ll look at doing that. Will need to dust down my USB to UART programmer stuff!
Looking at those output statements I’m wondering if removing the DS18B20 might improve matters pro tem. I assume, if removed, the status will revert to zero and remove a chunk of the transmission. Worth a try.

DickiePhitt · 2019-04-29T15:27:09.608Z

No, that made no difference

Regarding signal quality:
Checking historic data I notice that up until today the quality of inputs 1-3 have been at or around 100%, input 4 at around 50%.
Today inputs 1-3 quality has tumbled to as low as 34% for a time, input 4 again around half of this.
What might be causing this drastic variation? I can’t think of anything that has changed.

Gordon

Robert.Wall · 2019-04-29T15:57:30.555Z

Is it a network problem - either on your LAN or between your router and emoncms.org? (i.e, your ISP?) I don’t know how “quality” is arrived at; if it’s documented anywhere, I’ve no idea where to look. If it’s purely the number of received packets versus the expected number over the same period, then a transmission problem - or even a variable delay which causes packets to fall into the wrong ‘slot’ - would explain it.
[What I mean by that is, if you’re using PHPFINA at 10 s rate, the data is simply dropped into the “now” time slot as it arrives. So let’s say a packet arrives 9.9 s into the time slot consistently, all is well. but if one packet gets delayed by 150 ms, it misses its slot and goes into the next one. If the next packet isn’t delayed, it goes into its proper slot and overwrites the late one.]

Why ct4 should be different, I’ve no idea. I don’t know what goes on inside the ESP8266, nor how the data is sent over the Internet (i.e. how many packets, of what size). But there’s no law saying that the data needs to be sent in that order, so you could shuffle ct4 up into 2nd or 3rd place, if you wished.

You don’t need to send empty values for anything you’re not using, so you could comment out ct3, maybe vrms, and pulse.

DickiePhitt · 2019-04-29T18:00:43.118Z

If its a network issue I’m relatively relaxed as that can be checked easily enough when it happens again. The ‘quality’ dropping off does not appear to affect the ‘good’ inputs anyway. Its back at 100% now.
I’m ultimately looking to use all 4 inputs and vrms so pulse could go.
Its the absolute regularity of the dropping out/coming back that puzzles me. Looks to be the same period on as off - ie 50% - suspiciously the ‘quality’ is also around 50% of the good inputs.
For now the workaround is to use inputs 1-3 but I want to be sure I’ve not got a faulty unit so have asked openenergymonitor support for a view.

DickiePhitt · 2019-09-29T12:36:34.143Z

In case this happens to someone else I’ll detail the solution.
OpenEnergyMonitor kindly checked out the hardware they had sold me and pronounced it OK.
When I reinstalled I had the same problem but on a different input.
I then started a new node on Emoncms.org and moved each input over to it, using identical Process Lists to create new feeds. Have had no problems since.
Can only conclude that the problem was within the original node on the Emoncms.org site.

Robert.Wall · 2019-10-01T18:54:02.676Z

I must say I find that rather surprising. Is @TrystanLea aware of your conclusion?

graememk · 2020-09-17T20:46:55.327Z

Sorry to bump an old thread, but Im bumping into this issue too I think.

Have emonTX with wifi.

Installed and connected solar yesterday, so today was the first full day. and the solar feed (on CT2) seems to freeze updating.

Nighttime now and the power usage is tracking but the solar feed and the "house consumption, ie solar gen + CT1) is cutting out for 20- 30 minutes.

I added a second log to feed (a new one) and that has been running fine, It may not be the emontx all along.

Going to nuke it and set up again and see if it changes anything. Could it actually be a bug in emon cms? trying to narrow it down will be difficult, esp if i cant re create it

Edit: nuked it and set it up again it is running fine and steady after an hour. Now how on earth could i break a feed. Will keep an eye out for it in the future.

Robert.Wall · 2020-09-17T23:14:47.431Z

graememk:

Sorry to bump an old thread, but Im bumping into this issue too I think.

You’re going to be using a totally different sketch in your emonTx - assuming it’s a new one (bought since the end of last year).

graememk:

Have emonTX with wifi.

That’s the emonESP, or an emonTx with the ESP8266 module attached?

DickiePhitt · 2020-09-18T02:59:06.000Z

Just to say that since I changed node, as I described, I’ve had no further problems for nearly a year

graememk · 2020-09-18T15:08:32.326Z

Yep Emon tx with the emonesp.

It’s been perfect since I reset it too( the cms feed)

Robert.Wall · 2020-09-18T15:19:46.007Z

@TrystanLea
This sounds weird. Could it be a funny in emoncms.org, even though you looked a while ago EmonTX missing data on CT4 input at regular intervals - #7 by DickiePhitt

TrystanLea · 2020-09-18T15:38:54.933Z

I cant see anything that might cause it on emoncms.org. @graememk if you want to send me your account name in a private message, I can see if there is anything associated with your account if you like?

graememk · 2020-09-18T17:26:48.283Z

It was on a raspberry pi 2 that I have Emon cms setup on and doing the recording.

Only thing I might have changed was that I originally set the update interval to 5 seconds rather than the default 10.

I’ll setup another feed on an irrelevant input and see if I can replicate.

Today it has worked perfectly.

graememk · 2020-09-18T19:48:28.293Z

image1120×658 48.8 KB

I was able to replicate it, Yep it was setting a feed to 5 seconds. (this is on a emontx +wifi, currently just plugged in to the AC, havent got around to installing it yet)

Currently its an input over mqtt, but had the same behaviour when posting over the http to a emoncms.

I can post to emoncms.org if you like and see if I can replicate the bug there.

Dr_W · 2022-10-27T20:58:17.574Z

Looks like I’ve got the same issue. There is 20 min worth of missing data on a regular basis, and i only ever 50% data quality in the feed summary. All other feeds maintain ~100%.
This is the data from my newly added ‘solar diverted’ feed. This is CT on the live wire of a cable running from an mk2router to the immersion tank. Below is a night time data set (so no current to measure).

image1807×810 185 KB

Below is day time data set:

image1783×836 141 KB

The CT is connected to a TX, running a modified sketch from Robert Wall,
https://community.openenergymonitor.org/t/missing-temperature-feeds-on-emontx-and-emoncms-setup/17264/6?u=dr_w. The TX has 6 temperature sensors plugged in via a breakout board using an RJ45 cable.

Robert.Wall · 2022-10-27T21:16:04.353Z

Has your custom sketch run for the last 18 months, or have you only just put it to work?

Dr_W · 2022-10-27T21:37:03.847Z

Hi robert,
It’s been running since march 2021. No other problems that i’ve noticed, just this missing data when i added this new CT a few weeks ago.
I’m now wondering if i setup the feed wrong and didn’t set a time interval in the process list setup. I’ve added a new process duplicating the ‘solar divert’ feed but this time made sure i set a 10 s interval.

Is it possible to edit existing feeds to change the interval? Doesn’t seem to be an editable option in the feed setup.

Robert.Wall · 2022-10-27T22:57:25.337Z

Dr_W:

Is it possible to edit existing feeds to change the interval?

No. The PHPFINA feed works by storing the start time, and calculating the time for each data point from the interval and the record number. You’d need to rewrite the file interpolating or skipping values to “change” the interval of historic data.

Dr_W:

I’m now wondering if i setup the feed wrong and didn’t set a time interval in the process list setup.

You can hover your mouse over the name in Feeds to see what the properties are.

borpin · 2022-10-28T08:40:15.254Z

Dr_W:

just this missing data when i added this new CT a few weeks ago.

Check emonhub is set to log DEBUG and check in the window that you see the data coming in.

Then SSH in and open the log and look to see what emonhub is receiving as data.

Then check the MQTT Publish (if it is using MQTT) and check the data is published every time.

If the data is coming in OK and going out OK in emonhub, the issue will be emoncms.

Personally, I have seen the input MQTT service get overloaded if sent too many topics to process too quickly.

It is odd it is just that one feed, though.

[edit]
To follow the emonhub log use a command like this - the 2 -e create an OR in the grep.

Yours will probably look different!

pi@emonpi:~ $ tail -f /var/log/emonhub/emonhub.log | grep -e "emon/emonpi" -e "NEW FRAME : OK 5"
2022-10-28 09:54:07,437 DEBUG    RFM2Pi     24368 NEW FRAME : OK 5 140 0 0 0 140 0 240 91 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2022-10-28 09:54:07,534 DEBUG    MQTT       Publishing: emon/emonpi {"power1": 140, "power2": 0, "power1pluspower2": 140, "vrms": 235.36, "t1": 0, "t2": 0, "t3": 0, "t4": 0, "t5": 0, "t6": 0, "pulsecount": 0, "time": 1666947247.437131}
2022-10-28 09:54:12,492 DEBUG    RFM2Pi     24369 NEW FRAME : OK 5 139 0 0 0 139 0 1 92 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2022-10-28 09:54:12,666 DEBUG    MQTT       Publishing: emon/emonpi {"power1": 139, "power2": 0, "power1pluspower2": 139, "vrms": 235.53, "t1": 0, "t2": 0, "t3": 0, "t4": 0, "t5": 0, "t6": 0, "pulsecount": 0, "time": 1666947252.491709}
2022-10-28 09:54:17,443 DEBUG    RFM2Pi     24371 NEW FRAME : OK 5 139 0 0 0 139 0 244 91 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2022-10-28 09:54:17,715 DEBUG    MQTT       Publishing: emon/emonpi {"power1": 139, "power2": 0, "power1pluspower2": 139, "vrms": 235.4, "t1": 0, "t2": 0, "t3": 0, "t4": 0, "t5": 0, "t6": 0, "pulsecount": 0, "time": 1666947257.4434764}
2022-10-28 09:54:22,495 DEBUG    RFM2Pi     24372 NEW FRAME : OK 5 135 0 0 0 135 0 21 92 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2022-10-28 09:54:22,644 DEBUG    MQTT       Publishing: emon/emonpi {"power1": 135, "power2": 0, "power1pluspower2": 135, "vrms": 235.73000000000002, "t1": 0, "t2": 0, "t3": 0, "t4": 0, "t5": 0, "t6": 0, "pulsecount": 0, "time": 1666947262.495364}