Missing data in local database

Roberto_Zanasi · 27 September 2021 20:37

This is the graph created by the data viewer of my EmonPi. Data are transmitted from an EmonTX via the wifi module. It is evident that there is a periodic packet loss.

This is the graph created instead by the data viewer at emoncms.org, where my EmonPi saves all the data. No loss at all:

So it seems that this is not a network issue: I can access the EmonTX even during the “blackouts”, and all the data is transmitted to the remote site. What could I do to fix the problem? What tests could I perform?

Robert.Wall · 27 September 2021 21:37

How is the data from the emonTx reaching your emonPi? Is it using the 433 MHz ISM band radio and appearing in emonHub?

And how is the data from the emonTx going to emoncms.org? Does the ESP8266 Wi-Fi module send it via your router directly to emoncms.org?

If the answer to both questions is “yes”, then because I see a very regular pattern in the breaks in the data reaching your emonPi, I think the problem lies in the radio link between the emonTx and the emonPi - and I think there might be a local source of interference that is blocking the signal.

If the answer to both questions is “no”, then the problem appears to be internal to the workings of the emonCMS inside your emonPi, and I do not know enough about that to help you. I do not think this can be the case, because I am not aware of a mechanism in the ESP8266 to send the data to two different servers, neither am I aware of a mechanism in the emonPi to take data received via your LAN and send it onwards to emoncms.org (but I am open to correction here).

Is there any useful information in the emonHub log?

Roberto_Zanasi · 28 September 2021 14:29

I have an emonESP wifi interface on my emonTX, connected to my local network. So, data is directly sent from the emonTX to emoncms.org. Besides, data should reach my emonPI via the same local network (the 433 MHz ISM band radio is not able to reach the emonPI directly).

So the answers are no and yes

I should add that data sent from the emonPI reaches correctly emoncms.org, so the problem is located in the link emonTX — emonPI.

In the emonHub log I can see messages like the following ones:

2021-09-28 14:26:26,017 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 27 70 45 3 80 235 183 62 249 188 3 250 223 174 247 199 142 58 44 223 100 (-85)
2021-09-28 14:26:26,129 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 30 71 253 191 112 125 95 22 67 165 63 127 236 231 239 127 127 179 207 150 90 (-85)
2021-09-28 14:26:28,263 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 10 93 61 252 125 168 132 239 237 255 55 200 109 100 125 254 27 24 192 25 241 (-85)
2021-09-28 14:26:29,021 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 21 121 247 30 209 190 9 214 205 81 186 245 128 129 132 203 122 118 30 214 101 (-84)
2021-09-28 14:26:31,015 DEBUG    RFM2Pi     496038 NEW FRAME : OK 5 196 4 0 0 196 4 194 97 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2021-09-28 14:26:31,019 DEBUG    RFM2Pi     496038 Timestamp : 1632839191.02
2021-09-28 14:26:31,020 DEBUG    RFM2Pi     496038 From Node : 5
2021-09-28 14:26:31,021 DEBUG    RFM2Pi     496038    Values : [1138.26, 0, 1138.26, 233.49258, 25.8, 0, 0, 0, 0, 0, 0]
2021-09-28 14:26:31,022 DEBUG    RFM2Pi     496038 Sent to channel(start)' : ToEmonCMS
2021-09-28 14:26:31,022 DEBUG    RFM2Pi     496038 Sent to channel(end)' : ToEmonCMS
2021-09-28 14:26:31,131 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 25 151 253 68 81 249 237 25 145 97 203 188 49 45 222 126 212 27 110 18 253 (-85)
2021-09-28 14:26:31,213 DEBUG    MQTT       Publishing: emon/emonpi/power1 1138.26
2021-09-28 14:26:31,215 DEBUG    MQTT       Publishing: emon/emonpi/power2 0
2021-09-28 14:26:31,217 DEBUG    MQTT       Publishing: emon/emonpi/power1pluspower2 1138.26
2021-09-28 14:26:31,219 DEBUG    MQTT       Publishing: emon/emonpi/vrms 233.49258
2021-09-28 14:26:31,221 DEBUG    MQTT       Publishing: emon/emonpi/t1 25.8
2021-09-28 14:26:31,223 DEBUG    MQTT       Publishing: emon/emonpi/t2 0
2021-09-28 14:26:31,224 DEBUG    MQTT       Publishing: emon/emonpi/t3 0
2021-09-28 14:26:31,226 DEBUG    MQTT       Publishing: emon/emonpi/t4 0
2021-09-28 14:26:31,228 DEBUG    MQTT       Publishing: emon/emonpi/t5 0
2021-09-28 14:26:31,229 DEBUG    MQTT       Publishing: emon/emonpi/t6 0
2021-09-28 14:26:31,231 DEBUG    MQTT       Publishing: emon/emonpi/pulsecount 0
2021-09-28 14:26:31,233 INFO     MQTT       Publishing: emonhub/rx/5/values 1138.26,0,1138.26,233.49258,25.8,0,0,0,0,0,0
2021-09-28 14:26:35,883 DEBUG    RFM2Pi     496039 NEW FRAME : OK 5 143 4 0 0 143 4 190 97 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2021-09-28 14:26:35,887 DEBUG    RFM2Pi     496039 Timestamp : 1632839195.88
2021-09-28 14:26:35,888 DEBUG    RFM2Pi     496039 From Node : 5
2021-09-28 14:26:35,889 DEBUG    RFM2Pi     496039    Values : [1088.8110000000001, 0, 1088.8110000000001, 233.45525999999998, 25.8, 0, 0, 0, 0, 0, 0]
2021-09-28 14:26:35,890 DEBUG    RFM2Pi     496039 Sent to channel(start)' : ToEmonCMS
2021-09-28 14:26:35,890 DEBUG    RFM2Pi     496039 Sent to channel(end)' : ToEmonCMS
2021-09-28 14:26:35,978 DEBUG    MQTT       Publishing: emon/emonpi/power1 1088.811
2021-09-28 14:26:35,980 DEBUG    MQTT       Publishing: emon/emonpi/power2 0
2021-09-28 14:26:35,982 DEBUG    MQTT       Publishing: emon/emonpi/power1pluspower2 1088.811
2021-09-28 14:26:35,984 DEBUG    MQTT       Publishing: emon/emonpi/vrms 233.45526
2021-09-28 14:26:35,985 DEBUG    MQTT       Publishing: emon/emonpi/t1 25.8
2021-09-28 14:26:35,987 DEBUG    MQTT       Publishing: emon/emonpi/t2 0
2021-09-28 14:26:35,989 DEBUG    MQTT       Publishing: emon/emonpi/t3 0
2021-09-28 14:26:35,990 DEBUG    MQTT       Publishing: emon/emonpi/t4 0
2021-09-28 14:26:35,992 DEBUG    MQTT       Publishing: emon/emonpi/t5 0
2021-09-28 14:26:35,994 DEBUG    MQTT       Publishing: emon/emonpi/t6 0
2021-09-28 14:26:35,996 DEBUG    MQTT       Publishing: emon/emonpi/pulsecount 0
2021-09-28 14:26:35,998 INFO     MQTT       Publishing: emonhub/rx/5/values 1088.811,0,1088.811,233.45526,25.8,0,0,0,0,0,0
2021-09-28 14:26:36,840 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 31 125 153 124 155 107 135 203 185 233 62 117 237 222 10 251 128 241 31 137 226 (-85)

The “unreliable content” lines are suspect…

Robert.Wall · 28 September 2021 15:41

Read again what I wrote above. The ESP8266 sends the data to ONE specific address, which appears to be emoncms.org. As far as I’m aware there is no mechanism in the ESP8266 software to send it to both the emoncms.org URL AND the emonPi’s URL. So the fact that both are on the same LAN is not relevant as you have it set up at present.

So I believe you are wrong, the emonPi is receiving its data via the radio link. The log shows that there is some received signal at -85 dBm, but the example you’ve picked looks to me more like random noise. If you download the log to your computer and look at the times where data is arriving successfully, you should see messages every 10 s (roughly) that look like this:

 NEW FRAME : OK 15 11 0 0 0 38 0 22 0 22 0 0 0 0 0 157 6 0 0 171 6 0 0 151 6 0 0 0 0 0 0 48 117 48 117 48 117 1 0 0 0 (-75)

“OK 15” identifies the data as coming from the emonTx, “(-75)” on the end is the signal strength. -75 is rather low but OK, anything below -80 is marginal and would explain the drop-outs. “unreliable content” means the checksum on the data has failed, so that message is discarded.

I suggest you look at improving the radio link. One cheap and simple way is to provide the antenna with a “ground plane” - there’s a Topic here to read: Improving RF signal strength emonTH emonTX
You can have a ground plane at both ends of the link if you need to.

If that is impractical, then we need some help from a networking expert to advise on how to either get the ESP8266 to send the data to two destinations, or how to get your local emonCMS to forward the data to emoncms.org

It’s easy to send data received via emonHub onwards to emoncms.org, but the data sent by the ESP8266 doesn’t come by that route, it bypasses emonHub completely.

Roberto_Zanasi · 28 September 2021 16:02

I wrote that the ESP8266 sends data to my emonPI because of this page (visible in my emonPI input view):

After your message I looked at the emonESP configuration page. This refers to what you wrote:

emonESP posts data to emoncms.org.

This, however:

should mean that data from ESP8266 is sent to the emonPI via MQTT (correct me if I’m wrong, please).

If I’m correct, feeds are created from the MQTT input, and not from the radio one. Which sometimes works, but non with little blackouts like the ones I’m experiencing. For example, it worked more than 2 days ago for the last time:

Robert.Wall · 28 September 2021 16:11

I don’t understand MQTT - I’ve never needed to find out its relevance. We need somebody who knows to help. @TrystanLea created the system, so hopefully he’ll see this and advise you.

TrystanLea · 1 October 2021 12:20

Hello @Roberto_Zanasi it looks like you have both the HTTP upload to emoncms.org and the local MQTT method to your emonPi configured correctly. I cant immediately see why the data is reaching emoncms.org fine but not your local emonPi. Do you have configuration on your local network that might cause any issues? repeaters or some kind of partitioning of multiple routers? You could try entering the IP address of your emonpi directly in the MQTT server field just in case theres some hostname resolution issue…

Roberto_Zanasi · 1 October 2021 21:39

@TrystanLea Hi, thank you for your advice. I’ve changed the MQTT server, but I still see the “blackouts”. What else could I try?