Missing Data - 66% Quality

Emoncms
#1

Hi Guys,

I have an emonTx posting data via emonBase to my web server but have a large number of missing data points as shown in the graph below. The quality ratio is at 66% with the data feeds set at 10 seconds, is this behavior normal?

2017-04-27 12.11.04 pm.png1425×702 101 KB

The emonBase is connected via wifi but ethernet connection gives the same results. The quality figure is exactly the same on the emonBase when read locally.

Any ideas what could be causing the missing data?
As a note, I can not seem to create a negative reading from the CT’s, even if I turn them around. This means that night time standby consumption from my solar inverter is recorded as generation instead of consumption. Why is this?

Look forward to any help. Thanks, Sam.

Emoncms missing data
#2

I can’t help you with the networking problem, someone better informed will need to advise you there.

Are you reading real power, i.e., do you have an a.c. adapter and was it connected when you started your emonTx? Without that, it can only estimate apparent power, which is directionless.

Also, do you have an emonBase or an emonPi? (You mentioned both.) The emonPi likewise needs an a.c. adapter to measure real power and the direction of power flow, without it, it likewise can only estimate apparent power.

#3

Hi Robert, thanks for your swift reply.

Arr yes, I am using batteries so no a.c. adapter present. That will explain the apparent power readings, thanks for clearing that up for me.

I meant emonBase, I will edit my original post to save confusion.

That is my problem, I do not think this is a networking problem as reading the emonBase locally produces the same missing data problem.

Cheers, Sam.

#4

Here is the log:

2017-04-27 15:51:40,241 DEBUG    RFM2Pi     8516 NEW FRAME : OK 8 243 3 0 0 0 0 163 2 234 1 184 11 184     11 184 11 184 11 184 11 184 11 0 0 0 0 (-60)
2017-04-27 15:51:40,244 DEBUG    RFM2Pi     8516 Timestamp : 1493308300.24
2017-04-27 15:51:40,245 DEBUG    RFM2Pi     8516 From Node : 8
2017-04-27 15:51:40,246 DEBUG    RFM2Pi     8516    Values : [1011, 0, 0, 675, 4.9, 300, 300, 300, 300, 300, 300, 0]
2017-04-27 15:51:40,246 DEBUG    RFM2Pi     8516      RSSI : -60
2017-04-27 15:51:40,247 DEBUG    RFM2Pi     8516 Sent to channel(start)' : ToEmonCMS
2017-04-27 15:51:40,248 INFO     RFM2Pi     Publishing: emon/emontx3/power1 1011
2017-04-27 15:51:40,249 INFO     RFM2Pi     Publishing: emon/emontx3/power2 0
2017-04-27 15:51:40,250 INFO     RFM2Pi     Publishing: emon/emontx3/power3 0
2017-04-27 15:51:40,251 INFO     RFM2Pi     Publishing: emon/emontx3/power4 675
2017-04-27 15:51:40,253 INFO     RFM2Pi     Publishing: emon/emontx3/vrms 4.9
2017-04-27 15:51:40,254 INFO     RFM2Pi     Publishing: emon/emontx3/temp1 300
2017-04-27 15:51:40,255 INFO     RFM2Pi     Publishing: emon/emontx3/temp2 300
2017-04-27 15:51:40,257 INFO     RFM2Pi     Publishing: emon/emontx3/temp3 300
2017-04-27 15:51:40,258 INFO     RFM2Pi     Publishing: emon/emontx3/temp4 300
2017-04-27 15:51:40,259 INFO     RFM2Pi     Publishing: emon/emontx3/temp5 300
2017-04-27 15:51:40,260 INFO     RFM2Pi     Publishing: emon/emontx3/temp6 300
2017-04-27 15:51:40,262 INFO     RFM2Pi     Publishing: emon/emontx3/pulse 0
2017-04-27 15:51:40,263 INFO     RFM2Pi     Publishing: emon/emontx3/rssi -60
2017-04-27 15:51:40,264 INFO     RFM2Pi     Publishing: emonhub/rx/8/values 1011,0,0,675,4.9,300,300,300,300,300,300,0
2017-04-27 15:51:40,266 INFO     RFM2Pi     Publishing: emonhub/rx/8/rssi -60
2017-04-27 15:51:40,267 DEBUG    RFM2Pi     8516 adding frame to buffer => [1493308300.241262, 8, 1011, 0, 0, 675, 4.9, 300, 300, 300, 300, 300, 300, 0, -60]
2017-04-27 15:51:40,268 DEBUG    RFM2Pi     8516 Sent to channel(end)' : ToEmonCMS
2017-04-27 15:51:41,686 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 4 140 58 199 179 (-98)
2017-04-27 15:51:42,803 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 17 50 112 60 159 132 241 25 238 101 225 8 168 247 85 253 71 73 229 190 183 (-91)
2017-04-27 15:51:44,964 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 24 7 67 211 119 156 223 207 172 79 79 146 12 4 74 49 87 249 85 16 197 (-100)
2017-04-27 15:51:45,271 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 8 169 66 7 202 113 227 27 80 116 84 72 15 173 33 142 49 64 189 189 97 (-100)
2017-04-27 15:51:46,293 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 2 236 192 14 188 6 145 163 160 5 116 2 4 160 24 253 62 52 144 104 204 (-91)
2017-04-27 15:51:47,416 INFO     emoncmsorg sending: https://sample.co.uk/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1493308300.241262,8,1011,0,0,675,4.9,300,300,300,300,300,300,0,-60]]&sentat=1493308307
2017-04-27 15:51:48,249 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 0 222 20 132 109 12 233 8 21 97 8 121 4 17 7 98 208 38 93 81 175 (-100)
2017-04-27 15:51:48,702 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://sample.co.uk/emoncms
#5

I have disabled sendstatus as I am not using Emoncms.org, is that correct?

       sendstatus = 0                  # Enable sending WAN IP to Emoncms.org MyIP > https://emoncms.org/myip/list

Thanks, Sam.

#6

So it’s the r.f. link between the emonTx and the emonBase then? What is the distance between the two, and what sort of r.f. path have you got - how many walls, their construction, etc.
If you look at the serial data coming in, the signal strength (the last number in brackets) -60 is OK, but the other stuff marked ‘unreliable’ with signal strength down in the -90s looks to be pickup from something else a distance away operating on the same frequency. As there’s only 8 s worth there, and only one valid transmission, it’s hard to say what’s going on.

If you want to send to your server, I think all you need to do is replace the emoncms.org URL with yours (http://192.168…?) in the interfacer (or copy the interfacer and comment out the original emoncmsorg one).

#7

Hi Robert,

Great, thanks, that is much clearer now. There is a single wall between the two but I placing tried having both units in the same room and even increasing the distance but the signal strength does not improve. I guess, as you say, there is likely to be something operating on the same frequency. Is it possible to hard wire the emonTx to the emonBase so I don’t have to worry about the r.f. link?

Thanks for your help with this, Sam.

#8

You need to look at a bit more of the log to get a better idea of what is going on, but a single brick wall should not take too much out. Your best bet would be to reduce the receiver sensitivity. That would take your wanted signal down, but if you get it right, it will push the interfering signal below the detection threshold. I’ve never looked hard at the RFM69CW data sheet with regard to the receiver part, but from a quick skim read, I think it would be possible to do that in software. There are gain, AGC and threshold settings available. What would need to be done in the RFM69Pi sketch, I don’t know.

Indeed it is - over a short distance.
Hardware-wise: EmonTx V3.4 - OpenEnergyMonitor Wiki

Software:
EmonHub serial interfacer: emonhub/conf/interfacer_examples/directserial at emon-pi ¡ openenergymonitor/emonhub ¡ GitHub

#9

Thanks for your input Robert, I will have a play with the receiver sensitivity next week and let you know how I get on :slight_smile:

#10

Does the emonhub log file show that your hub is receiving the transmissions at the expected intervals? Note that the emonhub config doesn’t change how often the emontx sends new readings, it’s just updating how often they’re sent to emoncms. This will check whether the transmission is being sent/received as expected… I would do this by looking at the timestamps of the NEW FRAME messages in the emonhub.log and check that they’re spaced 10 seconds apart.

E.g. in my case I see:

[email protected]:~ $ tail -f /var/log/emonhub/emonhub.log | grep NEW
2017-05-03 17:29:05,379 DEBUG    RFM2Pi     478511 NEW FRAME : OK 10 227 0 0 0 0 0 0 0 186 94 184 11 184 11 184 11 184 11 184 11 184 11 1 0 (-29)
2017-05-03 17:29:15,252 DEBUG    RFM2Pi     478512 NEW FRAME : OK 10 223 0 0 0 0 0 0 0 169 94 184 11 184 11 184 11 184 11 184 11 184 11 1 0 (-31)
2017-05-03 17:29:25,120 DEBUG    RFM2Pi     478513 NEW FRAME : OK 10 221 0 0 0 0 0 0 0 148 94 184 11 184 11 184 11 184 11 184 11 184 11 1 0 (-28)
...
2 Likes
#11

Hi Andrew,

Yes the hub does appear to transmit data every 10 seconds, or sometimes every 9s! Here is a copy from the log:

2017-05-03 17:18:09,434 DEBUG    RFM2Pi     34156 NEW FRAME : OK 8 200 1 0 0 0 0 188 0 222 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-40)
2017-05-03 17:18:18,891 DEBUG    RFM2Pi     34157 NEW FRAME : OK 8 198 1 0 0 0 0 191 0 221 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-40)
2017-05-03 17:18:28,443 DEBUG    RFM2Pi     34158 NEW FRAME : OK 8 91 1 0 0 0 0 192 0 222 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-40)
2017-05-03 17:18:37,911 DEBUG    RFM2Pi     34159 NEW FRAME : OK 8 60 1 0 0 0 0 191 0 222 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-40)
2017-05-03 17:18:47,439 DEBUG    RFM2Pi     34160 NEW FRAME : OK 8 62 1 0 0 0 0 195 0 220 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-40)
2017-05-03 17:18:56,968 DEBUG    RFM2Pi     34161 NEW FRAME : OK 8 62 1 0 0 0 0 206 0 219 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-41)
2017-05-03 17:19:06,527 DEBUG    RFM2Pi     34162 NEW FRAME : OK 8 61 1 0 0 0 0 193 0 222 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-39)
2017-05-03 17:19:15,986 DEBUG    RFM2Pi     34163 NEW FRAME : OK 8 64 1 0 0 0 0 193 0 218 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-38)
2017-05-03 17:19:25,503 DEBUG    RFM2Pi     34164 NEW FRAME : OK 8 63 1 0 0 0 0 198 0 219 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-42)
2017-05-03 17:19:35,060 DEBUG    RFM2Pi     34165 NEW FRAME : OK 8 66 1 0 0 0 0 199 0 220 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-40)
2017-05-03 17:19:44,534 DEBUG    RFM2Pi     34166 NEW FRAME : OK 8 66 1 0 0 0 0 199 0 222 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-39)
2017-05-03 17:19:54,039 DEBUG    RFM2Pi     34167 NEW FRAME : OK 8 64 1 0 0 0 0 202 0 221 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-40)
2017-05-03 17:20:03,618 DEBUG    RFM2Pi     34168 NEW FRAME : OK 8 65 1 0 0 0 0 203 0 220 1 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-42)

I am hoping to try hard wiring the emonBase to the emonTx tomorrow to see if that eliminates the problem and then hopefully I will know for sure!

Thanks, Sam.

#12

Cool use of tail and grep :slight_smile:

If you have more than one node you can filter that further eg

[email protected](rw):~$ tail -f /var/log/emonhub/emonhub.log | grep "NEW FRAME : OK 5"
2017-05-03 17:48:57,914 DEBUG    RFM2Pi     8023 NEW FRAME : OK 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2017-05-03 17:49:03,025 DEBUG    RFM2Pi     8026 NEW FRAME : OK 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2017-05-03 17:49:07,981 DEBUG    RFM2Pi     8029 NEW FRAME : OK 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2017-05-03 17:49:13,053 DEBUG    RFM2Pi     8032 NEW FRAME : OK 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2017-05-03 17:49:18,095 DEBUG    RFM2Pi     8035 NEW FRAME : OK 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)
2017-05-03 17:49:23,080 DEBUG    RFM2Pi     8038 NEW FRAME : OK 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)

[one for the Linux tips & tricks thread @Jon ?]

1 Like
#13

In the meantime you could redirect that output to a temp logfile and if there are any drop outs, look to see if the packets are also missing from the temp log too.

First create a temp log file and set permissions for Pi user

sudo touch /var/log/test.log
sudo chown pi:pi /var/log/test.log

then use this command to pipe the output you are seeing to that test log file

tail -f /var/log/emonhub/emonhub.log | grep --line-buffered "NEW FRAME : OK 8" >> /var/log/test.log&

the trailing “&” should ensure the piped output continues to be logged once you close the terminal window. but before you do, to check the test log is being written to, try

tail -f /var/log/test.log

This temp file will be lost on reboot so you will need to recreate the temp log file and restart the piped output if you reboot.

Alternatively if it has happened recently you could list all the recent packets from the emonhub.log using

cat /var/log/emonhub/emonhub.log | grep "NEW FRAME : OK 8"

works for the rotated out logfiles too

cat /var/log/emonhub/emonhub.log.1 | grep "NEW FRAME : OK 8"
#14

It seems that your base station is receiving all the data, so there is some issue between it being received and posted to emoncms.

EDIT - As noted above maybe the dropouts were on a larger timeframe: Paul’s suggestion seems like a sensible step to take.

Can you post the [[emoncmsorg]] section of your config file, redacting the API key as appropriate? In particular I’m wondering if the sendinterval needs to be changed.

I personally prefer using the MQTT mechanism to get data to emoncms. I wrote a replacement script to post data to emoncms, emon_mqtt_logger.py, available in my emonhub repo here: GitHub - adpeace/emonhub: emonHub is the link between your inputs and emonCMS), as I found the PHP one that is provided to not be very reliable. However, that might require a bit more manual setup than you’re up for doing. I wrote a post about this a while back but didn’t get much interest, so am just using this myself. If others are interested I could write setup steps: this is more for emontx/hub installations rather than emonpi which is more turnkey.

#15

Thanks for your help here guys :slight_smile:
I am going on holiday on Friday for a week and have some solar installs to complete when I get home so it may be a couple of weeks until I have chance to experiment with your suggestions! Here is my config, I edited the sendinterval parameter to ‘10’ a few days ago:

[[emoncmsorg]]
    Type = EmonHubEmoncmsHTTPInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,
        url = https://example.co.uk/
        apikey = APIkey
        senddata = 1                    # Enable sending data to Emoncms.org
        sendstatus = 0                  # Enable sending WAN IP to Emoncms.org MyIP > https://emoncms.org/myip/list
        sendinterval = 10                # Bulk send interval to Emoncms.org in seconds
#16

I agree the emoncms’s phpmqtt_input.php module is not very good. Many, many moons ago the serial inputs were also handled in php before a python script replaced it as php isn’t very good at continuously looped code. From that OEMgateway was born and that led to emonHub. So I do not understand why we are using PHP for the mqtt_input and feedwriter daemons, it seems like a big step backwards to me, so I would like to hear more about your alternative, I have to admit I was previously thrown a little by it’s inclusion in emonhub not emoncms, suggesting it was an alternative route out of emonhub rather than an alternative route into emoncms.

Having had a very brief second look at your code, am I right in thinking it uses the http input api to post to emoncms? When I was looking at this same subject recently I was trying to call the PHP functions directly from Python using the OS module but didn’t get very far at that time as I’m not familiar with PHP and time was short (work commitments).

I really want to explore mqtt more but both the emonhub and emoncms implementations put me off due to poor reliability and inflexibility. I would need more than one account to work with mqtt within emoncms and I would want emonhub to subscribe to other topics in a variety of formats, not just provide a single topic for other softwares to publish too.

#17

Yes, that’s how it works. One advantage of this approach is that it is decoupled, so can be on a separate host to emoncms. I put it in emonhub mainly because it is a Python script, but it could be put into the emoncms repository instead.

I don’t fully understand your requirement regarding more than one account? I don’t think having emoncms connected directly to MQTT at all is necessary - an approach like I’ve taken decouples it into a completely separate script and just uses the emoncms APIs to input data. emonhub currently publishes to a set of topics under a given basetopic, but the layout under that AFAIK is not currently configurable. For my purposes I view emonhub only as an interface to get data from the serial line of the RF module published to MQTT; beyond that other processing can be done in external ‘microservice’ scripts. For example, I have another script similar to the one I linked to that puts data from an emonTH into a separate archive database; it just subscribes to the MQTT topic, no modifications were needed to emonhub to make that work.

#18

Ahh ok! So you were right to put it with emonhub I guess. My original assumption was in fact correct in that this is a bolt-on for emonhub to provide a route from mqtt to emoncms via http, not a direct “Python” alternative to the phpmqtt_input.php script.

What you are doing with this add-on IMO should be achievable by emonhub, that was the intention and why I am quite frustrated with the current implementation in the emon-pi variant. emonhub should be able to subscribe to any mqtt topic or publish to any topic. If that was the case you could direct your RFM2Pi data to be published to the MQTT topic(s) of your choosing for other local services to subscribe to and also send the same data to emoncms via http directly.

I have to admit I am still a little hazy on your need to publish the data to MQTT for another script to subscribe to that topic and send it to emoncms via http rather than just sending it by http. You said “I personally prefer using the MQTT mechanism to get data to emoncms.” but if I understand correctly, although arguably your data does do a little round trip via MQTT, it is ultimately using http to get data to emoncms not mqtt. So both publishing to MQTT and sending via http from emonhub would do the same in less steps, would it not?

Agreed, but it does seem desirable, many users just want to use MQTT without gaining any real benefit in some cases.

There could be value in a proper MQTT implementation for emoncms though, the major benefit of MQTT is it’s low overhead, so it would be ideal for use with a GSM/3G/4G mobile data network etc as it would keeps running costs low. It could also work well for LoRa and SigFox type networks, (both of which have been mentioned here on the forum).

This is essential to have MQTT inputs on an emoncms server that operates more than one account eg emoncms.org, I to operate a emoncms server for my client sites so the current phpmqtt_input.php is of no value to me as it only works for one account/user.

That sort of goes in the opposite direction to emonhub’s original purpose, the RFM2Pi is just one interface, and the intention was for emonhub to work seamlessly with emoncms (locally or remotely) and use the same core structure and the same target settings and configuration, init script, log files etc for many interfaces, including some very custom/personal as well as widely used interfaces.For example there have been “mySQL” and “CSV” interfacers for the original version to post directly to file, that got little interest too back then.

There have been several custom interfacers added to the “emon-pi” variant by other users including Smilics, Victron, ModBus, Graphite, SMASolar and BMW EV which if implemented correctly should allow posting to emoncms via http and/or mqtt

There is of course no reason you cannot just use emonhub as a RF interface only if you prefer, but it wasn’t designed to get RF data to emoncms, it was designed to make interfacing with emoncms simpler in general.

#19

I think it is a direct alternative. It runs outside emonhub as a separate process; I think it’s pretty much equivalent to the php_mqtt_input PHP script.

It’s mostly about decoupling: I have other applications that also subscribe to the topics and do things with the data. For example, I have a temperature archiving service that logs all readings into a postgres DB. I’m also looking at other services that will make use of the temperature data from emonTH. Using MQTT means I can add those services easily without them having to know about emoncms or emonhub themselves. Similarly, other data sources can publish temperature or power readings to MQTT and they can be consumed alongside those published by emonhub.

The architecture diagram at Boiler Control to Maintain a Set Temperature – Hacking at Home shows a glimpse of the kind of things I’ve been doing using that architecture.

That’s one potential benefit: for my use case the main benefit is the decentralisation/easier division of work between into separate services. As I’m adding additional functionality locally, I can do it without have to patch any of the core emonhub/cms. Once you add in SSL, which would be necessary for any publicly-visible emoncms instance, the overhead of MQTT is higher again (potentially not that different from just using HTTP with a compressed payload?).

I see; I’m not familiar with the multi-user stuff but I think that there’s no reason at least the Python script I’ve been using couldn’t be extended, but again I would imagine such a solution would use HTTP(S) to connect to emoncms rather than MQTT. So in that model, each client has a local emonhub to get data from the RFM2Pi, then the MQTT input module receives that data and has credentials for your emoncms server and uses the REST API to push data there with appropriate per-client credentials. So MQTT is entirely local to each client.

#20

Hi all,

I am still experiencing loss of data points even with a direct serial connection between my emonTX and emonBase. Anyone have any ideas what else I could explore in order to resolve this? Should I typically expect to enjoy 100% quality?

Interestingly the local and remote readings now differ with the direct serial connection. The figures are:

Local: 87% (629/723)
Remote: 83% (598/723)

Maybe that provides a clue as to the issue.

2017-05-16 05.34.42 pm.png1436×716 97.6 KB

Look forward to any help. Cheers, Sam.