Another quick question; my emonPi and three emonTX/ESP8266’s all report through to emoncms.org
What happens if the internet connection goes down? Do these boxes continue to collect local data from the CT sensors etc, and then upload it when the www comes back, or is the data lost, or do I need to do anything to re-send the “missing” data? Is there any difference between the way an emonPi and an EmonTx/ESP8266 handles this?
Would sending the emonTx/ESP8266 data to the emonPi (through the WLAN) instead of to emoncms.org improve things in this situation?
I can’t answer for the ESP8266 - it might store the data on-board if it can’t see a connection, or if it doesn’t receive an ‘ok’. (@pb66 can probably answer that.) But the emonTx itself certainly does not check for a response: once the data had been sent to the ESP8266, it’s forgotten about completely.
What you could do is both store the data on your emonPi and send it onward to emoncms.org. But then that still depends on your local WAN, if not the Internet. The downside is, you’ve got two almost (but not quite) identical emonCMSs, both of which need a set of inputs with their associated processing, feeds and dashboards that all need setting up and maintaining.
This morning there is a mysterious Node 22 showing up in my Inputs page, with its’ 3 feeds all “4 hrs” old. I certainly didn’t do anything at 04:00 this morning, so wonder whether these two events are a glitch? (I don’t have any emonTh units.)
You haven’t accidentally published your APIkey on the forum, have you? We try to keep a eye open for people who do that and edit it out, but there’s no guarantee we get there before somebody else sees it. And then of course they can send data to your account there.
Thanks Robert; the Node 22 data is showing on emoncms.org , yes. I am 99.99% sure that I have never publicised my API keys anywhere, and all inputs, feeds and dashboards are set to private as well.
In other news, I think that I have now finally resolved my psent / psuccess issues - like another forumite it was most likely the 5V DC input jack being slightly too big for the emonTx plastic cover & therefore not making a good connection. I drilled out the hole in the cover and have had 24-48 hours of continued data. Good news!
We’ll see if Trystan can shed some light on the phantom nodes.
Experience has taught me never to underestimate the capability of a power supply fault to generate totally confusing symptoms.
I’ll flag this to @Gwil - it warrants at least looking at when the next production batch is ordered. Either something is coming in out of tolerance, or there was never enough room in the first place.
Unfortunately the way you have it configured, the emonESP+Tx data is lost when there are network outages. AFAIK the esp has no ability to store data, it certainly cannot post more than one frame of data per http request (ie no bulk upload mode), there maybe retries in the http library but as far as the esp sketch goes, all it does on request success (or fail) is increment psent/psuccess as appropriate and move on to the next frame if one exists.
I have often wondered about the priority or order of events when a network connection is slow or intermittent, ie if the request takes 60s to get a confirmation reply, what happens during that 60s? My guess it life goes on and only the latest frame when the request has finished that will be sent, ie each frame of data overwrites the previous whether it has been sent (and received) or not.
This is where the RFM do have an advantage as they post direct to the emonPi with no external dependency (router/wifi/internet etc).
Another concideration here is the timestamping, even if an emonESP could sit on frames of data, it doesn’t add a timestamp, that happens when it lands at emoncms or emonHub, so if your network goes down when the sun’s up and comes back on when the sun has set, it would still appear that your PV is producing at midnight if that’s when your midday data lands.
The emonPi on the otherhand is running emonhub, so anything that gets sent to emonhub on the emonpi will be timestamped immediately and buffered until it can post the data to it’s target and obtain a confirmation that has been done successfully (as long as power is not removed).
RFM data goes through emonhub, however the emonESP data doesn’t. It is configured to post direct to emoncms, so if you point your emonESP’s to your local emonPi’s emoncms, yes it would remove any dependency on a working internet connection for the local emoncms, but it wouldn’t help with your remote emoncms.org account.
However, even then, you are dependent on the network router(s) for a complete connection. The nearest you can get to the RFM’s “direct” connection with an emonESP is to set up the emonPi as a access point and connect the emonESP’s direct, but that may then be subject to similar range issues as the RFM’s again and it only removes the need for the router, the way the data is sent to emoncms not emonhub still stands.
The ideal position would be for the emonESP to have the ability to post to a emonHub socket, that way emonHub can timestamp and buffer the data for multiple emoncms targets, local and or remote. (This was agreed at our staff meeting in Feb 2018 but hasn’t yet materialised).
There may be a way of sort of doing it via MQTT if you were to play with the publish topic of the emonESP and subscribe topic of emonHub, perhaps creating another mqtt interfacer instance, I have never tried and cannot say with any certainty if it would work or how well if it does work.
I will try and help with a couple of your other questions later, I was just taking a quick break whilst rain stopped play, now I must get back to the grindstone before the rain returns as I’m sure it shall
Hi Julian, is this with a cable that was supplied by the OpenEnergyMonitor shop?
Thanks for flagging, Robert. I’ve tested a batch of emonTx units we have in stock, no issues thus far powering them up with the cable we supply. That doesn’t rule out it being an issue with tolerance on some units or it not being compatible with some third party cables.
The emonPi’s “Node 5” is being delivered via the emoncms bulk input api from emonhub. so each node just has a number and the "Node " prefix is added by emoncms, the payload is CSV (comma separated variables) so the 11 values are identified by their index key (position in the CSV array), that’s why they are numbered and you can add your own “description” (aka input name). The emonESP however passes the data as a nodename plus an array of key:value pairs where each pair is an input name and value, so the emonESP inputs show up as a named node and individually named inputs.
Yes, almost definitely RF noise picked up by the emonPi RFM module. There are various ways you can reduce the chance of this happening. What would be by far the easiest method would be if it were possible to just switch off the RFM receiver when you’re not using it. The emonpi FW sketch actually does have the ability to do this but it is a hardcoded option (requires a recompile and flash the emonpi add-on board’s MCU) it was never an “active” option that could be set via the serial port like quiet mode or US mode etc.
So the next best (and by best I guess I mean easiest) way is to tighten the checks made by emonhub on the nodes you are not using (ie 0 thru 31 except 5), using the 2 phantom inputs you have already seen as examples, node 4 has no definition in emonHub.conf therefore anything that gets past the JeeLib CRC checks and presents itself as “4” is passed no matter what. If you were to add
[[4]]
[[[rx]]]
datacodes = b
to your emonhub.conf then any payloads that pass the JeeLib CRC checks as node “4” will get rejected unless they are exactly 1 byte in length, this will add another hurdle and greatly reduce the chance of stray data getting through.
Node “22” is already defined in emonhub.conf, but it has a more vague “datacode = h” (note the datacode is single not plural as in my example above). With this setting anything getting through as node “22” with an even number of bytes, whether it be 2 or 20 is passed, this should really instead be defined as
datacodes = h,h,h,h
that way only a payload of exactly 8 bytes will pass if identified as “22”. You can either just tighten it down to that “exactly 8 bytes” or use the same “exactly 1 byte” setting as my previous example.
You could use the same approach to effectively block all nodes 0 thru 31 except 5 by editing emonhub.conf in this way.
Some other idea’s, remove the emonPi antenna to make it deaf(er) ? Or try setting the RFM module to 868MHz (RFM modules are pretty inefficient when run at the wrong frequency with the wrong antenna length) or try a different RF group (like channels but not quite), the emonPi uses “210” by default but you can select 1 thru 255, maybe you can find a quieter “channel” as even before the JeeLib CRC checks, payloads need to match an expected preamble which includes the group, so any rogue inputs you see will have appeared to be group 210.
It has been discussed elsewhere on the forum, IMO the RFM modules of the emonPi and emonTx should be able to be enabled and disabled via the serial port, but that’s not in place yet. so one of the workarounds above might have to do for now.
One way to do that, so-to-speak, would be to attach a 50 Ω dummy load to the antenna port.
Quick, and easy to do, fairly inexpensive, and leaves the code intact should one desire to use the radio at a later date. Here’s an example that’ll handle 1 Watt. (i.e. it’ll handle full transmitter output power)
A pair of them cost 5.50 USD. (I didn’t see any listings for them in quantity of 1)
@pb66 - thanks for your comments; shame about the data being lost if the www connection goes down, but I can live with that. Sadly due to the various locations of cables and CUs etc, the RFM won’t work for me. I haven’t looked at the MQTT stuff yet, but fear that for now at least that is beyond my comfort zone.
@Gwil - yes; both the power brick and USB cable are from the OEM shop. I have had another two drop outs this morning, which is very frustrating. I’m as interested in recording data as I am in visualising real time data, and so this is going to make historic data unreliable. Turning off/on helps, but it doesn’t always work. Over the weekend I half hoped I’d used a third party power supply as that would have been comparatively easy to resolve, but I checked this morning and it is an OEM shop unit. At the end of last week I bought and set up an additional wifi access point, just in case it was a wifi issue, but that has proved not to be the issue. Any other thoughts / suggestions?
@pb66 - thanks for the info re: phantom nodes being picked up through RF noise and the emonPi RFM module. I’ve not yet played around in the .conf files so I will wait until I have an hour free to concentrate on that tomorrow. (I had removed all of the RFM antennas on all units on installation.)
@Bill.Thomson - thanks for the dummy load adapter idea - I have ordered a couple via eBay and will let you know how I get on when they arrive.
Hmmm, this is getting beyond unfunny; my emonPi is working fine and reporting through to emoncms.org .
The previously errant emonTx/ESP8266 in the garage says it reported 129 seconds ago, but with psent = 1, psuccess = 0 … & I’m not actually convinced that it has reported reliably / continuously or properly for 48 hours or more.
The previously solid emonTx/ESP8266 in the house now says it hasn’t updated for 5 hours.
The previously solid emonTx/ESP8266 in the loft (Hot Water) is fine.
Am waiting for the dummy load adapters to arrive via eBay, but am losing confidence in the whole thing which is disappointing …
It is odd that the data is getting to emoncms.org and not the local server.
@TrystanLea - does the ESP transmit to both sets of data via API calls or is the remote one API and the internal MQTT? Is there a hardcoded MQTT clientID by any chance?
@haffle, if you only run one of the EmonTX/ESP combos (reboot the EmonPi), what happens?
@pb66 - I could not find it (long thread) but has the RFM actually been disabled or is it still running? AIUI, the ESP can only setup one server by API (in this case emoncms.org), the other needs to be via MQTT. If the MQTT settings have not been setup, then the data must be getting through via RFM which is why it is flakey.
Just do one EmonTX to start as the base topic for each will need to be different. Also use the IP address of the EmonPi not just emonpi for the MQTT server setting.
You will possibly see duplicated data (RFM & MQTT) but that can be sorted out.
I haven’t knowingly done anything with the RFM - I just assumed that having used the ESP8266s that that would take care of the comms. (AIUI, the three EmonTx/ESP8266 “report” straight through to emoncms.org via wifi, and the emonPi does also. I don’t think the Tx’s communicate with the Pi at all - but I may be wrong).
@haffle, have you setup the MQTT on the ESP devices as per GitHub - openenergymonitor/EmonESP: ESP8266 WIFI serial to emoncms link ? Just do one EmonTX to start as the base topic for each will need to be different. Also use the IP address of the EmonPi not just emonpi for the MQTT server setting. You will possibly see duplicated data (RFM & MQTT) but that can be sorted out.
Sorry for being a bit dim, but why would I need or want to set up MQTT? I have to admit that I don’t really understand what it does, but if the Tx’s and the Pi are reporting straight through to emoncms.org is that not sufficient?
it is currently sending data to emoncms.org via an HTTP API call - this is why it works. However, it only sends it to that server not your EmonPi. I’m pretty certain the data you are seeing on the Pi is what is being received by RFM as you have not disabled it.
You have 2 choices:
Set the server on the EmonESP to send data to the EmonPi and then use the Sync module to ‘sync’ it to emoncms.org
Leave the emoncms.org as the ‘emoncms server’ and send data to the EmonPi via MQTT.
The MQTT setup should be pretty simple and just need the fields filled out, as MQTT is already running on the EmonPi. Just each EmonTX needs its own, unique, basetopic (after the emon e.g. emon/emontx1, emon/emontx2).
@haffle
Do you have a programmer, and are you happy to edit the emonTx sketch, compile and reload it?
If you are, you can disable the RFM by commenting out line 471 in the sketch:
send_rf_data(); // *SEND RF DATA* - see emontx_lib
Just put two slashes at the beginning of the line, recompile and upload to your emonTx.
(If you need to go back to using the RFM, take the slashes out.)
If it then fails to send anything to your emonPi, that’s a fairly solid indication that it is using the radio for that.
I’ve suggested elsewhere that perhaps there should be a mechanism for selecting the inputs you want to use rather than allowing anything discovered appearing.
The procedure to update the sketch in the emonPi is a lot more complicated than for the emonTx.
But if @haffle wishes to tackle that, the lines that need commenting are 263 - 265 in src.ino.
That will prevent reception but should allow transmission if an emonGLCD is ever added.
