Setting up an emonPi - some newbie questions

borpin · 2019-09-10T20:54:41.811Z

It is odd that the data is getting to emoncms.org and not the local server.

~~@TrystanLea - does the ESP transmit to both sets of data via API calls or is the remote one API and the internal MQTT? Is there a hardcoded MQTT clientID by any chance?~~

~~@haffle, if you only run one of the EmonTX/ESP combos (reboot the EmonPi), what happens?~~

borpin · 2019-09-10T21:10:15.832Z

haffle:

I haven’t looked at the MQTT stuff yet, but fear that for now at least that is beyond my comfort zone.

@pb66 - I could not find it (long thread) but has the RFM actually been disabled or is it still running? AIUI, the ESP can only setup one server by API (in this case emoncms.org), the other needs to be via MQTT. If the MQTT settings have not been setup, then the data must be getting through via RFM which is why it is flakey.

@haffle, have you setup the MQTT on the ESP devices as per GitHub - openenergymonitor/EmonESP: ESP8266 WIFI serial to emoncms link?

Just do one EmonTX to start as the base topic for each will need to be different. Also use the IP address of the EmonPi not just emonpi for the MQTT server setting.

You will possibly see duplicated data (RFM & MQTT) but that can be sorted out.

haffle · 2019-09-11T15:34:27.252Z

Thanks Brian.

@pb66 - I could not find it (long thread) but has the RFM actually been disabled or is it still running? AIUI, the ESP can only setup one server by API (in this case emoncms.org), the other needs to be via MQTT. If the MQTT settings have not been setup, then the data must be getting through via RFM which is why it is flakey.

I haven’t knowingly done anything with the RFM - I just assumed that having used the ESP8266s that that would take care of the comms. (AIUI, the three EmonTx/ESP8266 “report” straight through to emoncms.org via wifi, and the emonPi does also. I don’t think the Tx’s communicate with the Pi at all - but I may be wrong).

@haffle, have you setup the MQTT on the ESP devices as per GitHub - openenergymonitor/EmonESP: ESP8266 WIFI serial to emoncms link ? Just do one EmonTX to start as the base topic for each will need to be different. Also use the IP address of the EmonPi not just emonpi for the MQTT server setting. You will possibly see duplicated data (RFM & MQTT) but that can be sorted out.

Sorry for being a bit dim, but why would I need or want to set up MQTT? I have to admit that I don’t really understand what it does, but if the Tx’s and the Pi are reporting straight through to emoncms.org is that not sufficient?

TIA

Julian

borpin · 2019-09-11T15:46:18.075Z

If your EmonESP setup page looks like this

it is currently sending data to emoncms.org via an HTTP API call - this is why it works. However, it only sends it to that server not your EmonPi. I’m pretty certain the data you are seeing on the Pi is what is being received by RFM as you have not disabled it.

You have 2 choices:

Set the server on the EmonESP to send data to the EmonPi and then use the Sync module to ‘sync’ it to emoncms.org
Leave the emoncms.org as the ‘emoncms server’ and send data to the EmonPi via MQTT.

The MQTT setup should be pretty simple and just need the fields filled out, as MQTT is already running on the EmonPi. Just each EmonTX needs its own, unique, basetopic (after the emon e.g. emon/emontx1, emon/emontx2).

Robert.Wall · 2019-09-11T16:23:05.581Z

@haffle
Do you have a programmer, and are you happy to edit the emonTx sketch, compile and reload it?

If you are, you can disable the RFM by commenting out line 471 in the sketch:

send_rf_data(); // *SEND RF DATA* - see emontx_lib

Just put two slashes at the beginning of the line, recompile and upload to your emonTx.
(If you need to go back to using the RFM, take the slashes out.)

If it then fails to send anything to your emonPi, that’s a fairly solid indication that it is using the radio for that.

borpin · 2019-09-11T17:20:36.184Z

I’ve suggested elsewhere that perhaps there should be a mechanism for selecting the inputs you want to use rather than allowing anything discovered appearing.

Phantom Nodes in Inputs Getting Started

Continuing the discussion from Setting up an emonPi - some newbie questions: The issue of phantom nodes appearing on the Inputs page has been a recurring issue for a while. I see it occasionally. I wonder if the solution is to not add nodes automatically to Inputs, but rather EmonCMS keeps track of nodes that are available, and the user can then select which ones they would like to monitor & add Feeds for. This selectable node list could be maintained in memory rather than actually writing an…

borpin · 2019-09-11T17:24:53.407Z

@Robert.Wall, would it be easier to disable the receiver on the EmonPi?

Robert.Wall · 2019-09-11T17:55:43.169Z

In short, probably not.

The procedure to update the sketch in the emonPi is a lot more complicated than for the emonTx.

But if @haffle wishes to tackle that, the lines that need commenting are 263 - 265 in src.ino.
That will prevent reception but should allow transmission if an emonGLCD is ever added.

haffle · 2019-09-12T08:49:14.222Z

If your EmonESP setup page looks like this EmonESP/docs/emonesp.png at master · openenergymonitor/EmonESP · GitHub it is currently sending data to emoncms.org via an HTTP API call - this is why it works. However, it only sends it to that server not your EmonPi. I’m pretty certain the data you are seeing on the Pi is what is being received by RFM as you have not disabled it.

@borpin Brian, just to clarify my thoughts … I think that my three emonTx/ESP8266 are reporting straight through to emoncms.org , and not to my emonPi; the emonPi is also reporting through to emoncms.org , and all my Inputs and Feeds are created within & hosted by emoncms.org remotely. Assuming this is correct, I am unsure why a flaky RFM signal would cause data loss; it could well create phantom node / node data I suppose, but why would it cause data loss on Inputs and Feeds going straight to emoncms.org ? This is what my (robust) emonTx/ESP8266 set up page looks like. I haven’t looked at SSL or MQTT yet, hence a slight variance with your page above.

2019-09-12%20Hot%20Water1078×2132 228 KB

borpin:

You have 2 choices:

Set the server on the EmonESP to send data to the EmonPi and then use the Sync module to ‘sync’ it to emoncms.org

Leave the emoncms.org as the ‘emoncms server’ and send data to the EmonPi via MQTT.

The MQTT setup should be pretty simple and just need the fields filled out, as MQTT is already running on the EmonPi. Just each EmonTX needs its own, unique, basetopic (after the emon e.g. emon/emontx1 , emon/emontx2 ).

If all four units are posting straight through to emoncms.org , and my Inputs and Feeds are hosted remotely, what does MQTT give me? Again, sorry for being dim, but whilst moderately tech savvy & I have played with a Pi before, my skills are more with macOS than Raspberry Pi etc
.
.

Robert.Wall:

@haffle
Do you have a programmer, and are you happy to edit the emonTx sketch, compile and reload it?

If you are, you can disable the RFM by commenting out line 471 in the sketch:

send_rf_data(); // *SEND RF DATA* - see emontx_lib

Just put two slashes at the beginning of the line, recompile and upload to your emonTx.
(If you need to go back to using the RFM, take the slashes out.)

If it then fails to send anything to your emonPi, that’s a fairly solid indication that it is using the radio for that.

I don’t have a programmer, no. Are there any you suggest? Preferably macOS based?

I wonder (cf half hope) if the RFM dummy load widgets will prevent the need to turn off RFM through software?

Thanks all - you are being very helpful!

borpin · 2019-09-12T09:01:49.152Z

haffle:

I think that my three emonTx/ESP8266 are reporting straight through to emoncms.org , and not to my emonPi;

Assuming you have blanked out the API fields etc - yes you are talking to emoncms.org. But I think you said you are seeing data on the EmonPi ?

As I said, the EmonPi will automatically pick up any RFM data flying about - I still think this is what you are seeing on the EmonPi.

haffle:

the emonPi is also reporting through to emoncms.org

So you have set this up then on the EmonPi?

image786×316 16.1 KB

If not, then no, the EmonPi is not talking to emoncms.org.

If you are talking directly to emoncms.org from the EmonTX via EmonESP, then you really do not want to use sync - it will really confuse you.

My suggestion is to change the EmonESP to talk to the local EmonPi over WiFi. Ignore the RFM inputs that will appear and then use the sync module to sync with emoncms.org.

This should minimise and data loss and that will only happen if there are WiFi issues (rather than Internet drop out) or a problem with the EmonPi.

pb66 · 2019-09-12T09:18:07.754Z

Hi Julian, you are right on all points.

You do not need to use the local emoncms if you prefer to just use emoncms.org at this point, therefore you do not need MQTT at all at this stage.

And yes a dummy load on the emonPi should eliminate any remaining chance of stray RFM data getting in.

Regardless of the above, the RFM data (noise or spurious valid reception) will not interfere with your emonTx/emonESP data at emoncms.org.

The issue you seem to be having appears to be entirely due to the emonESP(s) not reliable posting to emoncms.org. Either due to network or power supply issues (or potentially but perhaps unlikely, a faulty unit?).

Do you log psuccess and psent like Ian did in the thread I linked? If you are not already doing so I suggest you do, even if just whilst we debug this and get you running, you can delete the feeds later as you are charged for each feed at emoncms.org.

The psent 1 and psuccess 0 is normal for the first attempt at posting, obviously the psuccess won’t be incremented until the emonESP gets that “ok” from emoncms.org after the data has landed at emoncms.org.

The fact you see psent 1 and psuccess 0 means either the emonESP has

just been powered up (still got a dodgy PSU or connection ?) or
just been deliberately reset in software due to 30 consecutive unsuccessful connection attempts
since gone off-line due to network issues lack of power or perhaps your router is blocking the connection, recently there was a case of an openEVSE being blocked by a BT router and I have seen a similar issue with another router.

By capturing the psucces and psent we can look at a graph to see if there is any pattern, ie is the emonesp connecting but not getting a valid reply from emoncms.org, driving up the failures until it’s reset at 30. Or is the connection ok at times but not others etc etc.

Getting the emonESP’s to reliably post to emoncms.org is indeed where you need to focus right now.

Robert.Wall · 2019-09-12T09:26:19.972Z

haffle:

I don’t have a programmer, no. Are there any you suggest? Preferably macOS based?

The programmer is the interface between a USB port and the emonTx. You’d need to set up the Arduino IDE on MacOS to use it. There are notes in ‘Learn’ Learn→Electricity Monitoring→Using the Arduino IDE

haffle:

I wonder (cf half hope) if the RFM dummy load widgets will prevent the need to turn off RFM through software?

The terminations won’t completely turn off the r.f. signal, but they should attenuate it significantly - hopefully enough so that it’s not received.

haffle · 2019-09-12T09:46:55.026Z

I think I need a bit of a lie down as I am confusing myself here

If I type in the LAN IP of my Pi (192.168.1.40) I can see on the emonPi Inputs (x2 CTs and a temp probe). There are no Feeds set up here.
If I go to emoncms.org I can see on the Inputs page data from the x3 Txs and x1 Pi; On the Feeds page there are a load of Feeds within each unit.
I have now created Log to Feed for x3 psent and x3 psuccess, and will monitor the data over the next few hours.
To be honest I am now sufficiently confused that I don’t remember how the Pi data even gets to emoncms.org , but the three Txs have been pointed to emoncms.org with an API key. (NB: The API key does get accepted but as and when I go back in to the set up page it is always a blank field; is this normal?)
The emonPi is located in my office - a good 50-60 metres away from the three Txs (with brick walls and barns in between as well.
Because I decided fairly early on to (try to) keep things simple, I opted for the remote emoncms.org rather than self hosting, knowing that that would mean some slight loss in features. (I do have webspace, so could reverse that decision at a later date.) So I have not yet looked at Sync or MQTT.
My modem is a Draytek Vigor 2760, which replaced the free PlusNet unit from the ISP. In between that and the Txs are a variety of Apple Airport Extremes and Expresses.

haffle · 2019-09-12T10:17:02.996Z

haffle:

To be honest I am now sufficiently confused that I don’t remember how the Pi data even gets to emoncms.org

I now remember - it was because I put the API key in the Config page within EmonHub.

EDITED TO ADD: When I go to 192.168.1.27 (house) and .21 (garage), no API key shows, and it also says “Connected No” on both. Neither are reporting any data to Inputs. The HW Tx shows no API key, but “Connected Yes”, and Input data is flowing.

On the first two, if I copy / paste the API back in, click Save and then Restart, they still don’t connect - but have in the past. Wifi in the garage is showing as RSSI dBm -57, and the house is -59. (The HWater is -79).

pb66 · 2019-09-12T11:47:15.901Z

Try the Fing app to see if you can see the emonESP’s and the route they take, ping them too.

I have used Fing on Android and just checked it is available for Apple devices but I have never used it myself (‎Fing - Network Scanner on the App Store).

haffle:

In between that and the Txs are a variety of Apple Airport Extremes and Expresses.

I can’t really help with this bit as I have no experience of either.

If you are able to access the emonESP settings via the network then I assume your LAN connection is ok. Does the Draytek Vigor 2760 have any sort of debugging? Is there a log for example? If there is, try searching for the emonESP IP no to see if there is any clues. Is there any sort of “white list” or any restrictions? Again I assume (not wise I know) that if one emonTx gets through, then maybe all should get through, but weirder things can happen.

Also try double checking both the apikeys and urls, making sure there are no spaces or line endings when you paste in the apikeys, perhaps try copy and pasting from the emonESP that is working rather than sourcing elsewhere or editing manually (be sure not to edit the working one accidentally though).

borpin · 2019-09-12T12:49:49.882Z

haffle:

I now remember - it was because I put the API key in the Config page within EmonHub.

Ok that makes sense.

EmonPi is not collecting any data into the local version of EmonCMS - everything is going to emoncms.org.

haffle:

no API key shows, and it also says “Connected No” on both

That could be why there is no data . Did you use the Read/Write API key from emoncms.org and not the local Emoncms version (they are different)?

haffle · 2019-09-12T13:30:24.092Z

A couple of thoughts; if I can access the three Txs through my LAN IPs, they are obviously connecting to the wifi network. What is the difference between being able to connect to wifi, and not being able to connect to emoncms.org ?

The Write API key I use is the one found on emoncms.org / [spanner] / My Account; I won’t put the number here, but it starts with 16xxx

If I look on emonpi.local / emonhub / Edit config, the write API key I put in on Day 1 is the same; is this right?

With Fine on iOS (new to me), I can ping the
House Tx: Av ping 33ms, Packet Loss 0.0%, Min Ping 3.0ms, Max ping 202ms, Std Dev 60ms
Garage Tx: Av ping 57ms, Packet Loss 0.0%, Min Ping 5.6ms, Max ping 322ms, Std Dev 94ms
Hot Water Tx: Av ping 24ms, Packet Loss 3.3%, Min Ping 3.3ms, Max ping 282ms, Std Dev 50ms
Office Pi: Av ping 10ms, Packet Loss 0.0%, Min Ping 4.3ms, Max ping 22ms, Std Dev 3.3ms

borpin · 2019-09-12T14:08:49.943Z

haffle:

What is the difference between being able to connect to wifi, and not being able to connect to emoncms.org ?

None.

haffle:

The Write API key I use is the one found on emoncms.org /

OK good.

haffle:

If I look on emonpi.local / emonhub / Edit config, the write API key I put in on Day 1 is the same; is this right?

Yes

Why therefore 2 of the EmonESPs are not connecting is a bit of a mystery .

Re-enter the server details in the EmonESP interface and see if that solves it. There have been some other issues with the EmonESP firmware talked about elsewhere.

haffle · 2019-09-15T06:56:51.944Z

A quick thought; might there be a correlation between the number of Inputs, calculations & Feeds that an EmonTx/ESP8266 is required to do and the reliability of data transfer? Is it a coincidence that my two “flakey” devices are the two with the most complicated setup?

h2196×1088 241 KB

.
.

g2194×1022 215 KB

borpin · 2019-09-15T08:26:19.641Z

haffle:

A quick thought; might there be a correlation between the number of Inputs,

Is this on Emoncms.org?