EmonTX 3 update to continuous Monitoring

John_White · 2023-07-12T02:22:18.409Z

Hi
Having updated my Pi box to the latest I decided I should also update my emontx3’s to the latest
So I used the UART to USB lead I purchased many years ago and connected each TX to the pi and used the update utility selecting PORT USB0 Hardware “emonTx3” Radio format “RFM69 JeeLib Classic” then chose the Firmware "emonTx v3.4 continuous sampling firmware, jeelib_classic, v2.4.0

Two issues appear to have arisen
1 - All boxes now appear to have been set to Node 15 - when I purchased the boxes I had requested that they be set to Node 7, 8, 9, 10, 11, 12, 13. These boxes had not been updated since 2016.
I have downloaded Arduino IDE onto my laptop and fumbled my way to validate that they are on Node 15. Please note I have no prior experience with Arduino.
My question is → Is the only way to fix the problem to download the sketch and change the node no. to XX i.e. 13 then compile it and then upload it ?? I have tried to figure this out without success so far.

I have noticed that I believe I will have to change the emonhub config as the data order and type are different ?? Am I correct in this ?? What is the data type “E1, E2, E3, E4” ??

Any help or pointing in the right direction would be appreciated - Please treat me as a beginner if it comes to Arduino discussion.

Regards John

Robert.Wall · 2023-07-12T13:47:18.756Z

John_White:

1 - All boxes now appear to have been set to Node 15

This is very likely the case. Node 15 is the new default.

John_White:

I have downloaded Arduino IDE onto my laptop and fumbled my way to validate that they are on Node 15. Please note I have no prior experience with Arduino.

This is one way to do it. I strongly suggest you stay with the Arduino IDE and not attempt to use platformio. The ‘obvious’ way would be to download the sketch into your laptop with the Arduino IDE on it, edit the default Node ID in the sketch and then compile and upload the changed sketch. You’ll also need to download and install some libraries in order to be able to compile the sketch. All the instructions are in ‘Docs’ here: Using the Arduino IDE — OpenEnergyMonitor 0.0.1 documentation

However, there should be an easier way. When you had your emonTx’s connected to your emonPi to update the sketch, you should have been able, under ‘Admin’ → ‘Serial Monitor’ to “talk” directly to the emonTx and configure the Node ID. which should now be stored (or capable of being stored) in EEPROM. The procedure is: Stop emonhub (the on-screen button) and start comms with the emonTx (another button). You might see something about “LCD”. Type ? and click ‘send’ to issue the command. The emonTx will give you a list of instructions for all the options available. You need n7 for the first emonTx, ( n8 for the second and so on ) and send that - it should respond and confirm, then importantly you need s to save the new NodeID to EEPROM. It should acknowledge this too.

(If it carries on printing lines like

OK 5 54 0 51 0 105 0 195 92 0 0 0 0 0 0 0 0 0 0 0 0 191 179 44 0 0 0 0 0 98 128 44 0 223 198 46 0 (-0)
OK 5 54 0 51 0 105 0 201 92 0 0 0   0 0 0 0 0 0 191 179 44 0 0 0 0 0 98 128 44 0 223 198 46 0 (-0)

to the window, you didn’t stop emonHub.)

You can restart emonHub, disconnect and reinstate your emonTx. Now, when you look at the emonhub log, you should see some data from Node 7, but as you have seen, it will be rejected because the data format is inconsistent.

What you probably have is your emonTx is sending data according to this:

[[15]]
  nodename = emontx3cm15
  [[[rx]]]
    names = MSG, Vrms, P1, P2, P3, P4, E1, E2, E3, E4, T1, T2, T3, pulse
    datacodes = L,h,h,h,h,h,l,l,l,l,h,h,h,L
    scales = 1,0.01,1,1,1,1,1,1,1,1,0.01,0.01,0.01,1
    units = n,V,W,W,W,W,Wh,Wh,Wh,Wh,C,C,C,p
    Whitening = 1 [Whitening = 0  for all _rfm69n variants.]

(There are 40 bytes there - count the decimal numbers that follow “OK 7” but not the one in brackets, like in the example just above ‘If it carries on…’)

If you have 40 bytes, copy and paste the above into your emonhub.conf file (via Emonhub →Edit config in emonCMS), changing [[15]] to the Node ID you want: [[7]] for the first, and make sure there are no more entries with the same Node ID in the file, i.e. delete or comment out the old Node 7 entry with a # at the start of each line.

If your emonTx is not sending 40 bytes, tell me how many as I’ll need to think again )

John_White:

What is the data type “E1, E2, E3, E4” ??

MSG is the serial number of the message
Vrms the voltage,
P1 - P4 the powers in watts
E1 - E4 the corresponding accumulated energies in watt-hours (not kWh).
T1 - T3 the temperatures
pulse the accumulated count of pulses.

John_White · 2023-07-13T14:14:18.409Z

Hi Robert
Thanks for the detail in your reply.

I have tried to use the Serial Monitor method and can’t seem to get it to respond to commands (I have stopped emonhub) screenshot below

image474×511 21.5 KB

So I surrendered and am trying to go the Arduino route. The file I believe I should be using is “emonTx34_CM.ino” I have changed the Node id, the Saved then tried to compile and no matter what I do I can’t seem to get past this error

image862×110 4.24 KB

The file structure i have is below

I must be missing something obvious here I think ??
Any thoughts or pointers ?

regards the Raspberry Pi imager > yes I did pre-set preferences before hand as follows
allow SSH
Username & Password
Configure wireless LAN
Local settings Australia - keyboard US
I hope this is what you were asking
John

Robert.Wall · 2023-07-13T15:09:14.178Z

John_White:

I have tried to use the Serial Monitor method and can’t seem to get it to respond to commands

It appears Trystan has stripped out all the calibration commands and changed most of the few remaining configuration commands (except ‘?’) for that sketch since I wrote it - and not told anybody. The new code is ‘reverse Polish notation’: 7i
And it replies i7. How perverse can you get?

John_White:

I must be missing something obvious here I think ??

Yes, I think you’re missing a new file, required for the Low Power Labs library, that hasn’t been added to the ‘Learn’ documentation yet: “RFM69.h”
N.B. This implies you’re using the new LPL radio format, not Jeelib’s ‘classic’ format.
Get it from GitHub - LowPowerLab/RFM69: RFM69 library for RFM69W, RFM69HW, RFM69CW, RFM69HCW (semtech SX1231, SX1231H) and install it in its own directory under your emonTx34_CM/libraries directory. Then it should compile.

Thanks for the settings for the Imager - my colleague @Bill.Thomson is going to look into this.

John_White · 2023-07-14T00:10:37.769Z

Hi Robert
I have made progress by putting that extra library in and when I received a similar error for SPIflash, I searched and found a library for that as well and then I am now stuck on this

image1233×391 15.4 KB

Could you help with another pointer please.

Also I note in the code the following line
if (digitalRead(DIP_switch1)==ON) EEProm.nodeID++; // IF DIP switch 1 is switched on (LOW) then add 1 from nodeID

I assume this means that I can limit the number of different sketches I need to make i.e. if I make a sketch for Node 10 then I can just use the switch on the board to get a node 11 ??

Thanks again John

Robert.Wall · 2023-07-14T01:00:59.924Z

John_White:

Could you help with another pointer please.

Did you have both files in the download (emonTx34_CM.ino and emonTx34_CM_config.ino) in the Arduino IDE when you compiled? If you look in the second, those functions are in that file.

John_White:

I assume this means that I can limit the number of different sketches I need to make i.e. if I make a sketch for Node 10 then I can just use the switch on the board to get a node 11 ??

Yes, so one sketch for 7 & 8, one for 9 & 10, one for 11 & 12, and one for 13.

John_White · 2023-07-14T01:33:29.393Z

Hi Robert

thanks for that, it has now compiled.

Final question i hope is which board should I choose, my guess is ""Arduino uno"or "“Arduino Mega ADK”

John_White · 2023-07-14T05:24:56.929Z

Hi Robert
I have found the Board Manager and installed what I think is correct and selected “Atmel atmega 328p Xplained mini” - correct?? ( I found this name on the board under the battery case.
I then thought I should close Arduino and reopen and then re-compile - unfortunately I have now got another error

image1290×429 23.3 KB

Going back to the Emonpi - what is the largest SD card it will support 64Gb or 128Gb smaller cards are getting hard to buy.
Thanks John

Robert.Wall · 2023-07-14T11:28:53.057Z

John_White:

which board should I choose, my guess is "“Arduino uno”

It’s Arduino Uno, not Mega.

John_White:

unfortunately I have now got another error

I’ve not seen that before - when I originally wrote the remote configuration, it would compile with no errors. I’ll need to look into this.
(It would be a lot easier for me if you could copy a screen grab rather than posting an image.)

[edit]
The emonTx V3.4 sketch released alongside the software for the emonPiCM The emonPiCM compiles perfectly for me. This leads me to suspect that your problem was introduced when Trystan made the Low Power Labs version.
[/edit]

John_White:

Going back to the Emonpi - what is the largest SD card it will support 64Gb or 128Gb smaller cards are getting hard to buy.

I can’t answer this - the Raspberry Pi website is surely the place to look. I suspect it will depend on which model of Pi you have.

John_White · 2023-07-14T11:54:56.463Z

Here is the Errors in txt
C:\Users\John\AppData\Local\Arduino15\packages\arduino\hardware\avr\1.8.6\cores\arduino\main.cpp: In function ‘main’:
c:\Data\OpenEnergyMonitor\emonTx34_CM\libraries\EmonLibCM\emonLibCM.cpp:415:20: warning: iteration 5 invokes undefined behavior [-Waggressive-loop-optimizations]
wh_CT[channel] = _wh;
^
C:\Data\OpenEnergyMonitor\emonTx34_CM\emonTx34_CM_config.ino:417:25: note: within this loop
for (byte n=0; n<6; n++)
^
C:\Data\OpenEnergyMonitor\emonTx34_CM\emonTx34_CM_config.ino:340:45: warning: ‘k2’ may be used uninitialized in this function [-Wmaybe-uninitialized]
case 1 : EmonLibCM_setPulseMinPeriod(k2);
^
C:\Data\OpenEnergyMonitor\emonTx34_CM\emonTx34_CM_config.ino:157:10: note: ‘k2’ was declared here
double k2, k3;
^
C:\Data\OpenEnergyMonitor\emonTx34_CM\emonTx34_CM_config.ino:309:123: warning: ‘k3’ may be used uninitialized in this function [-Wmaybe-uninitialized]
Serial.print(F(“Cal: k”));Serial.print(k1);Serial.print(F(" “));Serial.print(k2);Serial.print(F(” "));Serial.println(k3);
^
C:\Data\OpenEnergyMonitor\emonTx34_CM\emonTx34_CM_config.ino:157:14: note: ‘k3’ was declared here
double k2, k3;
^
Sketch uses 31964 bytes (99%) of program storage space. Maximum is 32256 bytes.
Global variables use 1219 bytes (59%) of dynamic memory, leaving 829 bytes for local variables. Maximum is 2048 bytes.

Robert.Wall · 2023-07-14T12:01:26.259Z

Those are warnings, not errors. Provided the input you give it is legal and in the correct format, all should be OK.

John_White · 2023-07-17T07:26:58.753Z

Thanks Robert
I have proceeded and tired it on 2 of the emonTx 3.4’s and they do not seem to work. The red light on the box now are different - it flashes ON 4 sec OFF 12 continuous.

I don’t know if this means anything but when I have gone to upload the firmware it seems to freeze everytime the first time. Then I stop it and do it a second and it will go straight thru.

I will be off line for a day or more so I won’t be able to answer, just so that you don’t think I have lost interest. I will reply as soon as back.

Robert.Wall · 2023-07-17T07:56:11.217Z

Unfortunately, because of a naming conflict, I can’t compile Trystan’s version of the sketch without totally rebuilding my system, so I’m restricted with the amount of support I can give you.

John_White · 2023-07-20T13:22:35.315Z

Hi Robert,
I have been able to solve my issue with node numbers. I went thru and re-updated each of my emonTx’s (to jeelib classic v2.4.0) by using the Raspberry Pi box that was updated to the latest using the update facility after reinstalling the 24Nov22 image. I then, using the install of Arduino IDE installed on my laptop used it’s Serial Monitor facility and it worked enabling me to setup nodes 8 - 12 saving each time to epromm.

I have taken a snapshot from the emonhub.log for each of the nodes as well as the Pi and 1 x Temp/humidity unit.
The voltage for Node 5 is roughly correct @ 232.55v (actual 241v) however the Tx boxes all report approx. 133 - 138v. which would seem well beyond a calibration issue.

Beyond this all of the nodes seem to be reporting very simular values for P1-4 & E1-4.

I have read a fair amount of docs on the website instruction section and I can’t see where to read to resolve the issue.

Beyond this I can’t see and definitive instructions for setting up the feeds with the continuous monitoring code.
I have attached a txt file sample of emonhub.log to explain.
John
emonhubsamplelog.txt (13.1 KB)

Robert.Wall · 2023-07-20T15:06:25.283Z

John_White:

the Tx boxes all report approx. 133 - 138v. which would seem well beyond a calibration issue.

I don’t see how it can be anything other than a calibration issue if the same emonTx’s were reporting near correct voltages previously, and provided they are using the same a.c. adapters.
Does the emonTx still report the same calibration values as it did here EmonTX 3 update to continuous Monitoring - #3 by John_White
The voltage calibration there is for the UK “Ideal Power” adapter. If that’s what you have still, it should be correct.

Have you changed the DIP switch to “USA” - this could explain it (it grossly affects the calibration, because the 120 V USA adapter outputs roughly, not exactly, the same voltage—which is what the emonTx measures—for 120 V as your Jaycar Australian one does for 240 V). Even, does the new sketch use the DIP switch? Can you provide the exact link to the sketch you downloaded, please, or even drop all the sketch files (.ino suffix) into your reply?

There’s got to be something wrong with the sketch - or Trystan’s made something not backwards-compatible when he changed everything round for LPL.

John_White:

Beyond this I can’t see and definitive instructions for setting up the feeds with the continuous monitoring code.

This should be the same - but until I see the sketch and know what it’s doing, I can’t be sure. The big change is you now get energy accumulated in the emonTx. You can either do as before and calculate watt-hours in emonCMS from the powers, or you can use the accumulated energy (‘E’ in Wh) values your emonTx now provides. Both should be the same - it largely depends on continuity of power, radio signals and your personal preference which you use - or you can record both and compare!
Voltages and powers should be identical to what you had previously, once we’ve sorted the calibration problem.

John_White · 2023-07-22T09:11:45.929Z

Hi Robert
To make sure I was giving you accurate information, I decided to start from scratch.

downloaded emonSD-10Nov22
used balenaEtcher to image a 32Gb sd card
inserted card started Pi registered and logged in
noted emonhub not recieving data from node5 pi
created node5 in emonhub config (attached)
decided to update as I had done previously using “Admin” “Update” “Update All” Log attached
update.txt (14.1 KB)
update_1.txt (63.4 KB)
emonhub not seeing Nodes at all log attached
emonhub.txt (16.6 KB)
updateEmonTx3.txt (4.6 KB)
emonhub_config.txt (6.0 KB)

I must be doing something really basic wrong but I can’t figure it out - FYI I have tried the above on 2 Pi boxes same result.

John

Robert.Wall · 2023-07-22T14:21:03.466Z

John_White:

I decided to start from scratch.

So everything I’ve done up ‘til now was a waste of time? Charmin’.

Why did you destroy the emonCMS, which was working, instead of sending me the details of the sketch inside the emonTx’s, which is where the source of the calibration problem was?

Let’s get a few basics understood.

The emonTx and the “front end” of the emonPi are essentially the same hardware.
Both send data serially to the Pi part of the emonPi, the emonTx by radio where it’s received by the ‘emon’ part and merged with the ‘emon’ part’s data, both then make their way into emonHub.
The quantity of this data and the format has evolved over time, so the contents of the data packet from an emonTx V2 is significantly different to that from an emonTx V3.4.
emonHub receives the data as byte stream. It requires a description of the order, sizes and format of the values and interprets the serial data packets based on this.
4a. If the description is precise and the number of bytes does not match, the data is rejected.
4b. If the description is “open”, then emonHub makes assumptions and the data might or might not be interpreted correctly.
The data description is in the file emonhub.conf.
5a. When you update emonCMS, this might be updated to have the latest formats & descriptions, in which case these won’t match the data from “legacy” sensor nodes (principally emonTx & emonTH).
5b. When you write a new image on the SD card, your old emonhub.conf is lost and you have only the latest formats & descriptions, and these will not match the data from “legacy” sensor nodes.
I don’t know the conditions when the sketch in the ‘emon front end’ is updated or not. I believe it is not now always automatically updated.

John_White:

decided to update as I had done previously using “Admin” “Update” “Update All” Log attached

I’m no emonPi expert, but I can’t see anything in there to indicate that the install and update was not successful.

John_White:

emonhub.txt (16.6 KB)

Lines 73 - 75 tell me it failed to establish serial comms with the ‘emon’ front end. emonHub.conf needs changing - it is probably the baud rate that’s wrong for the speed the sketch inside the front end is using.

2023-07-22 09:00:58,289 INFO     MainThread Creating EmonHubJeeInterfacer 'RFM2Pi'
2023-07-22 09:00:58,291 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 38400 bits/s
2023-07-22 09:01:00,294 WARNING  MainThread Device communication error - check settings

You won’t get very far until you get rid of this problem and data starts to flow.

This is the interfacer inemonhub.conf:

    ### This interfacer manages the RFM12Pi/RFM69Pi/emonPi module
    [[EmonPi2]]
        Type = EmonHubOEMInterfacer
        [[[init_settings]]]
            com_port = /dev/ttyAMA0
            com_baud = 38400
        [[[runtimesettings]]]
            pubchannels = ToEmonCMS,
            subchannels = ToRFM12,

Did you “downdate” your front end software in the emonPi to “RFM69 JeeLib Classic”, if that’s what you’ve now loaded into all your emonTx’s?
Or maybe I should ask, why don’t you want to use the LPL version in your emonTx’s, consistent with the CM front end receiving LPL in your emonPi?

John_White · 2023-07-23T09:39:15.165Z

Robert.Wall:

So everything I’ve done up ‘til now was a waste of time? Charmin’.

Why did you destroy the emonCMS, which was working, instead of sending me the details of the sketch inside the emonTx’s, which is where the source of the calibration problem was?

The main reason I started from scratch was that after shutting down and restarting the Pi nothing was talking to EmonHub. I obviously did something before shutting down but I couldn’t figure it out. I put that SD card aside and purchased 2 more cards to go step by step to try and figure out what went wrong. That was the primary reason I couldn’t answer your question. Apologies for pissing you off.

The reason I had stuck with the Jeelib Classic for the EmonTx was I did not understand that my emontx’s were compatible with the LowPowerLabs.

Thanks for the walk thru the basics - this helped me to understand the possible locations of my issues.

After loading the emontx’s with the LowPowerLabs firmware and also updating the emonPi firmware to the same + changing the [[EmonPi2]] com_baud to 115200 + adding a “nodeids = XX” to each node config everything now works again.

All of the voltages are now within 5% which I have adjusted a few and they are now close. I don’t know if it is the correct way but I did it in the scales section for each node config. I have checked some of the P1-4 and they are within range. I will have to go thru all of the nodes one by one.

Some of the P1-4 readings are negative even though I have double checked the direction of the CT clip. My solution has been to make the scale value “-1”. this has worked but the only other solution I tried was to put the clip on backwards which also worked. I had to also set the E1 scale to “-1” as well.
Not sure what is best practice ??

Question - regards the E1-4 readings, as they keep accumulating is there some way to zero them all to make a starting point (i.e. 31 July) for the data ?

Is the MSG data simply to record the number of lines of data collected ?

thanks for your help

Robert.Wall · 2023-07-23T14:24:35.642Z

John_White:

Not sure what is best practice ??

Whatever works for you. I would favour getting it right at source, i.e. reverse the c.t. on its cable; but if it’s hard to get at, then multiplying everything that depends on the current being the correct sign (power, energy) by -1.0 (or whatever fraction you need for accurate calibration) is ok.

Note that if you scale the voltage(s) in emonhub.conf to read correctly, this does not affect/change the calibration of power or energy. If only for this reason, it’s best to get the calibration right in the sketch with the voltage and current amplitude calibration constants, and leave emonhub.conf 'scales = ’ set to 1.0

John_White:

Question - regards the E1-4 readings, as they keep accumulating is there some way to zero them all to make a starting point (i.e. 31 July) for the data ?

I don’t know. I thought I had included this in a version of software that I’d written, but I can’t see it now.
The API function certainly exists in emonLibCM (and in emonLibDB) to set the value to any valid number (including zero).

John_White:

Is the MSG data simply to record the number of lines of data collected ?

Yes - or to know how many have gone AWOL.