The emonPiCM

The history of emonLib, and why we want Continuous Monitoring

The original emonTx was designed as a small, battery-powered device for monitoring whole-house power, which transmitted its data to the emonGLCD – a liquid crystal display. As such, it was essential that it operated at minimum power and the “discrete sample” monitoring mode was created (though it was never called that until much later). In this mode, current is sampled for approx. 200 ms, and the data sent by radio to the GLCD. The emonTx then goes to sleep for about 10 s, using very little power, before repeating the measurement sequence.

This works surprisingly well provided the power demand doesn’t fluctuate rapidly. But there’s a problem if a load is switched on and off rapidly, with only a few seconds in either state (e.g. an oven or hob). If the load is switched on while the emonTx is “asleep”, the energy consumed during that interval won’t be recorded, hence the average power measurement will be incorrect.

Converting the emonPi to continuous monitoring

A few years before emonLibCM became standard for the emonTx, the emonTx V3 had been introduced. V3 uses the RFM69CW radio, which is capable of handling the transmitted data as one complete message. The emonTx 3-phase sketch needs to use this capability because the processor is too busy taking and processing measurements to be able to handle the radio traffic one character at a time. Neither the 3-phase sketch, nor the standard emonTx sketch, now use JeeLib, but the transmitted message is totally compatible with JeeLib – or JeeLib in compatibility mode, as the mode for the original message format is now called.

Implementing continuous monitoring on the emonPi is difficult because of the need to handle incoming radio traffic while continuously monitoring two power channels (and optionally, reading as many as 6 temperature sensors.) The RFM69CW employs what is called “packet mode”. The transmitter can accept the entire message into its internal buffer, then transmit it independently of the processor. Meanwhile, the receiver can, independently of its processor, receive, validate and hold the message, ready for processing when the processor can handle it. What JeeLabs could not predict when they created the ‘classic’ JeeLib packet format was that Hope’s packet format for the RFM69CW would be different, making it impossible to use the RFM69CW’s ‘native’ format, yet be compatible with anything using the ‘classic’ JeeLib format.

Therefore, if you want to use emonPiCM and other sensing nodes sending their data by radio, there is no choice, the format that OEM uses must change. If you only have an emonPi, you can use CM if you wish, there is nothing else to change. If you have other nodes sending by radio and don’t want or need CM on the emonPi, or you use an emonBase, there’s no need to change.

Options

There appeared to be three main options:

  1. Use the LowPowerLabs library. It is well-established, it lives up to its name by using a low data rate of 4799 bits per second, which allows the transmitter power to be turned down. That’s eminently suitable for environmental monitoring, but the regulations regarding the use of the radio band mean that the emonTx running the standard CM sketch would be breaking the law if it transmitted more than once every 8.5 s.

  2. Write our own software library. As mentioned above, both the emonTx 3-phase sketch and the standard emonTx CM sketch use our own transmitter library, but an equivalent receiver library has not been written.

  3. Use the JeeLabs RFM69 driver. This is, compared against JeeLib, a very simple piece of software and is easy to understand. It has a couple of shortcomings: it does not check that the channel is free before transmitting, and there was a bug in the receiver code that resulted in an incorrect RSSI value being returned.

Decision

On the basis of these factors, the decision was made to go with the JeeLabs RFM69 driver, patched to correct the RSSI bug, for the emonPi and emonTH. For the mains-powered emonTx and emonTx Shield, we modified the OEM transmitter software to check that the channel is clear before transmitting.

The emonPiCM front-end sketch

The sketch that runs in the Atmel ATMega 328P primarily provides the energy monitoring function and handles the radio traffic coming in from other sensor nodes.

It uses emonLibCM, set up for two current channels, to obtain and process current and voltage readings and, from external sensors, temperature readings and the pulse input.

It is now possible to fully calibrate the emonPi. Calibration commands come in via the serial data connection from the Raspberry Pi. The interface has been designed so that these commands can be generated manually and routed to the front end via the Raspberry Pi, and so that it might be possible at some future time to set up and calibrate the front end via the emonHub/emonCMS GUI.

emonPiFrontEndCM.zip (55.6 KB)

MD5 Hashes:
emonPiFrontEndCM.zip 9de88226b6376cdaab49e5f97bcfcbab
emonPiFrontEndCM.ino 6be0b5c1c7a4d305b3970ed0d6e0806a
emonPiFrontEndCM_config.ino 7c2e2ba7937f5b7de47d9fecb6a92ff8
emonPiFrontEndCM.pdf 9dbe63138121b5842bb5eb1779a1f6be

Loading the front-end sketch

The file must be compiled with the “Export compiled binary” command and the resulting .hex file (NOT the one having with_bootloader in the name) transferred to the emonPi. The procedure to get it from there into the “emon” front-end processor is given in section 8 of the emonPi Wiki: EmonPi - OpenEnergyMonitor Wiki

Supporting sketches for monitor nodes

The sketches listed below are available. They send r.f. data in the new ‘RFM69 native’ format to the emonPiCM. The single-phase energy monitoring ones have also been updated to use emonLibCM and include a standardised on-line configuration and calibration section.

emonTx V3.4 single phase: EmonTxV34CM_rfm69n.ino
EmonTxV34CM_rfm69n.zip (71.4 KB)

MD5 Hashes:
EmonTxV34CM_rfm69n.zip 034f48fa8c65e7e3587f3359841539d6
EmonTxV34CM_rfm69n.ino 92f516f0547c5d440ae8a867d17cde8d
EmonTxV34CM_rfm69n_config.ino c0cb3167db3c7ae161f91e305cd57030
EmonTxV34CM_rfm69n.pdf 4e14bb8b09deac0a173eb669b064f0f7

 

emonTx Shield single phase: EmonTxShieldCM_rfm69n.ino
EmonTxShieldCM_rfm69n.zip (69.0 KB)

MD5 Hashes:
EmonTxShieldCM_rfm69n.zip f3c59f2fec041597c70912a557312acd
EmonShieldCM_rfm69n.ino 96ab9d0359ad61f7bdda81414f521fc1
EmonTxShieldCM_rfm69n_config.ino d0faf73e18e7a715a2b318423eb97f38
EmonTxShieldCM_rfm69n.pdf ef9be26c14c035e01e08903e69dbc439

 

emonTH: EmonTH_V2_rfm69n.ino
(note: This sketch offers no protection against r.f. collisions – protection may be enabled but battery life is shortened, possibly to below 75%, depending on r.f. traffic)

EmonTH_V2_rfm69n.zip (54.8 KB)

MD5 Hashes:
EmonTH_V2_rfm69n.zip d5bbacbbbeaa93d15daf30d9fb1055e0
EmonTH_V2_rfm69n.ino e420a993e2ccffcef19dbcd113a9a538
EmonTH_V2_rfm69n_config.ino a426e90d982a49c30d3140fe86bce2c3
EmonTH_V2_rfm69n.pdf cb5488cb4ee5811326ac7a3e28161dcd

 

emonTx V3.2: EmonTxV32CM_rfm69n.ino
EmonTxV32CM_rfm69n.zip (45.4 KB)

MD5 Hashes:
EmonTxV32CM_rfm69n.zip 9d8091c2219481c99fecbf824c91b0a9
EmonTxV32CM_rfm69n.ino 0266d4f8a5ccbe4f123674fa10a6e11b
EmonTxV32CM_rfm69n_config.ino 8be37cb658ea94664924212f7ce2c4e0
EmonTxV32CM_rfm69n.pdf 356a6735361c56a814fa22b9f0f8bcb2

 

emonTx_V2 (RFM12B or RFM69CW): emonTx_V2_CT123_Voltage_Temp_Pulse_rfm69n.ino
emonTx_V2_CT123_Voltage_Temp_Pulse_rfm69n.zip (59.5 KB)

MD5 Hashes:
emonTx_V2_CT123_Voltage_Temp_Pulse_rfm69n.zip 39e3e4a5ecdabb2bf54481259941c47e
emonTx_V2_CT123_Voltage_Temp_Pulse_rfm69n.ino df10b3a4e87d1f0f0e1e47501f0cfe50
emonTx_V2_CT123_Voltage_Temp_Pulse_rfm69n_config.ino 8bdaf7b9998734cbd6b7f2b9dce7511e
emonTx_V2_CT123_Voltage_Temp_Pulse_rfm69n.pdf fe88dcd340dfcbbc07cce4b2db1dd02f

 

emonTx 3-phase emonTx_3Phase_PLL_rfm69n
(all hardware variants, both Classic and RFM69 Native formats)
See:

GLCD

Existing sketches for the GLCD can be recompiled using a patched version of JeeLib. See below.

EMONBASE

There should be no need for an emonBase to use the new ‘RFM69 native’ format – it will only be required if and when the 'RFM69 native’ format becomes the standard. Until then, the only use case will be if it replaces an emonPiCM, thereby removing the need to convert the monitor nodes back to the ‘RFM69 classic’ format.

emonBase_rfm69n.zip (42.2 KB)

MD5 Hashes:
emonBase_rfm69n.zip 36c77071de009140440e51245dde66f1
emonBase_rfm69n.ino 1ee9c827c85e76762f37ff80067dfee7
emonBase_rfm69n_config.ino b01d0e4d902aa29f74a621c05d8dd562
emonBase_rfm69n.pdf 860269ae8f68f506303a85c13005b49a

 

Required Libraries

emonLibCM

 

emonEProm.zip (68.1 KB)

MD5 Hashes:
emonEProm.zip c1a8edaf7a180e28a193dbba2fee89de
emonEProm.cpp da16b3739adcf6a0f720051e7e08c149
emonEProm.h 66d5cbbf4cc932c80279acdcbae15289
emonEPromLib.pdf a5c5695012ad958453db11c482c91f4a

 

rf69.zip (38.0 KB)

MD5 Hashes:
rf69.zip 70ec1533b6d45a9c38716a8ae73b4ccf
rf69.h 8ce7152c65b7191cf1c34775e1af3507
spi.h d82e7b87d107d902025fb1b10645e3b3
rf69.pdf 679e6240c03ea6c4447451c3dee96f5b

 

rfm69nTxLib.zip (62.2 KB)

MD5 Hashes:
rfm69nTxLib.zip 463ffcb0fbd1e05c16882435e83a9999
rfm69nTxLib.cpp cea9746029b7a530c5fd940984ff8124
rfm69nTxLib.h 4382e1f74f0338791aeca77cece72f5e
rfm69nTxLib.pdf 28675fa24962f35611ba70483baa66eb

Libraries used by each sketch:

emonPi front end:

  • emonLibCM
  • emonEProm
  • rf69

emonTx V3.4 (single phase),
emonTx Shield (single phase),
emonTx V3.2 (single phase),
emonTx_V2: (single phase):

  • emonLibCM
  • emonEProm
  • rfm69nTxLib

EmonTH V2:

  • *power
  • *sleep
  • rfm69nTxLib
  • emonEProm
  • JeeLib (JeeLabs)
  • OneWire (Paul Stoffregen)
  • DallasTemperature (Miles Burton)

emonLibCM requires:

  • *Arduino
  • *Wprogram
  • *SPI
  • *crc16
  • OneWire (Paul Stoffregen)

emonEProm requires:

  • *EEPROM

* This is a standard Arduino library.

Sketches for emonGLCD

It is possible to convert JeeLib to operate in RFM69 Native mode, however great care must be taken as it means making a change to one of the JeeLib files itself. When this is done, most of the emonGLCD sketches should work with sensor nodes and emonPi that are using the RFM69 ‘native’ message format.

In the file RF12.h, near the top, you will find these lines:

/// RFM12B driver definitions

// Modify the RF12 driver in such a way that it can inter-operate with RFM69
// modules running in "native" mode. This affects packet layout and some more.

#define RF12_COMPAT 0

Change the last line to read

#define RF12_COMPAT 1

Now, for the IDE to recognise the change, you must completely exit the Arduino IDE, restart it and compile the sketch as normal.

If for any reason you need to revert to the JeeLib ‘Classic’ mode, you must restore the line to its original state and restart the IDE. Due to the way the Arduino IDE works, it is NOT possible to have two versions of the library - even in different directories.

Reference Documents

JeeLabs publications:

JeeLabs published 5 separate documents describing the RMF69 Native format. Those and a chart showing the various formats have been consolidated for convenience into a single PDF document:

JeeLib Radios RFM69 Native.pdf (859.5 KB)

The original JeeLabs documents are:

RFM69 on ATmega » JeeLabs:
https://jeelabs.org/2015/05/27/rfm69-on-atmega/index.html

Classic vs native packets » JeeLabs:
https://jeelabs.org/book/1522a/index.html

RF compatibility options » JeeLabs:
https://jeelabs.org/book/1522b/index.html

RF69 native on ATmega’s » JeeLabs:
https://jeelabs.org/book/1522c/index.html

Using RFM12’s with RFM69 native » JeeLabs:
https://jeelabs.org/book/1522d/index.html

Design of the “old” RF12 Driver:

Inside the RF12 driver » JeeLabs:
https://jeelabs.org/2011/12/10/inside-the-rf12-driver/

Inside the RF12 driver – part 2 » JeeLabs:
https://jeelabs.org/2011/12/11/inside-the-rf12-driver-part-2/index.html

Inside the RF12 driver – part 3 » JeeLabs:
https://jeelabs.org/2011/12/12/inside-the-rf12-driver-part-3/index.html

RF12 packet format and design » JeeLabs:
https://jeelabs.org/2011/06/09/rf12-packet-format-and-design/index.html

RF12 broadcasts and ACKs » JeeLabs:
https://jeelabs.org/2011/06/10/rf12-broadcasts-and-acks/index.html

Data Sheets:

RFM12B Radio Module:

RFM69CW Radio Module:

5 Likes

Thank you for posting this up @Robert.Wall I know this has been a significant amount of work for you over quite some time, with a lot of careful testing and documentation. @glyn.hudson @Gwil and I, and I’m sure everyone else here really appreciate your efforts in putting this together. Thanks again!

As a humble emonPi owner, can someone explain if all this good work is relevant to me? What prompted this effort, did something change in the emonPi hardware?

I understand the first 3 paragraphs about the emonTx. Does “Converting the emonPi to continuous monitoring” describe work that was done in the past, or a plan for the future?

‘Hope’ is the maker of the RFM radios, right? Is the RFM69CW what we have in the emonPi? What is emonPiCM and why would I want to use it?

The emonPiCM is an emonPi running Continuous Monitoring software in the “emon” front end, bringing it into line with the software that’s been standard in the emonTx V3.4 since August 2020.

I don’t understand what you’re trying to ask, or the point you’re trying to make. As the software has now been released and is available for download from the links above or from the emonPi’s Admin menu, it’s obviously in the past. For you, until you download it and use it, it is in the future. @TrystanLea has indicated that CM will become standard for emonPi’s bought from the shop at some stage - no decision has been made regarding when.

Yes to the first; yes and in the emonTx V3.4 for the second, and for the third, did you read all of the first two sections of the first post here, down to but not including “Options” ? If you’ve read that and still don’t understand, try to explain why you don’t understand.

Thanks, that’s clearer, especially the part about the new software being the future standard for emonPi.

It’s obvious to you where this project fits in because you’ve been working on it for a long time. For someone like me, coming to it cold, it would be good to start by saying “we’ve been working on some new software for emonPi which adds/improves features X, Y and Z”.

I am currently looking at deploying additional EmonTx V3.4.3 modules with the emonTx_3Phase_PLL Firmware, my existing one install is using V1.7 firmware, as the newer ones will use a different emonPi I am trying to decide if I use emonPiCM for my next install.

However if I try to build The emonTx_3Phase_PLL_rfm69n code V2.0.x for with #define RF_CLASSIC instead of #define RF_NATIVE I get

rfmTxLib.h: No such file or directory.

However I don’t see rfmTxLib in the list of libs above

what am I missing ?

rfmTxLib of course! It has been available for a long time - a year to be precise, but hasn’t been picked up by anyone and listed on Github…
I’ve added it to the 3-phase sketch thread: Update to 3-Phase PLL sketch - #22 by Robert.Wall

[Just to confirm I have this right, and hopefully save the next person some time!]

So for someone buying a brand new emonpi, and a emontx from the store today containing “latest continuous sampling firmware” and wanting to also use the emonpi in continuous sampling mode in, the steps to enable it are:

  1. Buy the USB to UART lead for flashing emontx

  2. Update emonTX to the “rfm69nTxLib” firmware (incompatible radio format with historic jeelib format)
    (ref. Update to 3-Phase PLL sketch - #22 by Robert.Wall)

  3. Flash emonpi to the “Native RFM68 (New)” Firmware
    (in the web U emonpi/admin/update)

  4. Edit the emonpi emonhub config so that node [[5]] contains the following:
    (in emonpi/config)
    → Edit Confg
    (ref emonPiFrontEndCM/emonPiFrontEndCM.ino at main · openenergymonitor/emonPiFrontEndCM · GitHub)

### For the "New" RFM69 firmware supporting continuous monitoring
[[5]]
    nodename = emonpi
  [[[rx]]]
       names = power1,power2,power1pluspower2,vrms,t1,t2,t3,t4,t5,t6,pulse1count,pulse2count,E1,E2
       datacodes = h, h, h, h, h, h, h, h, h, h, L, L, l, l
       scales = 1,1,1, 0.01, 0.01,0.01,0.01,0.01,0.01,0.01, 1, 1, 1,1
       units = W,W,W, V, C,C,C,C,C,C, p, p, Wh,Wh

As an aside, I’ve notice the emonpi “Update Firmware” option does not seem to be entirely sticky. After a few reboots it seems to sometime regress back to the old firmware, and re-updating it to the new one is needed to fix it. Is that a known issue / expect?

Thanks!

Welcome, Jonatan, to the OEM forum.

Until the CM software is officially adopted for the emonPi, then I’m afraid you’ll need to continue to rewrite the “emon” CM software in the Atmel '328P front end, because the default is to always overwrite it with the “new” update (but it isn’t ‘new’ to you, it’s an update to the discrete sample version).

I don’t know what’s on Github (I’ve tried but I can’t understand it), the release and as far as I know, the up-to-date version is here: The emonPiCM

Note, ANY sensor node working to the emonPiCM will need its software changing, according to that post.

I don’t think there’s any problem with the CM software, however it might be a good idea to fire up your system using the installed software, just to prove it works, before you change anything.

Also note, I can only support the Arduino IDE for compiling and uploading the sketches.

So:

  1. Buy the USB to UART lead for flashing emontx - Correct

  2. Update emonTX to the “rfm69nTxLib” firmware (incompatible radio format with historic jeelib format)
    (ref. Update to 3-Phase PLL sketch - #22 by Robert.Wall) Only if you have a 3-phase supply. You haven’t put where you are in the world, so I can’t guess.

  3. Flash emonpi to the “Native RFM68 (New)” Firmware
    (in the web U emonpi/admin/update) I don’t know that this is the correct front end software, I proceed as I’ve noted in the post where the software was released - linked above, using Remmina (other tools are available) to transfer the file to the emonPi.

  4. Edit the emonpi emonhub config… You’ll find details of the config file entry at the top of each sketch - for the sensor nodes as well as the emonPi’s front end itself.

Thanks for the speedy reply! For full context, I already had the emonpi using continuous sampling mode (via that "Native RFM68 (New)” Firmware) fine for several months. I just got the emonTx this week (happy to see it had the “CM fw” preinstalled!) and was scratching my head for a while as to why it would not connect! It all makes sense now I read it through over again, that CM on the Tx side does not imply “using the new protocol”. I’ve now put the emonpi back to stock FW for now and it is indeed working fine :+1:

I indeed only have single phase (in the UK) so it would be EmonTxV34CM_rfm69n I need then.

I hadn’t seen the full set of steps needed in a concise form so wanted to check the overall steps in the recipe is correct (even if I got some of the ingredients wrong :grimacing:)

FWIW my original goal is tracking PV import/export and also controlling redirect (to immersion heater) so I’m okay going back to the old radio protocol, as on balance CM isn’t quite as useful as I hoped:

  1. The E1 and E2 accumulators don’t track “import” and “export” separately, so it’s not possible to properly track how much was spent/returned on energy coming from/to grid at different rates. (I believe this would need to follow the “1 Wh bucket” principle to be representative of UK meters’ billing personality)
  2. The sampling frequency of 1/5 Hz is lower than the ideal responsiveness for PV redirect, but it had seemed rather complicated to increase it.
    I see adding RF reception (or 1-wire thermistors) is the main impediment to increasing sampling frequency. I have the option to connect the emonTx via serial cable so I might still do that anyway at some point.

Thanks!

That’s true, just as it’s true for emonLib and the Discrete Sampling emonTx and emonPi, but it would be relatively easy to separate import and export. Normally, that task is left to emonCMS. The CM library itself can report a lot faster than

(I don’t know where you got that number from). The true sampling frequency of the library depends on the number of c.t’s - for the emonPi with two, it’s around 3 kHz, the minimum reporting period is 0.1 s.

Sorry, wrong terminology! I mean the reporting frequency. I’m sending the grid import/export over MQTT to a Loxone controller where all the real logic (currently) lives, and the external messages to that are limited to 5s interval.

This is welcome news if it means that it is possible to use a burst or phase diverter (e.g. the famous Emley Mk2).
I assume this is all “tried and tested”, but I’ve not found anything via the Search to confirm this.

Could someone confirm that changing the firmware as above does indeed give a decent level of accuracy and fidelity.
Thanks in advance…

The software is exactly the same as used in the emonTx since October 2019, except that it scans 3 channels (1 voltage, 2 current) instead of 5 (1 voltage, 4 current) and thus scans slightly faster, which should lead to a small improvement where higher order harmonics are present.

It will be as accurate, or as inaccurate, as component tolerances and your calibration allows.