The emonPiCM

Hardware emonPi
10

Thanks for the speedy reply! For full context, I already had the emonpi using continuous sampling mode (via that "Native RFM68 (New)” Firmware) fine for several months. I just got the emonTx this week (happy to see it had the “CM fw” preinstalled!) and was scratching my head for a while as to why it would not connect! It all makes sense now I read it through over again, that CM on the Tx side does not imply “using the new protocol”. I’ve now put the emonpi back to stock FW for now and it is indeed working fine :+1:

I indeed only have single phase (in the UK) so it would be EmonTxV34CM_rfm69n I need then.

I hadn’t seen the full set of steps needed in a concise form so wanted to check the overall steps in the recipe is correct (even if I got some of the ingredients wrong :grimacing:)

FWIW my original goal is tracking PV import/export and also controlling redirect (to immersion heater) so I’m okay going back to the old radio protocol, as on balance CM isn’t quite as useful as I hoped:

  1. The E1 and E2 accumulators don’t track “import” and “export” separately, so it’s not possible to properly track how much was spent/returned on energy coming from/to grid at different rates. (I believe this would need to follow the “1 Wh bucket” principle to be representative of UK meters’ billing personality)
  2. The sampling frequency of 1/5 Hz is lower than the ideal responsiveness for PV redirect, but it had seemed rather complicated to increase it.
    I see adding RF reception (or 1-wire thermistors) is the main impediment to increasing sampling frequency. I have the option to connect the emonTx via serial cable so I might still do that anyway at some point.

Thanks!

11

That’s true, just as it’s true for emonLib and the Discrete Sampling emonTx and emonPi, but it would be relatively easy to separate import and export. Normally, that task is left to emonCMS. The CM library itself can report a lot faster than

(I don’t know where you got that number from). The true sampling frequency of the library depends on the number of c.t’s - for the emonPi with two, it’s around 3 kHz, the minimum reporting period is 0.1 s.

12

Sorry, wrong terminology! I mean the reporting frequency. I’m sending the grid import/export over MQTT to a Loxone controller where all the real logic (currently) lives, and the external messages to that are limited to 5s interval.

13

This is welcome news if it means that it is possible to use a burst or phase diverter (e.g. the famous Emley Mk2).
I assume this is all “tried and tested”, but I’ve not found anything via the Search to confirm this.

Could someone confirm that changing the firmware as above does indeed give a decent level of accuracy and fidelity.
Thanks in advance…

14

The software is exactly the same as used in the emonTx since October 2019, except that it scans 3 channels (1 voltage, 2 current) instead of 5 (1 voltage, 4 current) and thus scans slightly faster, which should lead to a small improvement where higher order harmonics are present.

It will be as accurate, or as inaccurate, as component tolerances and your calibration allows.

15

Hi Robert,

First, I want to say great work - specifically on joining all the dots on the radio formats. It’s great to be able to use RFM69 in it’s native mode which gives so many benefits. It was also fantastic to read all the information on how RF12 can be made to use the same RFM69 packet format, allowing us to have a mixed network of old & new with simple firmware update.
The great thing was you’ve provided a way to update all the OEM devices including GLCD to be able to continue to function. It really is an amazing effort, never mind the benefits of CM.
The little atmel chip can breath a nice sigh of relief now sending and receiving is offloaded to the RFM69… although I’m sure it is now sweating more doing CM instead!

However, I am confused as to why you moved away from the established command format , (reverse polish) to the extent you lock it out when you see it on the input stream.
I was looking at the code making sure I could still transmit data, and instead of being able to send ‘bytes’ as previously, it seems to send alphanumeric strings which is likely not what most applications need, nor is it compatible with the previous raspberry-atmel interface, making this not ‘plug and play’ with current emonhub send functionality.

Previously, we would send by sending decimal numbers between 0-255 , comma-separated, followed by ,s eg. 12,122,244,33,7,s would send a 4 byte packet to node 7.
I was looking at submitting a patch to put this functionality back so EMONGLCD (and other sending tools) could run without changes to emonhub or how they send, but I thought I’d ask first as it seems to have deliberately broken this backward compatibility and not sure the rationale why.

I use emonhub as my RF hub for sending and receiving to lots of JeeNodes, so this functionally is very useful (it saves me have yet another node connected for TX to the RF network when the emonpi is perfect for it).

Looking forward to understanding your thinking on this, and again, GREAT work… your research and fantastic explanation has helped me start to figure out how to get STM32 talking to my RF network in a compatible way.

Alan

16

I’d like to be able to answer properly - but to do that will mean taking me away from the emonTx V4 and looking back quite a long time to see where the Reverse Polish actually was, and what I did, which might give me a clue as to why. You are of course free to develop, and if you wish publish, your own version.

17

No problems Robert. I’m sorry I didn’t discover this thread sooner. It would have saved me a lot of time (thankfully only thinking time, not coding).

I’ll take a crack at getting your code to accept the previous notation for sending only so it can be plug and play. Do you have it on GitHub somewhere so I can clone it and then send a pull request if I get that far ?

Thank you again
Alan

18

No, it will never go on Github. I wasted far too much time trying to understand it, finally giving up. I see no advantage to me in it. If it’s on Github, I had no part in it. If you want me to comment or consider a change, the place to do so is on the release Topic or associated comments here.

19

Having looked again at your post, it looks as if you’re totally confused. Nothing has changed in the way the data is presented to the RFM69CW. Just as before, it accepts a message meaning whatever you want it to mean, the only change (which makes Native and Classic formats incompatible) is in the header, as is explained above and in the supporting documentation. Nothing affects the message you send to a JeeNode - provided that it too can receive and handle the Native format.

What has changed is the format of the command string to calibrate the emonTx and associated devices - with the notable exception of the emonGLCD software, which your complaint wrongly states has changed. The RP notation cannot be handled by the calibration software and sending a digit first would cause problems, this is why it is blocked. This was after consultation with Glyn & Trystan and with their full support and agreement.

20

Hi Robert,

We are talking about emonpiCM, correct ?
The emonhub sends commands today in the reverse polish format, notably using the ‘s’ postfix. This can easily be validated by looking about at the python code.
Currently you send a MQTT msg to emonhub/tx//values/msg/ with the BYTES you want to transmit, encoded specifically according to the emonhub TX definition.
Your change breaks this as it is no longer accepted, nor does it accept a decimal string which is converted into bytes for onward transmission.

I refer you to lines 235 - 256 of EmonHubJeeInterfacer.py for TX. In fact looking at the entire file it relies on the current implementation for any commands SENT to the uC.
This breaks inter-operability and will require changes to this code and additionally any existing software that uses it for tx. Not great I’m sure.
But, I think you’ve misunderstood my comment, so please do read it again and look at the current code to see how it works. Your CM code is a great improvement so I think getting it 100% compatible is a worthy endeavour.

However, I came on here to post some other news, which is that I’m finding the RF12b sending RFM69 packets to be buggy and unreliable. The simple idea to change the define in the code and restart / recompile does not work, except in the most simplistic cases.
For example - do this in the standard rfdemo sketch and you’ll find it doesn’t work. Even after manually sorting out the rf12_initialization. In short, the ONLY way I could get it to work is comment out the PRGMEM definition for help1 text. In that case it worked (after some other prodding). However, this is without even calling the help to execute the function to display the help text, so there is a rather nasty memory corruption bug somewhere in the RF12 libs. This took many many hours to bottom out.

I then verified this by trying on my own known code (for controlling rollerblinds), which would not get past a simple f12_initialise (hung) on the same board, and it works perfectly with the old packet mode. Flipping the ```RF12_COMPAT back to zero and no issues.

With the upcoming emonpi2 release, I guess we need to figure out the way forward. I think JCW has been pretty quiet over the years and moved onto other things… embello is pretty quiet and bugs don’t seem to get fixed. In my opinion it’s a dead codebase, and even the the jeenode/jeelib website is now found mostly on the wayback machine. You are now essentially have to maintain the RF69 code.

Given my tests, that conclude for me at least that RF12 doing native RFM69 packets is far from stable and therefore inter-operability not guaranteed, it may be time to look again at lowpowerlab, which is at least developed and supported.
Changing the code for a small handful of devices would not be super difficult and the decision forced to use a different packet format made, we may as well make something that is current.

That’s my 2c on this, but at the very least, this needs some more testing to make it work with current ecosystem of things that may have RF12 onboard (which from OEM is only the emonGLCD, right ?).

I’ve spent a good few hours on this at this point and unless you want to take the decision that sending will not work without changes upstream, I’d step back and re-evaluate.

Again, sorry I’m late to this party.

21

Oh, and by the way I’m happy to help on the emonGLCD side, as I think I’m the last person who touched it.
Oddly my emonGLCDs came with RFM69 so I’m not sure if there were some shipped with RF12 originally.
Anyway, the emonGLCD suggestion above does did not work on my RFM69 based board, it failing to transmit when it was applied (not surprising in fairness).
I’m happy to workup new RF code for it as needed (20 minutes playing and I’ve got the sketch transmitting with rf69.h based code).

22

That is the title of the Topic, but also where I was not sure what exactly, in some places, the subject was, because of the rather frequent use of “it”.

I have had nothing to do with that. If you check the references, it is completely down to the work of JCW and his associates. Some of the r.f. properties of the transmission are subtly different, so a careful study of the differences between the JeeLib and the RF69 libraries might be fruitful.

I think you’re absolutely correct there. I reported the RSSI bug when I found it - the response is still awaited. Also note there are hints that some of the RFM69 settings were less than optimal - again I’m not aware of any update. So I’m not holding on to any hope of support from that direction.

As far as I’m aware, the emonGLCD is obsolete =, and has been unavailable as a self-build kit for a good while (regrettably, but I fully understand the reason it was withdrawn). Converting it to use the RFM69 Native library (for transmission, if you require the extended capabilities) and the RF69 for reception, or the RF69 for both, should be simple, I considered it not worth the effort for the few systems that might, only might, still be running and needing to be converted.

I know I soldered up two emonGLCD kits, both had RFM12Bs, I also remember a post emonGLCD as an RF signal strength meter | Archived Forum that took a wire from the RFM12B to recover signal strength. Until I read this, I was under the impression that no emonGLCD kits included the RFM69CW, and all had included the RFM12B. @glyn.hudson or @TrystanLea are likely to be the only ones to know just how many of each were shipped, and I think they also must be drawn in to give their opinion on your contribution.

23

I think we largely concur.
The rfm69 emonglcd are an easy conversion and I’ll get that done. I guess the thing we need to agree on is the sending api. I’m not sure many folks use it. A few do but I think small numbers. The fact it was broken for a few years in the most recent release of emonhub (after the python3 conversion) leads me to that conclusion.

Maybe it’s time to revisit the topic as you say. Especially for emonpi2 which uses another avr which I have no familiarity with. Or perhaps makes sense to have the bridge between the two defined and then it can be uC independent.

I’ve also never used low power lab but it kind of seems nice to have a common ecosystem. Although the rf69.h simplicity and portability are nice. Essentially any device that can drive SPI can send and receive super simply.

I’ll take a look at low power lab some more to further my own knowledge

Out of curiosity do you know if all emonth shipped with rf69 ?

Maybe we are lucky and the vast majority of oem devices are rfm69 based ?

It’s not the frequency settings is the issue it really is down to buggy code in jeelib rf12 compatibility layer.

Sorry to drag you back into this world.

24

Oh and can you confirm that your rf12 based emonglcd in rf69 mode worked (at least for it to tx it’s temp every 30 secs) ?

25

I looked very hard before deciding that it might be a case of “The devil you know…” A serious worry is the 1% limit on band occupancy and how it interacts with the very low data rate of LPL, which clearly isn’t a problem with environmental sensors where things change slowly, but which could be restrictive where energy monitoring and the much faster rates of change are encountered. I might be wrong, but from memory LPL won’t handle the RFM12B.

The real bugbear was Hope’s decision to bake into hardware a message format that was just two data bytes transposed from JeeLabs’. Most inconsiderate :face_with_hand_over_mouth:

Again, most certainly not. The original was RFµ with a RFM12B, then the RM69CW

Yes, a decision was made to switch completely to the RFM69CW, I’m not sure when as a date, but around the time of the launch of the V3.4. But I have a V3.4 with a RFM12B - maybe it was an early one for testing. The V3.2 used the RFµ328 and a RFM12B, the V2 also always had a RFM12B.

No - I can’t remember, and I don’t want to side-track even further (though the hot weather precludes logical thought - as you might have noticed.)
I too am not impressed by the patch to JeeLib, it’s implementation (forced by the Arduino IDE’s concept of a library) means editing source code to swap? Why can’t a library have a path? Utterly and totally wrong and the worst possible hack, but I recognise it might have had a certain appeal. It’s why rfm69nTxLib exists, and I don’t propose to do anything to implement a receive version. My view is the future lies with the Hope data format and the RFM69CW. To me as a systems person, it makes complete sense to hand over the entire transmission and reception path to a subsystem: put a message in at one end and get it out at the other. Handling one byte at a time was the only way with the very limited capabilities of the RFM12B, but the basic concept was flawed - because it was pushed too far. It worked with emonLib because the processor was doing nothing for 99% of the time, so it could respond to interrupts on each byte sent or received.

26

I am now more enlightened on jeelib and rfm69nTxLib.
I’ve just compared the code for the send functions and rfm69nTxLib is, well, elegant.
I can easily see the data been sent in 2byte hunks via SPI.transfer where in jeelib, it’s a bit of a mess. I think there is a bug in jeelib around a memcpy when RF12_COMPAT is enabled, but damned if I can find it. That makes sense with the behaviour I’m seeing, and as the packet is 2bytes longer with RF12_COMPAT, I think there is a bug there somewhere. But I digress.

What I’ve learnt is worth gold the latest being just why rfm69nTxLib exists. It is much cleaner than jeelib and it is possible to get a RF12 to send RFM69 native packets reliably using it. We just can’t RX using it (yet).

I just need to put pieces of the jigsaw together now… so I’m going to shut up a while as I’ve a lot to be going on with the information you’ve given me, which again has furthered my understanding yet further.

I’ll be in touch and thank you again.

Alan

27

For what its worth, when I tried to use JeeLib receiving with the emonLibCM library running (i.e. doing what the emonPi has to do - measure and receive from another node), the success rate was not good - from memory about 20% of incoming data got through. It seemed pointless to continue.

28

Nor won’t – for the very simple reason that the processor (the '328P at least) can’t avoid dropping samples or incoming bytes when it has to handle each incoming character separately, along with sampling 30 - 40 samples per 20 ms cycle across the 3 ADC channels of the emonPi, and process and send the data onwards to wherever.

And that’s ultimately why, in the absence of the chip shortages and as they mentioned, G&T were intending to go with a STM32 solution for the emonTx V2 and emonPi2, rather than being forced into the AVR-DB in order to have a product to sell.

29

It’s all making sense now.
Because as you said it has to poll and suck data out 1 byte at a time from the RF12, if it’s busy doing other things, then tough… it’ll miss all the data. Which is why now I understand the RF12 sucks for this application. As you said earlier, you want all of this TX/RX offloaded to the hardware and the AVR deals with it when it’s ready.
Good thing there is no emonpi that has a RF12 afaik.
This leaves the cases of emonglcd that needs to RX and, well, being frank, my rollerblind controller jeenodes, which at this stage I’m tempted to rip off the RF12 and replace with RFM69… but where would the fun in that be.

Since your last messages I’ve been staring at the jeelib code trying to figure out where this transmit is going wrong… and I’m not further… Bed time now though…

I’ll keep you posted as I loose more hair.