The emonPiCM

I think we largely concur.
The rfm69 emonglcd are an easy conversion and I’ll get that done. I guess the thing we need to agree on is the sending api. I’m not sure many folks use it. A few do but I think small numbers. The fact it was broken for a few years in the most recent release of emonhub (after the python3 conversion) leads me to that conclusion.

Maybe it’s time to revisit the topic as you say. Especially for emonpi2 which uses another avr which I have no familiarity with. Or perhaps makes sense to have the bridge between the two defined and then it can be uC independent.

I’ve also never used low power lab but it kind of seems nice to have a common ecosystem. Although the rf69.h simplicity and portability are nice. Essentially any device that can drive SPI can send and receive super simply.

I’ll take a look at low power lab some more to further my own knowledge

Out of curiosity do you know if all emonth shipped with rf69 ?

Maybe we are lucky and the vast majority of oem devices are rfm69 based ?

It’s not the frequency settings is the issue it really is down to buggy code in jeelib rf12 compatibility layer.

Sorry to drag you back into this world.

Oh and can you confirm that your rf12 based emonglcd in rf69 mode worked (at least for it to tx it’s temp every 30 secs) ?

I looked very hard before deciding that it might be a case of “The devil you know…” A serious worry is the 1% limit on band occupancy and how it interacts with the very low data rate of LPL, which clearly isn’t a problem with environmental sensors where things change slowly, but which could be restrictive where energy monitoring and the much faster rates of change are encountered. I might be wrong, but from memory LPL won’t handle the RFM12B.

The real bugbear was Hope’s decision to bake into hardware a message format that was just two data bytes transposed from JeeLabs’. Most inconsiderate :face_with_hand_over_mouth:

Again, most certainly not. The original was RFµ with a RFM12B, then the RM69CW

Yes, a decision was made to switch completely to the RFM69CW, I’m not sure when as a date, but around the time of the launch of the V3.4. But I have a V3.4 with a RFM12B - maybe it was an early one for testing. The V3.2 used the RFµ328 and a RFM12B, the V2 also always had a RFM12B.

No - I can’t remember, and I don’t want to side-track even further (though the hot weather precludes logical thought - as you might have noticed.)
I too am not impressed by the patch to JeeLib, it’s implementation (forced by the Arduino IDE’s concept of a library) means editing source code to swap? Why can’t a library have a path? Utterly and totally wrong and the worst possible hack, but I recognise it might have had a certain appeal. It’s why rfm69nTxLib exists, and I don’t propose to do anything to implement a receive version. My view is the future lies with the Hope data format and the RFM69CW. To me as a systems person, it makes complete sense to hand over the entire transmission and reception path to a subsystem: put a message in at one end and get it out at the other. Handling one byte at a time was the only way with the very limited capabilities of the RFM12B, but the basic concept was flawed - because it was pushed too far. It worked with emonLib because the processor was doing nothing for 99% of the time, so it could respond to interrupts on each byte sent or received.

I am now more enlightened on jeelib and rfm69nTxLib.
I’ve just compared the code for the send functions and rfm69nTxLib is, well, elegant.
I can easily see the data been sent in 2byte hunks via SPI.transfer where in jeelib, it’s a bit of a mess. I think there is a bug in jeelib around a memcpy when RF12_COMPAT is enabled, but damned if I can find it. That makes sense with the behaviour I’m seeing, and as the packet is 2bytes longer with RF12_COMPAT, I think there is a bug there somewhere. But I digress.

What I’ve learnt is worth gold the latest being just why rfm69nTxLib exists. It is much cleaner than jeelib and it is possible to get a RF12 to send RFM69 native packets reliably using it. We just can’t RX using it (yet).

I just need to put pieces of the jigsaw together now… so I’m going to shut up a while as I’ve a lot to be going on with the information you’ve given me, which again has furthered my understanding yet further.

I’ll be in touch and thank you again.

Alan

For what its worth, when I tried to use JeeLib receiving with the emonLibCM library running (i.e. doing what the emonPi has to do - measure and receive from another node), the success rate was not good - from memory about 20% of incoming data got through. It seemed pointless to continue.

Nor won’t – for the very simple reason that the processor (the '328P at least) can’t avoid dropping samples or incoming bytes when it has to handle each incoming character separately, along with sampling 30 - 40 samples per 20 ms cycle across the 3 ADC channels of the emonPi, and process and send the data onwards to wherever.

And that’s ultimately why, in the absence of the chip shortages and as they mentioned, G&T were intending to go with a STM32 solution for the emonTx V2 and emonPi2, rather than being forced into the AVR-DB in order to have a product to sell.

It’s all making sense now.
Because as you said it has to poll and suck data out 1 byte at a time from the RF12, if it’s busy doing other things, then tough… it’ll miss all the data. Which is why now I understand the RF12 sucks for this application. As you said earlier, you want all of this TX/RX offloaded to the hardware and the AVR deals with it when it’s ready.
Good thing there is no emonpi that has a RF12 afaik.
This leaves the cases of emonglcd that needs to RX and, well, being frank, my rollerblind controller jeenodes, which at this stage I’m tempted to rip off the RF12 and replace with RFM69… but where would the fun in that be.

Since your last messages I’ve been staring at the jeelib code trying to figure out where this transmit is going wrong… and I’m not further… Bed time now though…

I’ll keep you posted as I loose more hair.

Back again.

Looking at the emonhub code, it seems someone has already created a new interfacer to talk using the new command format used in your CM code. It’s called EmonHubOEMInterfacer.
However there are small issues with it - both in the send functionality which has not been updated at all and still uses the byte based approach with a ‘s’ suffix for send. I’ll take a look at that today.

You may want to add a note to your original posting and emonPiFronEndCM code where you talk about editing emonhub.conf to edit it as follows :

[[RFM2Pi]]
    Type = EmonHubOEMInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyAMA0
        com_baud = 38400                        # 9600 for old RFM12Pi
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        subchannels = ToRFM12, emonCMS

        group = 210
        frequency = 433
        baseid = 5                             # emonPi / emonBase nodeID
        quiet = true                            # Report incomplete RF packets (no implemented on emonPi)
        calibration = 230V                      # (UK/EU: 230V, US: 110V)
#        interval =  20                       # Interval to transmit time to emonGLCD (seconds)

Again, I’ll take a look at getting send working again, but we are a lot closer than I thought.

FWIW, I’ve made some progress debugging JeeLib sending with RF12_COMPAT. It seems to only be a problem in rfdemo sketch as it somehow corrupts the groupid and increments it by one when the PROGMEM help text is enabled. It’s very weird but I’m putting this down to a corner case in this codebase for now. If the bug is indeed confined to RFDEMO and not Jeelib RF12 then we have a way to send and receive RFM69-format packets using RF12 as JCW intended, however it is not suitable for CM applications due to the interrupt-driven, slow nature of both send and receive. But it does allow a way forward on the older RF12 OEM devices that may have RF12 that need to send and receive (emonGLCD for example).

[Formatted by Mod]

True (I think) but that gives the wrong impression. The RFM69CW in the emonPi NOT running the CM code uses the RFM69 as if it were a RFM12B, i.e. with a 1-byte message buffer that is emptied as a result of the interrupt generated when the byte arrives. So it might just as well be a RFM12B, which is why the emonPi won’t do CM until you swap all the software in your system and adopt the Native message format, which allows the RFM69CW to handle reception and assembly of the message into a string in its buffer all on its own.

Absolutely.
That’s the end point I’m trying to get to:

emonPi - RFM69 - RF69 Native Packets - CM firmware
EmonGLCD - RFM69 - RF69 Native Packets - rf69.h library
EmonGLCD - RF12 - RF69 Packets - jeelib library (RF12_Compat)

I think you’ve the other cases covered pretty well (emonth, emonTx).

I’m making good progress on it.
The good news is we don’t need no stinkin’ reverse polish anymore - EmonHubOEMInterfacer.py was easily modified.

I’m just working on the last little bugs in the sending and should be there shortly. The good news is it required minimal changes in your emonPiCM firmware.

When I’m done, hopefully later on, I’ll get it on GitHub so you can have a look.

Then, we should be well on our way to getting the CM firmware adopted as the ‘default’ on emonPi (well, emonPi2 at least, with an option for emonPi1 users to do it).

I presume you realise that emonLibCM for the AVR-DB will be a lot different to emonLibCM for the '328P.

I didn’t realise that (doh!) … but as long as the serial API is the same between the PI and AVR and you continue to use EmonHubOEMInterfacer.py we should all be good.
It’s not as if you are going back to using a RF12 in emonPi2, so I assume the intent to stay with RFM69 packets.

It’s quite obvious that it will change dramatically, from handling 1 voltage and 5 currents on a single phase to 3 voltages and 12 currents across three phases or a split phase.
The API will stay as close as possible to the existing - but that’s as far as it will go. For example, each c.t. will need to know which voltage or voltages its load is connected to.

Note, there’s nothing in emonLibCM (old or new, or indeed in emonLib) that interacts directly with the radios, or any other means of communication. It’s the sketch that handles extracting the data that the library provides, formatting it and despatching it.

In that sense, operation is exactly analogous to the RFM69 Native mode: the sketch polls the device via its library (RFM69…lib or emonLib…) and if new data is present, retrieves it and does whatever is necessary with it.

Indeed, obvious now you’ve spelt it out :slight_smile: Those changes may (and likely) will necessitate changes in EmonHubOEMInterfacer.py, or even provision of another interfacer to talk to it. And that’s all fine.

As long as you try and preserve the sending function, and still spit received data out, all good.

Anyway… GOOD news.
I have literally, this second just got my emonGLCD (with RFM69, using rf69.h) sending and receiving native packets through your emonPICM firmware.
There were a few subtle changes required (in essence the 2 extra bytes (header + srcId) now being part of the payload), but easily handled.

Small code changes in the emonPiCM firmware - contained in only the handling of input after cmd T issued and addition of destID in the send function (as opposed to only 0 for broadcast).

I’ll get all of this on GitHub shortly for you to look over … then we’ll have to figure out how to handle RF12 based emonGLCD as I don’t have one, and ideally be nice to have a common codebase.

If you tell me what your suggested changes are, I’ll look at them. But I won’t be backtracking to test anything with an emonGLCD any time soon. I’ve had to stop work on the -DB due to the temperatures, and won’t be restarting until into next week, then I’ll be trying to make up for the lost time.

Here you go:

That is your code with my small mods added for your review.
If testing, you will also need the 2 interfacer files in the same repo - these both were broken as the send functionality was not tested after the python3 move. Please do review and consider adding the changes proposed so emonGLCD can work.

This is updated emonGLCD code for RFM69 based only. I include it so you can see how it sends and receives data. I do plan to make this adaptable to support both RF12 and RFM69. At this stage it is here to prove the entire end to end TX & RX stack using emonPiCM and emonGLCD works.

Let me know your thoughts.

That doesn’t help - can you attach or PM the files - with the change notes of course.

Hi Robert

You’re really should get onto Github, it makes collaboration with you much easier, and brings a multitude of collaboration features that posting code on a forum doesn’t. Your choice I guess, but it makes it harder for others to collaborate. And this way it’s easily merged into the emon codebase as Glen and co can also easily merge changes. With your blessing I’m hoping to make a GitHub branch that includes all your + my changes that Glyn can just merge in, making it an easy choice which F/W people can run on emonPi. Certainly going to get the interfacers committed as they are just broken at present.

Here is a zip of your emonPICM code. The only things I changed really were the handling of ‘T’ command so it is byte-wise to match how it’s always been done and what emonhub expects and the’ RF Data handler - outbound’ routine to accommodate destination node ID. Having everything with a dest of ‘0’ (broadcast’) creates a lot of extra traffic on the AVRs that RFM69 can filter out now. Both changes documented in the GitHub readme.md, but I’d advise a quick diff to ensure nothing else snuck in.

Again, please let me know your thoughts.

emonPiFrontEndCM.zip (64.1 KB)

I suggest you read the comments I’ve made previously in other Topics about Github. I’ve no intention of wasting any more time on it.

After many more hours, I can confirm that TX and RX of RF69 format packets through your library works well from RF12 devices.
There is a bug in RFDemo ( I think somewhere in the rf12 init from EEPROM, but it’s a really weird one, and the offset is that it increments the Group by 1 vs what you told it via RF12_init). Anyway that is a digression.

However, I’ve uncovered an issue in emonPICM. Every 5 seconds or so it appears to ‘tickle’ the shutdown GPIO, this leads to the display showing ‘Shutdown ? Hold 5 seconds’, and it does this every 5 seconds, despite navigating away from that page of the LCD.
Investigation shows the shutdown GPIO is being invoked and detected by the emonPiLCD.py control through this code :

   logger.info("Attaching shutdown button interrupt...")
    try:
        shut_btn = Button(17, pull_up=False, hold_time=5)
        shut_btn.when_pressed = preShutdown
        shut_btn.when_held = shutdown

The GPIO is being ‘tickled’ but not held for the 5 seconds that are needed to invoke shutdown, but this happens every 5 seconds without fail when running the CM firmware.

My investigation thus far shows it is somewhere in this block of code in the loop in emonPiCM:

  if (EmonLibCM_Ready())  
{
    single_LED_flash();                                                // single flash of LED on local CT sample

    emonPi.power1 = EmonLibCM_getRealPower(0);                       // Copy the desired variables ready for transmission 
    emonPi.power2 = EmonLibCM_getRealPower(1); 
    emonPi.power1_plus_2 = emonPi.power1 + emonPi.power2;
    emonPi.E1     = EmonLibCM_getWattHour(0);
    emonPi.E2     = EmonLibCM_getWattHour(1);
   
    emonPi.Vrms     = EmonLibCM_getVrms() * 100;                       // Always send the ACTUAL measured voltage.

    if (EmonLibCM_getTemperatureSensorCount())
    {
      for (byte i=0; i< MaxOnewire; i++)
        emonPi.temp[i] = allTemps[i];
    }
   
    emonPi.pulseCount = EmonLibCM_getPulseCount(0);
    emonPi.pulse2Count = EmonLibCM_getPulseCount(1);
    
    send_emonpi_serial();                                              // Send emonPi data to Pi serial using packet structure

    if (EEProm.showCurrents)
    {
      // to show voltage, current & power factor for calibration:
      Serial.print(F("|Vrms:")); Serial.print(EmonLibCM_getVrms());

      Serial.print(F(", I1:")); Serial.print(EmonLibCM_getIrms(0));
      Serial.print(F(", pf1:")); Serial.print(EmonLibCM_getPF(0),4);

      Serial.print(F(", I2:")); Serial.print(EmonLibCM_getIrms(1));
      Serial.print(F(", pf2:")); Serial.print(EmonLibCM_getPF(1),4);
      
      Serial.print(F(", f:")); Serial.print(EmonLibCM_getLineFrequency());
      Serial.println();
     }
    delay(50);

    // Save energy & pulse count values to EEPROM
    storeEValues(emonPi.E1, emonPi.E2, emonPi.pulseCount, emonPi.pulse2Count); 
}

Commenting that out and all is well. Now, that block of code is the guts of the CM, so I really don’t want to go down that rabbit hole when the man who built if may know why this is happening.

Any ideas ?

Thanks
Alan