EmonTX to Rpi - Direct Serial Connection

Sage-Energy · 2021-05-04T21:11:02.401Z

AMAZING!!! Wish I’d signed up for the forums sooner instead of suffering through thousands of browser tabs and confusing dead ends on the forums and docs…

We have inputs finally! Many thanks everyone!!!

pb66 · 2021-05-04T21:12:54.179Z

borpin:

Just created a PR for the docs to try and emphasise that.

The solution should be to fix the code, it wouldn’t be difficult. I have suggested this before to no avail. Now it’s so ingrained in the discussions it would just cause more complication to fix than to just live with it ( a common position!)

borpin · 2021-05-04T21:15:00.483Z

pb66:

The solution should be to fix the code, it wouldn’t be difficult. I have suggested this before to no avail.

Is there a PR sitting?

Sage-Energy · 2021-05-04T21:25:51.839Z

Here’s some thoughts on a step-by-step guide to help others (my offer stands if I can be of assistance contributing to something useful for others):

[It’s documented all over the web but we might as well try and include everything to make this easy, even for those new to RPis] add the blank “ssh” file in /boot/ to enable SSH

0a) include here a link to the instructions that already exist in the documentation about getting wifi set up

The emonSD image needs to be expanded to fill SD cards bigger than 16GB via: sudo emonSDexpand (rasbpi-config won’t work)
Default SSH username is: pi password: emonpi2016
The tweaks to the arduino three phase sketch are quite well commented in the sketch itself but for anyone not familiar with these things a step-by-step guide of these tweaks might be useful
Baud rate for three phase needs to be <=9600 (I assume this is still true?) and the com port in the interfacers needs to be /dev/ttyAMA0 for direct wired serial connection (the docs assume everyone will spot the USB device listed in the example code and know what to change this to…
I couldn’t figure out if the SD image already sorts out the UART/bluetooth conflict with the RPi 2/3 so I added “dtoverlay-disable-bt” to /boot/config.txt (I also couldn’t find an answer on whether the old rpi3 notation for this overlay still works on buster?) and prevented initialisation of the BT modem with “sudo systemctl disable hciuart” followed by a reboot

5a) link to the raspberry pi docs explaining the UARTs in the different pis: Raspberry Pi Documentation - Configuration

“miniterm dev/ttyAMA0 9600” shows the serial output and if you miss the instruction line at the top as it starts scrolling Ctrl+] quits miniterm
I gave up trying to use the pi to program the ATMEGA via avrdude as I was getting errors that made no sense (to me) despite following the instructions…

Sage-Energy · 2021-05-04T21:29:58.562Z

Also, in the github repo for the three phase sketch there’s this item mentioned under “issues”:

boarder01 commented on 18 Dec 2019

There is a little mistake in the code, file src.
You must invert the two function:like this:

calculateConstants();
calculateTiming();

This code is great. Thanks

TrystanLea added the bug label on 23 Jan 2020

Given that Trystan gave it the bug label, I assume it should be the way around Boarder01 suggests? The bug was flagged at the start of last year though…

Robert.Wall · 2021-05-04T21:34:17.459Z

The place to download the 3-phase sketch from is Update to 3-Phase PLL sketch - #21 by Robert.Wall
That’s a zip file with full documentation. I’m afraid I have no control over what appears on Github (even if I understood Github and had the time to figure out how it could be useful).

pb66 · 2021-05-04T21:50:21.344Z

borpin:

Is there a PR sitting?

Not that I can remember. This was over 5years ago and there were many more far more pressing failures causing crashing etc for several years before the emonpi variant cofr eas revisited. I spent many months coding and testing to arrive at a pre-release version and it was then rewritten and released within a week of initial review. There are soooo many things that needed sorting but they have been around so long now there’s no longer any point.

I don’t tend to submit PR’s for the same reason I don’t document other peoples work, because I would end up totally redoing things rather than “sticking plasters” or documenting “what is for some reason” and that would not get accepted, thus it would be a further waste of my time.

borpin:

It’s called ‘development’. Without it, we’d still be living in mud huts.

“Development” isn’t necessarily “good development”, I would rather a cozy & homely mud hut than a high tech home that isn’t fit for purpose! I’m a huge fan of improving things, but not of just changing things, even if it’s changing to “something better” if it doesn’t actually improve anything.

borpin:

Emonhub started as a means of decoding RF messages (AIUI) - everything else has been bolted on.

No!

“OEM Gateway” started as a means of decoding RF messages. Emonhub was designed to be a “hub” for many different technologies, initially it had to have JeeLib decoding to replace OEMG and fit in with emon devices. OEM then took it and ran it as a dumb RFM message forwarder and made it too difficult to pursue other the avenues. There were many different interfacers before the OEM emonpi variant of emonhub came about! emonhub was NEVER intended to be just an RF forwarder, the other bit were not just bolted on! the architecture was specifically designed to accommodate generic interfacers and “drop in” interfacers that could eventually be included in the main repo or be unique to any user. It has now “developed” into something that is very difficult to adapt to other uses!!

Sage-Energy · 2021-05-04T21:50:41.231Z

Robert.Wall:

The place to download the 3-phase sketch from is Update to 3-Phase PLL sketch - #21 by Robert.Wall
That’s a zip file with full documentation. I’m afraid I have no control over what appears on Github (even if I understood Github and had the time to figure out how it could be useful).

OH!! That’s good to know - I didn’t know that existed either… I’m running out of date firmware!!! Can all the links to this sketch on the OEM properties not point to the same location??

I don’t fully understand the nitty-gritty of github either but what’s the point in having a github repo if it does not include the latest version?

Robert.Wall · 2021-05-04T22:09:37.131Z

Sage-Energy:

what’s the point in having a github repo

I concur.
It must have a purpose, but I can’t see how it is any better than a plain web page with a list of links. The answer’s got to be somebody is making money out of it somehow. It certainly doesn’t serve my purposes better than a web page - in fact it’s a lot worse as it clutters my machine with a load of hidden files.

If you want a fully documented emonLibCM: EmonLibCM - Version 2.2.2
And an emonTH & emonTx V3.4 that won’t clutter emonCMS with “factory test” numbers each time they start up: Emon devices factory testing transmissions of "zero values" - #11 by Robert.Wall

Sage-Energy · 2021-05-04T22:51:49.095Z

EDIT: Stupid lesson when tired - quit miniterm in that minimised ssh window you’ve forgotten about before you decide the serial output to emoncms has been borked again!!

Ok, I updated the firmware (I just noticed some helpful code comments about emonhub.conf below the changelog section that I’d missed before too!) but now it’s ground to a halt again… perhaps it’s time for bed and a look with fresh eyes in the morning, but in case someone spots something obvious before I do, here’s the emonhub.conf relevant sections and log output again:

2021-05-04 23:41:24,848 INFO     MainThread EmonHub emonHub (emon-pi variant) v2.1.5
2021-05-04 23:41:24,851 INFO     MainThread Opening hub...
2021-05-04 23:41:24,853 INFO     MainThread Logging level set to DEBUG
2021-05-04 23:41:24,855 INFO     MainThread Creating EmonHubSerialInterfacer 'SerialDirect' 
2021-05-04 23:41:24,872 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 9600 bits/s
2021-05-04 23:41:24,875 DEBUG    MainThread Setting SerialDirect pubchannels: ['ToEmonCMS']
2021-05-04 23:41:24,889 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT' 
2021-05-04 23:41:24,896 DEBUG    MainThread Setting MQTT pubchannels: ['ToEmonCMS']
2021-05-04 23:41:24,909 DEBUG    MainThread Setting MQTT subchannels: ['ToEmonCMS']
2021-05-04 23:41:24,911 INFO     MainThread Setting MQTT node_format_enable: 1
2021-05-04 23:41:24,912 INFO     MainThread Setting MQTT nodevar_format_enable: 1
2021-05-04 23:41:24,914 INFO     MainThread Setting MQTT nodevar_format_basetopic: emon/

and:

[interfacers]
### Plain Text Wired SerialOut Interfacer
[[SerialDirect]]
     Type = EmonHubSerialInterfacer
      [[[init_settings]]]
           com_port = /dev/ttyAMA0   # or /dev/ttyUSB0 or/dev/ttyACM0 etc
           com_baud = 9600              # to match the baud of the connected device
      [[[runtimesettings]]]
           pubchannels = ToEmonCMS,

[[MQTT]]

    Type = EmonHubMqttInterfacer
    [[[init_settings]]]
        mqtt_host = 127.0.0.1
        mqtt_port = 1883
        mqtt_user = emonpi
        mqtt_passwd = emonpimqtt2016

    [[[runtimesettings]]]
        pubchannels = ToEmonCMS, 
        subchannels = ToEmonCMS,

        # emonhub/rx/10/values format
        # Use with emoncms Nodes module
        node_format_enable = 1
        node_format_basetopic = emonhub/

        # emon/emontx/power1 format - use with Emoncms MQTT input
        # http://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
        nodevar_format_enable = 1
        nodevar_format_basetopic = emon/

[nodes]

[[11]]
    nodename = emonTx_three_phase
    firmware = three_phase
    hardware = emonTx V3.4/V3.4/Shield
    [[[rx]]]
       names = powerL1, powerL2, powerL3, power4, Vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulsecount
       datacode = 0  ### using [SerialDirect] Interfacer
       scales = 1,1,1,1,0.01,0.01,0.01,0.01,0.01,0.01,0.01,1
       units = W,W,W,W,V,C,C,C,C,C,C,p

pb66 · 2021-05-04T23:15:39.992Z

Sage-Energy:

but I pushed the "RFM12, " sideways out of the way with a comment character and added in the “ToEmonCMS”

I’ve only just fully digested this info and recommend undoing that. In theory if an interfacer both pubs and subs to the same channel it creates a infinite loop. It’s great that hasn’t happened but I don’t know it can’t happen without looking at the code.

Sage-Energy · 2021-05-05T22:29:37.886Z

pb66:

recommend undoing that. In theory if an interfacer both pubs and subs to the same channel it creates a infinite loop. It’s great that hasn’t happened but I don’t know it can’t happen without looking at the code.

But the RFM2Pi interfacer is commented out as I’m using direct serial wired link only… Surely I don’t want to subscribe to a channel I’m not using - is there more to this that I’m missing?

borpin · 2021-05-06T06:51:06.238Z

Sage-Energy:

is there more to this that I’m missing?

Yes, the names of the pub channels should be different to the sub channels in the same interfacer. One channel is an input to the interfacer and the other an output from the interfacer. The channels are pipes between 2 (or more) interfacers. Push the water in at one point and it can come out at several other points.

So the serial interfacer is ‘publishing’ the data to channel XXX and the output interfacer (HTTP or MQTT usually) ‘subscribes’ to channel XXX for data to send on. Much like an MQTT Broker.

Sage-Energy · 2021-05-06T09:07:51.479Z

borpin:

Yes, the names of the pub channels should be different to the sub channels in the same interfacer. One channel is an input to the interfacer and the other an output from the interfacer. The channels are pipes between 2 (or more) interfacers. Push the water in at one point and it can come out at several other points.

So the serial interfacer is ‘publishing’ the data to channel XXX and the output interfacer (HTTP or MQTT usually) ‘subscribes’ to channel XXX for data to send on. Much like an MQTT Broker.

Elsewhere the suggestion has been made (which I have followed) to comment out the RFM2Pi interfacer to get the SerialDirect interfacer to work (to avoid two interfacers conflicting on /dev/ttyAMA0) but apparently (reading the interfacer docs that I originally missed) this interfacer is supposed to be necessary for GPIO wired serial too, yet it still works commented out with MQTT pub and sub ToEmonCMS?

Is this wrong too? What is the correct way to set up the whole interfacers section to work with SerialDirect over GPIO wires?

Just noticed the default MQTT interfacer in the docs has NO pubchannel defined (commented out) - is this the solution?

borpin · 2021-05-06T09:22:22.151Z

Sage-Energy:

What is the correct way to set up the whole interfacers section to work with SerialDirect over GPIO wires?

There is no ‘correct’ way, as each circumstance is different.

However, if you do not have the RFM card fitted, then obviously there is little point having that in the configuration.

For me;

Reading the incoming serial data needs

[[SerialTx]]
     Type = EmonHubTx3eInterfacer
      [[[init_settings]]]
           com_port= /dev/ttyAMA0
           com_baud = 115200
      [[[runtimesettings]]]
           pubchannels = ToEmonCMS,

           nodeoffset = 0
           nodename = Serial_PiZ

To send the data out via MQTT (using the JSON format)

[[MQTT]]
    Type = EmonHubMqttInterfacer
    [[[init_settings]]]
        mqtt_host = 127.0.0.1
        mqtt_port = 1883
        mqtt_user = emonpi
        mqtt_passwd = emonpimqtt2016

    [[[runtimesettings]]]
        subchannels = ToEmonCMS,

        timestamped = True
        node_JSON_enable = 1
        node_JSON_basetopic = emon/

So incoming has a pub channel (the channel it will publish data on) and the sending interfacer has a sub channel (the channel it will subscribe to to get data from). The 2 names match - it is the same channel (pipe).

You could add a second MQTT interfacer, subscribing to the same channel, to send the data to a second MQTT broker. Or else add an HTTP interfacer to send the data to emoncms.org as well.

pb66 · 2021-05-06T10:05:52.250Z

Sage-Energy:

but apparently (reading the interfacer docs that I originally missed) this interfacer is supposed to be necessary for GPIO wired serial too,

I’m not sure where that info is seeping but that is not correct at all.

Sage-Energy:

Elsewhere the suggestion has been made (which I have followed) to comment out the RFM2Pi interfacer to get the SerialDirect interfacer to work (to avoid two interfacers conflicting on /dev/ttyAMA0)

Absolutely correct, you do not want conflicts for the serial port.

A little background info to hopefully help you understand. The EmonHubJeeInterfacer (“[[RFM2Pi]]”) is an extended EmonHubSerialInterfacer (“[[Serial-Direct]]”) which uses the same serial protocol and is essentially the same except it also has a lot of additional features to control the RFM device attached to any JeeLib type devices (JeeLabs is an open source project centered around the rfm12 and rfm69 devices) and later additional items were then added specific to the emonpi.

So, you could use the RFM2Pi interfacer to serially communicate with an emontx if you really chose to, but since most of the features are redundant the simpler and more correct interfacer to use is the “serial-direct”.

The tx3e interfacer is as I said previously “a fix” rather than a planned development and doesn’t fit with either of the previous 2, it is different in several ways.

Sage-Energy:

What is the correct way to set up the whole interfacers section to work with SerialDirect over GPIO wires?

There is no such thing as a correct conf as “one size fits all” definitely doesn’t work here, no matter how hard the project hammers that route all it does it create confusion such as this.

Sage-Energy:

Just noticed the default MQTT interfacer in the docs has NO pubchannel defined (commented out) - is this the solution?

Well! That’s interesting! By commenting out the pub channel it might be defaulting to a hardcoded default in the interfacer. Which IF that’s what is happening is the correct way, the non-explicit way, shame the line is there at all rather than being commented. However! The pub and sub “channels” are essentially just queues of data (or pipes as Brian says but I’m not a fan of that term here), the “channel” names are arbitrary string ids only, you could use pubchannel = ToSomeNonExistentQueue, if you like because, as far as I know the MQTT interfacer isn’t complete and doesn’t subscribe to topics to be able to internally publish them. But if it can and you do, publishing to a queue named “toRFM2Pi” only matters to interfacers listening to that “channel” of which you have none, currently, since you have commented out the RFM2Pi section. The channel names have no relation to the interfacers per se.

My previous comment about undoing your channel change was to avoid what is generally a bad idea in software, which is to have the potential loop of talking to and listening to the same place, it didn’t seem to cause you an issue but I do not know if that is by chance or by design, so future updates eg to get MQTT working both directions might create an issue. But it isn’t (currently) anything you need to worry about, set to anything you like as it isn’t fully implemented, but maybe avoid the loop just in case Sorry if that passing comment has caused further confusion.

Sage-Energy · 2021-05-06T10:09:26.009Z

borpin:

So incoming has a pub channel (the channel it will publish data on) and the sending interfacer has a sub channel (the channel it will subscribe to to get data from). The 2 names match - it is the same channel (pipe).

Thanks Brian - that bit seems straightforward to me now that you explain it… I’m not familiar with MQTT stuff nor the architecture of emoncms, so I wasn’t sure whether “ToEmonCMS” was a ‘tank’ at the end of a ‘pipe’, or the pipe itself and whether MQTT needed two-way access to that data ‘tank’… I’ll recomment-out the MQTT pubchannel and hope for the best!

Looking at your setup I just had a thought - are the node_JSON_* lines in your MQTT interfacer what makes the EmonHubTx3eInterfacer (vs EmonHubSerialInterfacer) method work??

The default MQTT interfacer has different code here for “use with emoncms Nodes module”…

Sage-Energy · 2021-05-06T10:21:46.809Z

pb66:

Sage-Energy:

but apparently (reading the interfacer docs that I originally missed) this interfacer is supposed to be necessary for GPIO wired serial too,

I’m not sure where that info is seeping but that is not correct at all.

Sorry read it more carefully again (“The [[RFM2Pi]] interfacer section contains the settings to read from RFM69Pi / emonPi boards via GPIO internal serial port /dev/ttyAMA0.”) and it doesn’t actually say what I thought it said… it’s only talking about an RFM board connected via GPIO serial… whoops!

pb66:

A little background info to hopefully help you understand.

Thank you!! I’m not someone who can memorise things without understanding them first, so this is appreciated!

pb66:

the “channel” names are arbitrary string ids only

This helps to know too - thank you!

pb66:

My previous comment about undoing your channel change was to avoid what is generally a bad idea in software

Thanks for clarifying - now that I understand the basics of how this code functions I see how your general warning applies (I’m familiar with software design principles - though sometimes this knowledge is a bit rusty these days).

borpin · 2021-05-06T12:05:21.046Z

Sage-Energy:

Looking at your setup I just had a thought - are the node_JSON_* lines in your MQTT interfacer what makes the EmonHubTx3eInterfacer (vs EmonHubSerialInterfacer) method work??

No, just a different format out from the MQTT interfacer. One topic rather than a topic per reading. Plus it gives you named Inputs.

Robert.Wall · 2021-05-06T21:27:38.229Z

A post immediately above was duplicated in the topic Three Phase emonPi using an emonTx with a RPi Zero W mounted inside
A post relating to that was merged into that topic.