emonTx firmware build with Arduino IDE, and code changes for serial connection to emonPi

Hardware emonTx
,
1

Hi,

I’m trying to compile and upload the stock emonTx firmware using Arduino IDE. I know the doco says use PlatformIO, but I use Arduino IDE for my ESP8266 projects, so i’d rather not add another different IDE to the mix.

The first problem I’ve struck is what board type to use as the target for the build?

I’m guessing then I’m going to have to hunt down and install/download a whole bunch of dependencies/libraries, but that’s fine.

I can’t find any good doco on building emonTx firmware using the arduino IDE on the web. Maybe if I can get it working I can post a step by step guide.

What I’m trying to achive once I get the firmware building, is to experiment with direct serial connection to an emonPi via a serial-USB dongle.

I can see the bootup serial output from the emonTx. After startup, it sends (debug) which is the CT output etc on one line of serial text. Does this just get fed into the Pi, with an appropriate change in emonhub.conf? That’s the sort of thing I’m expecting, but I’m not sure if the data will need to me re-formatted in some way before the emonPi will like it.

Any hints or links to further reading would be much appreciated.

Michael.

2

So I’ve done some more reading on the forum…

It seems I don’t need to modify the emonTx code, as the NEW version sends out serial data in a format compatible with the ESP version…

The only catch is my emonTX’s are firmware version 2.30, and they seem to send the serial data in an older format… So is there a different serial interfacer configuration I can use in emonhub.conf to use the serail data as is?

Amy I also going to update the code on the emonPi to get all this working?

3

The instructions for the Arduino IDE and Libraries are in the ‘Learn’ section under ‘Using the Arduino IDE’. Obviously too late for you, but it might help anyone reading this thread later.

The board type is “Uno”, as it says. I don’t think you need to rewrite those instructions, but if there is something missing, please point it out and I’ll see what can be done.

As you say, if you download the present emonTx sketch from Github, that should send the serial output in an acceptable format. You can of course change it as you wish. I don’t have an ESP module, but my notes say:

When using the EmonSerialTx3eInterfacer in emonHub to connect to the emonTx using the output designed solely for the emonESP, you will need to tell it what node ID you would like it to use by using the “nodeoffset” setting in emonhub.conf e.g. if using the default node id of 8:

    [[SerialTx3e]]
         Type = EmonHubTx3eInterfacer
          [[[init_settings]]]
               com_port= /dev/ttyAMA0
               com_baud = 115200
          [[[runtimesettings]]]
               pubchannels = ToEmonCMS,
     
               nodeoffset = 8
4

Thanks Robert,

The info on [[SerialTx3e]] above is definitely what I needed, especially the nodeoffset bit.

I’ll have another look into the Arduino IDE option with those docs.

Turns out I have a v2.3 and v2.9 emonTx (I have 4 in total) that I need to convert to serial.

I could not get either of them to talk to avrdude despite them sending data on reset (9600 baud for the older firmware, and 115200 for the v2.9):

avrdude -v -p atmega328p -c arduino -P /dev/ttyUSB0 -b 9600 -U flash:w:emonTxV3_RFM69CW_latest_433.hex

avrdude: Version 6.3
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2014 Joerg Wunsch

     System wide configuration file is "/etc/avrdude.conf"
     User configuration file is "/home/mzc/.avrduderc"
     User configuration file does not exist or is not a regular file, skipping

     Using Port                    : /dev/ttyUSB0
     Using Programmer              : arduino
     Overriding Baud Rate          : 9600

avrdude: stk500_getsync() attempt 1 of 10: not in sync: resp=0x65
avrdude: stk500_getsync() attempt 2 of 10: not in sync: resp=0x6d
avrdude: stk500_getsync() attempt 3 of 10: not in sync: resp=0x6f
avrdude: stk500_getsync() attempt 4 of 10: not in sync: resp=0x6e
avrdude: stk500_getsync() attempt 5 of 10: not in sync: resp=0x54
avrdude: stk500_getsync() attempt 6 of 10: not in sync: resp=0x78
avrdude: stk500_getsync() attempt 7 of 10: not in sync: resp=0x20
avrdude: stk500_getsync() attempt 8 of 10: not in sync: resp=0x56
avrdude: stk500_getsync() attempt 9 of 10: not in sync: resp=0x33
avrdude: stk500_getsync() attempt 10 of 10: not in sync: resp=0x2e

avrdude done. Thank you.

The getsync() parts happen after the emonTx is reset. That’s where I presume the bootloader should talk to avrdude, but the ascii codes of the bytes above are just the initial txt sent by the emonTx when it boots. So I must be doing something wrong. I just have a FDTI serial adapter connected to the PC with USB, and to the emonTx’s serial port.

The other mystery I found was with the v2.9 firmware on one of my emonTx’s. I thought from my reading that that would have the serial ESP data built into it’s output, but all I get is “+++” prompt to enter serial config mode. I must be missing something there too.

Cheers,

Michael.

5

The main advantage of using emonHub’s SerialTx3e interfacer with the serial output intended for the emonESP is that it’s already there and negates the need to upload new FW, since not all your emonTx’s already have the v2.9 FW that advantage is somewhat diluted.

The emonTx’s key:value pairs output is not used fully in the SerialTx3e interfacer, the values are simply converted to an array (CSV) and the keys ignored in favor of the names defined in the [nodes] section of emonhub.conf.

If you are planning on connecting 4 emonTx’s serially you need to consider how they are identified, since the emonESP is intended for an emonESP connected to a single emonTx, there is no node id passed in the payload, therefore emonHub cannot tell the emonTx’s apart. Yes you can set the node id using nodeoffset but the node id will be bound to a serial address such as /dev/ttyUSB1 (4 emonTx’s will be ttyUSB0 thru ttyUSB3). But you will not have any control over which emonTx get’s what address and it can change at boot time on a first come (discovered) first served (addressed) basis.

For this issue you have 2 real options, you can either ensure you use genuine FDTI FT232RL based USB adapters (or really good clones) so that they can be programmed to have unique ID’s that you can set udev rules to control the assigned addresses or edit the sketches to include a leading node id in the payload, considering the key:value pairs are pointless, I would probably edit the sketch to output a space separated payload of values with a leading node id regardless, even if I intended to use FTDI brand adapters.

Another advantage of editing the sketch is you can remove the RFM bits, which may improve the reliability by not being dependent on redundant HW being functional.

The upload baud of all emonTx’s is 115200, the baud of 9600 for older emonTx’s refers to the runtime serial connection not the upload baud.

How does your USB programmer connect? the emonTx’s Rx and Tx are UNconventionally labelled, you need to connect Rx to Rx and Tx to Tx (not Rx to Tx and Tx to Rx as expected)

6

Have you found this:
https://wiki.openenergymonitor.org/index.php/EmonTx_V3.4#Flashing_Pre-compiled_.hex

I’ve never done it that way - for what I do (development and fault finding, mainly) the compiled file is useless, but I was under the impression that only 115200 has ever been used for that.

I’ve just now got Avrdude and your command failed for me at 9600, but succeeded at 115200, in an emonTx V3.4. after I’d put my full path. (The existing sketch is irrelevant - mine had a test sketch in it.) That ‘latest’ compiled version reported itself as V2.3, and begs the question: Is the compiled version the latest? I think not. I suggest you compile and upload using the Arduino IDE, you’ll then be able to see exactly what you are uploading.

Are you using the “Shop” programmer or a look-alike? Some look-alikes don’t work, and for various different reasons, a likely one is we need the RTS line as a hardware reset.

[Grrr… Paul, you’ve posted again while I was typing. (That’s our party trick.)]

7

Thanks Robert for the clue with RTS. As soon as I hooked that up to the Rst pin on the emonTx serial port, avrdude worked! I thought manually resetting while the avrdude was waiting to transfer the data would be equivalent. Apparently not!

The serial adapter I have is a FTDI adapter I bought for ESP8266 tinkering (unrelated to the emon stuff).

So, I can flash the boards now. That’s great!!! Now I’d really like to tinker with the code, disable RF etc. This will be especially important if the latest compiled fw is not really 2.9 :slight_smile: So, next step is to try to get all the dependancies working for arduino IDE, or I may have to use the other IDE that’s recommended and better documented.

Do you happen to know what board target to use in the arduino IDE? Is it just an arduino DUO (compatible)?

Cheers,

Michael.

8

Hi Paul,

Yes, I’ll use the std emonESP format, and disable RF once I get the build chain working. I have avrdude uploading firmware now, so that’s progress.

I’ve ordered some ‘genuine’ FTDI adapters from 3 different places (I only need 2), so hopefully I’ll get some with different serial numbers so they show up as the same /dev after a reboot.

Yes, I was aware of the (incorrect) :stuck_out_tongue: labels on the emonTx serial port. All good now. The trick to get avrdude to work was to connect RTS on serial adapter to RST on the emonTx serial port.

9

Thanks Robert, Not too late at all, I’m going back now to see if I can get a custom sketch building in the IDE.

Then hopefully I can hook it to the emonPi, edit the config file and see data being logged over serial!

10

Bingo… After adding the libraries as documented in the Learn Section of the openenergymonitor site, Bingo, the latest sketch builds and uploads fine… With the RTS-RST link and my FTDI serial dongle:

emonTx V3.4 Discrete Sampling V2.90

OpenEnergyMonitor.org



No EEPROM config

RFM69CW Node: 7 Freq: 433Mhz Group: 210



POST.....wait 10s

'+++' then [Enter] for RF config mode

CT 1 Cal 90.90

CT 2 Cal 90.90

CT 3 Cal 90.90

CT 4 Cal 16.67

RMS Voltage on AC-AC  is: ~0V

AC-AC NOT detected - Apparent Pwr measure enabled

Assuming VRMS: 230V

Assuming power from batt / 5V USB - power save enabled

NO CT's detected

No temperature sensor

CT1 CT2 CT3 CT4 VRMS/BATT PULSE

ct1:18774,ct2:0,ct3:0,ct4:0,vrms:500,pulse:0

ct1:3701,ct2:0,ct3:0,ct4:0,vrms:501,pulse:0

ct1:729,ct2:0,ct3:0,ct4:0,vrms:506,pulse:0

ct1:143,ct2:0,ct3:0,ct4:0,vrms:505,pulse:0

OK, next step, see if I can get the emonPi to recognize the data, and then disable the RF on the emonTx…

11

If they are genuine, they will have programmable serial numbers, using the official FTDI utility you can set a serial number of your choice. The serial numbers are not used by the OS by default, you will need to set up udev rules in the OS to ensure the same /dev addresses are used.

I was recommending NOT going that route. As I said, since you are editing the sketch you would be better off outputting the serial data as space separated values with a leading node id and use the standard SerialInterfacer not the SerialTx3eInterfacer. It’s a safer, easier and better established method.

Well it is the party season after all :christmas_tree::tada::wine_glass:

12

OK, well I might have a look into that. I’m not having much luck getting the ESP serial interfacer to work. Firstly because it did not exist on my pi. Added it now, but I’m in the middle of a apt-get update, and the emoncms gui says can’t connect to the database, please verify credentials. Hopefully that is transient and will be fixed once I reboot after all the updates finish. Otherwise, I have something else to fix :slight_smile:

I’ll look into the std serial interfacer version of things given I can now modify the sketch code and get it uploaded.

Cheers,

Michael.

13

Timing is the answer. I think I read somewhere that there’s a fairly narrow window during which the bootloader can accept the sketch. Don’t ask me what it is, I can’t remember.

14

OK, so I got the serialinterfacer version working, data getting logged to emonhub.
I have not (yet) fully removed all the startup serial output from the sketch. It seems that the emonhub gets very upset about this and I get SerialDirect thread is dead.

This death also seems easily triggered by unplugging the usb port on the Pi, so I’m a bit concerned about the overall stability of this setup.

15

What is with this SerialDirect thread is dead? I can’t find much info on this relating to serial connected emonTx’s anyway.

I’ve been getting this quite frequently since switching from RF to direct serial.

Initially I thought it was power related (powering the Pi and 2 x Tx’s from one 5V SUB pack gave 4.2V). They all now have individual mini USB direct to a beefier power pack, giving 5.2V when under load.

Then I thought it was the flaky eBay USB-seial adapters. I had a genuine FTDI one, and that seemed to be more stable. Now today some more ‘genuine’ FTDI ones turned up. Will run these for a while and see what happens.

What actually causes the serial direct thread to die? Is it the SerialDirect python parsing of the data that falls over if invalid data is sent? If that’s the case, surely a checksum can be added, and better pre-processing? I’m not a python programmer, so that file looks a little scary to start changing.

Is the 115200 baud rate and small unshielded wires between the Tx and Pi the issue? If that’s the case, maybe dropping to 9600 would help?

16

Just for reference, if anyone is trying to get emonTx’s connected to an emonPi over seial, using external USB-serial adapters, I’ll attach my modified emonTx src.ino for and the emonhub.conf from the emonPi for reference.

The trick to getting the code to download to the emonTx (from linux) using either the arduino IDE or with avrdude was to connect the RTS line on the serial adapter to the RST line on the emonTx.

Of course, Rx-Rx and Tx-Tx as the Rx pin is labeled Tx (and Tx labeled Rx) on the emonTx.

The library import to get the sketch to build in the arduino IDE is available in the learn section of this site.

In teh arduino IDE, the board type was Arduino Uno and the programmer type was Atmel STK500 dev board.

src.ino (26.8 KB)

emonhub.conf:

#######################################################################
#######################      emonhub.conf     #########################
#######################################################################
### emonHub configuration file, for info see documentation:
### http://github.com/openenergymonitor/emonhub/blob/emon-pi/configuration.md
#######################################################################
#######################    emonHub  settings    #######################
#######################################################################

[hub]
### loglevel must be one of DEBUG, INFO, WARNING, ERROR, and CRITICAL
loglevel = DEBUG
### Uncomment this to also send to syslog
# use_syslog = yes
####################################################################### 
#######################       Interfacers       #######################
#######################################################################

[interfacers]
### This interfacer manages the RFM12Pi/RFM69Pi/emonPi module
[[RFM2Pi]]
    Type = EmonHubJeeInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyAMA0
        com_baud = 38400                        # 9600 for old RFM12Pi
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        subchannels = ToRFM12,

        group = 210
        frequency = 433
        baseid = 5                              # emonPi / emonBase nodeID
        quiet = true                            # Report incomplete RF packets (no implemented on emonPi)
        calibration = 230V                      # (UK/EU: 230V, US: 110V)
        # interval =  0                         # Interval to transmit time to emonGLCD (seconds)

[[MQTT]]

    Type = EmonHubMqttInterfacer
    [[[init_settings]]]
        mqtt_host = 127.0.0.1
        mqtt_port = 1883
        mqtt_user = emonpi
        mqtt_passwd = emonpimqtt2016

    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,

        # emonhub/rx/10/values format
        # Use with emoncms Nodes module
        node_format_enable = 1
        node_format_basetopic = emonhub/

        # emon/emontx/power1 format - use with Emoncms MQTT input
        # http://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
        nodevar_format_enable = 1
        nodevar_format_basetopic = emon/

[[emoncmsorg]]
    Type = EmonHubEmoncmsHTTPInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,
        url = https://emoncms.org
        apikey = 50ee889c7fccb0a165231d76cc6df4c5
        senddata = 1                    # Enable sending data to Emoncms.org
        sendstatus = 1                  # Enable sending WAN IP to Emoncms.org MyIP > https://emoncms.org/myip/list
        sendinterval= 30                # Bulk send interval to Emoncms.org in seconds

[[SerialDirect0]]
     Type = EmonHubSerialInterfacer
      [[[init_settings]]]
           com_port = /dev/ttyUSB0      # or /dev/ttyAMA0 or/dev/ttyACM0 etc
           com_baud = 115200              # to match the baud of the connected device
      [[[runtimesettings]]]
           pubchannels = ToEmonCMS,

[[SerialDirect1]]
     Type = EmonHubSerialInterfacer
      [[[init_settings]]]
           com_port = /dev/ttyUSB1      # or /dev/ttyAMA0 or/dev/ttyACM0 etc
           com_baud = 115200              # to match the baud of the connected device
      [[[runtimesettings]]]
           pubchannels = ToEmonCMS,

####################################################################### 
#######################          Nodes          #######################
#######################################################################

[nodes]

## See config user guide: http://github.com/openenergymonitor/emonhub/blob/master/configuration.md

[[5]]
    nodename = emonpi
    [[[rx]]]
        names = power1,power2,power1pluspower2,vrms,t1,t2,t3,t4,t5,t6,pulsecount
        datacodes = h, h, h, h, h, h, h, h, h, h, L
        scales = 1.0588,1.0588,1,0.010588,0.1,0.1,0.1,0.1,0.1,0.1,1
        units = W,W,W,V,C,C,C,C,C,C,p

[[6]]
    nodename = emontxshield
    [[[rx]]]
       names = power1, power2, power3, power4, vrms
       datacode = h
       scales = 1,1,1,1,0.01
       units =W,W,W,W,V

[[7]]
    nodename = emontx4
    [[[rx]]]
       names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p

[[8]]
    nodename = emontx3
    [[[rx]]]
       names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       datacodes = h,h,h,h,h,h,h,h,h,h,h,L
       scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p

[[9]]
    nodename = emontx2
    [[[rx]]]
       names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p

[[10]]
    nodename = emontx1
    [[[rx]]]
       names = power1, power2, power3, power4, vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
       datacode = h
       scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
       units =W,W,W,W,V,C,C,C,C,C,C,p


[[19]]
   nodename = emonth1
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[20]]
   nodename = emonth2
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[21]]
   nodename = emonth3
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[22]]
   nodename = emonth4
   [[[rx]]]
      names = temperature, external temperature, humidity, battery
      datacode = h
      scales = 0.1,0.1,0.1,0.1
      units = C,C,%,V

[[23]]
    nodename = emonth5
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p

[[24]]
    nodename = emonth6
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p

[[25]]
    nodename = emonth7
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p

[[26]]
    nodename = emonth8
    [[[rx]]]
       names = temperature, external temperature, humidity, battery, pulsecount
       datacodes = h,h,h,h,L
       scales = 0.1,0.1,0.1,0.1,1
       units = C,C,%,V,p
17

What modifications have you made? Should any of these modifications be incorporated in in the standard direct serial FW?

Although if using directserial-serialtx3e emonhub interfacer no change to the emonTX FW should be required as the directserial-serialtx3e interfacer can read the standard emonTx FW key separated values: emonhub/conf/interfacer_examples/directserial-serialtx3e at emon-pi · openenergymonitor/emonhub · GitHub

18

I may well have made some chages to the std emonTx sketch that are already in that version. I was under the impression that the one that supported serial was older, so I started from the latest and made some changes.

Nothing too dramatic (commented at the top on the .ino file):

Radio DISABLED, most Radio code removed for code size savings.
Serial data (only) out serial port for SerialInterfacer use on emonPi.
DIP 1 subtracts 1 from nodeID if ON else ADDS one. Use DIP to select nodeID 7 or 9 (for my 2 serial connected boards).
DIP 2 used to enable/disable startup serial debug. No longer selects USA voltage. Default 240V setting hard coded. Can turn on debug when connected to the IDE or a terminal, and off when connected to the emonTx.
Removed code for +++ mode to change nodeID over serial (as I did not need that).

So basically it’s a gruesome cut down version that does what I needed to get serial working, and removing features I did not need (to maximise memory).

1 Like