Avoiding wireless connections - EmonTX serial RPiZero solution

[haffle (Julian Berry)]

Job Done!

Well almost; I now have four EmonTx communicating to my emonPi via Raspberry Pi Zeros; I have recreated the Inputs & Feeds etc, just need to re-build my Dashboards and Apps now.

For future reference, I have attached a “How-To for Dummies” that I have cobbled together, in case it help others if they struggle as much as I do.

In hindsight, I found the process as fascinating as it was frustrating, but I have learnt a lot in the process. Words like sketch, Arduino, avrdude and miniterm are now familiar, even if what they do isn’t always …

2020-10-22 emonTx - Raspberry Pi Zero W.pdf (3.4 MB)

Once again, Thanks All for your patience !!!

[borpin (Brian Orpin)]

Looks good. I have just noticed on one of the photos, the internal serial breakout (JP3). You could have put a header on that and connected up so no external cable!

[haffle (Julian Berry)]

Interesting; are the screw terminals effectively the same as the pins next to them (but in a different order)?

1 (5V) > pin #4 (5V)
2 (3.3v) > n/a
3 (GND) > pin #6 (GND)
4 (IRQ1) > pin#???
5 (ADC5) > pin #???
6 (Dig 5) > pin #???

[Robert.Wall]

No, only the ones sharing the same designation.

The UART connector is specifically for the programmer, the screw terminals are specifically for I/O, such as the temperature sensor and pulse input.

If you want the nitty-gritty details, take a look at the circuit diagram (ideally, you need to download and install Eagle CAD (free for personal use with the size of PCB we use) then you can see the same track highlighted on the pcb and on the circuit diagram.

[borpin (Brian Orpin)]

I think they are but I was thinking of this

[borpin (Brian Orpin)]

@haffle, @Robert.Wall, I think I have worked out why the writing of the firmware from the RPi directly didn’t work. The utility is using Python not Python3.

EMonTX Bad data after upgrade

TL;DR The utility avrdude-rpi still uses python which may be the issue @bwduncan @TrystanLea. @Martin_Harizanov - I note OEM fork is from you. Went back to the original repo and is still python as well. In detail… Interesting. Start ATmega328 serial upload using avrdude with latest.hex chmod: cannot access '/var/log/emoncms/firmware.log': No such file or directory Attempt: 1 avrdude -c arduino -p ATMEGA328P -P /dev/ttyAMA0 -b 115200 -U flash:w:/opt/openenergymonitor/emonpi/firmware/compile…

[haffle (Julian Berry)]

Interesting. And I am so pleased to learn that it was probably not all down to my ineptitude causing that grief the other week

[haffle (Julian Berry)]

Hmm, first drop out, despite the Pi Zero still being accessible via the IP address. Looking back at a Feed graph it must have gone wrong on Sunday.

Can one learn anything from this emonHub log? What might be the solution?

TIA

2020-11-03 04:35:34,878 WARNING  MainThread SerialTx thread is dead.
2020-11-03 04:35:34,880 WARNING  MainThread Attempting to restart thread SerialTx (thread has been restarted 1515 times...)
2020-11-03 04:35:34,892 INFO     MainThread Creating EmonHubTx3eInterfacer 'SerialTx'
2020-11-03 04:35:34,897 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 115200 bits/s
2020-11-03 04:35:34,911 DEBUG    MainThread Setting MQTT timestamped: True
2020-11-03 04:35:35,866 WARNING  SerialTx   Exception caught in SerialTx thread. Traceback (most recent call last):
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 32, in wrapper
    return func(*args)
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 99, in run
    rxc = self.read()
  File "/opt/openenergymonitor/emonhub/src/interfacers/EmonHubTx3eInterfacer.py", line 46, in read
    self._rx_buf = self._rx_buf + self._ser.readline().decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 19: invalid continuation byte

[borpin (Brian Orpin)]

This is a problem with the Serial data from the emonTX.

@haffle

MainThread SerialTx thread is dead.

It could be dying and trying to restart repeatedly.

Try resetting the EmonTX as a starting point I suggest.

@bwduncan, could this exception be handled more gracefully (error in log and discard frame)?

[haffle (Julian Berry)]

Thanks @borpin ; pressing the Reset button didn’t kick the Inputs in to life, so I powered the Pi Zero down through the EmonCMS / Admin page, waited a bit then powered both bits down and back on again. But that too has not cleared it.

There is always a chance that it is a physical issue - jumper lead coming dislodged etc, but that seems unlikely as the unit is screwed on to a purlin high up in a rodent / bird / cat free barn in close proximity to an Apple Airport Express.

[borpin (Brian Orpin)]

You can see from the emonhub log, the issue is the serial connection between the EmonTX and the Pi.

@haffle

pressing the Reset button didn’t kick the Inputs in to life

You need to look at the emonhub log and see what it is telling you.

[Bill.Thomson]

@haffle

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 19: invalid continuation byte

Not sure how relevant this is, but here’s some info on that particular error:

"for line in..." results in UnicodeDecodeError: 'utf-8' codec can't decode byte

[borpin (Brian Orpin)]

Except that there is no good reason, I suggest, why the data isn’t utf-8. @Robert.Wall?

[Robert.Wall]

What data are you referring to, exactly?

As far as I can tell, the emonTx sends to the serial port nothing outside the range [0x0a, 0x0d, 0x20-0x7f] and I think those characters lie wholly within the utf-8 character set.

If the baud rate has become wrong for an unknown reason, then I cannot predict how characters within the utf-8 set will be interpreted, but it’s feasible that the result of a wrongly interpreted character could well fall outside the utf-8 character set.

[Bill.Thomson]

@borpin

Except that there is no good reason,

Although that may be, something is afoot. Otherwise, there’d be no error, or a different error.

[haffle (Julian Berry)]

Thanks All. I’m afraid the stacksverflow.com thread is above my knowledge so haven’t changed any code. If this PiZero / emonTx is producing a duff character I wonder why the other three aren’t? They were all set up following the same steps.

Meanwhile, this is the current emonHub output:

2020-11-07 09:37:33,385 INFO     MainThread Setting MQTT node_JSON_enable: 1
2020-11-07 09:37:33,397 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2020-11-07 09:37:33,401 DEBUG    MainThread Setting emoncmsorg pubchannels: ['ToRFM12']
2020-11-07 09:37:33,403 DEBUG    MainThread Setting emoncmsorg subchannels: ['ToEmonCMS']
2020-11-07 09:37:33,404 WARNING  MainThread Setting emoncmsorg apikey: obscured
2020-11-07 09:37:33,406 INFO     MainThread Setting emoncmsorg url: https://emoncms.org
2020-11-07 09:37:33,408 INFO     MainThread Setting emoncmsorg senddata: 1
2020-11-07 09:37:33,420 INFO     MainThread Setting emoncmsorg sendstatus: 1
2020-11-07 09:41:35,600 INFO     MainThread Logging level set to INFO
2020-11-07 09:41:40,898 INFO     MainThread Logging level set to DEBUG
2020-11-07 09:41:40,913 DEBUG    MainThread Setting MQTT timestamped: True
2020-11-07 09:41:58,925 DEBUG    MainThread Signal 15 received.
2020-11-07 09:41:59,051 INFO     MainThread Exiting hub...
2020-11-07 09:41:59,330 INFO     MainThread Exit completed
2020-11-07 09:42:11,951 INFO     MainThread EmonHub emonHub (emon-pi variant) v2.1.5
2020-11-07 09:42:11,954 INFO     MainThread Opening hub...
2020-11-07 09:42:11,956 INFO     MainThread Logging level set to DEBUG
2020-11-07 09:42:11,958 INFO     MainThread Creating EmonHubTx3eInterfacer 'SerialTx'
2020-11-07 09:42:11,968 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 115200 bits/s
2020-11-07 09:42:11,983 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
2020-11-07 09:42:11,989 DEBUG    MainThread Setting MQTT timestamped: True
2020-11-07 09:42:12,000 DEBUG    MainThread Setting MQTT subchannels: ['ToEmonCMS']
2020-11-07 09:42:12,002 INFO     MainThread Setting MQTT node_JSON_enable: 1
2020-11-07 09:42:12,011 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2020-11-07 09:42:12,014 DEBUG    MainThread Setting emoncmsorg pubchannels: ['ToRFM12']
2020-11-07 09:42:12,016 DEBUG    MainThread Setting emoncmsorg subchannels: ['ToEmonCMS']
2020-11-07 09:42:12,018 WARNING  MainThread Setting emoncmsorg apikey: obscured
2020-11-07 09:42:12,040 INFO     MainThread Setting emoncmsorg url: https://emoncms.org
2020-11-07 09:42:12,042 INFO     MainThread Setting emoncmsorg senddata: 1
2020-11-07 09:42:12,044 INFO     MainThread Setting emoncmsorg sendstatus: 1

I am able to interact with the emonHub (& emonCMS etc) fine, so comms at a certain level are working, but the emonHub log suggests that the process(es) just get stuck after a short period following a soft Re-Start. No data has arrived on my emonPi’s emonCMS.

[borpin (Brian Orpin)]

@Robert.Wall

As far as I can tell, the emonTx sends to the serial port nothing outside the range [0x0a, 0x0d, 0x20-0x7f] and I think those characters lie wholly within the utf-8 character set.

Exactly my point - if the EmonTX only sends data within utf-8, then there should be no reason to change the char set.

@Bill.Thomson

Although that may be, something is afoot.

There is definitely something afoot, I just don’t think the char set is set wrong (see above).

@haffle

Meanwhile, this is the current emonHub output:

Hopefully that isn’t all you are getting

I think you are running the newer Image based on RaspberryOS, I wonder if something has changed in the serial configuration. We currently do;

openenergymonitor/emonhub/blob/a9e47bf684f62632a118196e5aec806efba64be4/install.sh#L33-L33

as part of the install.

However, if 3 are OK and one not, it implies a local problem.

First point is to check header soldering on the Pi and the cable crimps.

@haffle

I am able to interact with the emonHub (& emonCMS etc) fine, so comms at a certain level are working, but the emonHub log suggests that the process(es) just get stuck after a short period following a soft Re-Start.

I’ll say this again the connection to the Pi is via Wi-Fi and nothing to do with the connection between the Pi and the EmonTX.

What you could look at is the system log to see if there are any other issues.

sudo journalctl -f

[Robert.Wall]

@borpin

First point is to check header soldering on the Pi and the cable crimps.

And check at the emonTx end as well - get it out of its case and carefully look at everything.

Is there a local source of interference that could be getting onto that serial connection?

[haffle (Julian Berry)]

@borpin

Hopefully that isn’t all you are getting

I may have missed something, but it is; I have just logged back in and the EmonHub log of the PiZero in question is still the same as earlier today (see above). Clicking Restart brings up the same return.

2020-11-07 09:42:12,042 INFO     MainThread Setting emoncmsorg senddata: 1
2020-11-07 09:42:12,044 INFO     MainThread Setting emoncmsorg sendstatus: 1
2020-11-07 14:19:05,296 DEBUG    MainThread Signal 15 received.
2020-11-07 14:19:05,607 INFO     MainThread Exiting hub...
2020-11-07 14:19:05,745 INFO     MainThread Exit completed
2020-11-07 14:19:17,963 INFO     MainThread EmonHub emonHub (emon-pi variant) v2.1.5
2020-11-07 14:19:17,966 INFO     MainThread Opening hub...
2020-11-07 14:19:17,968 INFO     MainThread Logging level set to DEBUG
2020-11-07 14:19:17,971 INFO     MainThread Creating EmonHubTx3eInterfacer 'SerialTx'
2020-11-07 14:19:17,976 DEBUG    MainThread Opening serial port: /dev/ttyAMA0 @ 115200 bits/s
2020-11-07 14:19:17,995 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
2020-11-07 14:19:18,009 DEBUG    MainThread Setting MQTT timestamped: True
2020-11-07 14:19:18,011 DEBUG    MainThread Setting MQTT subchannels: ['ToEmonCMS']
2020-11-07 14:19:18,013 INFO     MainThread Setting MQTT node_JSON_enable: 1
2020-11-07 14:19:18,021 INFO     MainThread Creating EmonHubEmoncmsHTTPInterfacer 'emoncmsorg'
2020-11-07 14:19:18,025 DEBUG    MainThread Setting emoncmsorg pubchannels: ['ToRFM12']
2020-11-07 14:19:18,027 DEBUG    MainThread Setting emoncmsorg subchannels: ['ToEmonCMS']
2020-11-07 14:19:18,028 WARNING  MainThread Setting emoncmsorg apikey: obscured
2020-11-07 14:19:18,051 INFO     MainThread Setting emoncmsorg url: https://emoncms.org
2020-11-07 14:19:18,052 INFO     MainThread Setting emoncmsorg senddata: 1
2020-11-07 14:19:18,054 INFO     MainThread Setting emoncmsorg sendstatus: 1

LOG LEVEL: DEBUG 

I think you are running the newer Image based on RaspberryOS, I wonder if something has changed in the serial configuration. We currently do;

Typing cat /etc/os-release returns this:

pi@emonTxPiZero1:~ $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@emonTxPiZero1:~ $ 

@haffle

I am able to interact with the emonHub (& emonCMS etc) fine, so comms at a certain level are working, but the emonHub log suggests that the process(es) just get stuck after a short period following a soft Re-Start.

I’ll say this again the connection to the Pi is via Wi-Fi and nothing to do with the connection between the Pi and the EmonTX.

OK, hands up, I hadn’t realised this (or at least hadn’t internalised it!), as I had assumed (yes, I know, I know) that the two devices communicated via the Rx & Tx pins (and to a lesser extent the RST pin). But I know now

I will unplug the emonTx & PiZero & open the case and re-check the wires etc. The only thing that is different to the other PiZero’s that might be causing interference is the (un-insulated) corrugated tin roof that it is near; I will bring it down from the purlin and see if that helps.

sudo journalctl -f

This brings up the following (and then “loops” the last few lines). Not sure why it is unable to resolve host emonTxPiZero1 as is shows that name at the cursor prompt. But then this is again at the edge of my knowledge.

pi@emonTxPiZero1:~ $ sudo journalctl -f
sudo: unable to resolve host emonTxPiZero1: No address associated with hostname
-- Logs begin at Sat 2020-11-07 11:20:12 GMT. --
Nov 07 14:33:01 emonTxPiZero1 systemd[1]: emonPiLCD.service: Succeeded.
Nov 07 14:33:01 emonTxPiZero1 systemd[1]: emonPiLCD.service: Service RestartSec=100ms expired, scheduling restart.
Nov 07 14:33:01 emonTxPiZero1 systemd[1]: emonPiLCD.service: Scheduled restart job, restart counter is at 32335.
Nov 07 14:33:01 emonTxPiZero1 systemd[1]: Stopped emonPi LCD driver.
Nov 07 14:33:01 emonTxPiZero1 systemd[1]: Starting emonPi LCD driver...
Nov 07 14:33:01 emonTxPiZero1 systemd[1]: Started emonPi LCD driver.
Nov 07 14:33:02 emonTxPiZero1 sudo[31786]:       pi : unable to resolve host emonTxPiZero1
Nov 07 14:33:02 emonTxPiZero1 sudo[31786]:       pi : problem with defaults entries ; TTY=pts/0 ; PWD=/home/pi ; USER=root ;
Nov 07 14:33:02 emonTxPiZero1 sudo[31786]:       pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/bin/journalctl -f
Nov 07 14:33:02 emonTxPiZero1 sudo[31786]: pam_unix(sudo:session): session opened for user root by pi(uid=0)
Nov 07 14:33:08 emonTxPiZero1 python3[31783]: emonPiLCD logging to: /var/log/emonpilcd/emonpilcd.log
Nov 07 14:33:08 emonTxPiZero1 python3[31783]: I2C LCD NOT DETECTED on either 0x['27', '3f'] ...exiting LCD script
Nov 07 14:33:08 emonTxPiZero1 systemd[1]: emonPiLCD.service: Succeeded.
Nov 07 14:33:08 emonTxPiZero1 systemd[1]: emonPiLCD.service: Service RestartSec=100ms expired, scheduling restart.
Nov 07 14:33:08 emonTxPiZero1 systemd[1]: emonPiLCD.service: Scheduled restart job, restart counter is at 32336.
Nov 07 14:33:08 emonTxPiZero1 systemd[1]: Stopped emonPi LCD driver.
Nov 07 14:33:08 emonTxPiZero1 systemd[1]: Starting emonPi LCD driver...
Nov 07 14:33:09 emonTxPiZero1 systemd[1]: Started emonPi LCD driver.
Nov 07 14:33:14 emonTxPiZero1 python3[31810]: emonPiLCD logging to: /var/log/emonpilcd/emonpilcd.log
Nov 07 14:33:14 emonTxPiZero1 python3[31810]: I2C LCD NOT DETECTED on either 0x['27', '3f'] ...exiting LCD script
Nov 07 14:33:14 emonTxPiZero1 systemd[1]: emonPiLCD.service: Succeeded.
Nov 07 14:33:15 emonTxPiZero1 systemd[1]: emonPiLCD.service: Service RestartSec=100ms expired, scheduling restart.
Nov 07 14:33:15 emonTxPiZero1 systemd[1]: emonPiLCD.service: Scheduled restart job, restart counter is at 32337.
Nov 07 14:33:15 emonTxPiZero1 systemd[1]: Stopped emonPi LCD driver.
Nov 07 14:33:15 emonTxPiZero1 systemd[1]: Starting emonPi LCD driver...
Nov 07 14:33:15 emonTxPiZero1 systemd[1]: Started emonPi LCD driver.
Nov 07 14:33:20 emonTxPiZero1 python3[31831]: emonPiLCD logging to: /var/log/emonpilcd/emonpilcd.log
Nov 07 14:33:20 emonTxPiZero1 python3[31831]: I2C LCD NOT DETECTED on either 0x['27', '3f'] ...exiting LCD script
Nov 07 14:33:21 emonTxPiZero1 systemd[1]: emonPiLCD.service: Succeeded.
Nov 07 14:33:21 emonTxPiZero1 systemd[1]: emonPiLCD.service: Service RestartSec=100ms expired, scheduling restart.
Nov 07 14:33:21 emonTxPiZero1 systemd[1]: emonPiLCD.service: Scheduled restart job, restart counter is at 32338.
Nov 07 14:33:21 emonTxPiZero1 systemd[1]: Stopped emonPi LCD driver.
Nov 07 14:33:21 emonTxPiZero1 systemd[1]: Starting emonPi LCD driver...
Nov 07 14:33:21 emonTxPiZero1 systemd[1]: Started emonPi LCD driver.
Nov 07 14:33:27 emonTxPiZero1 python3[31850]: emonPiLCD logging to: /var/log/emonpilcd/emonpilcd.log
Nov 07 14:33:27 emonTxPiZero1 python3[31850]: I2C LCD NOT DETECTED on either 0x['27', '3f'] ...exiting LCD script
Nov 07 14:33:28 emonTxPiZero1 systemd[1]: emonPiLCD.service: Succeeded.
Nov 07 14:33:28 emonTxPiZero1 systemd[1]: emonPiLCD.service: Service RestartSec=100ms expired, scheduling restart.
Nov 07 14:33:28 emonTxPiZero1 systemd[1]: emonPiLCD.service: Scheduled restart job, restart counter is at 32339.
Nov 07 14:33:28 emonTxPiZero1 systemd[1]: Stopped emonPi LCD driver.
Nov 07 14:33:28 emonTxPiZero1 systemd[1]: Starting emonPi LCD driver...
Nov 07 14:33:28 emonTxPiZero1 systemd[1]: Started emonPi LCD driver.
Nov 07 14:33:34 emonTxPiZero1 python3[31871]: emonPiLCD logging to: /var/log/emonpilcd/emonpilcd.log
Nov 07 14:33:34 emonTxPiZero1 python3[31871]: I2C LCD NOT DETECTED on either 0x['27', '3f'] ...exiting LCD script
Nov 07 14:33:34 emonTxPiZero1 systemd[1]: emonPiLCD.service: Succeeded.
Nov 07 14:33:35 emonTxPiZero1 systemd[1]: emonPiLCD.service: Service RestartSec=100ms expired, scheduling restart.
Nov 07 14:33:35 emonTxPiZero1 systemd[1]: emonPiLCD.service: Scheduled restart job, restart counter is at 32340.
Nov 07 14:33:35 emonTxPiZero1 systemd[1]: Stopped emonPi LCD driver.
Nov 07 14:33:35 emonTxPiZero1 systemd[1]: Starting emonPi LCD driver...
Nov 07 14:33:35 emonTxPiZero1 systemd[1]: Started emonPi LCD driver.

[borpin (Brian Orpin)]

@haffle

emonPiLCD.service

That really doesn’t need to be running

sudo systemctl stop emonPiLCD.service
sudo systemctl disable emonPiLCD.service
sudo systemctl mask emonPiLCD.service

Try pressing the EmonTX reset, while watching this syslog.

@haffle

The only thing that is different to the other PiZero’s that might be causing interference is the (un-insulated) corrugated tin roof

No that isn’t going to be the problem.

@haffle

I may have missed something, but it is; I have just logged back in and the EmonHub log of the PiZero in question is still the same as earlier today

That implies there is nothing coming from the EmonTX.

How is this pairing powered? PiZero powering the EmonTX or the other way round?

Try a different power supply.

You could try stopping emonhub and connecting to the TX with miniterm.