ValueError: invalid literal for int() with base 10: 'OK'

brandock · 22 January 2023 05:52

@pb66 and @TrystanLea, this is probably a question for you, I’m guessing.

I recently did a write-up about using a Moteino as an RFM69Pi with the JeeLib Classic format. I also contributed to a topic, explaining how I used a Moteino LoRa repeating JeeLib Classic packets into the serial interfacer to bridge a long distance to an emonBase.

In these examples I was telling people to use Type = EmonHubSerialInterfacer in the emonHub configuration:

[[SerialDirect]]
     Type = EmonHubSerialInterfacer
      [[[init_settings]]]
           com_port = /dev/ttyUSB0 
           com_baud = 38400
      [[[runtimesettings]]]
           pubchannels = ToEmonCMS,
           subchannels = ToRFM12,

Then I set about writing up how I have, in the past, used a Moteino running the LowPowerLab RF69 library to collect packets from other such Moteinos and format it correctly for the emonHub serial interfacer.

When I went to test my old LowPowerLab-based sketch, it would not work with my emonBase unless I used the EmonHubJeeInterfacer. If I kept the Type=EmonHubSerialInterfacer the non-numeric ‘OK’ would cause errors:

ValueError: invalid literal for int() with base 10: ‘OK’

It seemed odd to me that I wasn’t getting those errors with the sketches I had recently been posting about, with the JeeLibClassic-to-USB and the RadioHead-LoRa-to-USB technique. These also produced the characters “OK” at the front of the serial output.

So I went back to re-test my JeeLibClassic Moteino USB-based RFM69Pi that I had so recently posted about. Sure enough, the “OK” problem tripped me up.

The TLDR for those out there that run into this problem is: use the EmonHubJeeInterfacer type instead. It works for all the USB-based interfacers I mention above.

But for me, I’d like to know why I was getting the emonHubSerialInterfacer to work yesterday (and in fact it is still working in the case of the LoRa setup I implemented last Fall and haven’t touched since) but it is not working for my tests today.

One other clue. I have two Moteinos I am testing with and I see two different errors using them with emonHubSerialInterfacer. I can’t say for certain at this time if which error I get is related to which Moteino I use, or what might be making that difference.

2023-01-22 04:40:43,279 WARNING  SerialDirect Exception caught in SerialDirect thread. Traceback (most recent call last):
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 31, in wrapper
    return func(*args)
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 98, in run
    rxc = self.read()
  File "/opt/openenergymonitor/emonhub/src/interfacers/EmonHubSerialInterfacer.py", line 90, in read
    c.nodeid = int(f[0])
ValueError: invalid literal for int() with base 10: 'OK'

and

2023-01-22 04:38:54,959 WARNING  SerialDirect Exception caught in SerialDirect thread. Traceback (most recent call last):
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 31, in wrapper
    return func(*args)
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 98, in run
    rxc = self.read()
  File "/opt/openenergymonitor/emonhub/src/interfacers/EmonHubSerialInterfacer.py", line 69, in read
    self._rx_buf = self._rx_buf + self._ser.readline().decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 0: invalid start byte

And here is the sketch I am using in my Moteino USB: RFM2Pi/firmware/RFM69CW_RF_Demo_ATmega328/RFM69CW_RF12_Demo_ATmega328 at master · openenergymonitor/RFM2Pi · GitHub

Robert.Wall · 22 January 2023 11:46

Only in the last couple of days, another user of the serial interfacers has had problems. It seems that there’s a requirement for some documentation describing the function and capabilities of each, so that people are able to choose the correct one based on the data format they have. I’ve already raised this in a private ‘staff’ topic.

In your case, has the interfacer itself changed, or could the version of Python have caused the error?

borpin · 22 January 2023 13:52

Can you check your version of emonhub? In the current version, that statement is not at line 69.

github.com

openenergymonitor/emonhub/blob/afa562bb371a2a426c80ed15a96d808efefc2fed/src/interfacers/EmonHubSerialInterfacer.py#L69


      
                  s = False
                  # raise EmonHubInterfacerInitError('Could not open COM port %s' % com_port)
              return s
          
          
def read(self):
              """Read data from serial port and process if complete line received.
          
          
    Return data as a list: [NodeID, val1, val2]
          
          
    """
          
          
    if not self._ser:
                  return False
          
          
    # Read serial RX
              self._rx_buf = self._rx_buf + self._ser.readline().decode()
          
          
    # If line incomplete, exit
              if '\r\n' not in self._rx_buf:
                  return

brandock · 22 January 2023 14:09

(I woke up this morning thinking, I wonder if I should have asked @borpin.)

borpin · 22 January 2023 14:11

Try a full update - emoncms is behind as is emonhub.

From the components tab (under Admin), you can see what you are at v the current versions of each component.

You might well still get the error, but need to be

brandock · 22 January 2023 14:15

No change to the interfacer. No updates at all to the emonBase between tests.

No change to the version of Python (nor the emonHub python script) between tests, so I don’t think so.

Python 2.7.16 (default, Oct 10 2019, 22:02:15)
[GCC 8.3.0] on linux2

borpin · 22 January 2023 14:17

Do you know the serial data sent is correct?

[edit]
I think (and I’m not sure as byte encoding is not my thing) but fundamentally,

Is not a utf-8 character - certainly it isn’t a numeral.

However, it should not bomb, so a try catch is needed (and in other serial based interfacers too probably)

brandock · 22 January 2023 16:25

This morning I can’t seem to recreate the exception for that 0x83 byte. Here is the output in the serial monitor using the RFM69CW_RF_Demo_ATmega328 sketch in both the Moteinos I am testing.

[RF12demo.14] O i15 g210 @ 433 MHz q1
OK 8 255 255 215 255 5 0 120 0 52 48 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-57) 
OK 23 0 223 95 13 15 0 0 0 (-62) 
OK 29 0 0 0 0 0 0 231 12 (-69) 
OK 10 123 2 218 5 17 0 0 0 227 47 184 11 184 11 184 11 184 11 184 11 184 11 0 0 (-73) 
OK 8 0 0 214 255 3 0 119 0 1 48 184 11 184 11 184 11 184 11 184 11 184 11 0 0 0 0 (-59) 
OK 21 200 0 0 0 59 1 25 0 (-58)

And here is the result in emonHub logs when using Type = EmonHubSerialInterfacer

2023-01-22 15:36:16,632 WARNING  SerialDirect Exception caught in SerialDirect thread. Traceback (most recent call last):
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 31, in wrapper
    return func(*args)
  File "/opt/openenergymonitor/emonhub/src/emonhub_interfacer.py", line 98, in run
    rxc = self.read()
  File "/opt/openenergymonitor/emonhub/src/interfacers/EmonHubSerialInterfacer.py", line 90, in read
    c.nodeid = int(f[0])
ValueError: invalid literal for int() with base 10: 'OK'

I was able to get logged in to the emonBase running the LoRa-to-serial Moteino at my father’s house. Mystery solved over there. Over there I had pointed the RFM2Pi JeeInterfacer to the USB port.

[[RFM2Pi]]
    Type = EmonHubJeeInterfacer
    [[[init_settings]]]
        com_port = /dev/ttyUSB0
        com_baud = 38400                        # 9600 for old RFM12Pi
    [[[runtimesettings]]]
        pubchannels = ToEmonCMS,
        subchannels = ToRFM12,

So there is less and less evidence that I have successfully implemented USB with the emonHubSerialInterfacer type, at least in a while.

Regarding the 0x83 byte that I haven’t been able to recreate this morning, I was wondering if it could have occurred last night when I was testing the emonBase_rfm69pi_LPL sketch in a Moteino. I noticed that the output sometimes has a pipe.

|Settings
|Radio: 1
|RF Band: 433MHz
|Power: 7 dBm
|Group: 210
|Node ID: 5

This occurs in the Arduino Serial Monitor when the sketch uses Serial.print(F(“whatever content”)). I did some testing to see if that would trigger the 0x83 starting byte error, but it did not seem to. At this point I am really not sure how that one came about.

My main question at this point is: what emonHub interfacer should be used for a USB-based RFM69Pi, since these will include the “OK” prefix in the output? Should the emonHubSerialInterfacer type be expected to work for this purpose going forward? My understanding is no. Instead, the emonHubJeeInterfacer would be the intended interfacer type for this use case.

Robert.Wall · 22 January 2023 17:49

Unless I’m being thick, that isn’t serial data of the sort any OEM Serial Interfacer will understand. It looks too much like byte values of our standard radio packet presented on the serial monitor as readable values.

Here are definitions of two serial output formats as produced by the 3-phase emonTx V3 sketch:

<nodeID> <decimalValue1> <decimalValue2> <decimalValue3> [etc] <decimalValue*N*><newline>

(The numbers are integers.)

 <varName1>:<decimalValue1>,<varName2>:<decimalValue2>,<varName3>:<decimalValue3>,[etc]<varName*N*>:<decimalValue*N*>,<newline>

Note there are no spaces between any of the elements. The numbers can be integers or floats.

It’s not a pipe. The vertical bar is simply a marker to indicate that it’s human-readable output not for decoding (and thus throwing errors because it’s not one of the expected serial formats).

This indicates the checksum on the radio packet was correct.

You are correct there.

TrystanLea · 22 January 2023 19:02

To confirm, the emonHubSerialInterfacer interfacer is not expected to work for this. It is only the EmonHubJeeInterfacer and newer EmonHubOEMInterfacer that include the code to decode these byte value string packets.

brandock · 22 January 2023 19:07

I spent some time going through my old notes and I think I’ve figured out how I got confused.

Over the years I’ve used the Moteinos in two ways:

With the EmonHubSerialInterfacer and sketches that do produce expected serial output formats. Like this format from 2015

    Serial.print(theData.nodeId);Serial.print(' ');
    Serial.print(theData.uptime);Serial.print(' ');
    Serial.print(theData.temp);Serial.print(' ');
    Serial.print(radio.RSSI);Serial.print(' ');
    Serial.print(theData.moisture);
    Serial.println();

As a replacement for the RFM2Pi hat, using the EmonHubJeeInterfacer of the default emonHub configuration and replacing ttyAMA0 with ttyUSB0.

When I went to resurrect (2) and post about it, I looked at my old notes about Interfacers, grabbed the settings for (1) and ran into the problems mentioned above.

I suspect that all the times I thought I had (2) working with EmonHubSerialInterfacer, I was actually doing (1) and not paying close attention, or forgetting how I had done it.

Note to self: The EmonHubJeeInterfacer does present the standard radio packets as serial data to the emonHub python script. The difference is that the JeeInterfacer in that script knows this is a list of byte values from the standard radio packets. The SerialInterfacer does not.

If nothing else, if I get confused about this again, I can go back and refer to this. Thanks all.