RX data length: 9 is not valid for datacodes

emonHub
1

Hi,

I’ve got an emonTx v3.4 connected to the serial on a Pi Zero, running EmonHub (on the latest “Raspberry Pi OS”).

My serial data is like:

MSG:276,Vrms:246.41,P1:467,P2:478,P3:480,E1:1095,E2:1121,E3:1117,pulse:1096

My emonhub.conf is like:

[interfacers]
[[SerialTx]]
    Type = EmonHubTx3eInterfacer
      [[[init_settings]]]
          com_port= /dev/ttyAMA0
          com_baud = 115200
      [[[runtimesettings]]]
          pubchannels = ToEmonCMS,

          nodeoffset = 1

[[MQTT]]

    Type = EmonHubMqttInterfacer
    [[[init_settings]]]
        mqtt_host = 10.0.0.10
        mqtt_port = 1883
        mqtt_user = USER
        mqtt_passwd = PASS

    [[[runtimesettings]]]
        subchannels = ToEmonCMS,

        # emonhub/rx/10/values format
        # Use with emoncms Nodes module
        node_format_enable = 1
        node_format_basetopic = emonhub/

        # emon/emontx/power1 format - use with Emoncms MQTT input
        # http://github.com/emoncms/emoncms/blob/master/docs/RaspberryPi/MQTT.md
        nodevar_format_enable = 1
        nodevar_format_basetopic = emon/

[[emoncms]]
    Type = EmonHubEmoncmsHTTPInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        # pubchannels = ToRFM12,
        subchannels = ToEmonCMS,
        url = http://10.0.0.20
        apikey = MYKEY
        senddata = 1
        sendstatus = 1


[nodes]

[[1]]
    nodename = emontx
    [[[rx]]]
       names = MSG, Vrms, P1, P2, P3, E1, E2, E3, pulse
       datacodes = L,h,h,h,h,L,L,L,L
       scales = 1,1,1,1,1,0.1,0.1,0.1,1
       units = n,V,W,W,W,kWh,kWh,kWh,p

but I just keep getting this, whatever I try:

2020-08-31 17:10:47,756 INFO     MainThread EmonHub emonHub (emon-pi variant) v2.1.5
2020-08-31 17:10:47,766 INFO     MainThread Opening hub...
2020-08-31 17:10:54,809 WARNING  SerialTx   1 RX data length: 9 is not valid for datacodes ['L', 'h', 'h', 'h', 'h', 'L', 'L', 'L', 'L']
2020-08-31 17:11:04,796 WARNING  SerialTx   2 RX data length: 9 is not valid for datacodes ['L', 'h', 'h', 'h', 'h', 'L', 'L', 'L', 'L']

I’m clearly missing something - can anyone see why?

Thanks,

Ian

2

The payload is already in decimal format when it comes via serial so you do not need to decode anything. So datacode = 0 or datacodes = 0,0,0,0,0,0,0,0,0, but the default for that interfacer should be 0 so you shouldn’t need to define any datacode(s).

The datacode s you have defined make emonhub expect 28 byte values but it receives 9 decimal floats values (correct myself before someone else does :slight_smile:), hence it complains.

3

Ahhh, ok! - thanks!

1 Like
4

I removed the datacodes line and it works now! :slight_smile:

EmonCMS has created a node for it, but all of the inputs are numerical - 0,1…9 - is that correct? - I thought it would use the names.

Is there any need/point defining names, scales and units in emoncms.conf? - does it not use them when posting them to emoncms?

Thanks,

Ian

5

If you are using the http bulk input api then yes that is normal. the buk api has been left behind due to the focus on MQTT over recent years and created a divergence. @TrystanLea did start on work to converge the 2 input methods (named and indexed) but I don’t think it was included in the move to v10 (see Development: Indexed Inputs and GitHub - emoncms/emoncms at indexedinputs).

If using http, then no, aside from personal notes the names are not used. If you use MQTT the names are used as topics and carried over to emoncms inputs automatically. But the scales are used for both, but rarely needed for decimal values over serial, they are more for use with integers when (for example) 240V is passed as 24000 to allow greater precision to hundredths of a volt.

You could find the scales useful for quick and dirty minor calibration adjustments and/or aligning pulse counts to watts where 1 pulse doesn’t equal 1 Watt.

6

Ah, interesting.

So am I better off removing the emoncms stuff from emonhub and having it just publish to MQTT, then link emoncms to MQTT?

Will emonhub read/process every single message on the serial interfacer?

Thanks,

Ian

7

If you rate auto naming inputs higher than more robust dat then you could. It is the OEM standard way of doing things to post to emoncms via mqtt locally and most of the time it’s ok. However, aside from the annoyance of the indexed inputs (but you can still use the description field for naming/describing) the http method is more robust (therefore can be more accurate) as it timestamps the data as it arrives via serial and that timestamp sticks regardless of how long it takes to get to emoncms, MQTT is timestamped at emoncms (that’s why MQTT is “ok” on local only). also, the order of the inputs landing at emoncms is fixed with http due to the indexed positions, where as MQTT splits up the once whole payload into individual topics that can arrive at emoncms in a different order or all at once. This is an issue when you have cross input processing in emoncms. For example if you sum ct1 and ct2 in ct2’s input, and then a ct1 reading get delayed the ct2 value will be added to the previous ct1 value as the ct1 value hasn’t landed at emoncms at the time of the calculation.

In short, MQTT is more trendy, named inputs are prettier, but if you want to ensure you get the best chance at solid reliable data, use http, even locally.

Yes, well as long as the interval isn’t ridiculously low eg 0.1 seconds etc. The MQTT side will just fire off all the values in every payload without timestamps, fixed ordering or confirmed receipt, the http interfacer will send in (user) defined batches and confirm reciept before sending the next batch.

8

Ahhh, ok! - that’s fine then!

Though from what I can see, emonhub seems to only be taking a reading every 30s or so, despite the emonTx taking a reading every 10s.

If I watch (and even refresh) the input screen on EmonCMS, the readings get <30s old before the next one appears.

Is there a way to speed it up and ensure it sees every reading?

Thanks,

Ian

9

No what you are seeing is the data being delivered in batches at 30s intervals, regardless of whether the emonTx interval is 10s or 5s etc, all the data from as many devices as you have connected since the last 30s interval are being buffered and passed in time order so that your input processing is predictable and methodically done as if the data all arrived instantaneously. This is basically a traffic calming method, when you have lots of devices and/or short intervals the target server may struggle with so many individual requests, emoncms.org certainly can with so many users, hence the 30s default setting. But rest assured, no data is dropped or lost with the http bulk upload api, that’s the main reason it is superior to the current MQTT inplementation.

Firstly it will see every reading! and in chronological order using the true timestamps.

You can set interval = 10 in the http interfacer’s runtimesettings in emonhub.conf. This setting is a simple throttling of outgoing requests, not the content of the batched requests, setting 1s would give you near realtime uploads but if 5 devices are posting 2s apart that would result in 5 requests every 10s and offer no advantage over a 10s interval sending all 5 every 10s in one request.

10

Makes sense - thanks!

11

Once this PR101 is accepted you can send timestamped MQTT JSON messages.

Again solved by that PR.

There is probably nothing to stop the creation of an HTTP interfacer that sends a JSON payload probably using the new bulk upload API. Best of both worlds.

12

@pb66 in case you have missed it, there are numerous reports of the HTTP interfacer stalling recently. It looks like it may have been caused by migrating to Python3, but so far the cause and the solution have evaded us.