New node data not showing in Emoncms.org

[lgheorghe (Liviu Gheorghe)]

Hello,

I have a emonpi installation which is used to log power, vrms and temperature to local emoncms and emoncms.org. I am using also 2 emonth’s for temperature and humidity, also logging local and emoncms.org. Everything works ok.

I recently added another node in my emonpi config - node 99 - for some temperature, humidity and voltage data from a remote device. The data is pulled form the remote device using a python script which send the data to my local emoncms.
The configuration added to my emonhub.conf is:

[[99]]
    nodename = lan_restarter
    [[[rx]]]
       names = temp1, ext temp, temp2, temp0, int temp,humidity,vcc
       datacodes = h,h,h,h,h,h,h
       scales = 0.1,0.1,0.1,0.1,0.1,0.1,0.1
       units = C,C,C,C,C,%,V

I see the inputs for node 99 in my local emoncms, I configured them to Log to feed - everything is fine with the local emoncms.

My issue is that I cannot see the data regarding node 99 in emoncms.org. All the other data from emonpi and the 2 emonth are sent successful to emoncms.org:

2018-01-29 08:42:11,347 DEBUG    RFM2Pi     15991 NEW FRAME : OK 23 223 0 55 0 71 1 15 0 1 0 0 0 (-50)
2018-01-29 08:42:11,350 DEBUG    RFM2Pi     15991 Timestamp : 1517215331.35
2018-01-29 08:42:11,351 DEBUG    RFM2Pi     15991 From Node : 23
2018-01-29 08:42:11,352 DEBUG    RFM2Pi     15991    Values : [22.3, 5.5, 32.7, 1.5, 1]
2018-01-29 08:42:11,352 DEBUG    RFM2Pi     15991      RSSI : -50
2018-01-29 08:42:11,353 DEBUG    RFM2Pi     15991 Sent to channel(start)' : ToEmonCMS
2018-01-29 08:42:11,354 DEBUG    RFM2Pi     15991 Sent to channel(end)' : ToEmonCMS
2018-01-29 08:42:11,547 DEBUG    MQTT       Publishing: emon/emonth5/temperature 22.3
2018-01-29 08:42:11,549 DEBUG    MQTT       Publishing: emon/emonth5/external temperature 5.5
2018-01-29 08:42:11,550 DEBUG    MQTT       Publishing: emon/emonth5/humidity 32.7
2018-01-29 08:42:11,552 DEBUG    MQTT       Publishing: emon/emonth5/battery 1.5
2018-01-29 08:42:11,554 DEBUG    MQTT       Publishing: emon/emonth5/pulsecount 1
2018-01-29 08:42:11,555 INFO     MQTT       Publishing: emon/emonth5/rssi -50
2018-01-29 08:42:11,557 INFO     MQTT       Publishing: emonhub/rx/23/values 22.3,5.5,32.7,1.5,1
2018-01-29 08:42:11,558 INFO     MQTT       Publishing: emonhub/rx/23/rssi -50
2018-01-29 08:42:11,984 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 0 34 0 55 0 71 1 15 0 1 0 0 0 39 163 0 32 32 0 32 0 (-106)


2018-01-29 08:42:28,725 DEBUG    MQTT       Publishing: emon/emonpi/power1 737
2018-01-29 08:42:28,727 DEBUG    MQTT       Publishing: emon/emonpi/power2 0
2018-01-29 08:42:28,728 DEBUG    MQTT       Publishing: emon/emonpi/power1pluspower2 737
2018-01-29 08:42:28,730 DEBUG    MQTT       Publishing: emon/emonpi/vrms 219.77
2018-01-29 08:42:28,731 DEBUG    MQTT       Publishing: emon/emonpi/t1 24.3
2018-01-29 08:42:28,733 DEBUG    MQTT       Publishing: emon/emonpi/t2 0
2018-01-29 08:42:28,735 DEBUG    MQTT       Publishing: emon/emonpi/t3 0
2018-01-29 08:42:28,736 DEBUG    MQTT       Publishing: emon/emonpi/t4 0
2018-01-29 08:42:28,738 DEBUG    MQTT       Publishing: emon/emonpi/t5 0
2018-01-29 08:42:28,739 DEBUG    MQTT       Publishing: emon/emonpi/t6 0
2018-01-29 08:42:28,741 DEBUG    MQTT       Publishing: emon/emonpi/pulsecount 0
2018-01-29 08:42:28,742 INFO     MQTT       Publishing: emon/emonpi/rssi 0
2018-01-29 08:42:28,744 INFO     MQTT       Publishing: emonhub/rx/5/values 737,0,737,219.77,24.3,0,0,0,0,0,0
2018-01-29 08:42:28,745 INFO     MQTT       Publishing: emonhub/rx/5/rssi 0
2018-01-29 08:42:28,809 DEBUG    RFM2Pi     15996 NEW FRAME : OK 24 240 0 0 0 41 1 28 0 1 0 0 0 (-63)
2018-01-29 08:42:28,811 DEBUG    RFM2Pi     15996 Timestamp : 1517215348.81
2018-01-29 08:42:28,812 DEBUG    RFM2Pi     15996 From Node : 24
2018-01-29 08:42:28,813 DEBUG    RFM2Pi     15996    Values : [24, 0, 29.700000000000003, 2.8000000000000003, 1]
2018-01-29 08:42:28,815 DEBUG    RFM2Pi     15996      RSSI : -63
2018-01-29 08:42:28,815 DEBUG    RFM2Pi     15996 Sent to channel(start)' : ToEmonCMS
2018-01-29 08:42:28,819 DEBUG    RFM2Pi     15996 Sent to channel(end)' : ToEmonCMS
2018-01-29 08:42:28,964 DEBUG    MQTT       Publishing: emon/emonth6/temperature 24
2018-01-29 08:42:28,965 DEBUG    MQTT       Publishing: emon/emonth6/external temperature 0
2018-01-29 08:42:28,966 DEBUG    MQTT       Publishing: emon/emonth6/humidity 29.7
2018-01-29 08:42:28,967 DEBUG    MQTT       Publishing: emon/emonth6/battery 2.8
2018-01-29 08:42:28,967 DEBUG    MQTT       Publishing: emon/emonth6/pulsecount 1
2018-01-29 08:42:28,968 INFO     MQTT       Publishing: emon/emonth6/rssi -63
2018-01-29 08:42:28,969 INFO     MQTT       Publishing: emonhub/rx/24/values 24,0,29.7,2.8,1
2018-01-29 08:42:28,970 INFO     MQTT       Publishing: emonhub/rx/24/rssi -63
2018-01-29 08:42:32,405 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 0 0 0 34 32 0 0 0 32 32 0 32 0 32 0 0 0 0 32 0 32 (-106)
2018-01-29 08:42:33,540 DEBUG    RFM2Pi     15997 NEW FRAME : OK 5 233 2 0 0 233 2 208 85 243 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 (-0)

2018-01-29 08:42:36,921 DEBUG    emoncmsorg Sending bulkdata, length: 8
2018-01-29 08:42:36,923 INFO     emoncmsorg sending: https://emoncms.org/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1517215328.45389,5,751,0,751,220.26,24.3,0,0,0,0,0,0],[1517215331.34766,23,22.3,5.5,32.7,1.5,1,-50],[1517215333.519791,5,758,0,758,219.77,24.3,0,0,0,0,0,0],[1517215338.478148,5,769,0,769,220.04,24.3,0,0,0,0,0,0],[1517215343.463304,5,738,0,738,220.11,24.3,0,0,0,0,0,0],[1517215348.485342,5,737,0,737,219.77,24.3,0,0,0,0,0,0],[1517215348.808746,24,24,0,29.700000000000003,2.8000000000000003,1,-63],[1517215353.540321,5,745,0,745,219.68,24.3,0,0,0,0,0,0]]&sentat=1517215356
2018-01-29 08:42:37,309 DEBUG    RFM2Pi     Discarding RX frame 'unreliable content'? 0 32 0 34 32 0 0 0 32 32 0 32 0 32 0 0 0 0 32 0 32 (-103)
2018-01-29 08:42:37,387 DEBUG    emoncmsorg acknowledged receipt with 'ok' from https://emoncms.org
2018-01-29 08:42:37,387 DEBUG    emoncmsorg Sending bulkdata, success: True
2018-01-29 08:42:37,389 DEBUG    emoncmsorg Current queue length: 0
2018-01-29 08:42:37,390 INFO     emoncmsorg sending: https://emoncms.org/myip/set.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y

What am I missing?

[TrystanLea]

Hello @lgheorghe

Your log does not seem to show either the node id 99 or nodename lan_repeater, can you copy in or attach an extended portion of the log? If you see the data locally it should be in there somewhere.

[pb66 (Paul)]

@lgheorghe

The data is pulled form the remote device using a python script which send the data to my local emoncms.

If your data is being sent direct to the local emoncms it is bypassing emonhub, therefore it will not be sent to other targets.

How does your Python script send it’s data? Where is the Python script running? How does it “pull data” from the remote device? Where and what is the remote device?

[lgheorghe (Liviu Gheorghe)]

Well… that was what I tried to say - in the /var/log/emonhub/emonhub.log there is no reference to node id 99 or nodename lan_restarter.
Data is coming in as seen in the pictures:

[lgheorghe (Liviu Gheorghe)]

The script is running on another linux machine and is sending the data to the IP address of the emonPi:

#!/usr/bin/env python

import urllib
import xml.etree.ElementTree as ET
import string

emoncms_prod = "http://172.17.1.106"
lr_url = 'http://1.1.1.161/st0.xml'
nodeid = 99
apikey_prod = "******"

doc = urllib.urlopen(lr_url).read()

tree = ET.fromstring(doc)

temp0 = float(tree.find('ia0').text)/10
vcc = float(tree.find('ia1').text)/10
temp1 = float(tree.find('ia10').text)/10
tempext = float(tree.find('ia11').text)/10
temp2 = float(tree.find('ia12').text)/10
tempint = float(tree.find('ia13').text)/10
humidity = float(tree.find('ia14').text)/10

print 'Temp0: ', temp0
print 'VCC: ', vcc
print 'Temp1: ', temp1
print 'Temp ext: ', tempext
print 'Temp2: ', temp2
print 'Temp int: ', tempint
print 'Humidity: ', humidity

path_prod = "/emoncms/input/post?node=99&fulljson={'temp1':"+str(temp1)+",'ext temp':"+str(tempext)+",'temp2':"+str(temp2)+",'temp0':"+str(temp0)+",'int temp':"+str(tempint)+",'humidity':"+str(humidity)+",'vcc':"+str(vcc)+"}&apikey="+apikey_prod

prod = urllib.urlopen(emoncms_prod+path_prod).read()
print prod

[lgheorghe (Liviu Gheorghe)]

The remote device is a cheap bord with a LAN connection and the possibility to connect other sensors - temperature, humidity, ACS712 current sensors, etc.
It’s connected to my wired local area network.

[TrystanLea]

Aha, are you posting locally via the http API? if so you are bypassing
emonhub completely and hence no log. To post to emoncms.org in this case
you will need to repeat the api call in your lan repeater script to post to
emoncms.org as well

[lgheorghe (Liviu Gheorghe)]

How do I send data to my emonpi in order to be processed by emonhub?
I used the examples provided in the Input API

[lgheorghe (Liviu Gheorghe)]

Repeating the API call to post to emoncms.org did cross my mind, but I want to use the emonpi as hub for a number of devices, all of them posting to emonpi/emonhub and the emonpi posting to emoncms.org.

[pb66 (Paul)]

@TrystanLea

To post to emoncms.org in this case

you will need to repeat the api call in your lan repeater script to post to

emoncms.org as well

OR

send your data to a socket on the emonPi emonhub, which will give you the facility to send the data to other targets, publish the data as mqtt and have the data buffered in case of WAN delays/outages.

OR

install emonhub at the source device too, this will give the option to send to both targets and buffer data locally before the LAN, which I’m guessing might be useful as you have a device called “lan_restarter”.

[lgheorghe (Liviu Gheorghe)]

@pb66

send your data to a socket on the emonPi emonhub, which will give you the facility to send the data to other targets, publish the data as mqtt and have the data buffered in case of WAN delays/outages.

is there a guide on how to implement this option?

[pb66 (Paul)]

Unfortunately not AFAIK.

Search the forums (both this and the archived one) for “emonhub socket interfacer”, there are a fair few examples and discussions out there.

[TrystanLea]

Never used the socket interfacer myself. I have added this to the emonhub issue list to look into: Document emonhub socket interfacer · Issue #46 · openenergymonitor/emonhub · GitHub

There does seem to be a number of search results when I try “emonhub socket interfacer” as you suggest @pb66 so it may all be out there.

Correction, my bad, @pb66 does actively use the socket interfacer, there will be more to come on this soon

[pb66 (Paul)]

Ahh! When you originally said “(thats two of the core developers of emonhub!?)” had never used them, I assumed you meant you and Glyn, I didn’t realise you were referring to me. Yes, I use a lot of socket interfacers, in many different forms, I would be lost without them.

[lgheorghe (Liviu Gheorghe)]

I found this post

MODBUS TCP/IP Python send to EmonCMS

Here is an example script that posts values 0-9 every 5 seconds, I’ll happily help you merge the two, but I’m not sure how you want to stage the data based on the discussion above, will you want to simply forward every reg as it is read “hot potato” style or will you collate the data into batches or build up a single complete frame. The best way IMO would to be send one complete packet. #!/usr/bin/env python import time import socket host = "localhost" port = 50012 nodeid = 13 interval = 5 …

which seems to be what I was looking for.
I have to questions:

1. if I understood correctly, you have to send to emonhub socket, timestamp+node id+data separated by spaces
2. didn’t quite understand what you meant by

@pb66

plus the emoncms reporter will need your apikey and url

[pb66 (Paul)]

@lgheorghe

if I understood correctly, you have to send to emonhub socket, timestamp+node id+data separated by spaces

you can, if you wish send it in that format and you set timestamped = true in the [[runtimesettings]] as per that example, or if timestamped = false (or omited, defaults to false) you can send as just nodeid val1 val2 val2 and a timestamp will be allocated by emonhub on receipt.

@lgheorghe

plus the emoncms reporter will need your apikey and url

That thread refers to original emonhub as opposed to the emonpi variant, there are no reporters in the emonpi variant. It simply means the url and write apikey need to be set to allow emonhub to post to a remote emoncms via http(s) (eg emoncms.org). you should have something like

[[emoncmsorg]]
    Type = EmonHubEmoncmsHTTPInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,
        url = https://emoncms.org
        apikey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        senddata = 0                    # Enable sending data to Emoncms.org
        sendstatus = 1                  # Enable sending WAN IP to Emoncms.org MyIP > https://emoncms.org/myip/list
        sendinterval= 30                # Bulk send interval to Emoncms.org in seconds

in your emonhub.conf, in which case you need to edit/add your own apikey.

[lgheorghe (Liviu Gheorghe)]

It seems there is an issue sending to the emonhub socket.
I set up the EmonhubSocketInterfacer:

[[LanRestarter]]
        Type = EmonHubSocketInterfacer
        [[[init_settings]]]
                port_nb = 50012
        [[[runtimesettings]]]
                timestamped = true

and a test node:

[[98]]
    nodename = nodeMCU
    [[[rx]]]
       names = temp1, ext temp, temp2, humidity
       datacodes = h,h,h,h
       scales = 0.1,0.1,0.1,0.1
       units = C,C,C,%

When sending to emonhub, timestamp nodeid val1 val2 val3 val4, i get the following message in emonhub.log:

2018-01-29 23:58:09,808 DEBUG    LanRestarter 1 NEW FRAME : 1517263080.0 98 3.8 1.6 5.2 80.3
2018-01-29 23:58:09,810 WARNING  LanRestarter 1 RX data length: 4 is not valid for datacodes ['h', 'h', 'h', 'h']

Changing timestamped = false in the [[runtimesettings]] and sending nodeid val1 val2 val3 val4, produces this message in emonhub.log:

2018-01-30 09:53:22,282 DEBUG    MainThread Setting LanRestarter timestamped: false
2018-01-30 09:54:30,967 WARNING  LanRestarter Exception caught in LanRestarter thread. Traceback (most recent call last):
  File "/home/pi/emonhub/src/interfacers/emonhub_interfacer.py", line 38, in wrapper
    return f(*args)
  File "/home/pi/emonhub/src/interfacers/emonhub_interfacer.py", line 87, in run
    rxc = self.read()
  File "/home/pi/emonhub/src/interfacers/EmonHubSocketInterfacer.py", line 119, in read
    c.nodeid = int(f[0]) + int(self._settings['nodeoffset'])
ValueError: invalid literal for int() with base 10: '3.1'

2018-01-30 09:54:30,994 WARNING  MainThread LanRestarter thread is dead.
2018-01-30 09:54:30,995 WARNING  MainThread Attempting to restart thread LanRestarter (thread has been restarted 0 times...
2018-01-30 09:54:30,998 INFO     MainThread Creating EmonHubSocketInterfacer 'LanRestarter' 
2018-01-30 09:54:30,999 DEBUG    MainThread Opening socket on port 50012
2018-01-30 09:54:31,000 ERROR    MainThread [Errno 98] Address already in use
2018-01-30 09:54:31,001 ERROR    MainThread Unable to create 'LanRestarter' interfacer: global name 'EmonHubInterfacerInitError' is not defined

[pb66 (Paul)]

Ok, for the first “timestamped” instance the issue only arises from the use of datacodes, delete the datacodes line from your conf like so

[[98]]
    nodename = nodeMCU
    [[[rx]]]
       names = temp1, ext temp, temp2, humidity
       scales = 0.1,0.1,0.1,0.1
       units = C,C,C,%

and it should work fine.

The reason is, you cannot reconstruct the “full” values from raw byte data using datacodes when you are already passing “full” values.

your data payload of

1517263080.0 98 3.8 1.6 5.2 80.3

is actually (correctly) in the form of

timestamp nodeid full_value1 full_value2 full_value3 full_value4

where as by defining

datacodes = h,h,h,h

you are telling emonhub the data will not be in the form of “full” normal values but it will arrive as a stream of 8 (4 x 2 byte signed ints “h”) byte values (0-255) and emonhub is replying that the 4 value payload it received (3.8 1.6 5.2 80.3) cannot be decoded (reconstructed in to 4 values) using the 8 byte value template you defined.

By simply not defining the datacode(s) setting it defaults to the norm of not decoding the data stream.

In the second instance, that’s my fault for working from memory. The setting should be timestamped = True or timestamped = False or omitted for a default of False.

The way Python works a capital “F” is required for a False, “false” is not False and the “false” string results in a True when tested.

This has gone un-noticed because I only use timestamped = True or timestamped = true when passing a timestamped payload (both result in a True because neither is False) and I generally omit the setting for payloads without a timestamp rather than defining False.

In fact more often that not I will just comment and un-comment the setting like so

        #timestamped = True

as deleting or adding the hash is easier/quicker than typing “True” or “False” and less prone to input error it would seem.

So your “false” setting resulted in a True and the nodeid val1 val2 was parsed as if it had a timestamp ie the nodeid was taken as a timestamp and val1 was taken as the nodeid.

[lgheorghe (Liviu Gheorghe)]

Changed timestamped = False, but there is still an issue:

2018-01-30 20:29:52,744 DEBUG    MainThread Setting LanRestarter timestamped: False
2018-01-30 20:31:12,355 WARNING  LanRestarter Exception caught in LanRestarter thread. Traceback (most recent call last):
  File "/home/pi/emonhub/src/interfacers/emonhub_interfacer.py", line 38, in wrapper
    return f(*args)
  File "/home/pi/emonhub/src/interfacers/emonhub_interfacer.py", line 87, in run
    rxc = self.read()
  File "/home/pi/emonhub/src/interfacers/EmonHubSocketInterfacer.py", line 119, in read
    c.nodeid = int(f[0]) + int(self._settings['nodeoffset'])
ValueError: invalid literal for int() with base 10: '6.8'

2018-01-30 20:31:12,450 WARNING  MainThread LanRestarter thread is dead.
2018-01-30 20:31:12,451 WARNING  MainThread Attempting to restart thread LanRestarter (thread has been restarted 1 times...

Then I changed timestamped = True and gave it another try:

2018-01-30 20:34:15,396 DEBUG    MainThread Setting LanRestarter timestamped: True
2018-01-30 20:35:05,579 DEBUG    LanRestarter 6 NEW FRAME : 1517337300.0 98 6.8 6.2 8.6 77.0
2018-01-30 20:35:05,581 DEBUG    LanRestarter 6 Timestamp : 1517337300.0
2018-01-30 20:35:05,582 DEBUG    LanRestarter 6 From Node : 98
2018-01-30 20:35:05,583 DEBUG    LanRestarter 6    Values : [0.68, 0.6200000000000001, 0.86, 7.7]
2018-01-30 20:35:05,584 DEBUG    LanRestarter 6 Sent to channel(start)' : ch1
2018-01-30 20:35:05,586 DEBUG    LanRestarter 6 Sent to channel(end)' : ch1
2018-01-30 20:39:52,423 DEBUG    LanRestarter 7 NEW FRAME : 1517337540.0 98 67.0 61.0 89.0 770.0
2018-01-30 20:39:52,425 DEBUG    LanRestarter 7 Timestamp : 1517337540.0
2018-01-30 20:39:52,427 DEBUG    LanRestarter 7 From Node : 98
2018-01-30 20:39:52,428 DEBUG    LanRestarter 7    Values : [6.7, 6.1000000000000005, 8.9, 77]
2018-01-30 20:39:52,429 DEBUG    LanRestarter 7 Sent to channel(start)' : ch1
2018-01-30 20:39:52,431 DEBUG    LanRestarter 7 Sent to channel(end)' : ch1

Now data seems to be accepted by emonhub, but it doesn’t appear in the inputs/feeds.
Don’t I have to configure pubchannels/subchannels in the EmonHubSocketInterfacer? I tried searching the documentation for their meaning, but didn’t find anything.

[pb66 (Paul)]

Can you provide more emonhub.log? Ideally I need to see what happens prior to the exception to understand what is happening. All I can determine from your snippet is that the payload probably had a “6.8” value in it somewhere.

Can you also test with the timestamped setting commented out or deleted to see if the default non-timestamped behavior is correct.

@lgheorghe

Don’t I have to configure pubchannels/subchannels in the EmonHubSocketInterfacer?

I expect that is most probably the case, this is something that is only made mandatory in the emonpi variant, however the same settings are used pretty much throughout so you could just copy them from the [[RFM2Pi]] section

        pubchannels = ToEmonCMS,
        subchannels = ToRFM12,

having said that it looks like the original defaults are still being found eg

2018-01-30 20:35:05,584 DEBUG    LanRestarter 6 Sent to channel(start)' : ch1

so you could just try adding the “ch1” channel to the emoncms and mqtt subchannels eg

        subchannels = ToEmonCMS, ch1