MODBUS TCP/IP Python send to EmonCMS

Integrations
, ,
1

Hi,

I’m trying to intergrate my energy meters, and I got the sample code from pyModbusTCP module, I ran a test script with this code from the examples folder,

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# read_register
# read 10 registers and print result on stdout

# you can use the tiny modbus server "mbserverd" to test this code
# mbserverd is here: https://github.com/sourceperl/mbserverd

# the command line modbus client mbtget can also be useful
# mbtget is here: https://github.com/sourceperl/mbtget

from pyModbusTCP.client import ModbusClient
import time

 SERVER_HOST = "192.168.1.175"
 SERVER_PORT = 502

 c = ModbusClient()

 # uncomment this line to see debug message
 #c.debug(True)

 # define modbus server host, port
 c.host(SERVER_HOST)
 c.port(SERVER_PORT)

while True:
# open or reconnect TCP to server
if not c.is_open():
if not c.open():
print("unable to connect to "+SERVER_HOST+":"+str(SERVER_PORT))

# if open() is ok, read register (modbus function 0x03)
if c.is_open():
   # read 10 registers at address 0, store result in regs list
      regs = c.read_holding_registers(0001, 10)
      # if success display registers
  if regs:
    print("reg ad #0 to 9: "+str(regs))

    # sleep 2s before next polling
  time.sleep(2)

Seems to be working and polling the correct data, my next thing is how do I take the data it has polled, and send it to emoncms? I know I need to use JSON to achieve that, not to savy on python.

thx.

2

Hi, can you post an example result and tell us more about your setup and requirements?

Where is this script going to run from?
Are you intending to send to emoncms.org?
Do you have a local emoncms?
Are you running emonHub?

Not strictly true, My first port of call would be to post it to a socket on emonHub, I assume you have a string of values in your result. that can easily be sent directly to emonhub via socket.

Then on receipt, emonhub can then allocate a timestamp and/or nodeid as required, and then take care of parsing for sending to emoncms. Depending on the format of the data you may find emonhub’s “datacodes” useful to parse the data and other features may prove useful too

The socket interfacer in emonhub is specifically there for hacking and tacking on scripts like this as a one-of or as a prototyping input before creating a bespoke emonhub interfacer.

If you want to send to emoncms.org directly from the one script, then JSON is the way to go.

3

Thanks @pb66 for the reply,

Here’s what I get when I run the script,

emoncms@SKUbuntu:~/pyModbusTCP/examples$ python read_bit_egx.py
reg ad #0 to 9: [1, 0, 0, 0, 0, 5191, 5206, 0, 0, 0]
reg ad #0 to 9: [1, 1, 0, 1, 32767, 5191, 5205, 3, 4, 0]
reg ad #0 to 9: [1, 0, 0, 0, 0, 5188, 5202, 0, 0, 0]
reg ad #0 to 9: [1, 0, 0, 0, 0, 5190, 5202, 0, 0, 0]
reg ad #0 to 9: [1, 0, 0, 0, 0, 5188, 5201, 0, 0, 0]
reg ad #0 to 9: [1, 0, 0, 0, 0, 5187, 5202, 0, 0, 0]
reg ad #0 to 9: [1, 0, 0, 0, 0, 5187, 5202, 0, 0, 0]
reg ad #0 to 9: [1, 0, 0, 0, 0, 5187, 5201, 0, 0, 0]
^CTraceback (most recent call last):
File "read_bit_egx.py", line 43, in <module>
time.sleep(2)
 KeyboardInterrupt
emoncms@SKUbuntu:~/pyModbusTCP/examples$

Im requesting the values from the registers from 0001 to 0010 using function code 0x03 to read the registers, the data is in mA. This energy monitor is a Veris H704-42 Branch Circuit Monitor and for each register, is a separate CT,

Where is this script going to run from? (Script will be running from a local installation of ubuntu, on the same server that emoncms is installed)

Are you intending to send to emoncms.org? Not intending to send to emoncms.org

Do you have a local emoncms?
Yes, local install of emoncms

Are you running emonHub?
Not running emonhub, my thought process as the script is printing the values, at the same time send to local emoncms using JSON,

Sounds like emonhub is the way to go, how would I send to emonhub with the script i used? Or does it rely on Python to begin with? Standalone? Couldn’t find much documentation.

4

So I have been tinkering around with python, have learned quite a bit, but now im stuck again, and getting an error. here’s the code I used. Same code as my original post, but added the httplib.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# read_register
# read 10 registers and print result on stdout

# you can use the tiny modbus server "mbserverd" to test this code
# mbserverd is here: https://github.com/sourceperl/mbserverd

# the command line modbus client mbtget can also be useful
# mbtget is here: https://github.com/sourceperl/mbtget

from pyModbusTCP.client import ModbusClient
import time
import sys
import string
import httplib



SERVER_HOST = "192.168.1.175"
SERVER_PORT = 502

c = ModbusClient()

# uncomment this line to see debug message
#c.debug(True)

# define modbus server host, port
c.host(SERVER_HOST)
c.port(SERVER_PORT)

# Domain you want to post to: localhost would be an emoncms installation on your own laptop
# this could be changed to emoncms.org to post to emoncms.org
domain = "localhost"

# Location of emoncms in your server, the standard setup is to place it in a folder called emoncms
# To post to emoncms.org change this to blank: ""
emoncmspath = "emoncms"

# Write apikey of emoncms account
apikey = "43b8xxxxxxxxxxxxxxxxc0ec754716b2"

# Node id youd like the emontx to appear as
nodeid = 10

conn = httplib.HTTPConnection(domain)

while True:
# open or reconnect TCP to server
if not c.is_open():
    if not c.open():
        print("unable to connect to "+SERVER_HOST+":"+str(SERVER_PORT))

# if open() is ok, read register (modbus function 0x03)
if c.is_open():
    # read 10 registers at address 0, store result in regs list
    regs = c.read_holding_registers(0001, 10)

    # if success display registers
    if regs:	    
        print("reg ad #0 to 9: "+str(regs))

# Create csv string
        csv = ",".join("reg ad #0 to 9: "+str(regs))


# Send to emoncms
        conn.request("GET", "/"+emoncmspath+"/input/post.json?apikey="+apikey+"&node="+str(nodeid)+"&csv="+csv)
    response = conn.getresponse()
   
    print response.read()


# sleep 2s before next polling
time.sleep(2)

and I get this error when I run it.

emoncms@SKUbuntu:~/pyModbusTCP/examples$ python read_register_egx.py
reg ad #0 to 9: [1, 0, 0, 0, 0, 5251, 5265, 0, 0, 0]
Error: Format error: csv value is not numeric

reg ad #0 to 9: [1, 0, 0, 0, 0, 5252, 5263, 0, 0, 0]
Error: Format error: csv value is not numeric

reg ad #0 to 9: [1, 0, 0, 0, 0, 5254, 5267, 0, 0, 0]
Error: Format error: csv value is not numeric

reg ad #0 to 9: [1, 0, 0, 0, 0, 5253, 5267, 0, 0, 0]
Error: Format error: csv value is not numeric

^CTraceback (most recent call last):
 File "read_register_egx.py", line 76, in <module>
 time.sleep(2)
KeyboardInterrupt
emoncms@SKUbuntu:~/pyModbusTCP/examples$ ^C

Not sure what I’m doing wrong, is it the way the CSV string is combined? or what the issue is.

When i open emoncms inputs tab, I get this, seems like it is trying to post and does refresh with new data, but value does not pass through.

Pasted image736×390 10.3 KB

Edit - munged API key - Moderator, BT

5

Won’t give you numeric CSV because you are effectively saying

csv = “,”.join(“reg ad #0 to 9: [1, 0, 0, 0, 0, 5253, 5267, 0, 0, 0]”)

which will probably give you something like

reg, ad, #0, to, 9:, [1, 0, 0, 0, 0, 5253, 5267, 0, 0, 0],

Even the numeric values will be taken as text due to the double comma.

You can either remove the square brackets from the string regs using

csv = str(regs)[1:-1]

or the better way is to use join as you have done, but on the list elements not the list as a string.

csv = “,”.join(regs)

I haven’t tested either of these so forgive me if I’ve made any errors, but hopefully it’s enough to help get you sorted.

I was going to adapt your script for posting to an emonHub socket, but I will wait to see how you get on with this, if this is the way you prefer to go.

1 Like
7

Thx for the response @pb66 I have dumbed it down to only 1 modbus register and it posts to emoncms and works as expected, Can you assist me with getting a list and posting that list of values from 42 registers and post them to emoncms? Registers are sequential from 1-42 registers. Im unsure how to go about creating a list and converting that list to csv and getting it posted to emoncms.

Heres the code I used to get 1 register to post successfully on emoncms.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# read_register
# read 10 registers and print result on stdout

# you can use the tiny modbus server "mbserverd" to test this code
# mbserverd is here: https://github.com/sourceperl/mbserverd

# the command line modbus client mbtget can also be useful
# mbtget is here: https://github.com/sourceperl/mbtget

from pyModbusTCP.client import ModbusClient
import time
import sys
import string
import httplib



SERVER_HOST = "192.168.1.175"
SERVER_PORT = 502

c = ModbusClient()

# uncomment this line to see debug message
#c.debug(True)

# define modbus server host, port
c.host(SERVER_HOST)
c.port(SERVER_PORT)

# Domain you want to post to: localhost would be an emoncms installation on your own laptop
# this could be changed to emoncms.org to post to emoncms.org
domain = "localhost"

# Location of emoncms in your server, the standard setup is to place it in a folder called emoncms
# To post to emoncms.org change this to blank: ""
emoncmspath = "emoncms"

# Write apikey of emoncms account
apikey = "43b84xxxxxxxxxxxxxxxx0ec754716a1"

# Node id youd like the emontx to appear as
nodeid = 10

conn = httplib.HTTPConnection(domain)

while True:
    # open or reconnect TCP to server
    if not c.is_open():
        if not c.open():
            print("unable to connect to "+SERVER_HOST+":"+str(SERVER_PORT))

    # if open() is ok, read register (modbus function 0x03)
    if c.is_open():
        # read register 2 from veris meter
        regs = c.read_holding_registers(2)

        # if success display registers
        if regs:	    
            print(str(regs))


  # Send to emoncms
            conn.request("GET", "/"+emoncmspath+"/input/post.json?    apikey="+apikey+"&node="+str(nodeid)+"&json="+str(regs))
 	    response = conn.getresponse()
   
	    print response.read()


    # sleep 3s before next polling
    time.sleep(3)

Edit - munged API key - Moderator, BT

8

Based on your previos code, I would have expected

regs = c.read_holding_registers(0001, 42)

to give you the 42 registers starting from register 1, but I’m guessing a bit there I haven’t looked at the ModbusClient.

As for posting the data via the JSON api, if it worked for a list of 1 value, it should also work for a list of 42 values. But the code you have posted as working, seems to post using square brackets where curly brackets are expected so it sounds like emoncms isn’t fussy or you have found a loop hole.

To make the request url conform with the api docs (syntactically correct or not) you would need to add the curly brackets to your string formation and remove the square brackets from the list as I previously demonstrated, eg

…+“&json={”+str(regs)[1:-1]+“}”)

1 Like
10

Its telling me the csv value is not numeric when I use the

+&json={"+str(regs)[1:-1]+"}")

I have also tried using csv, to join them and strip the brackets and get the same error, csv value is not numeric, as the response from emoncms.
…

csv = ",".join(str(regs))[1:-1]

  # Send to emoncms
       conn.request("GET", "/"+emoncmspath+"/input/post.json?apikey="+apikey+"&node="+str(nodeid)+"&csv="+csv)
response = conn.getresponse()
   
print response.read()

Heres what it looks like on the output, error is coming from emon cms, because the post does happen, it creates my inputs, but fails to send data, could it be because the values have a space after the comma?

1, 2, 0, 2, 32767
Error: Format error: csv value is not numeric

1, 5, 0, 5, 32767
Error: Format error: csv value is not numeric

^CTraceback (most recent call last):
 File "read_register_egx.py", line 75, in <module>
time.sleep(3)
KeyboardInterrupt
emoncms@SKUbuntu:~/pyModbusTCP/examples$ ^C
11

Try constructing your url as a string so you can print it along with the response eg

# Send to emoncms
            
            URL = "/"+emoncmspath+"/input/post.json?apikey="+apikey+"&node="+str(nodeid)+"&json={"+str(regs)[1:-1]+"}"
            print(URL) 
            
            conn.request("GET", URL)
 	    response = conn.getresponse()

	    print response.read()

That should let you see exactly what is being sent.

12

heres the output, it is the spaces, is what it seems. any way to remove them?

emoncms@SKUbuntu:~/pyModbusTCP/examples$ python read_register_egx.py


/emoncms/input/post.json?apikey=43b84xxxxxxxxxxxxxxxxxxxx54716a1&node=10&json={1, 222, 152, 269, 27043}
Error: Format error: csv value is not numeric

/emoncms/input/post.json?apikey=43b84xxxxxxxxxxxxxxxxxxxx54716a1&node=10&json={1, 222, 152, 269, 27043}
Error: Format error: csv value is not numeric

^CTraceback (most recent call last):
File "read_register_egx.py", line 80, in <module>
time.sleep(3)
KeyboardInterrupt
emoncms@SKUbuntu:~/pyModbusTCP/examples$

Edit - munged API key - Moderator, BT

13

I have just checked the API docs and think you have crossed 2 API’s. You can submit CSV as

csv=123, 456, 789

or the same in JSON would be

json={‘1’:‘123’,‘2’:‘456’,‘3’:‘789’}

Try using the CSV api again

…+“&csv=”+str(regs)[1:-1])

I really think emonhub would have been easier, you haven’t gotten as far as handling timeout’s and/or non-“ok” responses, and there isn’t much time for error with 2 or 3 second intervals, emonhub would batchsend your packets at a more manageable rate and handle non-deliveries by buffering and resending (original emonhub version) until delivery is confirmed.

14

Take care not to publish your write API key, as it leaves your emonCMS vulnerable to writing by anyone who has the key.

15

Thank will try when I get home. Stuck in traffic now and will test.

I’m willing to learn new techniques, if emonhub is the way to go, please assist. In setting it up. Im not sure where to start.

I appreciate it.

16

It’s a fake api key, I’ve been changing the characters in it.

17

Still getting same Error @pb66,

/emoncms/input/post.json?apikey=43b84401f6dad8a43effxz0ec754716a1&node=10&csv=1, 40, 30, 50, 26214
Error: Format error: csv value is not numeric

 /emoncms/input/post.json?apikey=43b84401f6dad8a1056gfdxec754716a1&node=10&csv=1, 40, 30, 50, 26214
 Error: Format error: csv value is not numeric

^CTraceback (most recent call last):
File "read_register_egx.py", line 80, in <module>
time.sleep(3)
KeyboardInterrupt
emoncms@SKUbuntu:~/pyModbusTCP/examples$

I would like to checkout emonhub Paul, not much documentation is available, can you guide me in the right direction?

EDIT: Whats odd, is I copied and pasted the url python used to send the data

http://localhost/emoncms/input/post.json?apikey=43b84401f6dad8a1056gfdxec754716a1&node=10&csv=1, 40, 30, 50, 26214

and it worked, it posted the data, and works as expected, could it be the httplib in python? doesnt covert the spaces to %20? or not sure why it causes issue. I used firefox for this, works as expected. But i would like to persue emonhub instead.

18

I will take a look at sending to a socket for you tomorrow, there is a guide to installing emonhub within the emoncms install docs, but most of that will not apply to you as you are not using serial or RFM, and using sockets isn’t covered there.

Try searching the forums (incl the old forum) for “Socket Interfacer” or I will find some links for you tomorrow.

In the mean time if you want to try using “join” on the CSV you may be able to confirm your suspicions, as it should provide a space free CSV string, I was pretty confident emoncms wasn’t fussy about the spaces and your use of the url in a browser seems to confirm that.

#Send to emoncms
            CSV = ",".join(regs)
            URL = "/"+emoncmspath+"/input/post.json?apikey="+apikey+"&node="+str(nodeid)+"&csv="+CSV
            print(URL) 
            
            conn.request("GET", URL)
 	    response = conn.getresponse()

	    print response.read()
1 Like
19

Thx @pb66 I will look through forums on emonhub socket interfacer.

as for python another errror came up. I am using python 2.7, not sure if that makes a difference or not. I know py2 and py3 syntax for the most part the same, but very picky if you run py2 scripts in py3.

emoncms@SKUbuntu:~/pyModbusTCP/examples$ python read_register_egx.py

Traceback (most recent call last):
File "read_register_egx.py", line 67, in <module>
CSV = ",".join(regs)
TypeError: sequence item 0: expected string, int found
skopytin@SKUbuntu:~/pyModbusTCP/examples$

EDIT: fixed it with this.

CSV = ','.join(str(v) for v in regs)

and does update, and now my output is missing the spaces, inbetween values and is updating emoncms correctly.

20

I haven’t made the switch, i still use 2.7 by default and without thinking about it, had assumed your code to be 2.7 by the lack of brackets on the print statement.

That makes total sense, my apologies.

So it is actually the “httplib” module that creates an issue when there are spaces in the csv rather than emoncms, that too makes better sense, not saying it’s right or not, but I have spaced out csv for better readability in the past and hadn’t hit that issue with emoncms before.

Later today i will look at editing a copy of your script for use with an emonhub socket interfacer.

EDIT - If you do go ahead and install emonhub, I would install the original not the emonPi variant for this, and maybe be aware that on Ubuntu there maybe an issue with the emonhub init service under systemd (ref Emonhub Systemd Ubuntu 15.10 thread)

EDIT2 - I had found this PDF yesterday, I will post it here for ref (so I can close the browser page) it lists the full list of registers.
http://www.veris.com/docs/comms/mb_pmap/h663_h704_100a_pm_10101.pdf

EDIT3 - I forgot to ask something about the device, That PDF says “100 amp model”, is that the overall or per channel? Just I noticed the registers were “16bit Integers” and the measure is mA, giving a range of less than +/- 33 amps, not sure why it would report negative amps either, are they “16bit unsigned integers” ie 65amps. (Just thinking out loud to myself, I’m sure it’s documented somewhere)

21

Your problem is the spaces. Remove them and I think it will be fine.

22

Thx Paul, I’m running into another issue, and that is overwhelming the Modbus TCP/IP device with DDOS attack essentialy, reqeusting all 42 registers at one time, and python freaks out as well, and I can’t pull more than 15 registers at one time before the modbus device freaks out which is normal, on other automation systems that i have done, we poll a modbus register, only 1 register every 1 second.

Before I build timers and delays, etc. in the script, I’ll wait for your script for emon hub, thx for the help.

EDIT: I have installed emonhub, and I ran into the ubuntu issue, and i left to figure it out for today.

EDIT 3: the veris meters I have are 50 amp CT’s, and I used unsigned 16 bit to get it to work correctly in my previous integrations.

The board i’m sure is capable of aquiring 100 amps CT’s, infact that is the acquisition board I have but I only have 50amp CT’s, and not sure it goes about handling 100amp CT’s,