Community
OpenEnergyMonitor

Community

UDP Broadcast

udp
Tags: #<Tag:0x00007f16b6f825e0>

(Daniel Bates) #1

Hey folks,

What’s the easiest way to broadcast emoncms data strings via UDP?

It’s for a data visualisation project :slight_smile:

Cheers,

Dan

EDIT: This was the result of exploring python script:

import requests
import socket
import time

print("Simple code to grab data from emonCMS and then UDP broadcast it every 10 seconds.")
print("Free to use. OpenEnergyMonitor.org")

#EMONCMS API READ KEY
path = "apikey.txt"
with open(path) as f:
        read_data = f.read()
f.closed
apireadkey = read_data
print("Got API Key, continuing")

#COMMENT IN YOUR REQUIRED EMONCMS HTTP REQUEST. ONE AT A TIME.
#API URLs

emoncmsipaddress = "emoncms.org"
feedid = "43361"
feedids = "43361,43363,43564,98473,98474,98476"

#---single feed value url---
#emon_api_url = "https://" + emoncmsipaddress + "/feed/value.json?id=" + feedid +"&apikey=" + apireadkey
#---multiple feed values---
emon_api_url = "https://" + emoncmsipaddress + "/feed/fetch.json?ids=" + feedids + "&apikey=" + apireadkey

print("Getting data via:" + emon_api_url)

#socket code___
UDP_IP = "127.0.0.1" #target IP
UDP_PORT = 6400

interval = 10 #seconds

def calc_next_time(time_now, interval):
	time_next = (((time_now//interval)*interval)+interval)
	return time_next

time_next = calc_next_time(time.time(),interval)

while(True):
	
	# Get the current time
	time_now = time.time()
	
	# Check if it's time to do another transmission
	if time_now >= time_next:

		# Request the data from emonCMS
		response = requests.get(emon_api_url)

		# Parse the data if the response was good
		if response.status_code == 200:
			datalist = response.json()
			
			# Convert any single value returned to a list of one value 
			if not isinstance(datalist,(list)):datalist=[datalist]
			
			# Make a CSV string of floats rounded to 3 decimals max 
			stringydata = ','.join([str(round(i,3)) for i in datalist])
			
			# Send the data over UDP
			sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
			sock.sendto(stringydata.encode('utf-8'), (UDP_IP, UDP_PORT))
			print (str(time_now) + "\tData sent: " + stringydata)
			
			# set the time of next loop
			time_next = calc_next_time(time_now,interval)
	
	# Don't loop too fast
	time.sleep(0.1)

re:edit: including @pb66 's ‘harmonised interval’ method.


(Paul) #2

Hi Dan, I’m afraid I can’t help you much but i would like to add a note here as I think this would be useful for future emonHub development.

There has been previous development into receiving UDP packets in emonHub previously for OWL energy monitors (see New user alert - Owl Intuition PV capture/logging)and I did manage to push some data to adjust voltage settings etc alas the attachments are no longer attached to that thread and I cannot seem to find them with a search.

There are several “owl” threads on both this forum and the archived forums and undoutable there will be more on UDP buried in there somewhere.

Hopefully you might find something useful, now for the bit that won’t help you much. Hopefully in the not too distant future (but certainly not in the next couple of months or so) there will be “OWL” and therefore “UDP” implementation in-built to emonHub. The reason for this note is to remind me that perhaps a generic “UDP” interfacer would be worth while and the “OWL” interfacer can inherit from that.


(stephen krywenko) #3

i did something in UDP push of raw data from emontx shield on a esp8266 and mcp3008 … if that is of any interest to you Finished - UDP PUSH emontx. based on mcp3xxx and esp and just process the raw data on the receiving computer in realtime … am sure it can be adapted to other board types . but the biggest benifit of mcp3XXX is that it can process a higher number of CTs at a much higher frequency


(Daniel Bates) #4

@pb66 A brief look into UDP broadcasting for arduino, esp and pi reveal simple implementations… Seems it’d be easy to do for emonHub.

@stephen Thanks, looks good. I used the MCP3208 for the emonDC project, great chip.

What I wanted was to grab values from a system already set up, so perhaps there is another way. I want to get the numerical values from an emoncms dashboard then I could localhost UDP to my desktop program.
I’m using this as a working example https://energy.aesengr.com/ but can’t find the specific java that grabs the numerical values… Thoughts?


(Paul) #5

Sorry, yes. i only addressed part of the problem.

I would use the emoncms “fetch” api

https://myserver.com/feed/fetch.json?ids=1,2,3&apikey=abc123

this would return the current values for feedids 1, 2 & 3 as a list of numerical data eg

[123.45, 45.678, 99.9]

Where I was going with the “oneday in emonhub” was that there will be an interfacer that “fetches” data from emoncms as above and that data can be routed to any other interfacer target eg publish it to mqtt or transmit it over RFM for a emonGLCD or indeed transmit it over UDP (or to OWL) if the interfacer is there.

For now you could write a small pythn script that fetches the data from emoncms via an api call and transmits over UDP in the format you need. I say Python because that’s what I tend to do these little script in so that they can be reused in emonhub interfacers. But you can use your language of choice.

Please post back here if you do this so that I may use it when I come to implement something in emonhub.


(Daniel Bates) #6

Great, Thanks. Python looks a very handy programming language. Do you have an example script I can get started with?
Typing in “https://myserver.com/feed/fetch.json?ids=1,2,3&apikey=abc123” to python returns ‘invalid syntax’, I could of assumed it wouldn’t be so simple :wink:


(Daniel Bates) #7

Getting there…
got this far now

cd /usr/local/bin/
python3.6 pip install requests
python3.6

>>> import requests
>>> response = requests.get("https://emoncms.org/feed/list.json?userid=718&apikey=b903*************60b9dca2") 
# note: change “userid=718” to your userid, available from "https://emoncms.org/user/view" in the top left. note: change “apikey” key string to your READ api key.
>>> data = response.json()
>>> print(data)
# feed list printed here

I’m working from macOS Terminal… not sure of a better option for python on Mac. EDIT: IDLE looks good. Using ‘modules’ in my workflow is fairly simple…
Attribution: This page helped.


(Daniel Bates) #8

grabbed from emoncms every 10 seconds, posted via UDP…
done:

import requests
import socket
import threading

print("simple code to grab data from emonCMS and then UDP broadcast it every 10 seconds")

#requests code
response = requests.get("https://emoncms.org/feed/value.json?id=USERID&apikey=APIREADYKEY_APIREADYKEY_APIREADYKEY")

data = response.json()

print("data from emoncms server:", data)
print()


#socket code
UDP_IP = "192.168.43.185"
UDP_PORT = 1234

print("UDP target IP:", UDP_IP)
print("UDP target port:", UDP_PORT)
print("data to send:", data)

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
sock.sendto(bytes(data, "utf-8"), (UDP_IP, UDP_PORT))
print ("first data sent", data)

#threading code
def printit():
  threading.Timer(10.0, printit).start()
  response = requests.get("https://emoncms.org/feed/value.json?id=USERID&apikey=APIREADYKEY_APIREADYKEY_APIREADYKEY")
  data = response.json()
  sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
  sock.sendto(bytes(data, "utf-8"), (UDP_IP, UDP_PORT))
  print ("data sent", data)

printit()

Python’s ace.

Real-time 3D modelling and visualisation from emonCMS next :wink:


(Paul) #9

That’s not a command it’s a http request. if you use that in a browser bar (substituting the apikey and 3x feedids) it will return the 3 current values.

Where you have used

#requests code
response = requests.get("https://emoncms.org/feed/value.json?id=USERID&apikey=APIREADYKEY_APIREADYKEY_APIREADYKEY")

to return one value you could use

#requests code
response = requests.get("https://emoncms.org/feed/fetch.json?ids=FEEDID1,FEEDID2,FEEDID3&apikey=APIREADYKEY_APIREADYKEY_APIREADYKEY")

to return multiple feed values (You use “USERID” as a variable for a feedid, but I used "FEEDIDn " for clarity)


(Daniel Bates) #10

That’s useful thanks.
So it wasn’t as simple as copying a http request and expecting it to work as a python command, but it was pretty simple.
This error did pop up from time to time, after leaving it run for a while:

Exception in thread Thread-66:
Traceback (most recent call last):
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py”, line 916, in _bootstrap_inner
self.run()
File “/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py”, line 1182, in run
self.function(*self.args, **self.kwargs)
File “/Users/admin/Documents/Python Scripts/threading example.py”, line 34, in printit
sock.sendto(bytes(data, “utf-8”), (UDP_IP, UDP_PORT))
OSError: [Errno 64] Host is down


(Daniel Bates) #11

I tested this out of interest. Do you know the exact format? I’m not having any success with it.

I’m using this to get historical data:

response = requests.get("https://emoncms.org/feed/data.json?id=43361&start=1446624000&end=1446824000&interval=10&apikey=b9033cf185b17ea192d3****60b9dca2")

And I get returned this in python:

data from emoncms server: []

Do I need to format or parse the data in a particular way?

[Edit - Apikey part-obscured (RW)]


(Paul) #12

When I recommended the “fetch” api I believed you were running this continuously and that just the latest data from several feeds was required to keep it updated. The “fetch” api won’t return historical data.

All the feed and input api’s are documented on the pages accessible via the link at the top right of the feed and inputs pages of emoncms. or at https://emoncms.org/site/api#feed and https://emoncms.org/site/api#input

The issue with the example you give is most likely the timestamps, I believe they need to be unixmillis

response = requests.get("https://emoncms.org/feed/data.json?id=43361&start=1446624000000&end=1446824000000&interval=10&apikey=b9033cf185b17ea192d3****60b9dca2")

The response you get from this will be a list of timestamp value pairs eg [[1446624000000,100][1446624010000,101][1446624020000,101.9876]..... and so on, I assume it will need parsing but what that might entail will depend on what you need or want to do with the data after it’s landed.


(Daniel Bates) #13

I have two aspects to my project.
To use ‘fetch.json’ or ‘value.json’ or ‘timevalue.json’ in realtime, then post to UDP.
To use ‘data.json’ in to get historical data.

Using ‘data.json’ or ‘average.json’ in python hasn’t worked for me, the unix timestamps I’m using in the script are identical to the browser csv export unix timestamps. I’ve toyed with the interval, no change.
Perhaps it’s a datatype issue… Perhaps the ‘requests’ lib I’m using is not the correct one? Or there’s some small user error I haven’t found yet. I don’t know.


(Daniel Bates) #14

Updated.
This formats a list from multiple feeds into a single string, comma separated.
Multiple feed values can be sent via UDP in one line in UTF-8 format.
The “%.3f” defines the float resolution.

import requests
import socket
import threading


print("simple code to grab data from emonCMS and then UDP broadcast it every 10 seconds")

#requests code___

#---multiple feed values---
emon_api_url = "https://emoncms.org/feed/fetch.json?ids=43361,43363,43564,98473,98474,98476&

response = requests.get(emon_api_url)
datalist = response.json()

print("data from emoncms server:", datalist)

formattedlist = []
for i in datalist:
    formattedlist.append("%.3f"%i)

print("Formatted:", formattedlist)

stringydata = ""
for result in formattedlist:
    stringydata = stringydata + str(result) + ","
stringydata = stringydata[:-1]


print("String:", stringydata)
    
print()


#socket code___
UDP_IP = "192.168.43.185"
UDP_PORT = 6400

print("UDP target IP:", UDP_IP)
print("UDP target port:", UDP_PORT)
print("Data to send:", stringydata)

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
sock.sendto(bytes(stringydata, "utf-8"), (UDP_IP, UDP_PORT))
print ("Data sent")



#threading code___
def printit():
    threading.Timer(10.0, printit).start()
    response = requests.get(emon_api_url)
    datalist = response.json()
    formattedlist = []
    for i in datalist:
        formattedlist.append("%.3f"%i)
    stringydata = ""
    for result in formattedlist:
        stringydata = stringydata + str(result) + ","
    stringydata = stringydata[:-1]
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
    sock.sendto(bytes(stringydata, "utf-8"), (UDP_IP, UDP_PORT))
    print ("Data sent", stringydata)
    
printit()

(Daniel Bates) #15

I’ve edited the original post with the solution.
Works for single of multiple feed requests.

Notes: We did not manage to successfully
A: Get all feed values by using a USERID as you suggested. So the multifeed script I made requires each feed id to be entered manually, that’s OK.
B: Get historical values using ‘data.json’ or ‘average.json’. This is likely just user error.

Cheers,
Dan


(Paul) #16

Great you got it sorted.

I think you’ve misunderstood me there. I was saying you can get as many feeds as you specify by using the fetch api (as you have now done), My reference to “USERID” was that you had used the “USERID” variable in place of a “FEEDID” whilst using the “feed/value.json” api, I was merely correcting the terminology, if you had it working, you must have actually been using a feedid despite calling it “USERID”.

feed/value.json uses singular “id=feedid” (not USERID)

feed/fetch.json uses plural “ids=feedid1,feedid2,feedid3” etc

If you don’t mind me asking why did you choose to use threading rather than just looking up the feed values, broadcasting and then sleeping for the remainder of the 10secs?


(Daniel Bates) #17

Cool :slight_smile:

I found an example using the threading module, so used that.
Here was the sample.
https://stackoverflow.com/questions/3393612/run-certain-code-every-n-seconds
What do you think?


(Paul) #18

I’ve never seen that approach before, to me it seems like overkill for a simple loop timer.

Working from your example I would have done something like this

#!/usr/bin/env python

import requests
import socket
import time

apireadkey = "abc123abc123abc123"

feed_ids = "12300,12301,12302,12303"
#feed_ids = "12300"
# The fetch api works for a singular feed id too so it works in both scenario's
# Uncomment either one of the 2 lines above to try
emon_api_url = "https://emoncms.org/feed/fetch.json?ids=" + feed_ids + "&apikey=" + apireadkey

#socket code___
UDP_IP = "192.168.43.185"
UDP_PORT = 6400

time_next = 0

while(True):
	
	# Get the current time
	time_now = time.time()
	
	# Check if it's time to do another transmission
	if time_now >= time_next:

		# Request the data from emonCMS
		response = requests.get(emon_api_url)

		# Parse the data if the response was good
		if response.status_code == 200:
			datalist = response.json()
			
			# Convert any single value returned to a list of one value 
			if not isinstance(datalist,(list)):datalist=[datalist]
			
			# Make a CSV string of floats rounded to 3 decimals max 
			stringydata = ','.join([str(round(i,3)) for i in datalist])
			
			# Send the data over UDP
			sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
			sock.sendto(stringydata.encode('utf-8'), (UDP_IP, UDP_PORT))
			print ("Data sent: " + stringydata)
			
			# set the time of next loop
			time_next = time_now + 10
	
	# Don't loop too fast
	time.sleep(0.1)

I’ve posted it here mainly for me to come back to if and when I do a UDP interfacer in emonHub. So it would be handy to know if it works if you do get a chance to try it out. You should just need to edit to your apikey and feedid(s)


(Daniel Bates) #19

I’m up late, so gave it a whirl.
Unfortunately these errors are thrown back quite often.

Traceback (most recent call last):
File “/Users/admin/Documents/Python Scripts/test1.py”, line 48, in
sock.sendto(stringydata.encode(‘utf-8’), (UDP_IP, UDP_PORT))
OSError: [Errno 64] Host is down

Or the last line of the error is

OSError: [Errno 65] No route to host

Your code gets jammed up when there’s an error it seems. Perhaps you can keep your code running in the event of an error?
Seems like the thread method I found insulates the rest of the program and keeps it running.
From what I’ve read there is a potential problem with my code as it is.
The code creates a new thread each time it executes the timed operations.
Thread-1, Thread-2, Thread-3 and so on…
This is OK, the thread is closed again after each occasion, so it doesn’t spiral out of control with countless threads running…
But it is apparently inefficient, it’s better to just keep a thread active for even a timed function like this…
Especially important for quick repetitive operations so I read.
It’s a simple change to the code to keep the thread active.
https://stackoverflow.com/a/18180189
However the ordering of the timer.start() means the problem with the error is back.

I note also, the threading module has inbuilt drift compensation, which I’ve this evening learnt is quite important!

I’m learning from scratch, so appreciate this exchange.


(Paul) #20

You’re right about my script, it isn’t production ready, I was just trying to establish what was the min required. As a rule most Python code should be wrapped in try/excepts to catch errors and here there are lots of possible hazzards to test for or handle, and for error messages to be created, (for me) this would ultimately be handled in emonhub, so a rough and ready script is fine for my purposes. But I can try and help with refinements if it helps you.

I was just not keen on spawning another thread (even if it was only the one) as emonhub is already threaded and I would want to avoid threads starting more threads (it gets messy).

The errors that your seeing (although they should indeed be handled) are essentially the result of the socket lines lifted straight out of you script, if the host is busy or not available I wonder if you have your own script running at the same time and one or the other is blocking. I don’t really have anything set up to test this.

When I wrote that I did wonder if the socket should really be opened, used then closed rather than leaving it open. That is how the sockets created in emonhub are used by multple clients, they connect, do what they need to and close.

No route to host normally means there is no network or possibly a dns lookup issue, the latter is unlikely when using an IP address.

The “drift compensation” is why I opted for using the actual time rather than a sleep for so many secs, The “insulation” you refer to can be a double edged sword, any exceptions raised within the thread are not likely to get relayed out to the calling thread so if you have any issues you could be working blind, all you’ll know is it fails. This has been a headache when debugging emonhub when threads crash out.