RFM69Pi distance range and Weather Underground integration?

Getting Started
,
1

Hi,

I have just learnt about openenergymonitor.org today in a blog post
Open source energy monitoring using Raspberry Pi - Raspberry Pi
And I am very glad about the reading because otherwise I might have not known of your existence, mission and goals.

I have been browsing the different products, guides and shop items and have a couple of questions:

1.- What is the range of the RFM69Pi 433Mhz Raspberry Pi Base Station Receiver Board? I should like to have an idea of far apart can the emonTH and the emonBase be. I could not see that detail in the guide https://guide.openenergymonitor.org or wiki RFM69Pi V3 - OpenEnergyMonitor Wiki
So… 10 meters in straight line with no obstacles? 100 meters?

2.- Remote logging. Is there data integration with Weather Underground? May data be posted to WU has an they have API that may be used to upload to their wesite as well? That way, there would be nice integration with other Personal Weather Stations. Yes, I know my approach is a bit more focused on climate than energy. Hope you are not against it.

Regards to every forum user!

1 Like
2

That depends on the levels of interference on that band. 10 m should be no problem at all, but I would not like to say the same about 100 m. You should find more about the range on the JeeLabs website. Note we use the RFM69CW, not the higher powered version.

I can’t answer your second question, but I know there are users of WU here, so hopefully someone else will pick this up.

3

To my knowledge most of the WU integrations have previously pulled weather data from WU to post on emoncms so that weather data can be viewed and recorded alongside energy use and production to map efficiency etc.

I wrote a WU integration for the original emonhub that did just that, and I can recall a couple of discussions where users have had existing WU weatherstation setups that posted to WU and the they also queried their own weatherstation data via WU for uploading to emoncms.

There have also been discussions about direct connections between emonhub and specific branded weather stations for uploading to emoncms, but I’m unsure if anything materialized for that.

There is also node-red, again I am aware of grabbing data from WU but do not recall any instance (here) of posting to WU.

I think the main stumbling block for passing data to WU (via an emon setup) is interfacing with the weather station, there are so many and they do not use a common protocol, as many of them already post to WU and others are more readily hackable to do so as WU is a more common target (than emoncms) for weather data.

Where is your weather data coming from? ie type of hardware and current communication setup.

Are you looking to develop something for the purpose or just asking if something is already available?

4

Thank you for your replies and insight to both, appreciated.

@pb66 In order to reply your questions, please let me tell you the idea is grabbing temperature readings from the DS18B20 external temperature sensor in emonTH and then transmitting it to emonBase. Or, as an alternative, grabbing readings from the same DS18B20 and passing the data via ethernet. Both options are described in the guide so it would be ok with both. In any case, the sensor is DS18B20, which is the one emonBase is configured with from what I have read in the guide. So data would not come from a davis vantage pro 2 weahter station or anything related.

In regards to second question, I have more of a hobbyist approach so prefer to create something by applying guides and code available and a bit of tweaking to get things done. Emoncms looks amazing and feeding temperature readings there is just great. Now, is those DS18B20 readings could be fed into the Weather Underground site, I think more people could benefit from it.

Kind regards,

5

In addition to the RFI that Robert mentioned, any obstacles in the path, e.g. buildings, walls, trees, etc. will affect the maximum usable range as well.

6

I would say the easiest way to do what you ask would be to let the emonTH → emonBase → emonCMS function normally and as intended, then with that data being available via the feeds api, you can write a small script or use node-red to query the data from emoncms as simply as

https://emoncms.org/feed/fetch.json?ids=101,102,103,104,105&apikey=abc123

where “101”,“102”, “103”,“104” and “105” are the feed ids for the metrics you want to share with WU and “abc123” is your emoncms apikey (in this case it can be either the readonly or the readwrite apikey)

Then parse the result eg

[15.6,16.1,16.7,21.1,50.0]

into an out-going WU api call such as

https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php?ID=MYPWSID&PASSWORD=pa55w0rd&dateutc=now&tempf=60&temp1f=61&temp2f=62&indoortempf=70&indoorhumidity=50&action=updateraw

where “MYPWSID” and “pa55w0rd” are the ID and password for the Personal Weather Station (PWS) you create at WU and tempf, temp1f, temp2f, indoortempf and indoorhumidity,are the values retrieved from emoncms above, the temperature need converting from Celsius to Fahrenheit as WU only accepts F where as emoncms values are usually C (this can be altered in the sketch/firmware or even via the input processing if you prefer to persist data in F), the humidity is a percentage in both cases.

Info on the PWS api can found at http://help.wunderground.com/knowledgebase/articles/865575-pws-upload-protocol which also lists all the possible fields you can include.

This script could be run every 5mins by creating a cronjob or you would need to look at using a start up service and putting the code in a loop, the latter is only really worth doing if you are intending to update a lot more frequently, in which case you would need to use the “rapidfire” WU PWS api instead (details at the bottom of the linked PWS api page). As temperature and humidity are quite slow moving a 5min update is more than ample as WU only updates the data every 5min unless the “rapidfire” api is used for “realtime” data, I think “rapidfire” is more aimed at capturing rain fall and wind speed direction and speed etc,

7

So I gave this a try…

this is a python script that will do what you want, it’s quite rough around the edges as there is no error handling etc and the temperature conversion from C to F is triggered by just checking to see if “temp” is in the field name.

import urllib2

emoncms_url = "https://emoncms.org"
emoncms_apikey = "abc123abc123"
emoncms_feeds = "101,102,103,104,105"
wu_pws_fields = "tempf,temp1f,temp2f,indoortempf,indoorhumidity"
wu_pws_url = "https://weatherstation.wunderground.com"
wu_pws_id = "MYPWSID"
wu_pws_psk = "pa55w0rd"

emoncms_feeds = emoncms_feeds.replace(" ","")
wu_pws_fields = wu_pws_fields.replace(" ","").split(",")

emoncms_request = emoncms_url+'/feed/fetch.json?ids='+emoncms_feeds+'&apikey='+emoncms_apikey
print emoncms_request

emoncms_reply = urllib2.urlopen(emoncms_request)
emoncms_reply = emoncms_reply.read()
print emoncms_reply
emoncms_reply = str.split(emoncms_reply[1:-1],",")

emoncms_feeds = emoncms_feeds.split(",")

wu_pws_string = wu_pws_url+'/weatherstation/updateweatherstation.php?ID='+wu_pws_id+'&PASSWORD='+wu_pws_psk+'&action=updateraw&dateutc=now'
for i in range(len(emoncms_feeds)):
        v = emoncms_reply[i]
        if "temp" in wu_pws_fields[i]:
                v = str((float(v)*1.8)+32)
        wu_pws_string += ("&"+wu_pws_fields[i]+"="+v) #emoncms_reply[i])

print wu_pws_string

wu_pws_reply = urllib2.urlopen(wu_pws_string)

print wu_pws_reply.read()

copy the above code to a new file named “wu_pws.py” (in Pi’s home dir) and edit the details at the top, you can test it with

python ./wu_pws.py

you should get output like

pi@raspberrypi:~ $ python ./wu_pws.py
https://emoncms.org/feed/fetch.json?ids=101,102,103,104,105&apikey=abc123abc123
[14.37,14.37,14.37,19.2,56.3]
https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php?ID=MYPWSID&PASSWORD=pa55w0rd&action=updateraw&dateutc=now&tempf=57.866&temp1f=57.866&temp2f=57.866&indoortempf=66.56&indoorhumidity=56.3
success

if all is well then for this to run every 5mins create a crontab

crontab -e

and at the bottom add

*/5 * * * * /usr/bin/python /home/pi/wu_pws.py >> /dev/null

let us know how you get on if you try it

8

May not be relevant but I have the older RFM12Pi board and it successfully receives from a emonTX with an RFM12B board at 433Mhz over a distance of 7m through a timber floor and several timber walls. The emonTX is also received by a emonGLCD at a distance of 10m in the same environment.
Hope this is helpful in some way.
If you find a solution to the WU question I would also be interested.

9

Gary makes a good point. Wood doesn’t attentuate RF energy as much as concrete or stone does.
Especially concrete with lots of steel (re-bar) in it. (tends to act like a Faraday cage)

1 Like
10

And dry stone or concrete is less bad than wet. You particularly need to avoid foiled-surfaced insulation.

I’ve had an emonTH (on battery power - rechargeable) working over 30 m line-of-sight across grass - at 868 MHz.

If it’s any help, the GLCD is on the opposite side of a stone and rubble wall that’s about 700 - 800 mm thick. Whether the r.f. is going in through the window and bouncing around inside the room I don’t know, but it’s worked fine for a few years.

Selection_014.jpg839Ă—379 51.1 KB

11

Quite likely, as the higher the frequency, the more reflective it behaves. (e.g. radar)

12

Another point - you can slow the data rate, which will reduce the bandwidth, and you should then see a worthwhile improvement in the range.

I knew I’d seen something over at JCW’s: NameBright - Coming Soon

13

@pb66 Thank you for your detailed and explanatory messages plus python script. Thumbs up for your support! I think that will be appreciated not just be me but by many like-minded users. I do intend to run it and do as many tests as needed. As of now, I am waiting for the mail order with the sensor, breadboard etc. to arrive. I have already learnt how to set it up so as soon as it arrives I will be hands on it. And I will report back here. Guaranteed. Thank you for the cron command and instructions as well. It has helped me to understand the meaning of the different paramenters. Great!

I have a couple of doubts, if you don’t mind…

&dateutc=now
The guide for the PWS upload protocal indicates:

if the weather station is not capable of producing a timestamp, our system will accept “now”.

Couldn’t the emoncms timestamp be used? I mean, as it would be closer in timee to the actual reading time by the sensor. Or whatever reading time may be grabbed and is closer to actual reading time. Well, that must sound a bit paranoic, sorry for that. But with a 3G modem connection with rather slow pings (e.g. 8.8.8.8 at 0.2ms or 0.4ms) time differences might matter.

In regards to update frequency, WU developer plan allows 500 calls per day, which would roughly translate into a a 3-minute update interval (1440 min / 3-min update). So running the script every 3 or 5 minutes would be really good and accurate as well. As you rightly say, that interval seems ample enough. RapidFire frequency update rate would show a fantastic figure for real-time monitoring.

Yes, you’re reading my mind. Good! I was thinking of adding the script to systemd (Debian) so that it runs at boot-time. I learning about it and doing some tests at the moment. Plus, in case of a power outage (a fact that must be taken into consideration), the script would run automatically again during start-up.

@garylovesbeer, @Bill.Thomson, @Robert.Wall Thank you for your informative remarks as well. I think in the end it may all be a matter of trial and error. Place emonTH further away, run a test and loop, till you enter no-signal-back lands.

Regards to everyone & waiting for the mail order,

14

You could indeed supply a date and time to WU PWS but I see little point unless you are using “rapidfire” to update very frequently or you are updating/rebuilding historical data. Temperature and humidity are so slow moving, does it matter if the 5min updates are not pinpoint accurate timewise?

The data will have passed via so many points eg the emonTH sending via RF every 60seconds’ish, the RF sends can then get delayed by a few seconds if the airwaves are congested, emonhub then gives it a “current” timestamp to the nearest millisecond, if posting to emoncms via MQTT this timestamp is ignored and another timestamp is given on arrival at emoncms or if using http the emonhub timestamp is used, either way that timestamp is then converted to an INT ie rounded down to the next whole second and passed for processing. If you are using fixed interval feeds the timestamp will then get distorted again somewhat to.

So using the emoncms datapoint timestamp isn’t accurate, but it does complicate the query request somewhat as you would need to query each feed separately rather than use the “fetch” api to get them all in one go, then which of the timestamps would you use?

I did consider using a timestamp generated by the WU_PMS.py script since we are effectively saying “the data at this time is…” but since the WU PWS only accepts a correctly formatted timestamp (not a unix timestamp), it adds additional “conversion” complexity and potential error if not formatted correctly, I felt the &dateutc=now was the best way forward but you can go in any direction you choose, my script was just an experimant and a springboard to maybe help get you started.

You are right in saying

however I do not think that applies to the WU PWS, (I don’t think you even need a developers account). The WU interfacer developed for emonhub queries WU at 3min intervals for that reason, I don’t recall seeing a PWS posting limit (other than the 2.5s rapidfire) but I’m sure there probably is one, there may be a fair usage per user but as a user can have several PWS’s I expect that to maybe be quite lenient.

There is actually little point in posting data at less than 5min intervals since the data is only updated at WU every 5mins unless “rapifire” is used. This can be tested by setting the above cronjob to run every 1 min, only every 5th update will appear in the PWS data list.

You wouldn’t need both a cronjob and a startup service. A cronjob can only be triggered every 1min max and the timing isn’t guaranteed to be accurate (which would become more of an issue on much smaller intervals). Therefore setting up a cronjob is the easiest method with a 5min interval and a move to a startup service would only be needed if you decide to post at much smaller intervals via WU’s “rapidfire” api. But again each to his own, If you prefer, you could create a startup service and edit the script to sleep for 300secs in each loop, or to avoid network delays disturbing the “5min” routine you could introduce a “if 300secs since last send” check and let it loop faster.

15

A slightly different perspective. I have a Maplin weather station from which I collect data using pywws which has a built-in mechanism for posting to WU. I have modified a pywws template to pass the weather data I want to record within Emoncms to it via MQTT. I have also passed the data to Emoncms via the http API as well.

1 Like