Read from local emoncms and post data to emoncms.org with node-red

copderoc · 2017-07-12T17:25:29.085Z

Hi all,
I am trying to mirror some of my emoncms local server data to emoncms.org server with node red.
I have a working sketch running, I can read from local server and post data, but I can only read 1 feed and post it to 1 node of the remote server.

I would like to read 4 feeds of my local emoncms and post them to the same node of emoncms.org, i don’t know how to merge the data into one string, can you help me?

Captura de pantalla 2017-07-12 a les 19.24.45.png857×329 26.2 KB

copderoc · 2017-08-22T14:40:31.000Z

Hi all, finally i found a solution to this, I build a string with the node red join node, and then it’s posted to emoncms.org.

401010×656 85.4 KB

Bill.Thomson · 2017-08-22T16:50:53.073Z

Hi Ernest,

You can set up emonhub to forward data to emoncms.org as well as your local instance of emoncms.

Was there a reason you decided not to do that?

copderoc · 2017-08-22T21:34:49.994Z

Hi Bill,

My first intention was to use the emonhub mirroring, but I can’t use emonhub because I have arduinos posting directly to a local emoncms server with a raspi and emonpi. This is the topic talking about that.

Thanks.

Bill.Thomson · 2017-08-23T13:06:36.958Z

Here’s a bash script that reads feed data from a local instance of emonCMS
and forwards it to emoncms.org via curl.

#!/bin/bash
#
RO_KEY='your-local-emoncms-read-only-apikey'
RW_KEY='your-emoncms.org-read-write-apikey'
#
# change FF to the numerical id of the feed you wish to read data from
#
A=$(curl -s "http://127.0.0.1/emoncms/feed/value.json?id=FF&apikey=$RO_KEY")
B=$(curl -s "http://127.0.0.1/emoncms/feed/value.json?id=FF&apikey=$RO_KEY")
C=$(curl -s "http://127.0.0.1/emoncms/feed/value.json?id=FF&apikey=$RO_KEY") 
#
A=${A//\"} # strip quotes from json string
B=${B//\"}
C=${C//\"}
#
# change NN to the number of the node you want to send data to
#
curl -s "https://emoncms.org/input/post?node=NN&csv=$A,$B,$C&apikey=$RW_KEY" > /dev/null 2>&1
#

get-data.txt (638 Bytes)

Delete the .txt extension, or change it to .sh

copderoc · 2017-08-24T05:22:02.426Z

Hi Bill and thanks for your time, I am lost here, I don’t know what is a bash script and where to put it?
How will it mirror the data without api keys, or server ips? Sorry I am a novice

Thanks.

Bill.Thomson · 2017-08-24T12:01:31.000Z

copderoc:

Sorry I am a novice

Not a problem, we were all newbies once.

Here’s some info to get you started with bash scripts:

ryanstutorials.net

What is a Bash Script? - Bash Scripting Tutorial

What is a Bash script and how do we format and run ( execute ) them. Section 1 of an 8 section introduction to Bash scripting.

copderoc:

How will it mirror the data without api keys

I did forget to include that part. I’ve edited my post to include the API keys.

copderoc · 2017-08-31T14:45:03.272Z

Thanks Bill,

I got it! I just created get-data.sh script on my raspi and when I execute it I get the inputs on emoncms.org without node-red, only one more question, how to auto execute the script every 1s to have the emoncms.org inputs updated?

Thanks again.

Bill.Thomson · 2017-08-31T15:22:25.492Z

copderoc:

how to auto execute the script every 1s to have the emoncms.org inputs updated?

Keep in mind there’s a limit to how often you can send data to emoncms.org/
If memory serves, that limit is once every 10 seconds.

Bill.Thomson · 2017-08-31T15:27:09.827Z

copderoc:

how to auto execute the script every 1s to have the emoncms.org inputs updated?

Here’s a Python script that should do the job.
I’ve attached APS as well as the Python script.
You’ll need to install Advanced Python Scheduler with pip.

APScheduler-2.1.2.tar.gz (40.8 KB)
no need to unzip the tarball. pip can install APS directly from the archive.
There is a newer version of APS available, but I prefer this one. The newer one (v3) isn’t as simple to use as this one (v2) is, so I stayed with the older version.

jobsked.txt (377 Bytes)

copderoc · 2017-08-31T20:23:17.953Z

Thanks Bill, I have it working.

It’s autoupdating correctly but if I close the ssh terminal window it stops running. How to configure it to autoexecute the python script, also if the raspi is rebooted?

Do you thing this method is better than the node-red one? More clean? More secure?

Thanks.

Bill.Thomson · 2017-08-31T21:00:32.570Z

copderoc:

How to configure it to autoexecute the python script, also if the raspi is rebooted?

Here’s info on how to autostart the Python script. Using that method, it’ll restart after a reboot.

copderoc:

better than the node-red one?

I can’t say, really. The best tool to use is usually the one you’re most familiar with.
So, for me, that turns out to be bash and Python, vice NodeRED.

Cleaner?
As in simpler? Yes.
But for those who aren’t up to speed on writing bash and Python scripts,
NodeRED is usable. i.e. NR makes it easy for a user to get the job done.

More secure?
Because it’s not as complex as NodeRED, I’d say at least somewhat.
But if somebody wants to exploit a machine/OS/app etc. they’re going to do it anyway.

copderoc · 2017-08-31T22:39:04.771Z

Thanks Bill,

I get it working!

seco · 2018-04-25T20:41:32.774Z

I think this situation you should use Python as an alternative scripting language which is simpler and easier for beginners.
But if you want to work with Bash, this tutorial will be a great start for bash scripting.

One thing to mention regarding naming your Bash file, you don’t need to rename it from .txt to .sh and that’s because the only thing needed is to make it executable regardless the extension. even if the file without the extension at all.

Regards,

Bill.Thomson · 2018-04-25T22:33:34.103Z

seco:

One thing to mention regarding naming your Bash file, you don’t need to rename it from .txt to .sh

Quite true. As you said, an extension isn’t necessary. But, it’s common convention to name a bash script with a .sh extension. That way, months later when you go back to look at it, modify it, etc. you know immediately its a shell script. Similar in principle to naming a Python script with a .py extension.

Christian298 · 2018-05-08T20:39:22.892Z

Good day Bill,I’m Christian. I have a data logger with raspberry pi that takes data by serial port(ttyAMA0) and after a script post everything on local emoncms and it works properly.l have followed the OEM guide for post data on emoncms.org,changing emonHub.conf, but some problems appeard:
"
2018-04-26 19:58:32,873 INFO EmonHub Pre-Release Development Version (rc1.2)
2018-04-26 19:58:32,875 INFO Opening hub…
2018-04-26 19:58:36,895 ERROR Unable to create ‘RFM2Pi’ interfacer: Attempting to use a port that is not open
"
Could your script works in my project?
Which is folder installation and owner?

Thank you in advance for your information.
WBR
Christian

pb66 · 2018-05-09T09:55:34.751Z

What is it you are trying to do?

Christian298:

I have a data logger with raspberry pi that takes data by serial port(ttyAMA0) and after a script post everything on local emoncms and it works properly.l have followed the OEM guide for post data on emoncms.org,changing emonHub.conf, but some problems appeard:

Your other thread

No input data in emoncms Emoncms

Good morning to all, I installed emoncms on my raspberryi pi following the good guide published on github. In the past the system worked properly with Wheezy but with Jessie I’m experiencing many errors. 2018-04-14 17:52:17.133|ERROR|input_controller.php|Format error, json value is not numeric on this directory “/var/log/emoncms.log” and following on http://192.168.1.100/emoncms/admin/view I get the data by serial port and checking with minicom everything is ok. Could it be a pr…

shows a script that posts direct to emoncms, therefore editing emonhub.conf will have no effect since the data is not flowing through emonhub, it is going direct from your script to emoncms.

if you are simply trying to post data to 2 emoncms instances you could just add a second target to your script so that it posts to 2 emoncms’s directly.

The error you see in emonhub is most likely because you are using the serial port for your datalogger and emonhub is still configured to use the serial port for a rfm2pi or emonpi, removing the “rfm2pi” section from emonhub.conf should resolve that.

Bill’s script would undoubtedly work, but it is going around the houses a bit, there is no reason to post the data to one place and then use another script to read the data from there and post it elsewhere.

The easy option would be to edit your first script to post to 2 places, the best option would be to change your script so that it sends to an emonhub socket and then emonhub can send it to as many emoncms’s as you like and offer error logging and data buffering to boot.

Christian298 · 2018-05-09T20:09:28.882Z

Good day Paul,the first thread concerned the data post on local emoncms and was solved.
This is related to issue that I have now for to post the data on emoncms.org.

" removing the “rfm2pi” section from emonhub.conf should resolve that"
done but not solve.

Thank you Paul I add the new line in my personal script,as you suggested me in your reply and now I can see the data on emoncms.org.
SUPER
Paul, has been a fight this installation…but now I’m winner!!!
Many thanks again