Hello,
This is perhaps a beginner question, but I am struggling. I am using an emonpi, which is one of the very early versions that were shipped as part of the kickstarter campaign.
I have successfully created a number of feeds locally to the emonpi, and up until now I have just been publishing very basic data to remote emoncms. These are power1, power2, power1pluspower2, and vrms, and they are published to emoncms.org quite successfully.
I am now trying to create some more advanced data analytics on the remote emoncms site, but I am finding that the list of available operations in the inputs processing section of emoncms is much more limited than on the local emonpi. For example, operations such as “Reset to Original” appears to be completely missing from the drop down list on emoncms.
I have already done all of the input processing that I want on the local emonpi, so all I really want to do is find a way to take a local feed and publish it to the remote emoncms site. Is this possible, and if so please could someone guide me?
Hello Bill,
Thank you for the link and I appreciate you taking the time to reply, but I’m afraid that doesn’t quite solve my problem.
I have already been able to get the raw inputs to the emonPi appear in emoncms, but it is the lack of operations (i.e. things like “Reset To Original”) on emoncms that prevent me from doing what I want.
It’s the feeds from emonpi that I would like to publish, because I have already successfully done the processing on them that I require.
Do you know how to achieve that please, and are you perhaps able to guide me?
Correct, there are several processes not available on emoncms.org, Which processes are available where is not documented any where since all the process documentation is within emoncms, ie if you don’t have the process, you don’t have the documentation and are none the wiser to it’s absence, this is also true about redis dependent processes, if you do not have Redis installed you would be unaware of the processes you are missing.
It makes sense to only do the processlists once so sharing the processed data makes total sense, there is a “publish to MQTT” process in the local emoncms but emoncms.org doesn’t accept mqtt inputs yet, so that’s not much help unless you use a third party script to forward the mqtt to emoncms.org.
For the record, what you want should be as easy as defining the relevant feedids in the emonhub.conf. The original plans for the emoncms http interfacer included the facility to use the same url and apikey to retrieve any number of feed values, (that’s what the feed/fetch api was specifically added to emoncms for and also why the interfacers were made bidirectional) and direct them to where ever you like via other interfacers using routing features. for example publish to MQTT, send to another emoncms via http, transmit over RFM to a emonGLCD or display on the emonPi LCD or save to text file etc etc or even all of the above at once, alas this functionality was not included with the “emonpi variant” of emonhub. Until such time as it is included you will have to use a bespoke script as Bill suggests.
It was a bit of a marathon, but I think I have managed to get something working. I wrote and debugged it on a desktop linux box, and I had quite a lot of arguments with sed along the way. All good fun in hindsight!
I shall now have a go at moving it over to run on the emonpi. I think I’ll try using cron to run the job every 10 seconds, and see whether that works for me.
If successful, I’ll try to tidy up the script and re-publish it to make it useful for other people.
This has been a bit of a learning curve, so I am keen to share what I have done with the forum in the hope that it might be useful one day. Some tidying up is required first though, because it’s currently a complete mess.
With thanks to Bill and Paul, I have learned a lot from this adventure and now have a script which publishes my chosen local emonpi feeds to emoncms dot org every ten seconds.
I discovered that the scheduler script written by Bill updates every second, and so I adjusted that to every ten seconds to be a bit more friendly to the emoncms server.
I also put together a full set of instructions which detailed the modifications that I made to my emonpi, and I wanted to share them with the forum, but sadly it seems that I cannot post attachments because my account is too new. If someone on the forum would like to adjust my account privileges then I will happily post my work for others to use.
Thanks again for the help. I shall now turn my attention to the next little challenge of logging and visualizing the data coming from and going to the Powerwall 2.
I checked your user level. You should be able to post attachments.
The system won’t allow certain extensions. .py and .sh are two that come to mind.
To get around that, you can either change the extension to .txt, or zip the file first then attach it.
Nope, you’ve got it. I likely fat-fingered that one. It should be 10 and I managed to leave out the zero.
I’ve fixed the script in my post. Thanks for finding the error.
OK, here is the script that I wrote. Once again, a lot of credit and huge thanks should go to Bill for his guidance on this topic. I merely adapted it to use sed, added some debugging statements to help new users, expanded the scope of what Bill originally wrote, and added some comments / instructions. Hopefully it will help someone at some point in the future.
{FILE LINK REMOVED BECAUSE IT WAS REVISED AND UPDATED LATER IN THIS THREAD}
Question for ya rich, why’d you decide to use sed?
I’ve used the script for over a year with no issues. There shouldn’t be any CR/LFs in your feed data, the only thing you should have to do is strip is the quotes. I used the bash A=${A//\"} construct to keep it as simple as possible.
Hi,
I went through a ton of pain trying to get to the end result, and I am sure that there are cleaner methods than what I ended up with. For a long time I was having huge problems because the value of A was overwriting the first few characters of “curl -s https”, and it was driving me mad. I pondered whether there was a CR at the front of the string that was captured into A, and that’s why I went off down the avenue of using sed to remove it. Looking back, it may have been a rat hole, but I certainly learned a lot about sed in the process!
I was also originally “source”-ing the script rather than running it with “./script.sh” and that may have been some of the problem. The use of the concatenation in my assembly of the final URL was part of my ongoing debugging effort to figure out what was going on, and I didn’t circle back to re-introduce other methods of assembly at the end. The use of the concatenation also helped me to add new feeds into the post operation on a line by line basis, which was easier for me at the time.
I guess it’s the difference between a beginner and an expert, so you clearly have the advantage over me there.
But your points are all valid about the variables, so I have updated the files:
You’re probably right about that. If you come across that difficulty in the future, use the dos2unix command to strip the carriage returns from the file. Every Linux distro I’ve ever used has had the dos2unix utility included as part of the distro, or available as an installable package There’s also a counterpart to dos2unix called unix2dos, which as the name suggests, does just the opposite of what dos2unix does.
