Thanks, @stephen. I knew I’d seen something about this, but I could not remember who had contributed.
@Helmich_de_Lange
You can still send the temperature sensors that are monitoring the piles directly to the emonPi for display and logging, and I’d suggest that separating out the heat energy calculation from the emonPi would be a better way to proceed, and even though it might add a little to the overall project cost, I think you would gain overall by using two separate but proven systems.
@Helmich_de_Lange I also wrote some code to read multiple DS18B20 sensors which may be more cost effective for you
Thanks for your comments! Love the simple node solutions, keep that in mind for the future. It seems that we need to expect a reasonable error in the flow calculations.
Because heatcompost piles are not a proven concept yet we like to have:
It seems that an emonpi can offer that and because we already like to use emonpi for storage and dashboarding it would be nice if we can stick to that hardware for reading the flowsensors.
I digged deeper in the community threads and found the “socket interface”.
It lookes like i can place my current python code (that read 16x DS18B20 and flow sensors connected to the GPIO) on the emon pi and use the socket interface to send the sensor values to emonHub.
Can this also be done by using the MQTT-interface or does that bypass emonHub?
I tried to get the socket interface working on a raspberry pi with emonpi-image using:
https://github.com/openenergymonitor/emonhub/blob/emon-pi/configuration.md
https://community.openenergymonitor.org/t/directly-connecting-to-optical-pulse-counter-with-rpi/1294/2
https://community.openenergymonitor.org/t/emonhub-receiving-weird-data-where-is-it-coming-from/839/4
changes i made to emonhub.conf:
[[mysocketlistener]]
Type = EmonHubSocketInterfacer
[[[init_settings]]]
port_nb = 50012
[[[runtimesettings]]]
pubchannels = ToEmonCMS,
#timestamped = True
[[30]]
nodename = sockettest
[[[rx]]]
names = val_A
datacodes = h
scales = 0.01
units = C
But i got stuck on the dataformat:
The goal is to send a float value of 25.35 or integer 2535:
s.sendall(‘30 25.35\r\n’)
emonhub conf:
datacodes = f
scales = 1
emonhub outputs:
2018-07-31 08:42:50,446 DEBUG mysocketlistener 1 NEW FRAME : 30 25.35
2018-07-31 08:42:50,448 WARNING mysocketlistener 1 RX data length: 1 is not valid for datacodes f
s.sendall(‘30 2535\r\n’)
emonhub conf:
datacodes = h
scales = 1
emonhub outputs:
2018-07-31 08:45:40,800 WARNING mysocketlistener 2 RX data length: 1 is not valid for datacodes h
s.sendall(‘30 25 35\r\n’)
emonhub conf:
datacodes = h
scales = 1
emonhub outputs:
2018-07-31 08:38:23,410 DEBUG mysocketlistener 1 NEW FRAME : 30 25 35
2018-07-31 08:38:23,411 DEBUG mysocketlistener 1 Timestamp : 1533026303.41
2018-07-31 08:38:23,412 DEBUG mysocketlistener 1 From Node : 30
2018-07-31 08:38:23,412 DEBUG mysocketlistener 1 Values : [8985]
Finally i got some input values but they are not right.
Can someone point me in a direction that i am missing?
Datacodes are used to reconstruct values from their byte components eg 192 93 and datacode h = 240. The values you are including in your payload are full human readable numbers that require no decoding or reconstruction. So you can use datacode “0” as in zero decoding or you can just omit the datacode(s) line as the default datacode for the socket interfacer is already “0”.
[[mysocketlistener]]
Type = EmonHubSocketInterfacer
[[[init_settings]]]
port_nb = 50012
[[[runtimesettings]]]
pubchannels = ToEmonCMS,
#timestamped = True
[[30]]
nodename = sockettest
[[[rx]]]
names = val_A
units = C
Publishing to mqtt will indeed bypass emonhub.
out of curiosity - how large of building are you trying to heat with your 10 - 100 m3 “biomeilers”
I guess it modern update to a very old heating system common in Roman times. the compost/manure was in the crawl space under the floor and vent stacks in the wall. which kept the house warm during the winter months
That did the trick! Thnx.
Nice, didn´t knew that of the romans, do you have a source?
Well, how much power is the big question, at the moment I have one (60 cubic meter) producing around 25 Watt/cubic meter.
I hope to get the proces more active and reduce losses to get more then 50W per cubic meter. That would be enough to heat my house.
In germany they are building a site with multiple biomeiler setups to find the best solution. The idea is that this measurement system is placed there to collect data.
oh what part of Germany do you live in, you might be able to check it out personally my wife comes from Saarbrucken, and that is where I first learned about it, in and around the all those Old roman cities sites in and around Trier. i do not remember which site it was exactly where i first saw it. but they show the complete construction of it and the deconstructed floor where they found the slave boy who was found in there pit under the house and the vent stacks in the walls all depicted on informational boards…
but and interesting method by Rudolf Doernach 1983 book - “handbuch fur bessere Zeiten” but the idea was sort of simular to yours. but instead of only compost. his was also a biodigester he had a +2000 l biodigester tank that he buried in compost . to maintain the biodigester internal temp and he would produce natural gas from waste bio material burn it for hotwater, cooking and heating purpose I believe from his book he states 5 gallons of bio material per day was enough to make all the bio gas he required for heating ,cooking and hotwater needs
My home is in holland, the biomeiler community spreads over whole europe.
Sounds like Rudolf Doernach describes the same method as
Jean Pain - Another type of garden
Will search for the book, maybe he publishes more numbers. Thnx
How does emonpi organize the DS18B20 sensors hardware-adresses so that T1 is always linked to the same hardware-address? Also after a reboot.
Can emonpi store the hardware adresses somewhere?
EDIT: one of the suggested topics below pointed me to the answer:
https://learn.openenergymonitor.org/electricity-monitoring/temperature/DS18B20-temperature-sensing-2
It seems to me that the socket interfacer only accepts a string of values in a specific sensor order. Is there a way to send the hardware adresses through the socket interfacer?
Otherwise the local python (sensor reading) scripts would require a config or INI file that stores that order. Is there any change that an config file like that can be editted through an emon dashboard item?
curious does not your python script always read them in the same order, as the order alway comes up the same on esp or arduino as it based on the “serial number” - based on the hexdecimal 4 steps over from the beginning .
you sure you do not want to build a node that communicates with the pi. ( either esp(wifi), serial, rs485, Ethernet( arduino) or other method ) then trying to do all on the PI
you could use stuart’s above it interface directly emoncms/emonpi as it would be the simplest
I would help you more directly but I do not use emoncms/emonpi as I have my method talking directly using MQTT to wifi connected touch screens and openwrt router as a broker storing my historic data as it passes through the router locally in RRD format and/or passing it on at higher detail to a influxdb to be view using grafana
I’m strugling with the same options. But I really like to make use of the benfits of emonHUB because emonHUB “can buffer the data, so that when posting to emoncms.org and the internet goes down (for example) emonHub can buffer the data from your script and post it to emoncms.org later, after the network comes up again”
Also posting to different/multiple emonCMS.org accounts would be easy to setup using the intented use of openenergymonitor and therefore easier to configure by the biomeiler community instead of programming.
I also like to replace sensors without getting the senors inputlist re-ordered.
So the setup of @Stuart sounds perfect but i like to convert it to the socket interface, as far as I could found that’s the only option to make benefit of emonHUB.
Any comments on this are usefull for me.
@stephen. I had a look at grafana, looks nice of course. But then I have to program the whole local backbone, storage, backup, dashboarding, sync-with-the-cloud, etc. myself. I like the idea that with emonpi/emonCMS that whole part is covered by a userguided, community supported and webinterface platform.
Disclaimer: I have not looked at the Raspberry Pi temperature-reading script.
I think that is only possible if the algorithm for reading the sensors has a list of addresses which you program as fixed values. (See note below.) If your algorithm searches for the sensors each time it starts up, then a missing sensor will be ignored and those that are found afterwards will all move down by one position in the list. This is how the sketch in the emonTx and emonPi front end works. You do not want this.
I wrote an explanation of how the sensor search works: https://learn.openenergymonitor.org/electricity-monitoring/temperature/DS18B20-temperature-sensing-2
Note: You can of course hold the list of sensor addresses in a disc file - if the file does not exist, search for the sensors and write the file, else if it exists, use the file. If it’s in human-readable form, you can edit it by hand if a sensor goes faulty and has to be replaced.
@Robert.Wall
Your note describes exactly how I programmed it this weekend.
We can edit this INI file through ssh or SCP. If someone does know a hack or tweak to edit this file through the emonpi webinterface (like emonhub.conf) or via a dashboard option I love to hear that.
I thought emonhub supported sub to mqtt topics. if not you probably could make it sub easy enough using socat and writing the topic to virtual serial and direct the emonhub to monitor the virtual serial port.
first run
socat -d PTY,link=/dev/ttyS10,group=tty,mode=770
write perl script or similar to this : ( probably not 100% accurate as I am not 100% familiar and probably needs a bit of tweaking )
#!/usr/bin/perl -w
open(SUB, “/usr/bin/mosquitto_sub -t /emon/# |”);while ($emon = <SUB>) {
$emon =~ tr{/}{ }; # removes / and replaces with space
print "$emon ";
system(“echo $emon > /dev/ttyS10”);
}
I think that should work fine with a little tweaking of the socat command and the perl script . unless emonhub does not support multiple serial ports for some reason ( edit: but then again you could simply write to the default serial if it does not cause an overlap issue)
Correct, the socket interfacer is a very basic input that accepts a space separated string of values
No, not easily, you would need to alter a significant amount of emonhub code to do that.
What you could do is add an entry to emonhub.conf [nodes] section under the node id you have chosen for your temp sensor node and either use the names = line for the ids if you wish to use the ids as mqtt topics and input names or add another line “ids =” which emonhub would ignore and get your script to read the emonhub.conf directly. The emonhub.conf can already be edited via the emoncms web pages.
eg1 (using example settings from earlier in this thread)
[[30]]
nodename = sockettest
[[[rx]]]
names = abcd1234
units = C
would publish the temp value to a mqtt topic and input name of “abcd1234” whilst your script could also read the emonhub.conf and use the “names =” line as the list of ids, or
eg2
[[30]]
nodename = sockettest
[[[rx]]]
names = val_A
units = C
ids = abcd1234
keep them seperate, where the name is used for mqtt topic and input name whilst the ids line is read by your script to read and position the values, emonhub would ignore the additional line and it could all be edited/maintained via the emoncms emonhub config page.
The config file is read using the ConfigObj Python module, which is already installed and functional if emonhub is working.
Something like
from configobj import ConfigObj
list_of_ids = ConfigObj(/path/to/emonhub.conf)['nodes']['30']['rx']['ids']
for id in list_of_ids:
# do something with id
might work (untested) I do not know for sure that 2 scripts can access the same file, but unless you add some code to pick up changes whilst running, your script will only read the file when started.
You have 2 options, add some code to your script to check the conf every n seconds and potentially experience clashes with emonhub and/or emoncms (might well be fine though) OR if changes are going to be infrequent you can restart your script some how to pick up the changes.
Thanks all for your thoughts they are really helpfull!
I choosed to placed an human readable INI file in /home/pi/data because that place is read-writeable.
Made an extra webpage in www/emoncms/ini/ini.php to edit the INI file by using this Simple PHP editor of text files
Sadly this page is not behind the emoncms login, any ideas how to do this?
But now we can edit the ini file through any webbrowser.
The python code is reading the ini file and saving new sensor hardware addresses to that ini file.
Replacing a sensor only requires small editting of the ini file.
So we’re at the point that we also like to log remotely to emoncms.org.
On site are 5 rpi’s sending data to the socketinterfacer of one emonpi.
I configured 5 extra nodes in emonhub.conf including sensor values names for the values.
[[31]]
nodename = Biomeiler1
[[[rx]]]
names = tf1_c,tf1_h,tf2_c,tf2_h,t1,t2,t3,t4,t5,t6,t7,t8,t9,t10,t11,t12,t13,t14,t15,t16,pulsecount1,pulsecount2,F1,F2,P1,P2
scales = 1
units = C
On the local emonCMS we’re getting the sensor values nicely organized under node “biomeiler1” and labelled with the names.
But on emonCMS.org all the sensor values are listed under “node 31” with key names “1”,“2”,“3”,“4,” instead of the values names out of emonhub (tf1_c,tf1_h,tf2_c,tf2_h,t1,t2,t3,t4,t5 etc.)
In the emonhub log i also see that the json data array is exported without the values names but on emoncms.org i cannot find a place to config the value names as i can do in emonhub.conf.
How should i config emoncms.org or emonhub in a way that the inputs list between the local emoncms and emoncms.org look the same?
This is an age old issue in emoncms where 2 different methods (indexed csv and name:value pairs) are used side by side and the same db field is used for either the index or the name. Therefore if you were to set that field on emoncms.org to “pulsecount1” (for example) it would no longer be found as input “21” and a new input would be created if you posted via the bulk upload (as emonhub does). These 2 methods are therefore not interchangeable. For now all you can do is add the name to the description field on emoncms.org.
Take a look at the Development: Indexed Inputs thread. There is a possible solution in the pipeline but it will not be that soon, first we need to test as a feature branch, then eventually bring those changes into the main emoncms core and then emoncms.org runs behind the main repo and doesn’t always get all new features,
Thnx, I understand the difference between the indexed csv and name:value pairs method. Have seen that it creates new inputs after changing the valuename.
-So emonhub always uses the “bulkupload” for sending data to emonCMS.org?
-And the bulkupload always uses the indexed CSV format?
So there is not a chance that we can change the emonhub export format to json (name:value pairs) method.
and how about using nodejs for that? (with respect for the “buffer” functionallity of emonhub)
Currently yes
Well since emoncms offers 2 ways of doing the same job, it should be down to emoncms to do both methods at the same time, changing emonhub doesn’t “fix” the problem, it just makes emonhub accommodate the problem. Plus the intention behind emonhub using the bulk upload and the emoncms bulk upload api in general was to keep the payload as small and fast as possible.
The intention is that emonhub will continue to use the csv bulk upload and in addition to that it will also upload the input names on any changes so you get the best of both worlds.
There are no plans to use anything but Python for emonhub currently.
The main problem with using key:value pairs compared to csv in emonhub is the size of the buffer will increase dramatically for the same amount of data due to the replication of key names in every payload. On my installs I set huge buffer sizes as sometimes clients (eg schools) can take a network down for weeks (during hols) for maintenance etc, I rarely lose data even after a month outage it catches up and for more mainstream emonPi/SD users, there is a lot going on in that image and for emonhub to demand so much ram for buffering might impact performance.