Community
OpenEnergyMonitor

Community

Interfacer Questions

(John Banks) #1

I have two instances of emonTx/RPi running emonSD Oct 2018 ver 9.9.8 – Node 12 & Node 13.

In each case, the emonTx is serial direct connected (3 short leads on the GPIO pins of the RPi).

Each has the rootfs & /home/pi/data in partitions on a USB HDD.

Each uses the EmonHubEmoncmsHTTPInterfacer to send data to watchman.

watchman is an RPi on the local network also running emonSD Oct 2018 ver 9.9.8. It has a USB HDD connected but no emonTx.

Nodes 12 & 13 are plagued with crashes. They both crashed yesterday and Node 13 crashed again today. (I fsck the USB HDD’s using Raspbian Lite to recover – which resets the free block count).

I also have Node 11 & Node 14 running – identical in every way (ver 9.9.8 and USB HDD) except they do not use the EmonHubEmoncmsHTTPInterfacer as they are not sending data to watchman.

Nodes 11 & 14 and watchman have been crash free for weeks/months.

EmonHubMqttInterfacer …

This is included in the emonhub.conf file for all 4 nodes and watchman.

Is it needed in my simple installations? – not using Node-Red, etc

Also what MQTT services or solely related services could be stopped?

EmonHubMqttInterfacer is the only interfacer in watchman’s emonhub.conf file. Does it even need that Interfacer?

watchman does not have the file /var/log/emonhub/emonhub.log (the Download Log button does not work) but strangely the web interface page for EmonHub does show the following …

-- Logs begin at Thu 2016-11-03 17:16:45 GMT, end at Sun 2019-03-17 20:07:28 GMT. --
Mar 17 00:51:17 watchman systemd[1]: Started emonHub service description.
Mar 17 00:51:24 watchman emonhub.py[360]: 2019-03-17 00:51:24,483 INFO     MainThread EmonHub emonHub emon-pi variant v2.1.2
Mar 17 00:51:24 watchman emonhub.py[360]: 2019-03-17 00:51:24,484 INFO     MainThread Opening hub...
Mar 17 00:51:24 watchman emonhub.py[360]: 2019-03-17 00:51:24,485 INFO     MainThread Logging level set to DEBUG
Mar 17 00:51:24 watchman emonhub.py[360]: 2019-03-17 00:51:24,485 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
Mar 17 00:51:24 watchman emonhub.py[360]: 2019-03-17 00:51:24,488 DEBUG    MainThread Setting MQTT subchannels: ['ToEmonCMS']
Mar 17 00:51:24 watchman emonhub.py[360]: 2019-03-17 00:51:24,489 DEBUG    MainThread Setting MQTT pubchannels: ['ToRFM12']
Mar 17 00:51:24 watchman emonhub.py[360]: 2019-03-17 00:51:24,489 INFO     MainThread Setting MQTT nodevar_format_enable: 1
Mar 17 00:51:24 watchman emonhub.py[360]: 2019-03-17 00:51:24,490 INFO     MainThread Setting MQTT node_format_enable: 1
Mar 17 00:51:24 watchman emonhub.py[360]: 2019-03-17 00:51:24,490 INFO     MainThread Setting MQTT nodevar_format_basetopic: emon/
Mar 17 00:51:40 watchman systemd[1]: Stopping emonHub service description...
Mar 17 00:51:40 watchman systemd[1]: Stopped emonHub service description.
Mar 17 00:51:40 watchman systemd[1]: Started emonHub service description.
Mar 17 00:51:42 watchman emonhub.py[1104]: 2019-03-17 00:51:42,919 INFO     MainThread EmonHub emonHub emon-pi variant v2.1.2
Mar 17 00:51:42 watchman emonhub.py[1104]: 2019-03-17 00:51:42,920 INFO     MainThread Opening hub...
Mar 17 00:51:42 watchman emonhub.py[1104]: 2019-03-17 00:51:42,921 INFO     MainThread Logging level set to DEBUG
Mar 17 00:51:42 watchman emonhub.py[1104]: 2019-03-17 00:51:42,921 INFO     MainThread Creating EmonHubMqttInterfacer 'MQTT'
Mar 17 00:51:42 watchman emonhub.py[1104]: 2019-03-17 00:51:42,924 DEBUG    MainThread Setting MQTT subchannels: ['ToEmonCMS']
Mar 17 00:51:42 watchman emonhub.py[1104]: 2019-03-17 00:51:42,925 DEBUG    MainThread Setting MQTT pubchannels: ['ToRFM12']
Mar 17 00:51:42 watchman emonhub.py[1104]: 2019-03-17 00:51:42,925 INFO     MainThread Setting MQTT nodevar_format_enable: 1
Mar 17 00:51:42 watchman emonhub.py[1104]: 2019-03-17 00:51:42,925 INFO     MainThread Setting MQTT node_format_enable: 1
Mar 17 00:51:42 watchman emonhub.py[1104]: 2019-03-17 00:51:42,926 INFO     MainThread Setting MQTT nodevar_format_basetopic: emon/

And that is all - it does not move forward with time passing which is expected.

EmonHubEmoncmsHTTPInterfacer …

This contains the line … sendinterval = 10

I think I got that from the Guide. However the emonhub.log (on the web pages) for nodes 12 & 13 show that data is only sent to watchman every 30 secs and watchman acknowledges receipt.

Whist, on the other hand, I’ve observed that watchman only updates its phptimeseries every 60 secs. Is this expected behaviour?

As it may help understanding, here is the syslog for a typical cycle of Node 13 sending to watchman …

13 syslog cycle.zip (2.0 KB)

And here are the log lines just containing fatal, error or warn from the watchman daemon.log starting from the last time I powered watchman up …

Mar 17 00:51:31 watchman supervisord[548]: 2019-03-17 00:51:31,117 WARN For [program:tuxtunnel], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
Mar 17 00:51:40 watchman mosquitto[1082]: Stopping network daemon:: mosquittostart-stop-daemon: warning: failed to kill 425: No such process
Mar 17 00:51:41 watchman emonPiLCD[1109]: Stopping system emonPiLCD daemon:start-stop-daemon: warning: failed to kill 421: No such process
Mar 17 00:51:45 watchman supervisord[1204]: 2019-03-17 00:51:45,931 WARN For [program:tuxtunnel], redirect_stderr=true but stderr_logfile has also been set to a filename, the filename has been ignored
Mar 17 00:51:19 watchman raspi-config[353]: Checking if shift key is held down:Error opening '/dev/input/event*': No such file or directory
Mar 17 00:51:28 watchman redis-server[564]: *** FATAL CONFIG FILE ERROR ***
Mar 17 00:51:35 watchman emoncms_mqtt[880]: Can't connect to database, please verify credentials/configuration in settings.php<br />Error message: <b>Connection refused</b>
Mar 17 00:51:28 watchman redis-server[564]: *** FATAL CONFIG FILE ERROR ***

Comments or suggestions?

PS - Until a few days ago I had /var/log mounted on the USB HDD /home/pi/data partition. There were several crashes that self re-booted and from the persistent logs I found that there were many NUL’s across the log line at crash time. On 3 occasions immediately before the crash, syslog records MQTT publishing. On the other occasion, it was Serial Tx being sent. Might this be relevant?

0 Likes
(John Banks) #2

UPDATE …

Node 13 has crashed again today

I tried removing the EmonHubMqttInterfacer and the feeds stopped updating - so I’ve answered my own question - it is needed.

Notice from a Forum message this evening that an update is available … I’ll try that and I’ve turned persistent logging back on again.

0 Likes
(Paul) #3

Where did you get that setting? I thought it should be just “interval” and went to the emonhub repo to check. It’s a bit bizzare! The only places that “sendinterval” appears is in a readme and 3 config files, it doesn’t get used anywhere in the code itself, ie it’s apparently redundant (see https://github.com/openenergymonitor/emonhub/search?q=sendinterval&unscoped_q=sendinterval)

You could try changing that line to use just “interval = 10” and see if that works.

You are using the emonSD image, albeit on a hdd, it is configured to reduce wear to the sdcard. So yes emoncms is configured to use the feedwritter that buffers data and only puts it to disk every 60s.

There are settings in settings.php to turn this off, but I do not know for sure it’s “that simple” as there is also a service to stop and disable and I do not know what happens “mid-buffering period” ie if you’ll lose data.

There have been some changes to the way emonhub logs so I cannot answer your questions about that with any confidence.

Based on the discussion we have had in your other thread around rc.local and these issues, I really think you would be better off building your own image to use on hdd. I would put money on most of your issues being caused or at least aggravated by the low-write optimisations that you simply don’t need.

I cannot think of any obvious link between the use of the http interfacer and the crashes, but I am very disconnected with how things are done in the emonpi variant of emonhub so cannot say it’s not possibly linked.

0 Likes
(John Banks) #4

Paul …

Many thx

Actually I’m quite happy for watchman to update only every 60 secs. Eventually all 4 Nodes will be broadcasting to watchman – so quite a lot of data.

I cannot remember from where I got my original template for the HTTPInterfacer.

The GUIDE (Section 2) has a picture showing …

sendinterval=30

Whereas in the Advanced Notes, at the end of Section 2 c), the example does not have an ‘interval’ line.

So thx for yr pointer - I’m going to delete my ‘interval’ line.

As an editorial comment, I’ve written a script whereby I get a daily email from watchman reporting the delay time in writing updates to its phptimeseries files – more than 60 secs means a Node has crashed. This helps me get on any ‘crash cases’ without too much delay.

But it may come down to me following yr suggestion that I build my own image – is there a guide for this?

PS: May not be relevant but I’m not using emonPi’s but emonTxV3’s serial direct connected to RPI3’s which are NOT fitted with RFM69Pi transceivers.

0 Likes
(John Banks) #5

@pb66

Update … my Node 12 crashed 3 times yesterday.

After each crash, I examined the USB HDD under Raspbian Lite and each time fsck did a correction of the free block counts on sda1 rootfs and on sda2 /home/pi/data.

As I had set /var/log not to mount as a tmpfs, it was possible to explore further.

Each crash was identified in daemon.log by a line containing many many NUL’s

Each crash occurred at almost the same point in the ‘processing cycle’ when either emonhub.py or feedwriter was running.

The last few daemon.log lines for each …

Prior Cycle CRASH 1.txt (11.5 KB)

Prior Cycle CRASH 2.txt (14.3 KB)

Prior Cycle CRASH 3.txt (14.0 KB)

Crash 3 was a bit different in that, by chance, I had an open puTTy session at the time. The session was still live apparently after the crash. But doing systemctl status apache2 returned Bus error and cat /var/log/daemon.log returned Input/output error.

I have 2 other identical instances running for months without a problem – identical hardware incl USB HDD and running emonSD image Oct 2018 at ver 9.9.8. The only difference is – they do not broadcast via the HTTPInterfacer to watchman - a local server on the network with identical hardware incl USB HDD and also running emonSD image Oct 2018 at ver 9.9.8.

Any comments?

My thoughts are to try – keeping rootfs on the SDHC with only /home/pi/data on the USB HDD but with /var/log also on the USB HDD if that is possible.

0 Likes
(Brian Orpin) #6

Hi John,

Correct me if I am wrong, but at the base level, you have 2x EmonTX that you wish to collect the data for. The 2X instances of Emoncms attatched to each are just passing the data on, you are not actually using the local Emoncms for any data analysis?

If so, all you need to install is the emonhub. It takes the data off the serial and sends it (either via HTTP or MQTT) to ‘watchman’ for collecting the data. The HDD and Emoncms is, I feel, a complication you really do not need.

I do something similar in that I have an OrangePi Zero (running DietPi) with an RFM board. Emonhub runs on that collecting the data off the RFM board, onto the serial interface, then sends it on (in my case via HTTP). No data is stored locally and it is rock solid.

Another alternative would be an EmonESP.

Is the log file mounted to tempfs on 12/13 by any chance?

That is because of the generic nature of the EmonSD (being addressed elsewhere). You only need emonhub to pull data off a serial interface and ‘watchman’ is not getting any data on that interface but emonhub is installed (and running).

[edit] @ian has recently used the EmonESP so if you wanted to go that way (and free up 2X Pis) he may be able to help.

0 Likes
(John Banks) #7

Brian …

Thx for yr inputs which has prompted me to think a bit harder.

The installation I’m monitoring is at my son’s place – 30kWp with panels and the Inverter in a field at the back of the house which is 100 metres away from the incoming grid meter.

I’ve progressively installed 4 ‘monitoring nodes’ – at the Inverter, at the Pool (7.5 kW air source heat pump), at the car charger and at the incoming grid meter.

These are identical for ease of maintenance – emonTxV3’s serial direct connected to RPI3’s which are NOT fitted with RFM69Pi transceivers and running the emonSD image Oct 2018 regularly updated. I’m happy with what it says on the emon tin and apart from using USB HDD’s, all pretty bog standard.

When the ESP unit first became available, I tried it but struggled and opted for site wide wi-fi by installing 2 external antennas – which has proved to be a pretty robust solution.

Things were slowly progressing until a couple of months ago. About the time that image version 9.9.5 started moving up to 9.9.8, I introduced a 5th identical node – watchman. Its primary purpose is to be the system/site reporter. For example, only watchman has generation data from the Inverter node and chargeable import and export data from the Grid Meter node.

As a result of one or other change or in combination, I’ve had a lot of crashes with the 2 nodes I initially set broadcasting to watchman. These are now back at my place in ‘intensive care’ – normally I work remotely using Dataplicity which is great.

Both are running a freshly burned emonSD image updated to latest ver 9.9.8. They and all other nodes are running the fixes I’ve found necessary (cmdline.txt and rc.local).

Node 13 has been running OK for a few days with persistent logging (change to fstab). Node 12 ran for a couple of days from the SDHC but within hours of being switched to USB HDD it crashed. I examined the USB HDD under Raspbian Lite and fsck did a correction of the inode & free block counts on sda1 rootfs and on sda2 /home/pi/data.

It was my by now familiar crash - identified in daemon.log by a line containing many many NUL’s. And the immediately prior log line showing emonhub.py running & MQTT publishing.

I scanned the other logs and found 2 warnings for redis-server …


1191:M 24 Mar 17:32:05.198 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

1191:M 24 Mar 17:32:05.198 # Server started, Redis version 3.2.6

1191:M 24 Mar 17:32:05.198 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

1191:M 24 Mar 17:32:05.213 * The server is now ready to accept connections on port 6379

I found and implemented a solution …

[https://www.techandme.se/performance-tips-for-redis-cache-server/]

Probably clutching at straws? – but Node 12 is up & running once again.

I have a spare set of hardware – I’ll try what you suggest – two questions …

Q1 – For the remote nodes - how do I uninstall emoncms or just stop the emoncms related services. I like the idea of switching off services as I could do that only when I ‘m ready to switch a node to broadcasting to watchman.

Q2 – How do I uninstall emonhub or just stop the emonhub related services for watchman.

Pls advise the link to yr Diet Pi blog

I realise that, with this alternative approach, I’ll lose some data security – feed data will only be on watchman. And the wi-fi network always being up will be key.

Many thx for yr inputs.

0 Likes
(Brian Orpin) #8

Ah yes, I remember now. Just one thing - why not monitor at the meter end (or have I asked that before and forgotten the answer…).

I’d be inclined to start with a fresh image (DietPi / Raspbian Lite), ditch the HDD and just install the emonhub https://github.com/openenergymonitor/emonhub. You need to decide if you are going to transfer the data by HTTP API or MQTT. Personally I am not a fan of MQTT in this circumstance so I’d use the HTTP API but YMMV.

You should be able to just stop and then mask the emonhub service (systemctl mask <service name>)

Link to the item on EmonCMS - https://tech.borpin.co.uk/2018/10/13/emoncms-on-dietpi/ This doesn’t cover the emonhub only setup though, just an install of EmonCMS on a DietPi base image.

0 Likes
(John Banks) #9

Brian …

Thank you very much for that input.

My test of DietPi (using yr blog) is going well until I came to installing emonhub …

I did the OEM git download but it would not run – no /home/pi/data directory.

DietPi have software package 99 called EMONHUB but the link to that is broken.

They also have a package EmonPi which sends data only to emoncms.org not to a local server.

Is it necessary for me to set up user pi and the /home/pi/data directories?

Your further suggestions would be most welcome.

Thx

John (Banks)

0 Likes
(Brian Orpin) #10

As I said that post was not aimed at EmonHub install. All the Emoncms stuff in that post should be ignored as you don’t want it (unless you want emoncms locally).

It is that long since I installed it - 2016 - I needed to check.

You just need to adjust the paths.

Was this the last command you used?

sudo ./install.systemd

If so just edit that file and change the path to /root/emonhub

You may need to disable the serial console via the dietpi-config menu system. Not sure if it enabled or disabled by default now.

0 Likes
(John Banks) #11

Brian …
Not totally straightforward …
I changed the GIT_PATH in install.systemd and commented out the lines checking for a directory /home/pi and got the following …

[email protected]:~/emonhub# ./install.systemd
Emonhub installation script for emonPi
cp: cannot create regular file '/lib/systemd/system/emonhub.service.d/': Not a d                                        irectory
mv: cannot move '/root/emonhub/conf/emonpi.default.emonhub.conf' to '/home/pi/da                                        ta/emonhub.conf': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/emonhub.service → /l                                        ib/systemd/system/emonhub.service.

I ran it a second time which shows that a couple of things worked …

[email protected]:~/emonhub#  ./install.systemd
Emonhub installation script for emonPi
mkdir: cannot create directory ‘/etc/systemd/system/emonhub.service.d/’: File exists
cp: cannot create regular file '/lib/systemd/system/emonhub.service.d/': Not a directory
mv: cannot move '/root/emonhub/conf/emonpi.default.emonhub.conf' to '/home/pi/data/emonhub.conf': No such file or directory
useradd: user 'emonhub' already exists

I may be way off beam here (very much a beginner) but would it be a good idea? to create user pi and/or the directory /home/pi with sub-directories - backup, data & usefulscripts
… particularly bearing in mind I want to try adding emoncms as well - just as an ‘experiment’.

Many thx again

0 Likes