Data not sending after network loss

This morning, before I headed out for the day, my wifi went a bit squiffy and the emonPi dropped connection for a few minutes, 3 times. I have read a lot about this and I was under the impression that it would re-send to emonCMS after the connection was restored. However, that doesn’t seem to be happening. See the image below where there’s missing data.

Is there anything I should check for why this is? Or is it intended behaviour?

Sadly my emonhub.log is now rolled - I hadn’t realised that it keeps only 2 hours of this log. I wonder if that should be increased?

Can you tell us more about your setup?

You mention emonPi, if you are posting locally to a self-hosted emoncms (same device) it should not be dependent on a network connection. If posting to emoncms.org then sadly no, the emonpi variant of emonhub doesn’t buffer data it just fires the data off every 30secs (can be changed in the conf) and deletes without confirming the data was received or that the network is even up.

Original emonhub buffers, and only deletes from the buffer once emoncms confirms a successful post.

Your emonhub.log is held in ram on a read only file system, if you have rebooted it is lost, however if it has been “logrotated” you may find it as emonhub.log.1 or something similar. What do you get with ls -la /var/log/emonhub ?

EDIT - just read your other post about sending to 2 emoncms’s so I see you are using a emonPi, so yes, if sending to emoncms.org, this is to be expected (although probably not intended) but it should not be the case for self-hosted.

Interesting. Yeh, that’s right that I’m sending to 2 external emoncms instances (emoncms.org and my own hosted on my VPS). And I also have it posting to the emoncms on the emonPi itself.

It’s a shame there’s no external sync, but not a big problem really. It’d be good to get that one day for sure.

Thanks for the clarification!

I found this frustrating as I don’t use emonPi to log/visualise logged data and instead wanted to collect on an external node.

Ended up customising emonHub and phpmqtt_input.php to use the current time a reading is logged and configured MQTT to use durable queues.

Let me know if you want further details to implement yourself

Oh sweet! Yes, could you give me the details please? That would be very useful.

I too would like to see how you have done it. That is very much the point of emonHub, to collect and timestamp the data, buffer if necessary and send it where you want, the way you want, when you want.

Most of that functionality had been removed from the original and experimental versions for the “emonPi variant” and I would like to see it return.

The approach I took was to simply concatenate a timestamp to the reading, separated by a , for all messages transported by MQTT. This will obviously impact any other emoncms destinations that you’re emonpi provides data to over MQTT.

Here are the diff outputs that identify the changes I made. I think i’ve captured everything related to this but let me know if you run into any issues and i’ll dig a little deeper to find anything I may have missed:

First off, are the changes to EmonHub that receive the readings and place on MQTT queue:

[email protected]:/var/www/emoncms $ cd /usr/share/emonhub
[email protected]:/usr/share/emonhub $ git diff
diff --git a/src/interfacers/EmonHubMqttInterfacer.py b/src/interfacers/EmonHubMqttInterfacer.py
index d25958c..35ed085 100644
--- a/src/interfacers/EmonHubMqttInterfacer.py
+++ b/src/interfacers/EmonHubMqttInterfacer.py
@@ -120,7 +120,9 @@ class EmonHubMqttInterfacer(EmonHubInterfacer):
                         varstr = str(cargo.names[varid-1])
                     # Construct topic
                     topic = self._settings["nodevar_format_basetopic"]+nodestr+"/"+varstr
-                    payload = str(value)
+                    # MTC:modified to prefix value with current time - phpmqtt_input.php modified accordingly
+                    now = int(time.time())
+                    payload = str(now)+":"+str(value)

                     self._log.info("Publishing: "+topic+" "+payload)
                     result =self._mqttc.publish(topic, payload=payload, qos=2, retain=False)

Next are the corresponding changes to phpmqtt_input.php that takes messages of the MQTT queue and persists to emoncms. Note the setting of a client identifier for the Mosqiuitto client connection that is required in order for messages to be reliably delivered and should be different for the mosquitto client in phpmqtt_input on each emoncms destination. I also reduced the timeout when i was trying to troubleshoot issues, but not sure if this is strictly necessary. The same is probably true for un-assigning values/time variables. Think I was suspected a memory leak somewhere, but left as is since it’s stable but could probably be improved/tidied up:

[email protected]:/var/www/emoncms/scripts $ git diff
diff --git a/scripts/phpmqtt_input.php b/scripts/phpmqtt_input.php
index b5ec231..a953c3c 100644
--- a/scripts/phpmqtt_input.php
+++ b/scripts/phpmqtt_input.php
@@ -1,5 +1,5 @@
 <?php
-
+//gc_enable();
     // TBD: support user target in message schema
     $mqttsettings = array(
         'userid' => 1
@@ -86,7 +86,7 @@
     require_once "Modules/process/process_model.php";
     $process = new Process($mysqli,$input,$feed,$user->get_timezone($mqttsettings['userid']));

-    $mqtt_client = new Mosquitto\Client();
+    $mqtt_client = new Mosquitto\Client('emonpi_phpmqtt', false);

     $connected = false;
     $last_retry = 0;
@@ -108,7 +108,7 @@
             $last_retry = time();
             try {
                 $mqtt_client->setCredentials($mqtt_server['user'],$mqtt_server['password']);
-                $mqtt_client->connect($mqtt_server['host'], $mqtt_server['port'], 5);
+                $mqtt_client->connect($mqtt_server['host'], $mqtt_server['port'], 60);
                 $topic = $mqtt_server['basetopic']."/#";
                 echo "Subscribing to: ".$topic."\n";
                 $log->warn("Subscribing to: ".$topic);
@@ -177,7 +177,20 @@

                 if (isset($route[2]))
                 {
+                    // MTC:modified /home/pi/emonhub/src/interfacers/EmonHubMqttInterfacer.py to pass value prefixed with time - time:value
+                    $values = explode(":",$value);
+                    $time = $values[0];
+                    $value = $values[1];
+
                     $inputs[] = array("userid"=>$userid, "time"=>$time, "nodeid"=>$nodeid, "name"=>$route[2], "value"=>$value);
+
+                    $values = null;
+                    $time = null;
+                    $value = null;
+
+                    unset($values);
+                    unset($time);
+                    unset($value);
                 }
                 else
                 {

In terms of transporting data to remote emoncms instances, i chose to leave that to MQTT by installing MQTT on the destination instance, configuring emoncms to use the local instance and configuring MQTT to connect to the emonpi and subscribe to messages:

connection emonpi
address 192.168.0.23:1883
cleansession false
topic # out 2 emon/ emon/

I’m pretty sure i made changes to MQTT on the emonPi, but they aren’t versioned so here’s a dump of my current config so you can compare to what you have. You’ll almost certainly need the persistence configuration elements e.g. max_queued_messages, persistent_client_expiration, upgrade_outgoing_qos, persistence, persistence_location, etc.:

[email protected]:/etc/mosquitto $ cat mosquitto.conf
# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example

pid_file /var/run/mosquitto.pid

autosave_interval 360
max_queued_messages 1000000
persistent_client_expiration 31d
max_inflight_messages 10
allow_duplicate_messages false
upgrade_outgoing_qos true

persistence true
persistence_location /home/pi/data/mosquitto/

max_connections 20
connection_messages false
log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d

allow_anonymous true
password_file /etc/mosquitto/passwd

# Bridge to raspi
connection raspi
start_type automatic
address 192.168.0.22:1883
cleansession false
topic # in 2 emon/currentcost/ emon/currentcost/
keepalive_interval 60

Note that I also have a bridge configured here to another MQTT instance on a raspberry pi that I interface with a currentcost monitor using python and dump messages to MQTT so that data finds it’s way into emoncms too. Here are MQTT relevant snippets from that script if of interest:

...
use Net::MQTT::Simple;
...
my $now;
my $mqtt1       = Net::MQTT::Simple->new("127.0.0.1");
...
        # log to mqtt
        $now = time();
        $now -= 3;

        $mqtt1->publish("emon/currentcost/watts", "$now:$watts");
        $mqtt1->publish("emon/currentcost/temp", "$now:$temp");
...

This solution has proved to be pretty reliable for me, and can withstand network outages at either source or destination, or destination nodes being shutdown. Data is buffered by MQTT and delivered when destinations become available.

Let me know if anythings unclear, and i’ll try and clarify further

Thank you for sharing all that Toby.

My interests were an insight into how you implemented the buffering and it’s characteristics, I won’t actually be installing it as I do not use an emonPi or emonSD image, I use original emonHub as it still has all those core elements.

The “timestamp prefix” is exactly how the data is/was passed around emonHub, I have just gone looking for the code in the “emonPi variant” to reference and found that too has been stripped out, so data coming into the “emonPi variant” of emonhub is no longer timestamped at source for any data and now relies on picking up a time stamp somewhere en-route (like your solution) or just be timestamped on arrival at emoncms, whenever that maybe,

Your solution is probably the best way of reintroducing those features without rewriting emonhub again, and is a great insight into MQTT persistence could also prove really useful for smaller wifi/mqtt devices eg emonESP that cannot do their own buffering.

I assume your solution is dependent on a MQTT broker being on the local machine, if publishing to a remote broker the issue would still exist (unless using a bridge).

Does the persisting/buffering of MQTT work for just emoncms topics or will it also affect, for example openHAB, in that once a network connection comes back up you could “theoretically” find your house lights flashing on and off like a disco with the inrush of backdated data unless some form of counter measure was implemented there?

Hi Paul,

The approach I took was to leverage the reliable messaging capabilities provided by MQTT rather than writing anything fancy to do this myself!

Time stamping readings as they are received by EmonHub overcomes the data loss that would be encountered had they simply been buffered by MQTT then batch processed by emoncms using the timestamp when processed.

I don’t think (although have not checked) that this solution is specific to the emonPi variant, however MQTT is a pre-requisite for both data capture and logging nodes whether these are in the same location as in emonPi, or on different machines.

As I recollect, either the MQTT publisher or subscriber typically specifies how traffic should be handled by specifying the level of QoS to be applied to messages, but I think I had issues somewhere and opted to force MQTT to reliably deliver all messages with the upgrade_outgoing_qos true config setting as this wasn’t an issue for me. Suspect with more time this could be refined to work properly.

The other versions simply wouldn’t need it, they all timestamp data on receipt when the data doesn’t already have a timestamp. That allows pre-timestamped data from other sources and also uploading of historic data.

They also all buffer data on a per target basis, if any/all targets were unreachable, each would get it’s full set of data in order and independently, when it became available again.

Your solution certainly does the job though.