EmonHub Development

No problems at all, It feels like I’ve been dragging my feet a bit. although I have been spending some time on emonhub over the last week or 2.

I have been looking closer at emonhub and trying to “get into it” again, but that hasn’t been easy, I seemed to have lost a few repo’s and branches. Stuff I have explored, partially fixed or already made progress with previously doesn’t seem to be to hand, not sure whether to spend time looking for it or just start over.

The list is pretty complete, there are other things that we haven’t yet discussed (on my hit list) and there are some things like the threaded exception handling that need looking at, at the very least I would like to see the @decorator replaced with a better implementation, but I’m not convinced it’s working as expected. There are posts that show the thread is dead issue after the fix was released and there are logs that show frequent restarting of the threads without any tracebacks.

Overall the error handling needs to improve, I can improve the common code but the read and write functions will be the responsibility of each interfacer, I would like to try and make the core as robust as possible and reduce the checking needed in each interfacer if possible.

Use of textual names throughout emonhub is something I would like to see implemented sooner rather than later.

I’m not sure I understand why scales and datapoints have been removed from the cargo object whilst names and node name have been added.

We also need to look at creating a systemd service unit.

and the way serial commands are sent from the main thread when changing Jee Settings could cause a crash if 2 threads are accessing the same serial port at the same time.

The way lists and single word settings are handled in the emon-pi variant need addressing to, it is not necessary to use a trailing comma if implemented correctly. I have also found a better way of handing boolean settings.

Hopefully the pace will pick up a bit as I get more familiar with the code again.

Can you also look at the timestamp processing?

ref Best way to guarantee times when using BULK load with PHPTIMESERIES? - #2 by pb66

Personally I would like to see the legacy mode updated to be “absolute” so that omitting a “mode” means the supplied timestamps are used without adjustment. The only alternative would seem to be adding “&time=0” to the end of each request (minor, but annoyingly pointless if we are trying to make the bulk upload as minimal as possible). The current “&sentat=” is open to potential errors.

perhaps we could change these lines

        // Legacy mode: input/bulk.json?data=[[0,16,1137],[2,17,1437,3164],[4,19,1412,3077]]
        else {
            $time_ref = time() - (int) $data[$len-1][0];
        }

to something like

        // Legacy mode: input/bulk.json?data=[[0,16,1137],[2,17,1437,3164],[4,19,1412,3077]]
        elseif ((int) $data[$len-1][0] <1000000000);
        {
            $time_ref = time() - (int) $data[$len-1][0];
        }
        // New default "absolute" timestamp mode: input/bulk.json?data=[[1519749672,16,1137],[1519749682,16,1437,3164],[1519749692,16,1412,3077]]
        else
        {
             $time_ref = 0;
        }

by checking if the timestamp of the last packet is in the range of a complete unix timestamp or just an offset, we could set the timeref to 0 if absolute timestamps are used.

Thanks Paul, Great, I’ve added the additional items you have highlighted to the list including the timestamp processing.

Note this thread about emonLCD and mqtt.

We will also need to think about a “Nodes legacy mode” mqtt by frame in the QoS1 mqtt interfacer.

Also we should create a logrotate entry to put in /etc/logrotate.d/emonhub so that emonhub.log files are not rotated out so frequently. We could concider reducing the size of the files from 5mb to 3mb if a potential 10mb is too much, but we should let emonhub manage it’s own log files and only use logrotate to compress emonhub.log.1.

Revise JeeLib confirmation messages ref “confirmed sent packet size: → ack” in Inputs Nodes Value only rssi Error after update of Emoncms version low-write 9.8.28 - #16 by pb66.

Also need to re-implement or fix the ability for the emonhub.log to display the emonPi/rfm69pi firmware revision again.

@TrystanLea - I would like to invite @beaylott to this discussion, do you have any issues with that?

sure that would be great

Thanks Trystan, in hind sight I hope this is the thread all the right answers are in, we do have several :slight_smile:

Ok… looks like I stumbled on something you were already quite exercised about! Our use cases are actually a lot like some of the ones you ( @pb66 ) describe where we are running emonhub on its own and posting to a remote emoncms instance over WAN. I like the idea that emonhub will work in resource constrained environments where neccesarry.

The first thing I might offer is that there are memory efficient ways of buffering the keys in addition to the values. My first thought was you could beef up (or create?) a Buffer class so that you can transparently store Cargo objects in it but these are actually stored in the Buffer with the relatively static names/scales/datatypes etc. and values separated so the memory use is almost the same as currently. This could also be made to deal with where the node changes. There are various data structures which would be suitable for this including some built-in to Python already. You could also embellish this class with various methods as you discuss above to do other things.

Obviously linking the name of the variable to its value would be desirable in some circumstances (using inputnames or otherwise). This is what I was after in my forum post but I can also see ways around this (like doing a input post the first time …although this needs the names to be passed in the Buffer as above). I could see how this could be made backwards compatible as well so that the current bulk upload without input names could be used.

Secondly, I don’t personally have any strong opinion on this interfacer/reporter distinction. To me there does seem to be a distinction as if you look at the code of the ‘other’ interfacers it anticipates (in its style and structure) the use of the HTTP and/or MQTT interfacer… so it would seem these are separate things. I haven’t put a lot of thought into that though.

I have also put quite a lot of thought into the multi user MQTT → emoncms business as this would be quite a desirable addition for us. I think you potentially need to separate the input requirement (so just a MQTT subscriber for displaying the input values in the tab) from the need to store values in feeds. The former can operate much as the single user version does now. The latter needs to be highly concurrent and go straight to the Redis or SQL database - I have called it ‘connector’ before in conversation with Trystan, but its an asynchronous server component which would take stuff off the broker and push it to the databases directly. I looked at writing something in Python or Go previously. You could also move authentication for MQTT to the broker (or even to a gateway proxy) as many brokers will be able to re-use the current authentication store. And you can also make use of the ACLs.

Hi Ben

Busy few days so sorry there’s been no response from Trystan or I.

I hear what your saying about storing the cargo object and Trystan was wanting a similar thing. This has been discussed and there seem to be little or no benifit to it, whilst the csv route is already there, works fine, allows humans to read/debug the content and it could have other uses in the near future eg a separate “traffic log” rather than trying to find the packets in the error log we could keep a track of all payloads that pass through emonhub for debugging or for “replaying time” to rebuild a lost/corrupt emoncms instance etc.

Buffering the cargo objects could also complicate the names thing as a namechange could easily be placed in the buffered csv without any special handling.

The use of both input names and index is something that Trystan is looking at in emoncms, this has been an obstacle fro many years, dividing users into either index or named input groups as the 2 methods cannot interact so making emoncms handled both methods on the same inputs is a huge step forward and makes things much easier for so many applications as it removes the hard choice.

That’s good to hear as although I’m trying to remain open on this, the more explanations I hear why it is un-necessary, the more essential it becomes in my mind. The distinction you suggest doesn’t really exist as there is good reason to have both http interfacers and reporter(-like interfacer)s and the same for MQTT, the distinction is most prominently the buffer as Trystan says, but yes you could have buffers on both as he has proved. The actual distinction is on the type of job it does. It either reports data in a secure, safe and confirmed way with no loss of data (hence the buffer) or it has a (potentially) 2 way interface where data is just read or written in realtime, status-like, send’n’forget with no (buffer or) confirmation before deletion etc.

Your comments about the MQTT in emoncms might be better suited to a discussion about the emoncms mqtt implementation. I’m unsure how that might work as there is no place for input processing if MQTT is depositing data directly into a database, and if it was to do that, then perhaps emonhub should deposit the data directly and not bother with publishing and subscribing the data to MQTT at all (between emonhub and emoncms).

Another point to consider with these reporter-like interfacers is the response from emoncms. Currently emonhub looks for a response equal to “ok”.

This can be upset by the $display_errors = true; setting as some non-fatal errors can sneak through with a successful request resulting in a response along the lines of “some horrendous error detectedok” where there is an “ok” appended to the end, but the whole response doesn’t equal “ok” so emonhub continues to try and repost the same data.

A (unacceptable) quick and dirty fix could have been to make emonhub check to see if the response contains “ok” but a response like “some error - failed to post any data - look at the logs for info” could result in data being accidentally deleted as it contains “ok”.

This hasn’t been a big issue in the past, but if we are making big changes, now would be the time to review this.

Also…

On occasions a “NaN” will slip through emonhub from a sketch and emoncms will refuse that request as it cannot handle the NaN, if you then do not notice the issue for several hours/days, the only way to get around it is to restart emonhub and loosing all the buffered data. It would be much better if emoncms could consume that data and just ignore the “NaN” otherwise we need to add some filtering in emonhub that maybe tricky or block data coming in.

emoncms already handles “null” we could easily just add another case for “NaN”.

I hear what your saying about storing the cargo object and Trystan was wanting a similar thing. This has been discussed and there seem to be little or no benifit to it, whilst the csv route is already there, works fine, allows humans to read/debug the content and it could have other uses in the near future eg a separate “traffic log” rather than trying to find the packets in the error log we could keep a track of all payloads that pass through emonhub for debugging or for “replaying time” to rebuild a lost/corrupt emoncms instance etc.

Perhaps I wasn’t clear … I actually was proposing that the data continues to be stored in a CSV string, it just is wrapped in an object with the same interface but which also intelligently stores the names.

EDIT: I posted this not fully differentiating in my mind emonhub and emonbase. I understand emonhub software runs on emonbase.

Can I make another suggestion regarding RF.
As much to make a record of fact there’s been issues with 433Mhz RF in noisy environments. (I’m also predicting noise issues for a setup I’m soon building).

  1. To design for a easy transition between Wire and RF for each connected device, using 4-wire single-core telephone cable.
  2. In the case of RF to have sent 3 sets of data each time for the 3 most recent packets, 30 seconds worth for 10s updates: this could be an optional mode perhaps, although if this route was taken I’d see advantage in making it default unless there’s a battery consumption issue.
    -A: to have memory on the sending side so the 3 most recent packets can be stored and sent each interval, applies to node sensors and the emonTx.
    -B: to have emonhub parse the received information correctly relative to timestamps, and bulk upload if necessary.
    -C: emonhub made sensitive to version information from transmitting devices. Such that received information can be parsed correctly from legacy and future devices. Also, future devices could communicate two-way with emonhub for settings and modes, in which case version recognition could be useful again.

Does that make sense? I think each step is quite simple to implement.

We also need to be sure that scaling cannot trip up on infinity, nulls and NaN’s etc, or any non-numeric values that might sneak in.

see Discarding RX frame 'unreliable content'? - After Upgrade to latest version 9.8.28 - #22 by pb66

Some interesting points regards locking threads to avoid attempting to access the same device from multiple threads. (see Missing threading.Lock() functionality in emonhub?)