EmonHub Development

emonHub
42

great, that sounds good.

I will have a think about the emoncms changes and the topic/payload formats you mention and the apikey question.

The fetch via mqtt should be possible but I think quite complex to implement, not sure will have to think about that as well.

yes agreed

yes i guess so… thats a complex action to add

Il try and summarise all of our points in a list so that we can refer back to it as we work on it. It might be good to know which one of us is going to work on each point or whether we both need to explore a point and come back to it once we’ve both had a chance to test ideas. It looks like a big job, how much time do we need to give ourselves to do the job properly? it looks like it will take much longer than a couple of weeks!! especially given other commitments

43

Not necessarily an immediate concern, but just something to bear in mind whilst structuring the more pressing stuff as something we may want down the line. We can use the http “fetch” api, I just assume that at some point someone will want a fully mqtt setup, it seems daft to go to so much effort to ensure emonhub is posting to emoncms by mqtt not http, and then need to use http for something else.

Maybe, but we maybe able to look at expanding the bulk api to help emonhub with this task, maybe change the api to accept text input values as names rather than rejecting all non-numerical data (eg [[ts,id,v1,v2,v3],[ts,id,“power1”,“power2”,“Power3”][ts,id,v1,v2,v3]]). this way emonhub can just buffer the name changes just as it would any values.

Yes I guessed it would be a significant amount of work, I really have no idea at this point how long it might take. But I do know it will be slow getting started due to needing to familiarize myself with the code again. I doubt much will progress that far at all in 2 weeks, but once we’re rolling it will get quicker, just as well really as I’m sure more work will become apparent as we progress. I already had a snagging list for the experimental version when you released the emon-pi version. I will need to find those notes too.

44

Hello Paul, apologies for the delay with this. Reading through the above again here is my attempt at summarising the key items raised for our ongoing reference.

  • Establish our approach: interfacers, reporters or reporter-like interfacers
  • Establish our approach: seperate interfacer files vs core interfacer file.
  • Review EmonHub internal message queue implementation
  • Consider process chain, standardise on correct approach for use of run, read, send, action, add, flush, process_post.
  • Consider interfacer naming
  • Buffer persistance: Option to save or load a buffer from disk?
  • Review rx & tx node definition in emonhub.conf used by emon-pi variant
  • Document and explain distinction between QoS1/QoS2 unbuffered/buffered interfacers/reporters.
  • Document further use of socket interfacer
  • Consider backwards compatibility when switching from reporters to interfacers or vice versa.
  • MQTT: What should the format of the generic MQTT Interfacer be?
  • MQTT: Specific format MQTT Interfacers can re-use the generic interfacer
  • MQTT: Keep a per-topic MQTT Interfacer QoS1 unbuffered
  • MQTT: how do we handle on_message case
  • MQTT: Implement a bulk mode, buffered QoS2 MQTT reporter-like interfacer
  • MQTT: Pass MQTT data through emonhub from ESP devices
  • EmonHub HTTP interfacer that can call the emoncms fetch api to retrieve data into emonhub
  • Emoncms changes: Implement the input/post and input/bulk formats in MQTT, perhaps extend to other api’s e.g fetch.
  • Emoncms changes: option to have indexed inputs, and seperate sending of inputnames.
  • Emoncms changes: support multiple emoncms accounts from the mqtt_input script.
  • Emoncms: review timestamp processing
  • Review threaded exception handling
  • Review error handling in core interfacers
  • systemd service unit
  • simultaneous tx of settings resulting in potential serial crash
  • Review list and single word settings in emon-pi variant
45

From this my next emonhub related step will be to investigate the indexed inputs option for emoncms and the possibility of sending an inputnames parameter.

You mentioned above a wish to find some time to familiarise yourself with the code. Let me know when you would like my input again.

46

No problems at all, It feels like I’ve been dragging my feet a bit. although I have been spending some time on emonhub over the last week or 2.

I have been looking closer at emonhub and trying to “get into it” again, but that hasn’t been easy, I seemed to have lost a few repo’s and branches. Stuff I have explored, partially fixed or already made progress with previously doesn’t seem to be to hand, not sure whether to spend time looking for it or just start over.

The list is pretty complete, there are other things that we haven’t yet discussed (on my hit list) and there are some things like the threaded exception handling that need looking at, at the very least I would like to see the @decorator replaced with a better implementation, but I’m not convinced it’s working as expected. There are posts that show the thread is dead issue after the fix was released and there are logs that show frequent restarting of the threads without any tracebacks.

Overall the error handling needs to improve, I can improve the common code but the read and write functions will be the responsibility of each interfacer, I would like to try and make the core as robust as possible and reduce the checking needed in each interfacer if possible.

Use of textual names throughout emonhub is something I would like to see implemented sooner rather than later.

I’m not sure I understand why scales and datapoints have been removed from the cargo object whilst names and node name have been added.

We also need to look at creating a systemd service unit.

and the way serial commands are sent from the main thread when changing Jee Settings could cause a crash if 2 threads are accessing the same serial port at the same time.

The way lists and single word settings are handled in the emon-pi variant need addressing to, it is not necessary to use a trailing comma if implemented correctly. I have also found a better way of handing boolean settings.

Hopefully the pace will pick up a bit as I get more familiar with the code again.

47

Can you also look at the timestamp processing?

ref Best way to guarantee times when using BULK load with PHPTIMESERIES? - #2 by pb66

Personally I would like to see the legacy mode updated to be “absolute” so that omitting a “mode” means the supplied timestamps are used without adjustment. The only alternative would seem to be adding “&time=0” to the end of each request (minor, but annoyingly pointless if we are trying to make the bulk upload as minimal as possible). The current “&sentat=” is open to potential errors.

perhaps we could change these lines

        // Legacy mode: input/bulk.json?data=[[0,16,1137],[2,17,1437,3164],[4,19,1412,3077]]
        else {
            $time_ref = time() - (int) $data[$len-1][0];
        }

to something like

        // Legacy mode: input/bulk.json?data=[[0,16,1137],[2,17,1437,3164],[4,19,1412,3077]]
        elseif ((int) $data[$len-1][0] <1000000000);
        {
            $time_ref = time() - (int) $data[$len-1][0];
        }
        // New default "absolute" timestamp mode: input/bulk.json?data=[[1519749672,16,1137],[1519749682,16,1437,3164],[1519749692,16,1412,3077]]
        else
        {
             $time_ref = 0;
        }

by checking if the timestamp of the last packet is in the range of a complete unix timestamp or just an offset, we could set the timeref to 0 if absolute timestamps are used.

48

Thanks Paul, Great, I’ve added the additional items you have highlighted to the list including the timestamp processing.

49

Note this thread about emonLCD and mqtt.

We will also need to think about a “Nodes legacy mode” mqtt by frame in the QoS1 mqtt interfacer.

50

Also we should create a logrotate entry to put in /etc/logrotate.d/emonhub so that emonhub.log files are not rotated out so frequently. We could concider reducing the size of the files from 5mb to 3mb if a potential 10mb is too much, but we should let emonhub manage it’s own log files and only use logrotate to compress emonhub.log.1.

51

Revise JeeLib confirmation messages ref “confirmed sent packet size: → ack” in Inputs Nodes Value only rssi Error after update of Emoncms version low-write 9.8.28 - #16 by pb66.

Also need to re-implement or fix the ability for the emonhub.log to display the emonPi/rfm69pi firmware revision again.

52

@TrystanLea - I would like to invite @beaylott to this discussion, do you have any issues with that?

53

sure that would be great

55

Thanks Trystan, in hind sight I hope this is the thread all the right answers are in, we do have several :slight_smile:

58

Ok… looks like I stumbled on something you were already quite exercised about! Our use cases are actually a lot like some of the ones you ( @pb66 ) describe where we are running emonhub on its own and posting to a remote emoncms instance over WAN. I like the idea that emonhub will work in resource constrained environments where neccesarry.

The first thing I might offer is that there are memory efficient ways of buffering the keys in addition to the values. My first thought was you could beef up (or create?) a Buffer class so that you can transparently store Cargo objects in it but these are actually stored in the Buffer with the relatively static names/scales/datatypes etc. and values separated so the memory use is almost the same as currently. This could also be made to deal with where the node changes. There are various data structures which would be suitable for this including some built-in to Python already. You could also embellish this class with various methods as you discuss above to do other things.

Obviously linking the name of the variable to its value would be desirable in some circumstances (using inputnames or otherwise). This is what I was after in my forum post but I can also see ways around this (like doing a input post the first time …although this needs the names to be passed in the Buffer as above). I could see how this could be made backwards compatible as well so that the current bulk upload without input names could be used.

Secondly, I don’t personally have any strong opinion on this interfacer/reporter distinction. To me there does seem to be a distinction as if you look at the code of the ‘other’ interfacers it anticipates (in its style and structure) the use of the HTTP and/or MQTT interfacer… so it would seem these are separate things. I haven’t put a lot of thought into that though.

59

I have also put quite a lot of thought into the multi user MQTT → emoncms business as this would be quite a desirable addition for us. I think you potentially need to separate the input requirement (so just a MQTT subscriber for displaying the input values in the tab) from the need to store values in feeds. The former can operate much as the single user version does now. The latter needs to be highly concurrent and go straight to the Redis or SQL database - I have called it ‘connector’ before in conversation with Trystan, but its an asynchronous server component which would take stuff off the broker and push it to the databases directly. I looked at writing something in Python or Go previously. You could also move authentication for MQTT to the broker (or even to a gateway proxy) as many brokers will be able to re-use the current authentication store. And you can also make use of the ACLs.

60

Hi Ben

Busy few days so sorry there’s been no response from Trystan or I.

I hear what your saying about storing the cargo object and Trystan was wanting a similar thing. This has been discussed and there seem to be little or no benifit to it, whilst the csv route is already there, works fine, allows humans to read/debug the content and it could have other uses in the near future eg a separate “traffic log” rather than trying to find the packets in the error log we could keep a track of all payloads that pass through emonhub for debugging or for “replaying time” to rebuild a lost/corrupt emoncms instance etc.

Buffering the cargo objects could also complicate the names thing as a namechange could easily be placed in the buffered csv without any special handling.

The use of both input names and index is something that Trystan is looking at in emoncms, this has been an obstacle fro many years, dividing users into either index or named input groups as the 2 methods cannot interact so making emoncms handled both methods on the same inputs is a huge step forward and makes things much easier for so many applications as it removes the hard choice.

That’s good to hear as although I’m trying to remain open on this, the more explanations I hear why it is un-necessary, the more essential it becomes in my mind. The distinction you suggest doesn’t really exist as there is good reason to have both http interfacers and reporter(-like interfacer)s and the same for MQTT, the distinction is most prominently the buffer as Trystan says, but yes you could have buffers on both as he has proved. The actual distinction is on the type of job it does. It either reports data in a secure, safe and confirmed way with no loss of data (hence the buffer) or it has a (potentially) 2 way interface where data is just read or written in realtime, status-like, send’n’forget with no (buffer or) confirmation before deletion etc.

61

Your comments about the MQTT in emoncms might be better suited to a discussion about the emoncms mqtt implementation. I’m unsure how that might work as there is no place for input processing if MQTT is depositing data directly into a database, and if it was to do that, then perhaps emonhub should deposit the data directly and not bother with publishing and subscribing the data to MQTT at all (between emonhub and emoncms).

63

Another point to consider with these reporter-like interfacers is the response from emoncms. Currently emonhub looks for a response equal to “ok”.

This can be upset by the $display_errors = true; setting as some non-fatal errors can sneak through with a successful request resulting in a response along the lines of “some horrendous error detectedok” where there is an “ok” appended to the end, but the whole response doesn’t equal “ok” so emonhub continues to try and repost the same data.

A (unacceptable) quick and dirty fix could have been to make emonhub check to see if the response contains “ok” but a response like “some error - failed to post any data - look at the logs for info” could result in data being accidentally deleted as it contains “ok”.

This hasn’t been a big issue in the past, but if we are making big changes, now would be the time to review this.

Also…

On occasions a “NaN” will slip through emonhub from a sketch and emoncms will refuse that request as it cannot handle the NaN, if you then do not notice the issue for several hours/days, the only way to get around it is to restart emonhub and loosing all the buffered data. It would be much better if emoncms could consume that data and just ignore the “NaN” otherwise we need to add some filtering in emonhub that maybe tricky or block data coming in.

emoncms already handles “null” we could easily just add another case for “NaN”.