EmonHub Development

  1. Yes,
  2. Great
  3. Yes.
  4. Agreed.
  5. We do need to give this considerably more thought, luckily we do not need to make a decision right now. A couple of further points to consider here. We can easily remove the code such as send_request, connect_serial and on_message etc etc to a emonhub_common.py file if we decide to retain both interfacers and reporters and want to share code, sharing elements of code needn’t be the reason to lump them all in together and also we need to decide about potential blocking from trying to post a huge buffered backlog, does a reporter-like interfacer even have a read function? If it doesn’t, is it really a proper interfacer?
    [edit] a Third point is that the buffering may well be expanded to use databases or write to disk so the differences between interfacers and reporter-like interfacers may even become greater. Perhaps the buffer code should be unique to reporters only.
  6. Much of this depends on the outcome of the above, something else to note here is that emon-pi users will undoutable update via the emonPi update script so it is possible to make regex changes to the emonhub.conf file at the same time as pulling in different code so changes in interfacer/reporter name will not require a manual change or user involvement/understanding, although any changes will need to be well documented as there will be a lot of posts and guides that refer to older settings, this latter part is unavoidable regardless of how we apprach this as there will be changes, regardless of the level of backwards compatibility
  7. We do
  8. Although we have a fairly clean sheet here, it is important to get right, and my current thoughts are to match the functionality of the serial and socket interfacers where a simple string of values are published or subscribed to on a common base topic with pub and sub topic extensions, all 3 being defined in the conf, with perhaps a switch to use the last topic level and first value in a frame as a nodeid/name to allow some separation, we will need to try/test a couple of things here, the key is to keep it basic and generic, we can always add another mqtt interfacer to do something particular, but a complex generic interfacer could block some implementations, this is the current situation with the “emonhubMQTTinterfacer” in the emon-pi variant.
  9. Essentially yes BUT! There remains the questions about what it is actually called in the end and whether it is a interfacer, a reporter or a reporter-like interfacer to be thrashed out.

As for the release of an interim step, I’m not sure I would go that route, but I am not going to debate or even think about it too much as I’m sure your mind is set.

Thanks Paul, great, I think we’re making progress with understanding the scope of the work required and compiling a kind of todo list of things to work on here. We’ve outlined both significant changes to emoncms (indexed inputs, mqtt bulk format) and emonhub. What do you think is the next step? How do we break this down into manageable chunks?

Not entirely sure but I do know I need to get into the code, currently, between working, the forum and these really long discussions, I am not finding any time for emonhub coding, so for me the priority is to set up a test environment and start playing, re familiarizing myself with the code based on what we have so far.

I’m primarily working from memory at the moment so perhaps once I get rolling, some answers to the outstanding questions might start flowing.

You are right there are the changes to emoncms to consider as well, it would be good to get a “emonhub” branch set up so we can start on those changes but NOT chamnge anything in the master/stable branches until we get both emonhub and emoncms’s emonhub branch to a place we are happy with, we do not want to make rash changes to emoncms that then dictate a less than agreeable implementation in emonhub, however it would be very useful to have the changes there in emoncms “emonhub” branch when we get to test emonhub.

Regarding the emoncms “mqtt” api, what are your thoughts on basically making “mqtt” a parallel route to similar/same api calls?

eg the bulk upload of mqtt would be to /emoncms/input/bulk and the payload would be [[ts,id,v1,v2,v3],[ts,id,v1,v2,v3]] authentication and identification are something we need to think about too, I would like to see the apikey in there somewhere, I’m guessing it should be in the payload so it isn’t exposed to unauthorized users (that do have access to the mqtt server) although a apikey or username or userid in the topic tree would then allow for an ACF with the apikey or password granting access to that one branch. I know this isn’t needed for the emonpi install but I bring it up now so that we might move closer to a multi-user mqtt implementation in the future, currently there can only be one mqtt account per emoncms server.

I would also like to add a emoncms http interfacer (non-reporter-like) that uses the “fetch” api so we can publish emoncms feed data via emonhub rather than using the “connect send and die” publish to mqtt process. Could we also do this via mqtt? if we mimic the http api’s in mqtt, in theory we can use the fetch api, but instead of “relpying” to the calling http request it could trigger a publish to topic (eg emoncms/outbound/username/reference) where the “reference” is a unique reference for this query that emonhub supplied and can identify to (thinking out loud a bit here I know what I’m after, but not the best way to do it yet). This would allow users to use emonHub to

Regards the “bulk” reporters or reporter-like interfacers, are we agreed that they will use indexed CSV format to minimize RAM used for buffering and bandwidth used for passing the payload to emoncms? Do we therefore want to look at updating emoncms input names with a api call on changes detected to the node/input names in emonhub.conf (and restart of emonhub)?

great, that sounds good.

I will have a think about the emoncms changes and the topic/payload formats you mention and the apikey question.

The fetch via mqtt should be possible but I think quite complex to implement, not sure will have to think about that as well.

yes agreed

yes i guess so… thats a complex action to add

Il try and summarise all of our points in a list so that we can refer back to it as we work on it. It might be good to know which one of us is going to work on each point or whether we both need to explore a point and come back to it once we’ve both had a chance to test ideas. It looks like a big job, how much time do we need to give ourselves to do the job properly? it looks like it will take much longer than a couple of weeks!! especially given other commitments

Not necessarily an immediate concern, but just something to bear in mind whilst structuring the more pressing stuff as something we may want down the line. We can use the http “fetch” api, I just assume that at some point someone will want a fully mqtt setup, it seems daft to go to so much effort to ensure emonhub is posting to emoncms by mqtt not http, and then need to use http for something else.

Maybe, but we maybe able to look at expanding the bulk api to help emonhub with this task, maybe change the api to accept text input values as names rather than rejecting all non-numerical data (eg [[ts,id,v1,v2,v3],[ts,id,“power1”,“power2”,“Power3”][ts,id,v1,v2,v3]]). this way emonhub can just buffer the name changes just as it would any values.

Yes I guessed it would be a significant amount of work, I really have no idea at this point how long it might take. But I do know it will be slow getting started due to needing to familiarize myself with the code again. I doubt much will progress that far at all in 2 weeks, but once we’re rolling it will get quicker, just as well really as I’m sure more work will become apparent as we progress. I already had a snagging list for the experimental version when you released the emon-pi version. I will need to find those notes too.

Hello Paul, apologies for the delay with this. Reading through the above again here is my attempt at summarising the key items raised for our ongoing reference.

  • Establish our approach: interfacers, reporters or reporter-like interfacers
  • Establish our approach: seperate interfacer files vs core interfacer file.
  • Review EmonHub internal message queue implementation
  • Consider process chain, standardise on correct approach for use of run, read, send, action, add, flush, process_post.
  • Consider interfacer naming
  • Buffer persistance: Option to save or load a buffer from disk?
  • Review rx & tx node definition in emonhub.conf used by emon-pi variant
  • Document and explain distinction between QoS1/QoS2 unbuffered/buffered interfacers/reporters.
  • Document further use of socket interfacer
  • Consider backwards compatibility when switching from reporters to interfacers or vice versa.
  • MQTT: What should the format of the generic MQTT Interfacer be?
  • MQTT: Specific format MQTT Interfacers can re-use the generic interfacer
  • MQTT: Keep a per-topic MQTT Interfacer QoS1 unbuffered
  • MQTT: how do we handle on_message case
  • MQTT: Implement a bulk mode, buffered QoS2 MQTT reporter-like interfacer
  • MQTT: Pass MQTT data through emonhub from ESP devices
  • EmonHub HTTP interfacer that can call the emoncms fetch api to retrieve data into emonhub
  • Emoncms changes: Implement the input/post and input/bulk formats in MQTT, perhaps extend to other api’s e.g fetch.
  • Emoncms changes: option to have indexed inputs, and seperate sending of inputnames.
  • Emoncms changes: support multiple emoncms accounts from the mqtt_input script.
  • Emoncms: review timestamp processing
  • Review threaded exception handling
  • Review error handling in core interfacers
  • systemd service unit
  • simultaneous tx of settings resulting in potential serial crash
  • Review list and single word settings in emon-pi variant

From this my next emonhub related step will be to investigate the indexed inputs option for emoncms and the possibility of sending an inputnames parameter.

You mentioned above a wish to find some time to familiarise yourself with the code. Let me know when you would like my input again.

No problems at all, It feels like I’ve been dragging my feet a bit. although I have been spending some time on emonhub over the last week or 2.

I have been looking closer at emonhub and trying to “get into it” again, but that hasn’t been easy, I seemed to have lost a few repo’s and branches. Stuff I have explored, partially fixed or already made progress with previously doesn’t seem to be to hand, not sure whether to spend time looking for it or just start over.

The list is pretty complete, there are other things that we haven’t yet discussed (on my hit list) and there are some things like the threaded exception handling that need looking at, at the very least I would like to see the @decorator replaced with a better implementation, but I’m not convinced it’s working as expected. There are posts that show the thread is dead issue after the fix was released and there are logs that show frequent restarting of the threads without any tracebacks.

Overall the error handling needs to improve, I can improve the common code but the read and write functions will be the responsibility of each interfacer, I would like to try and make the core as robust as possible and reduce the checking needed in each interfacer if possible.

Use of textual names throughout emonhub is something I would like to see implemented sooner rather than later.

I’m not sure I understand why scales and datapoints have been removed from the cargo object whilst names and node name have been added.

We also need to look at creating a systemd service unit.

and the way serial commands are sent from the main thread when changing Jee Settings could cause a crash if 2 threads are accessing the same serial port at the same time.

The way lists and single word settings are handled in the emon-pi variant need addressing to, it is not necessary to use a trailing comma if implemented correctly. I have also found a better way of handing boolean settings.

Hopefully the pace will pick up a bit as I get more familiar with the code again.

Can you also look at the timestamp processing?

ref Best way to guarantee times when using BULK load with PHPTIMESERIES? - #2 by pb66

Personally I would like to see the legacy mode updated to be “absolute” so that omitting a “mode” means the supplied timestamps are used without adjustment. The only alternative would seem to be adding “&time=0” to the end of each request (minor, but annoyingly pointless if we are trying to make the bulk upload as minimal as possible). The current “&sentat=” is open to potential errors.

perhaps we could change these lines

        // Legacy mode: input/bulk.json?data=[[0,16,1137],[2,17,1437,3164],[4,19,1412,3077]]
        else {
            $time_ref = time() - (int) $data[$len-1][0];

to something like

        // Legacy mode: input/bulk.json?data=[[0,16,1137],[2,17,1437,3164],[4,19,1412,3077]]
        elseif ((int) $data[$len-1][0] <1000000000);
            $time_ref = time() - (int) $data[$len-1][0];
        // New default "absolute" timestamp mode: input/bulk.json?data=[[1519749672,16,1137],[1519749682,16,1437,3164],[1519749692,16,1412,3077]]
             $time_ref = 0;

by checking if the timestamp of the last packet is in the range of a complete unix timestamp or just an offset, we could set the timeref to 0 if absolute timestamps are used.

Thanks Paul, Great, I’ve added the additional items you have highlighted to the list including the timestamp processing.

Note this thread about emonLCD and mqtt.

We will also need to think about a “Nodes legacy mode” mqtt by frame in the QoS1 mqtt interfacer.

Also we should create a logrotate entry to put in /etc/logrotate.d/emonhub so that emonhub.log files are not rotated out so frequently. We could concider reducing the size of the files from 5mb to 3mb if a potential 10mb is too much, but we should let emonhub manage it’s own log files and only use logrotate to compress emonhub.log.1.

Revise JeeLib confirmation messages ref “confirmed sent packet size: → ack” in Inputs Nodes Value only rssi Error after update of Emoncms version low-write 9.8.28 - #16 by pb66.

Also need to re-implement or fix the ability for the emonhub.log to display the emonPi/rfm69pi firmware revision again.

@TrystanLea - I would like to invite @beaylott to this discussion, do you have any issues with that?

sure that would be great

Thanks Trystan, in hind sight I hope this is the thread all the right answers are in, we do have several :slight_smile:

Ok… looks like I stumbled on something you were already quite exercised about! Our use cases are actually a lot like some of the ones you ( @pb66 ) describe where we are running emonhub on its own and posting to a remote emoncms instance over WAN. I like the idea that emonhub will work in resource constrained environments where neccesarry.

The first thing I might offer is that there are memory efficient ways of buffering the keys in addition to the values. My first thought was you could beef up (or create?) a Buffer class so that you can transparently store Cargo objects in it but these are actually stored in the Buffer with the relatively static names/scales/datatypes etc. and values separated so the memory use is almost the same as currently. This could also be made to deal with where the node changes. There are various data structures which would be suitable for this including some built-in to Python already. You could also embellish this class with various methods as you discuss above to do other things.

Obviously linking the name of the variable to its value would be desirable in some circumstances (using inputnames or otherwise). This is what I was after in my forum post but I can also see ways around this (like doing a input post the first time …although this needs the names to be passed in the Buffer as above). I could see how this could be made backwards compatible as well so that the current bulk upload without input names could be used.

Secondly, I don’t personally have any strong opinion on this interfacer/reporter distinction. To me there does seem to be a distinction as if you look at the code of the ‘other’ interfacers it anticipates (in its style and structure) the use of the HTTP and/or MQTT interfacer… so it would seem these are separate things. I haven’t put a lot of thought into that though.

I have also put quite a lot of thought into the multi user MQTT → emoncms business as this would be quite a desirable addition for us. I think you potentially need to separate the input requirement (so just a MQTT subscriber for displaying the input values in the tab) from the need to store values in feeds. The former can operate much as the single user version does now. The latter needs to be highly concurrent and go straight to the Redis or SQL database - I have called it ‘connector’ before in conversation with Trystan, but its an asynchronous server component which would take stuff off the broker and push it to the databases directly. I looked at writing something in Python or Go previously. You could also move authentication for MQTT to the broker (or even to a gateway proxy) as many brokers will be able to re-use the current authentication store. And you can also make use of the ACLs.

Hi Ben

Busy few days so sorry there’s been no response from Trystan or I.

I hear what your saying about storing the cargo object and Trystan was wanting a similar thing. This has been discussed and there seem to be little or no benifit to it, whilst the csv route is already there, works fine, allows humans to read/debug the content and it could have other uses in the near future eg a separate “traffic log” rather than trying to find the packets in the error log we could keep a track of all payloads that pass through emonhub for debugging or for “replaying time” to rebuild a lost/corrupt emoncms instance etc.

Buffering the cargo objects could also complicate the names thing as a namechange could easily be placed in the buffered csv without any special handling.

The use of both input names and index is something that Trystan is looking at in emoncms, this has been an obstacle fro many years, dividing users into either index or named input groups as the 2 methods cannot interact so making emoncms handled both methods on the same inputs is a huge step forward and makes things much easier for so many applications as it removes the hard choice.

That’s good to hear as although I’m trying to remain open on this, the more explanations I hear why it is un-necessary, the more essential it becomes in my mind. The distinction you suggest doesn’t really exist as there is good reason to have both http interfacers and reporter(-like interfacer)s and the same for MQTT, the distinction is most prominently the buffer as Trystan says, but yes you could have buffers on both as he has proved. The actual distinction is on the type of job it does. It either reports data in a secure, safe and confirmed way with no loss of data (hence the buffer) or it has a (potentially) 2 way interface where data is just read or written in realtime, status-like, send’n’forget with no (buffer or) confirmation before deletion etc.