I took this up privately with Trystan a week ago. This is what he wrote:
"Good question, clearly the auto node feature is seeing a lot of node data that matches the data length of the template decoders but extends over a much wider / invalid node range. There is no filter on valid node id’s for this feature.
Question is why is there so many apparently valid messages coming through to emonhub… the code on the emonPi should be filtering some of this, though spurious nodes has always and continues to be a problem that probably deserves more attention."
I was running an endurance test on emonLibDB, which precluded updating the emonPi. I stopped the test (at just over 1¼ million reports - at 10 s intervals - without stopping or resetting) and updated emonCMS, whereupon it hasn’t given a problem - but this was only a week ago, so by no means conclusive. I cited “I have 517 spurious node definitions in my emonhub.conf, 509 being unwanted emonTx4 and one an emontxshield.” (And I’d cleared all the spurious ones out a few weeks before.)
But one thing that was different for me - it continued using the original Node name.
Is the “new” input actually writing to the original Feed with the original processing steps?
I was thinking it wasn’t quite clear enough, if the screenshots above are from emoncms.org this could be an issue caused by the relatively recent update to emonhub that sends input names via the emoncms http interfacer. If this is local then the issue may instead be caused by the new autoconf feature.
Either way I would recommend moving the input processing to the new input names as these are more descriptive and deleting the old inputs.
Hi there, I don’t think I’ve done any updates to the emonhub unless it has done them automatically (without my say-so)?
I have several years back data, is there some way to consolidate this? I’m also not sure what it means to “move input processing to the new input names”? I cannot see a way of doing this.
I think my main concern is almost anything I think might be the right answer is likely to trash something else! Or prevent me importing/consolidating the old and “new” data. And I’m still not really any the wiser as to why a powercut caused the input names to all change without my say-so?
Ok so looking around and thinking about this a bit more.
I think I need to create a new feed, with all the same settings as the original feed, but talking to the input that is the same as the old input but with all different names.
I need to ‘post process’ the data that the “new” input stored (if it exists?).
And finally I need to consolidate, maybe by post processing the old data into the new data feed?
The problem is I really have exactly zero confidence in my ability to do any of these steps, or what the right order is…
It’d be really helpful if anyone could advise me on whether what I say above is at least vaguely sensible sounding?
Also, would anyone be able to tell me if I’m losing data at the moment? To reiterate, my “feeds” are 4.6 days out of date, and not updating, but I do have “up to date” input data. Is the input data being recorded in such a way that I can re-create the feed post-hoc? Or is it gone for ever?
I may yet need to dump OpenEnergyMonitor/EmonCMS/EmonPi/EmonTX altogether and change to something else as I am finding myself overwhelmed by the admin side of things; literally no idea what to do or where to go with this. All I can really see is it was working, then we had some power cuts, then for no really good reason I can see it broke itself and no longer really works, and in spite of going through the documentation, and asking on the forums, nothing really makes that much sense or has any easy answers
Part of my problem is that because I’ve only implemented this once, and not over and over again in lots of situations, I really don’t feel I know and understand it that well. But I am sure there has to be a product that is simpler out there. Failing all the above, does anyone have a suggestion for something with CT clamps that maybe has a lot less features but basically draws graphs and/or has a cloud service that does all the pretty stuff? I need power monitoring and I don’t have any at the moment… so I need to look at other ideas to solve this
Scratch that. It hadn’t finished. But the fact it offered a link to an extant file made me think (reasonably) it had finished. I refreshed the page, and now the file is more like 150MB. Good. Ok so now I can probably semi-safely risk breaking everything, at least.
If I knew how to raise an suggestion, I’d point out that the UI ought to perhaps say what message is displayed in the logging once it is finished, and maybe suggest not displaying the .tar.gz link UNTIL that had appeared in the log.
But … is there any way to recreate the data missing for 4 days? I really guess boiling it down, it’s about how “input” data is stored? Does it not get stored until it becomes a “feed”, or is it stored raw as input data? If it’s the latter, how would I go about post processing the input data, into “feed” data? So that my graphs become complete again?
I kept quiet thinking Trystan would come back, but he hasn’t.
I’m afraid the database is the Feed, so if data is not going into a Feed, it’s not being stored. I think your 4 days of data will be gone forever.
Now, would you believe that nobody tells even us Moderators and major contributors about changes to the system? - somehow we’re expected to read all the documentation every day to find out changes that have been sneaked in. I today or yesterday found one that might have helped prevent your problem - it’s at the top of the file emonhub.conf, which you can get at via Setup → Emonhub.
#######################################################################
####################### emonhub.conf #########################
#######################################################################
### emonHub configuration file, for info see documentation:
### https://github.com/openenergymonitor/emonhub/blob/emon-pi/configuration.md
#######################################################################
####################### emonHub settings #######################
#######################################################################
[hub]
### loglevel must be one of DEBUG, INFO, WARNING, ERROR, and CRITICAL
loglevel = DEBUG
autoconf = 1
### Uncomment this to also send to syslog
# use_syslog = yes
#######################################################################
####################### Interfacers #######################
#######################################################################
if you change autoconf = 1 to autoconf = 0 as I understand it, it should not have created the new set of inputs. I haven’t tried to test this, so I’m wary of suggesting you change it, but again as I understand it, it enables a semi-automatic first configuration, and once you have got that, it should never be needed again. I think it’s this “feature” that caused all the trouble for you.
Indeed. But the UI doesn’t say to expect that line, and presents the download link before that line has appeared in the log. Leading to the situation I found myself in, where I was downloading a duff file, because it hadn’t finished (but I didn’t know it hadn’t finished).
