Duplicate Input Appears Spontaneously - Result: Lost Data

Neil · 2019-08-23T11:50:41.900Z

Hello,

This anomaly has occurred several times over a period of months and is not limited to the OARH input. Note the OARH input, the one that is being logged, is the input I setup originally (but is not being updated), the input not being logged appeared spontaneously and is being updated.

I use MQTT to transfer the data to emoncms from another RPi.

Each time this happens I have to Log data from the ‘new’ input to a new feed and delete the original feed in order to get it to work again, loosing all data from the original feed.

I know that some ‘radio’ traffic can cause inputs to appear, this has happened in the past, I just delete the inputs. I don’t know if this might be related to my problem.

How can I prevent this from happening and also how can I ‘save’ the data stored in the malfunctioning OARH feed?

Thanks

16%20AM1994×1044 172 KB

pb66 · 2019-08-23T13:05:13.018Z

This is an old known but unresolved issue, it is less common these days but very annoying when it does happen.

See duplicated inputs · Issue #1198 · emoncms/emoncms · GitHub where I have raised a issue and I have just added this thread to that issue.

In the interim, you do not need to lose historical data, nor do you need to create any input processing or feeds. If you just delete the new duplicate input, the original will resume updating. Not an ideal resolution I know, but until the issue is resolved, at least all you lose is the data posted during the time between the duplicate being created and you noticing and deleting it. You could create a little cronjob that periodically checks the feed has been updated and if not, run the input/clean api call. This will remove any inputs that do not have at least one process and the original input would start to update again. again i know this is a workaround, but I do not know when the issue will get resolved.

Neil · 2019-08-24T13:21:25.404Z

Thanks Paul,

I deleted the new input and the original input immediately started updating. I saw @borpin’s comment at https://github.com/emoncms/emoncms/issues/1198

Got to be honest I have not seen this for a very long time and I pass a lot of MQTT data.

In my case it happens too often to ignore but I’m using the node-RED emoncms node to pass data to the emonpi. I wonder if this may be part of the problem? I am already setup to pass the data using node-RED MQTT node but am not using it at the moment.

I think I’ll switch to MQTT (rather than the node-RED emoncms node) and see if this makes any difference. I’ll report back here with any tangible results.

Thanks for your help

borpin · 2019-08-27T11:44:14.781Z

3 posts were split to a new topic: Duplicate inputs possibly caused by Emoncms Node-Red node

pb66 · 2019-08-26T08:50:48.949Z

There are multiple issues to address here, yes the incoming data could be better perhaps, but since we encourage data to be sent from here there and everywhere, the inputs must be robust enough to handle rogue data and/or occurrences.

This issue is as old as I can remember, I used to see it often with http inputs, not so much recently though. It was a prolific problem when the emonpi variant of emonhub had issues and it was just as bad when the emoncms mqtt input had issues.

Looking just at emoncms, there are 2 parts to the issue, firstly it shouldn’t create the duplicate input(s) and secondly, when it does, it shouldn’t continue to update the wrong (duplicate) input. If the duplicate inputs never occur (due to a more robust input module) then obviously the latter part of the issue is moot, but the fact that this can occur becomes a much bigger problem when emoncms then continues to update the wrong input. If it simply reverted to the original after creating the duplicate, there would be a minimal interruption in the data and no user intervention required except to perhaps delete any duplicate inputs every so often, annually or so?

The issue with updating the wrong input is another manifestation of a common issue that effects other parts of emoncms too (see Apps feedlist issues) where emoncms defaults to using the last found item if there are multiple items of the same name. Since this is an issue for duplicated inputs, this behaviour could perhaps be modified to always use the first found, that would minimise the impact of the occasional duplicated input.

A more aggressive approach would be to delete all but the first occurrence of any given input, after all the keys are supposed to be unique within that node/device. Obviously I would prefer to see the duplicate inputs not happen at all rather than them getting automatically removed, but however it’s done it must be fixed within emoncms for emoncms to be in control of it’s own reliability.

This duplication of inputs may “not be a problem” currently, for some or even most users, but whenever an input issue does arise it tends to result in duplicate inputs and therefore, lost data and unhappy users.