When re-establishing data upload after an outage (power, WiFi, server), IoTaWatt queries Emoncms with an input/get&node= request to determine the time of last input post. The response is parsed and upload is resumed after the most recent time returned in the response array. This works perfectly with Emoncms.org.
Lately, there have been some incidences of Rpi Emoncms instances restarting as much as a day behind after only a relatively short lapse of power to the Rpi.
I suspect that the Rpi version may not write-thru the latest context for the inputs to SDcard. At the risk of exhibiting my ignorance of the internals of Emoncms, my suspicions are fueled by the talk of low-write technology in the pi version, which I understand is needed to reduce wear on the SDcard, and also that I believe Redis is used to keep some of the database in memory.
Question(s):
Is the above scenario possible?
If so:
Is it avoidable though installation options?
or
Can the effects be mitigated by insuring the SD inputs are updated at a tolerable but not abusive rate of 30 minutes or so?
Emoncms does buffer (if it is enabled) in Redis (default is 300s) to reduce SD Card wear and any power failure would result in the buffered data being lost.
However, when you do the query, it will (should) give you the last data point written to disk, but in this case that should not matter as you can fill in from where it last stopped.
What API call do you use to get the last data point?
What API data format do you use to send the data?
I use input/get&node=. To get the json list of all inputs in the node with last post time.
IoTaWatt uses the bulk post protocol to send one or more frames of data with time stamps.
Yea, so it’s not so much a data integrity problem as an operational issue in that it takes a long time to resend a day’s worth of data. A day being the apparent result of the query. If the Redis cache writes through to SD every 5 minutes, that’s probably not the cause of the anachronism. It is odd though that it has never come up over several years with the enterprise Emoncms.org.
Just to understand though, would the feeds also update on SD at the same time as the inputs, or are they written through real-time?
I think I can see where the issue is arising from. Redis persistence is disabled on the emonSD image in order to reduce generally unneeded write load. On power cycle the redis database is empty, the input list and feed list and other cached items are reloaded to redis as needed. The feeds cache can reload last time and values from the feed data itself but there is no separate persisted store of the last time and values for inputs, so the input list generally shows n/a at first reload, this might then be the cause of the behaviour that you are seeing.
It’s a bit of a difficult one to fix, ideally the last time and values would be reloaded via any linked feeds but there’s no guarantee the feeds will be present to read the values back from…
One way around it would be to point Iotawatt as a specific feed/timevalue as the history time reference, but this adds an unfortunate setup step for users and development complication.
Another would be to persist the redis database at an interval say every few hours, but then after a reboot kWh feeds would start from the value x number of hours old and continue up from there rather than reading the last value from the data files themselves…
Sorry for bringing up an old topic but wondering if a solution was found for this?
All I really need is a way to manually persist the redis database before I turn off my EmonPi for planned electrical work. Can I just use the BGSAVE command in the redis cli after pausing the Iotawatt upload?
I found if I set the upload from date to the current day in Iotawatt the KWh feeds on emoncms get doubled for the period of time from midnight up to the power failure.