Frequency of logging

Another day, another question :grin:

Now I’ve got an emonTx set up and delivering data to my emonBase, I’m wondering about how frequently I should log the data to storage. I understand that the emonTx polls and reports data every 10s and at present, I’ve simply let that be logged to the SD card.

But that seems like overkill. I can’t imagine I need or want that granularity. It will fill the storage and/or wear it out too quickly. I did a quick search here and on the web and looked in the FAQ but didn’t find anything relevant. So what intervals do people record at? What reasons are there for recording at particular intervals? I don’t have any particular needs in mind at the moment - I just want a general record to help with planning usage and other facilities.

I currently have some temperature and humidity logging that records at hourly intervals, and solar generation daily totals. The solar data doesn’t seem like nearly frequent enough. I guess my gut feel is that logging once a minute is probably the most I’m likely to want, and logging once an hour is the least I’d feel might be adequate. But I don’t have a good feel for how to use the data, so I’d appreciate any experience anybody can offer.

Once I get sorted, I expect I’ll copy data from the emonBase to a PC at say daily intervals and incorporate it in my normal backups. So I could also remove old data from the emonBase at daily or less frequent intervals.

What do you want to miss?

The reason for putting it that way is, the emonTx by default measures a sample for 200 ms every 10 s. If you have something like an induction hob that switches on and off rapidly, you’re likely to sample sometimes when it’s on and sometimes when it’s off. If you go to the extreme and (say) sample every 10 minutes, you’re more likely than not to miss a kettle being boiled - but if you catch it, emonCMS will assume it took 10 minutes to boil and record the energy on that basis.

So in the paragraph “I currently have some temperature and humidity logging …” you’ve more or less put your finger on the problem: how often you should record the data depends on how quickly conditions change.

There are some errors in the CM library that I’m working on, when that’s done it will get around the sampling problem because it records continuously (hence Continuous Monitoring) and reports the true average over the period. That’s been tested up to 5 mins, it should function properly over much longer.

1 Like

A phpfina feed with a fixed interval of 10s will occupy around 12Mb per year (4b per datapoint x 8640 points per day x 365 days) so it’s not as crucial as you might think to conserve space unless you have many hundreds of feeds or a very small SDcard. As for the write frequency, all feed data is buffered and written to disk periodically (each 60s by default) so disc wear is also not so much of factor when choosing a feed’s fixed interval.

But, if you do wish to minimize disc writes and space used regardless then the raw power data is the most valuable source data and that cannot easily be reconstructed once lost, so (for example) you could save just your power data as 10s to retain granularity in case of a future change in plan etc and then review your other feeds eg the power to kWh feeds can be reconstructed at any time from the power data using the post-process module. Plus the calculations performed by the power to kWh process are held in memory by Redis and continue to get updated on each input update even if the fixed feed interval is greater than the source updates.

eg a processlist like so

Log to feed “power1” phpfina 10s
Power to kWh “power1_kWh” phpfina 60s

being updated every 10s by your emonTx would record the raw source power data close to original as that feed would get updated every 10s whilst the power to kWh value would get calculated and updated in RAM only and then on every 6th input update that current calculated value would get written to the feed once every min. The same would apply to a fixed interval of an hour or a day etc.

There is an accuracy vulnerability though because if the Pi is rebooted or Redis flushed inbetween the min/hour/daily fixed intervals the change since the last min/hour/daily fixed interval is lost.

IMO you are better to just log at a higher resolution than you need unless space is really tight as the above example (using a 60s interval over a 10s interval) will only save you 10Mb per annum (that may be significant if you have hundreds of feeds).

The raw power data “log to feed” process doesn’t do any averaging etc so if you were to save your 10s incoming data every 60s you will just have a 10s snapshot each minute that may not reflect the true picture, the unused values that are not saved to disc would be lost forever, you cannot regain that resolution in the future.

You may be surprised how revealing that extra granularity is, it will help recognize loads and possibly even reveal issues otherwise hidden from view, this greater resolution is the reason using CT’s is so much more useful than pulsecounting alone. Low res data is only good for recording consumption/generation etc whilst high res power data is better suited for debugging, control and management of consumption/generation. It basically boils down to whether you are just keeping records or actively managing consumption/generation.

Of course it’s for you to decide what fits your needs best, but disposing of such detail might not have a huge benefit (space and resources wise) but at some point some detailed historical data might prove useful.

The emonTH and temperture data in general is usually at lower resolution because a) temperature change is generally slower than changes in energy consumption, and b) the emonTH’s are battery powered so getting batteries to last 6x longer is more significantly more useful than reducing the data’s size on disc.

[edit - just for clarity, the suggestion of saving the power to kWh data to a feed of a lower interval is theoretical
(ie untested by me) and with the emonPi/SD in mind, it is totally dependent on Redis being used so the “last update” is NOT retrieved from the feed directly].

2 Likes

Thanks Robert and Paul both. That clarifies things a lot. I don’t have all that many inputs on the emon system, so I think I may as well leave the logging at default for now and see how it goes.

I do have a solar diverter, and we do have an induction hob, so there’s two good reasons there until the CM sketch is ready for reliable use. I also have a EV chargepoint, but no EV as yet, so that’s for the future.

The temp/humidity logging I have is completely separate and is rather basic - extract a CSV file on a USB stick once a year! But there are twenty sensors embedded in my walls and roof and it’s an off the shelf system that was embedded when we built the house. It uses a 9V battery.