New current and history logs

The current release of IoTaWatt has a capacity limitation of about 1.6 years of data that can be stored locally. This is governed by limitations in the Fat32 file system used in the internal SDcard. My home IoTaWatt has 10 months of data, so it was time to do something about that. The changes that are in test now will extend the datalog life to a maximum of about 15 years. I’ll explain and illustrate how it works.

The original log remains in place, exactly as it was, but instead of just filling up, it will wrap around a little after a year. The wrap will be transparent to you as a user, except that a new tab in the status display will show the date ranges for the log, and as new entries are added, you will see the current log start date/time increment up as older entries are overwritten. Here’s my log display:
image
So that brings us around to the new history log. This log is exactly like the current log, except that it has 60 second resolution instead of 5 second as in the current log. You will notice that the date ranges in my history log are in even minutes.

The new history log is built from the current log when you upgrade to a release that has it. The build process happens as a background activity, and takes about an hour for each month of data in your log. In my case, it took about ten hours to build the log.

Once again, this is a seamless operation. Sampling continues at a respectable rate, the current log continues to grow, and the history log is built as time allows between all that. If you have a power failure or otherwise reset while the log is being built, no problem, it will pick up where it left off after restart. In fact, the process that builds the history file is always running and adding to the history log as new data becomes available in the current log.

There are a couple of huge benefits to this scheme:

  1. Conversion is seamless. The existing log format is preserved and there is no “conversion” process.
  2. Data logging capacity is dramatically increased, and SD storage requirements are modest.
  3. Because the history log is in one minute increments, there is absolutely no difference in the data presentations of the local graph application for standard views. The finest resolution needed for a daily display is two minutes.
  4. The resolution of the data presented is seamlessly adjusted to use the finest resolution available - even within the data points of a single graph.

Here’s an example from a test system I have running right now. I’ve artificially set the capacity of the current log to one hour. (yours is about a year). So to look at power over a day, 23 hours is at 60 second resolution, and 1 hour is at 5 second resolution:
image
Now if I graph power for a day I get this:


OK, I started this new last night so not a whole day, but you see the plot. Looks normal, and the detail level is consistent. The history log contained all of the data points requested and so they were serviced from that log. (IoTaWatt always serves up the average power for the requested interval).
Now, lets say I drill down to look at the most recent couple of hours:

Notice that the high resolution detail starts at about 11:25. That’s where higher resolution data is available from the current log. Below that, the data is provided as one minute averages. Here’s where it kicks in in the rav CSV data:
.

For the average user, this resolution won’t be an issue unless you drill down to the sub-minute level on historical data that is more than a year-ago. But regardless, the visualization is seamless and adjusts to the data available in the logs.

1 Like