How to limit data size

Hi. Is there a way to limit of set the size that the data files can grow to or archive old data?

@pb66 or @Paul are the best people to answer this.

I do not think it is possible to limit the size of the data files. You can of course copy the files to another medium - USB flash drive or external disk drive, but I cannot tell you how to ‘empty’ the files and still retain the data structure.

I assume you are referring to the phpfina and phptimeseries feed types? You can of course back-up your data but there is currently no set or automated way to reduce the feed’s size on disk, the phpfina’s file size is set by the passing of time, when the feed is created the starting point is recorded and every passing of the chosen interval will add 4bytes, even if you do not post for a week and then post 1 datapoint with the current time it will pad out the file with (secs_since_last_datapoint / interval) x 4 bytes. So the size of a phpfina feed’s file size is directly related to how old it is and interval chosen so a 10 secs fixed interval should be around 3.15 12.6mb per year.

The only way to reduce the size of an existing feed is to remove data from the front end of the file and edit the meta file to suit. this involves manually editing files on a live system. bear in mind any error in your calcs will “timeshift” your data.

Now I’m worried! For some reason my phpfina is four times the 3.15 MB / year. Time to get out my calculator

EDIT: I calculated a different number:
4 bytes every 10 seconds equals 24 bytes/min
times 60 is 1440 bytes/hour
times 24 is 34560 bytes/day
times 365 is 12614400 bytes/year or 12.03 MB/yr

Doh! I forgot the 4bytes per datapoint when I did the math ie

10 sec interval = 6 datapoints per min x 60mins x 24hrs x 365days = 3.15
What I failed to do was x 4byte = 12.6mb

Well spotted!

Thanks for the info. Lets say I want to clear out the data every year. Is the dashboard and feed values held in the database and the data kept in the files ? Can I delete the files once a year and the dashboard will still work with new values. Or do I need to keep files with no data in them and replace the data files once a year?

Regards.

Each feed has a unique feed id and that is used not only as the feed file name but also as a reference to that feed in all the input processing, multigraphs and dashboards etc.

If you create a new feed it gets a new id so you would need to go through every place that feed is referenced and change from the old id to the new id.

If you do not replace the feed it’s fixed start point relative to now will dictate it’s size.

HOWEVER!!!

If you were to create a new feed and then fully stop the emoncms sever, and rename the new files to the old file name (and remove/rename the old files) when you restart emoncms it would not know any different, the old feed ids would still be in service, but they would not contain any historic data as they were only just created. It’s a little messy and you have stuff like redis working against you, but correctly it should work no problem.

Agree @pb66, as we have discovered previously, deleting or amending data files with Redis running can cause unexpected results.
It would help if the Jessie package updated Redis to a more recent version which included more tools, such as CLIENT PAUSE which was introduced in v2.9.50.
We are currently still using v2.8.17.

Paul

Thanks for the information. So what are the steps to follow to clean out the data and still have the charts work without having to redo them. I want to write a script that will stop the engine and the delete / copy etc as required to clean out the data and then restart the engine so that only new data will show.

Kind Regards.

The last paragraph of my last comment loosely outlines the steps, you would need to experiment to find what suits your needs.

You could start by looking at the feeds api in emoncms and these 2 threads

Deleting old data

Log to feed interval