I’m setting up a new emoncms. How best to import historical readings?
Can I extract the data from melcloud from before I set up my first emoncms and import that somehow too?
Thanks for any answers. Maybe @MyForest or @Timbones have already solved this? 
Hey @mjr
I have a solution, but you won’t possibly won’t like it.
I store all my scraped data from MELCloud as raw JSON files. I have millions of them. I have a job that says “where have I got up to in processing these files into EmonCMS?” and most of the time the answer is “about a minute ago”, but it also copes when I replay the entire thing from the start. I do that very rarely, but it’s set up this way specifically for the sort of situation you find yourself in.
So the answer could be to just grab the old data (via the API) into folders and walk over it.
However, it sounds like you’d also like to get the old, fine-grained data from MELCloud from a long time ago. I haven’t found a way to do that. The fine-grained data is only for about 24 hours because they loop round in their data structure from what I can see. That’s why I rushed so much to get my scraping working when they turned on my heat pump and why I hold on to those millions of JSON files.
If I were in your shoes (and I was as sceptical about software and longevity of databases as @MyForest ) I’d export all the old data from EmonCMS to JSON files and then process them back into the new instance. The way I see it, no database format is going to survive any appreciable length of time so text files are the only safe place to put things. Of course I may be wrong so you can take that comment with a pinch of salt. Having said that, I can still use my solar readings from 2011 because I put those into text files.
David
