Migrating timestore on one computer to phpfina on another computer (across LAN)

I have an installation of emoncms that has been running non-stop since 2013, so it has a lot of data, stored in timestore. I have not updated any software on this computer since 2013. it is configured with a hard drive and stores all data locally. It is still running live and accumulating data all the time.

I have now set up a new Raspberry Pi (3), also configured with HDD, and successfully installed emonCMS version 9.8.28. This is “empty” (no inputs, no feeds).

What I want to do is to migrate the timestore data to a currently supported storage engine on my new Raspberry Pi 3. I have found instructions and script at Upgrading mysql, timestore, phptimestore feed engines to phpfina and phptimeseries | Archived Forum but this seems to assume that you are migrating on the same computer.

What would be the best strategy for this migration? Both computers are on the same LAN so it is easy to copy files from one computer to another and also read databases on the other computer.

Where is the new data coming from?

I had hoped someone with more experience of this might have answered you by now, but I’ll offer you my tuppence worth in hope it helps.

I would expect the best route is to stop both servers. copy the working mysql and feeds data to the new Pi (replacing the new mysql table). Run the checks for mariadb to update the old mysql tables to new mariadb tables (search this forum for mariadb for more info on that). Once you have the feed data and mysql sorted you can run the conversion on your phptimestore data on the new Pi and hopefully then you’ll have a running “phpfina” based setup. remember to change the password in your settings.php if you transpose the old table into the new setup.

once you can log in and view all your existing data you can resume posting

The reason I asked where the data originates is because if it comes via emonhub, you can increase the buffer size and stop it sending while you duplicate the data and/or create a second remote target emoncms so that you cab resume sending to the old server once you have duplicated the data, whilst buffering the new data until you have converted the phptimestore data and tested it.

The advantage of setting up a second remote in emonhub and resuming/continuing to post data to the old set up is that you can start over again if you hit a problem the first time round. You can have a dry-run if you like. So if you are (for example) currently using an “apt-get” installed emoncms version 8.3 (or 8.4? i forget now) with the (now retired) “emoncms-rfm2pi-module” you could set up emonhub on the new device and point it at both the old and new emoncms instances to ease the transition.

The reason I have suggested these methods is that the newer “backuo module” doesn’t do phptimestore and is specific to the emonpi and the older “backup script” which does do phptimestore, doesn’t do the dashboards and multigraphs etc. Plus I do not know for sure that the phptimestore conversion script will work first time with mariadb, it should I guess, but I doubt that is proven.

You should also be aware that the dashboard module has under gone major transformation so you may need to reselect all the feed is all your dashboard widgets.

I’ve recently done something similar to move 30 odd accounts from one server to another (no phptimestore to convert though) and I managed to do it with no loss of data due to the buffering in emonhub.

@OffGridder can you do the conversion on the original Pi first? Can you run this script there?
When it asks if youd like to delete the timestore feeds I would answer no until your sure you can access the data in the phpfina format.

Which version of emoncms do you have on the original machine, I wonder if it supports PHPFina?

Thank you both for the responses.

I’m not actually sure what version of emoncms I have on the old computer. The web pages do not display a version number (it just says “Powered by openenergymonitor.org”). However, on the feeds page, the range of available engines is MYSQL, TIMESTORE, PHPTIMESERIES, GRAPHITE.

I have not installed emonhub on the new computer but I could certainly do that. So far I have followed the instructions at emoncms/docs/RaspberryPi at master · emoncms/emoncms · GitHub up to but not including the emonhub installation part.

I am reluctant to run anything at all on the old computer as it is currently my “live production” system, and I have noticed that if I ask it to do any other work, one consequence is that I start getting corrupt values in my timestore feeds. I get that problem too if I have too many real-time tracking graphics open at once, or run any sort of batch job.

I think I will more or less follow Paul’s suggestions. I’ll update when I make some progress - but that may be a while.

Thanks again.

I’ve just dropped a PR to try and make the backup module more general - It could do with a dev branch on this module for this work though. It is only a start to what needs done but the export script works when called from the command line. You could easily add in any other folders - was Timestore simply a folder to backup? The only other thing to watch would be that I commented out stopping the feedwriter service for the moment. Not sure if this was part of the early EmonCMS or not.