Hoping the community might be able to provide some advice. First the (long) backstory…
I’ve been running Emoncms since 2014, originally on a Pi 1 running Wheezy, which was later migrated to a Pi 2 running Jessie. This was was a ‘full-fat’ manual installation and I had kept it mostly up-to-date with developments including migrating to the emon-pi version of emonHub. At the time of the incident it was running Emoncms 9.9.6.
Unfortunately the disk developed bad sectors and trashed the volume. I managed to image it using ddrescue, although even with multiple passes numerous sectors could not be recovered. After writing the image to another disk, I ran fsck multiple times to repair the hundreds of errors, eventually resulting in a filesystem that would mount. I decided to return the disk to the Pi to see if it would boot, which it did, and after reinstalling Apache and PHP Emoncms was up and running although damaged. After downloading and replacing the Dashboard module I was able to confirm that the Time Store data appeared intact.
I have built a new instance of Emoncms 10 in a Linux container running on Proxmox, using the standard build script. RF is handled by a Jeelink USB passed through to the container and is working nicely. I would now like to try and transfer the feed data from the old instance to the new, but do not trust the damaged installation enough to use it as a complete backup source. For instance, packetgen.myd is missing from the database and I don’t know what else is affected.
What is the best option for transferring the feed data manually without a database backup? There are around 30 phpfina and phpfiwa stores in total…
I can presumably recreate the feeds in the new instance with the same IDs, and then copy the phpfina and phpfiwa stores across and fix the rights, however are there any things to consider with the low-write version?
Alternatively is it possible to use the API to bulk send data from the old instance to the new?
After finding the ‘usefulscripts’ package, I managed to achieve this with a few surprises along the way, so shall share the experience…
Data Conversion
I’d forgotten that PHPFIWA is deprecated and not supported in the low-write environment, so it was necessary first to convert the data. There is handily such a tool in the usefulscripts package but the first sticking point was that the scripts threw mysqli errors, because they have been refactored to support the object-based settings.php that presumably was introduced in v10. This is a slightly strange choice, because anyone who needs to convert from FIWA to FINA at this stage is almost certainly running v9 or earlier, so it would have been better to leave them untouched or create two separate packages.
integritycheck.php was used to check the … integrity of the data. One unimportant store was damaged so I removed it
remove_spike.php was used to trim some errant datapoints before conversion, as this is only possible on FIWA stores
convertdata was used to convert all the stores to FINA. First, the check_emoncms_feeds_for_conversion.php script was used as a final check, and then the phpfiwa_to_phpfina.php script was run to perform the conversion. Unfortunately this failed at line 172, where there is an untested condition:
$redis->hSet("feed:".$outid,"engine",5);
Since this was a full-fat standalone v9 installation there was no redis running. The script does test for presence of redis and sets a boolean accordingly so there should really be an IF statement around line 172 but I just remmed it out to resolve the issue.
All feed stores were then successfully converted and replaced (overwritten)
Data Transfer
All of the above took place on the old v9 instance. I then moved over to the new v10 instance and cloned the current usefulscripts package into the home directory.
backup.php was configured with the Emoncms root page (for v9 this is http://IPADDRESS/emoncms) and API key of the old instance we wanted to extract data from
The script then copied all of the data stores across along with the Feed configuration, easy as that.
All that was left to do was to re-point the inputs at the Feeds and rebuild the graphs. The Dashboards can be copied across using the new Sync feature.