There have been a couple of instances lately where users have fallen foul while trying to update.
One of the issues I believe is there is no robust mechanism to backup Emoncms. By this I mean both system, data and settings. This should also work for all users not just EmonSD and EmonPi.
Whilst there is a backup module available (not part of core install), my experience is that this is not robust.
I believe this should be a core part of the offering when using EmonCMS and would like to see it in the roadmap.
Its a good idea @borpin, are you thinking an extension or improvement of the existing backup module as used on the emonpi for use more generally? or perhaps a manual process, steps to follow if you dont have the backup module with all of its dependencies?
The backup module depends on service-runner which uses redis in order to run the slow backup creation process in the background. Redis and service-runner are optional and not present on all systems. What is the minimum system requirement that the backup process can require?
Bit of both I think. I am not familiar with the EmonPi backup as I don’t have one. Is it the Backup Module? if so, I found it to be very EmonPi specific.
I am perhaps thinking of something more akin to the export process that effectively creates a snapshot of the system that could be restored. So perhaps a ‘snapshot’ mechanism is a better description.
The aim of the snapshot mechanism is for an object to be created, that is easily downloaded or saved, which enables a system to be restored in the event of failure.
I’m not sure how best to achieve that - whether it be by improving what is there or clean sheet of paper.
As an aside, the very start of the update process should warn users to take a backup (snapshot).
I remember a Dropbox script that one of the Pauls did I think, which may be an alternative route.
Great, that what the backup module is designed to do. Perhaps making it less pi specific is a good first step and to then evaluate from that point how to improve further?
I’ve adapted the existing backup script to automatically upload the archives to Amazon S3 for storage. May be a good idea to make this a configuration option going forward (other cloud providers are available!)
That would be nice @stuart could you share how you do this?
I took a look at your pull request @borpin for removing hardcoded path’s in the backup module this morning and have been testing it both on my laptop and an emonpi. I made a few modifications to get it to work and have been improving a number of other aspects of the backup module more generally while Im at it (sidebar, automatic logout after import, error message if import file not selected). All rolled into a branch called ‘multienv’
Import is now working fine for me on a non emonSD install. After applying the php upload config in the backup module readme, fixing the import and export script to read the emoncms database name from settings.php and only stopping and starting services if they exist.
Not sure about the menu visibility @borpin, here’s what it should look like:
The Backup entry appears top-left after selecting from the Setup menu.
I would really like a way to plug in a USB pen drive into the emonbase/emonpi and backup periodically to the pen drive. The backup module would need to scan for removable media and automatically mount disks on the pi, which could be complicated…?
… and perhaps stream the export rather than build the tar on the emonSD SD card. There are php libraries to export a compressed zip as a stream but I imagine a fair bit of work to implement.
The main steps for a manual backup are (this is essentially what the backup module script does):
Export the emoncms mysql database, e.g on a emonSD install:
mysqldump -u emoncms -p emoncms > emoncms.sql
password: emonpiemoncmsmysql2016
The mysql database contains emoncms meta data such as the user table, the list of feeds, feednames etc. If your using the standard emoncms feed engines there will be no timeseries feed data in the emoncms database.
Copy the content of /var/lib/phpfina, phptimeseries & phpfiwa (if it exists) or /home/pi/data/phpfina… (location on emonSD installations). This is the feed data.
Copy /home/pi/data/emonhub.conf
Copy /var/www/emoncms/settings.php
Optional: manually create a tar.gz file to hold the backup, this is what the assembled backup should look like:
One thought - it would be worth loading the contents of the config file so you can see before you start what the config is. It could also do a few checks on the paths etc.
The main heading would be better named Backup & Restore - I’d actually call the sections the same instead of Export & Import.
Could this not be included in the emoncms ‘core’ modules, the scripts in the scripts folder and the settings etc pulled into the main settings.php file?
Now I can see the page I’ll sort out the settings and try to create the backup file.
@TrystanLea, I have tried it and it is silently failing - no errors in backup log or main log. Not even any friendly INFO entries. It is loading the config file.
I’ve tried to work out where it is failing without success.
I should say that I have had to put it in it’s own folder outside of a home directory but it is where $homedir points to.
The export script works fine when run from the commandline.
