Development Roadmap 2019 - Robust Backup Process

(Brian Orpin) #1

Continuing the discussion from OpenEnergyMonitor Development Plan Outline:

There have been a couple of instances lately where users have fallen foul while trying to update.

One of the issues I believe is there is no robust mechanism to backup Emoncms. By this I mean both system, data and settings. This should also work for all users not just EmonSD and EmonPi.

Whilst there is a backup module available (not part of core install), my experience is that this is not robust.

I believe this should be a core part of the offering when using EmonCMS and would like to see it in the roadmap.

(Trystan Lea) #2

Its a good idea @borpin, are you thinking an extension or improvement of the existing backup module as used on the emonpi for use more generally? or perhaps a manual process, steps to follow if you dont have the backup module with all of its dependencies?

The backup module depends on service-runner which uses redis in order to run the slow backup creation process in the background. Redis and service-runner are optional and not present on all systems. What is the minimum system requirement that the backup process can require?

(Brian Orpin) #3

Bit of both I think. I am not familiar with the EmonPi backup as I don’t have one. Is it the Backup Module? if so, I found it to be very EmonPi specific.

I am perhaps thinking of something more akin to the export process that effectively creates a snapshot of the system that could be restored. So perhaps a ‘snapshot’ mechanism is a better description.

The aim of the snapshot mechanism is for an object to be created, that is easily downloaded or saved, which enables a system to be restored in the event of failure.

I’m not sure how best to achieve that - whether it be by improving what is there or clean sheet of paper.

As an aside, the very start of the update process should warn users to take a backup (snapshot).

I remember a Dropbox script that one of the Pauls did I think, which may be an alternative route.

(Trystan Lea) #4

Great, that what the backup module is designed to do. Perhaps making it less pi specific is a good first step and to then evaluate from that point how to improve further?

(Brian Orpin) #5

Ok so my point is that this module is really unloved and it needs some TLC and I’d like to see that as part of the development stream.

I raised an issue last February about the hardcoded paths and since then service-runner has been updated but nothing has been done about this module.

(Trystan Lea) #6

Good point here @borpin Web page suddenly returning PHP source code? I think it would also be good to outline a manual backup procedure in the event that parts of the system are already failing.

(stuart) #7

I’ve adapted the existing backup script to automatically upload the archives to Amazon S3 for storage. May be a good idea to make this a configuration option going forward (other cloud providers are available!)

(Trystan Lea) #8

That would be nice @stuart could you share how you do this?

I took a look at your pull request @borpin for removing hardcoded path’s in the backup module this morning and have been testing it both on my laptop and an emonpi. I made a few modifications to get it to work and have been improving a number of other aspects of the backup module more generally while Im at it (sidebar, automatic logout after import, error message if import file not selected). All rolled into a branch called ‘multienv’

Export and import are working on the pi
Export is working on my non-pi laptop install, but not yet a complete import.

Will continue working on it later this week and update with progress then.

(Brian Orpin) #9

No bother - it was just an initial stab. I’ve had a look and I have a few comments but I will put something together tonight and post to the PR. :smile:

(Brian Orpin) #10

I have updated this on my system and now it does not appear in the menu. Tried rebooting the machine, no improvement.

Nothing in the log.

(Trystan Lea) #11

aha, it needs the emoncms master branch, but you should still see a menu item under Setup > Backup

(Brian Orpin) #12

Yes I’m on master.

(stuart) #13

No problem, I’ll wait for you to merge all the changes in and then look at a pull request.

(Trystan Lea) #14

Import is now working fine for me on a non emonSD install. After applying the php upload config in the backup module readme, fixing the import and export script to read the emoncms database name from settings.php and only stopping and starting services if they exist.

Not sure about the menu visibility @borpin, here’s what it should look like:

The Backup entry appears top-left after selecting from the Setup menu.

(Trystan Lea) #15

I would really like a way to plug in a USB pen drive into the emonbase/emonpi and backup periodically to the pen drive. The backup module would need to scan for removable media and automatically mount disks on the pi, which could be complicated…?

… and perhaps stream the export rather than build the tar on the emonSD SD card. There are php libraries to export a compressed zip as a stream but I imagine a fair bit of work to implement.

(Trystan Lea) #16

The main steps for a manual backup are (this is essentially what the backup module script does):

  1. Export the emoncms mysql database, e.g on a emonSD install:

    mysqldump -u emoncms -p emoncms > emoncms.sql

password: emonpiemoncmsmysql2016

The mysql database contains emoncms meta data such as the user table, the list of feeds, feednames etc. If your using the standard emoncms feed engines there will be no timeseries feed data in the emoncms database.

  1. Copy the content of /var/lib/phpfina, phptimeseries & phpfiwa (if it exists) or /home/pi/data/phpfina… (location on emonSD installations). This is the feed data.

  2. Copy /home/pi/data/emonhub.conf

  3. Copy /var/www/emoncms/settings.php

Optional: manually create a tar.gz file to hold the backup, this is what the assembled backup should look like:

A backup with this structure can be imported with the backup module.

(Brian Orpin) #17

It occured to me how the services were being handled. Relates to this post I’ve just done, but there is a mixed environment out there right now.

I’ll have another go at pulling it in but your view is different to mine! Backup used to appear under setup.

Interestingly I do not see schedule either

drwxrwxr-x 15 www-data www-data 4096 Jan 15 17:20 .
drwxrwxr-x  9 www-data www-data 4096 Jan 17 16:30 ..
drwxrwxr-x  3 www-data www-data 4096 Jan 17 17:16 admin
drwxrwxr-x  7 www-data www-data 4096 Jan 11 22:44 app
lrwxrwxrwx  1 www-data www-data   20 Jan 15 17:20 backup -> /root/backup/backup/
drwxrwxr-x  7 www-data www-data 4096 Jan 17 16:13 dashboard
drwxrwxr-x  6 www-data www-data 4096 Dec 27 11:20 device
drwxrwxr-x  3 www-data www-data 4096 Oct 12 22:36 eventp
drwxrwxr-x  5 www-data www-data 4096 Jan 11 22:50 feed
drwxr-xr-x  6 www-data www-data 4096 Jan 11 22:44 graph
drwxrwxr-x  4 www-data www-data 4096 Jan 11 22:50 input
drwxrwxr-x  4 www-data www-data 4096 Oct 12 22:36 process
drwxrwxr-x  4 www-data www-data 4096 Oct 12 22:36 schedule
drwxrwxr-x  2 www-data www-data 4096 Oct 12 22:36 time
drwxrwxr-x  5 www-data www-data 4096 Oct 12 22:36 user
drwxrwxr-x  6 www-data www-data 4096 Oct 12 22:36 vis

Symlink folder not working on a Lighttpd setup
(Brian Orpin) #18

I solved the issue - it was permissions.

One thought - it would be worth loading the contents of the config file so you can see before you start what the config is. It could also do a few checks on the paths etc.

The main heading would be better named Backup & Restore - I’d actually call the sections the same instead of Export & Import.

Could this not be included in the emoncms ‘core’ modules, the scripts in the scripts folder and the settings etc pulled into the main settings.php file?

Now I can see the page I’ll sort out the settings and try to create the backup file. :smile:

(Brian Orpin) #19

@TrystanLea, I have tried it and it is silently failing - no errors in backup log or main log. Not even any friendly INFO entries. It is loading the config file.

I’ve tried to work out where it is failing without success.

I should say that I have had to put it in it’s own folder outside of a home directory but it is where $homedir points to.

The export script works fine when run from the commandline.

(Trystan Lea) #20

Thanks for trying @borpin, could it be session related? The backup module requires administration privileges it is currently hidden with any other level of access. You may need to disabled rememberme cookie based login if your using that…?