Here’s one way to solve that problem.
NOT saying implementing a backup solution is easy, just addressing the USB drive part of the problem
as that info in and of itself might prove useful for other projects.
1 Like
But how do you deploy that on the SD Image for some people to setup is my point?
I didn’t say it couldn’t be done, I said it wasn’t easy to deploy for all.
Some might want to go to USB, some a NAS some Google Drive etc etc etc.
Um, I believe I said just that. 
Rather that the
1 Like
I have to admit I’ve not had a lot of success with mounted drives, either usb (stick) or usb mechanical drives attached to a pi. Even network connected drives have not been reliable for me - largely due to a rogue router deciding to change IP addresses even though they were fixed. Inevitably there are also system updates etc that often override settings or move files that tend to break backups.
Rsync seems very robust and as long as there is a wifi connection it seems to work.
Thinking out aloud - I wonder if there might be a revenue stream here for Megni/OEM to offer a remote backup service?
Unfortunally my programming skills are 0
I think keeping it simple to just USB would be a start.
As for finding the USB drive. when the backup is setup just look for a file on the root of the USB or even get the end user to create a file called “backup” on the USB much like the ssh file needed on the root of the SD card.
1 Like
I do like the idea of a nice automated backup to USB drive, that would definitely be a nice feature.
I currently use the sync module to backup to an emoncms installation that I have installed on my laptop (ubuntu OS) I appreciate this is not exactly a simple solution… I haven’t needed to use it to restore a system yet, the USB importer tool has worked a treat alongside the fsck commands when something does go wrong. Though Im aware of @promy’s recent experience with a couple of feeds that tripped up restoring using this approach Running but corrupted emonpi - #6 by promy, perhaps the importer script could be more robust and skip the corrupted feeds somehow
Whilst this is not an automated backup solution to USB, it may be a starting point?
I’m running the July 2020 emon image updated to v10.2.6 on a Raspberry Pi.
Thanks to you OEM guys there is an export backup script that can be run from the Admin webpage or from an SSH terminal with the command …
./opt/emoncms/modules/backup/emoncms-export.sh
This saves the backup to the SDHC.
I’ve taken it a bit further by then copying the backup to a USB flash drive. Excuse any verbosity as I’ve just copied & pasted from my Build Notes
A 32Gb or 64Gb flash drive is suitable but it should be USB 3.0 spec for the write speed.
The first task is to prepare the USB flash drive and to ensure it is correctly remounted at every reboot …
Do sudo mkdir /mnt/BACKUP
Do sudo chmod -R 777 /mnt/BACKUP … to add full write permissions
Plug in the USB Flash Drive
Do sudo blkid … which will identify the USB Flash Drive as being /dev/sda1
It is necessary to use the UUID=xxxxxxxxx form of drive identification and so make a copy note of UUID=xxxxxxxxx
Now create a filesystem on the USB Flash Drive …
Do sudo mkfs -t ext4 /dev/sda1
Then check it with sudo fsck /dev/sda1
Now make changes to /etc/fstab so the USB Flash Drive filesystem is mounted at start up/reboot … must use the UUID identification …
Copy the original fstab into the first (/boot) partition of the SDHC (as a precaution) …
sudo cp /etc/fstab /boot/fstab.ORIGINAL then …
sudo cp /boot/fstab.ORIGINAL /boot/fstab.NEW ready for some changes …
Do sudo nano /boot/fstab.NEW and add the following line …
UUID=xxxxxxxxx /mnt/BACKUP ext4 defaults,noatime,nodiratime 0 2
Save & exit
Finally …
Do sudo cp /boot/fstab.NEW /etc/fstab
And then sudo reboot
After the reboot, df -hT should reveal the USB Flash Drive is mounted
Admittedly that’s all a bit of a pain but you only have to do it once.
Finally, create the enhanced backup script which makes an actual Backup on the SDHC and then copies it to the USB Flash Drive
Do sudo nano /home/pi/regular.backup.sh
Then enter the following and save/exit …
#!/bin/bash
cd /opt/emoncms/modules/backup
./emoncms-export.sh
sleep 5
sudo rsync -axv /var/opt/emoncms/ /mnt/BACKUP
exit
Make the script executable with …
sudo chmod a+x /home/pi/regular.backup.sh
You can now do a backup at any time with …
./home/pi/regular.backup.sh
Or the script can be incorporated into a cronjob.
To set up a cronjob which runs at 1 minute after midnight each Sunday:
Do crontab -e and add the following and then save/exit …
#
1 0 * * sun /home/pi/regular.backup.sh
The script will not survive a major emon image update.
It’s been running successfully for me for months together with an enhancement – sends an email to say each backup successfully happened.
And I’ve never yet needed to use the backup because of an SDHC failure – famous last words?
This may be of interest.
Is there any reason you wouldn’t want to keep it on, and run it from, the USB drive?
@Bill.Thomson
No reason at all. That’s a good idea.
Copy the script to the USB flash drive and run it from there - and change the crontab entry accordngly.
Thanks for sharing @johnbanks looks good!
How often do you remove old backups, looks like it adds a new backup file every day?
If we were to include something like your solution as standard, how important is it to have a complete backup in a compressed archive vs an incremental backup that just transfers over the latest data?
@TrystanLea
As an aside I mentioned using USB 3 flash drives for speed but only the RPi 4 has a USB 3 bus which can take full advantage and it is only part of the story.
Speed is in 2 parts:
- The time taken by the OEM script to create the backup on the SDHC (takes about 24 mins in my case of which 10 mins is compressing the archive: compressed size = 333Mbytes)
- The time taken to copy this compressed backup to the USB flash drive (takes about 3 mins and I’m NOT running an RPi4)
And yes – my backups progressively increase in number but doing it weekly reduces the problem.
I’m certainly going to try @icenov’s suggestion posted above in this thread. And it does seem smart to retain the last few backups be they daily or weekly.
Re complete vs incremental backups – my instinct is to keep it simple – use complete backups.
In my opinion, the current OEM script is great – simple to use and serving many purposes – transferring to another RPi, after a major emon image update, disaster recovery (assuming the SDHC is still readable), etc.
Provided as an option - copying that compressed archive to a USB flash drive would additionally benefit many users but not those few who are more comfortable with NAS, cloud, etc solutions.
1 Like
Thx
I’ve now tried @icenov’s suggestion to remove old backups.
It worked but I did have to add a backslash before the final semi-colon …
find /var/opt/emoncms/backup/ -name '*.gz' -type f -mtime +23 -exec rm -f {} \;
(I’m using +23 because I want to retain the last 3 of my weekly backups)
This command can readily be incorporated into a script and run as a cronjob.
And if you use a USB flash drive as a second backup location, this command can also readily be adapted to remove old backups from that location.
PS: On a separate matter - would greatly appreciate any suggestions re my problem with DemandShaper/openEVSE - see my separate Forum post.
Sorry @johnbanks, “It worked but I did have to add a backslash before the final semi-colon …”
Not sure how the backslash was lost, but yes it is needed…
Edit - looks like it was there in the original post, but is removed when rendered. I’ve edited the post and put in the backticks which quote it correctly.
1 Like
I’ve changed it a little bit:
find /var/opt/emoncms/backup/ -name *.gz -type f -mtime +8 -execdir rm – ‘{}’ ;
Just curious since (after my recent sd card degradation problem) i also want some sort of backup: the export script stops the feedwriter , but are the measurements during the backup lost, or are they written after the feedwriter restarts?
No. the whole point of feedwriter is it transfers data held in Memory (REDIS) to disk in one go at regular intervals -redused disk wear. While the backup is running, the feed data is stored in memory and written once feedwriter is restarted.
Actually I think the backup process could start feedwriter while the compression is running as the data file access has finished at that point. @TrystanLea.
I’d like to offer the backup method I’ve been using for the last year.
It backs up directly to a USB flash drive. I then from one of my other machines (MacOS) do an scp of the contents of the flash drive. On MacOS I then perform the tar and compress of the files. I figure this method saves SD card writes from the tar and compress processes.
The script is based on the /opt/emoncms/modules/backup/emoncms-export.sh script so the resulting archive is compatible with the /opt/emoncms/modules/backup/emoncms-import.sh script which I believe is used by the GUI.
The mods I have made include
- Updated error habndliing
- Used variable names from /opt/emoncms/modules/backup/config.cfg vs hard coded paths
- Wasn’t backing up database dump
- Wasn’t backing up phpfiwa (at the time)
- Added progress messages
- Modified the copy’s to preserve the original file timestamps.
The script doesn’t backup the node-red configuration as I don’t use node-red but it shouldn’t be hard to add.
Hope this is useful to someone.
Peter
Script below …
#!/bin/bash
# This script is based off the emoncms-export.sh script and copies the same files to an attached USB Flash drive.
# It assumes the Flash drive is mounted at the /media/usbdisk/backup mount point.
#
# To automate the process do the following:
#
# Issue the following command
# sudo mkdir -p /media/usbdisk/backup
#
# Add the following line to the /etc/fstab
#
# LABEL="BACKUP" /media/usbdisk/backup vfat defaults 0 2
#
# A vfat (msdos) formated Flash drive will be auto mounted under /media/usbdisk/backup if it is formatted with a label of BACKUP
#
# A crontab entry
#
# 00 17 * * * sudo /opt/emoncms/modules/backup/emoncms-copy.sh >>/var/log/emoncms/usbcopy.log 2>&1
#
script_location="$(dirname "$0")"
date=$(date +"%Y-%m-%d")
echo "=== Emoncms export start ==="
date
echo "Backup module version:"
cat $script_location/backup-module/module.json | grep version
echo "EUID: $EUID"
echo "Reading $script_location/config.cfg...."
if [ -f "$script_location/config.cfg" ]
then
source "$script_location/config.cfg"
echo "Location of databases: $database_path"
echo "Location of emonhub.conf: $emonhub_config_path"
echo "Location of Emoncms: $emoncms_location"
echo "Backup destination: $backup_location"
echo "Backup source path: $backup_source_path"
else
echo "ERROR: Backup $script_location/config.cfg file does not exist"
exit 1
sudo service feedwriter start > /dev/null
fi
backup_location="/media/usbdisk/backup"
echo "Backup destination: $backup_location"
#----------------------------------------------------------------------------------------
# Remove Old backup files
#----------------------------------------------------------------------------------------
if [ -f $backup_location/emoncms.sql ]
then
sudo rm $backup_location/emoncms.sql
fi
if [ -f $backup_location/emonhub.conf ]
then
sudo rm $backup_location/emonhub.conf
fi
if [ -f $backup_location/settings.ini ]
then
sudo rm $backup_location/settings.ini
fi
if [ -f $backup_location/settings.php ]
then
sudo rm $backup_location/settings.php
fi
if [ -d $backup_location/phpfina ]
then
sudo rm -r $backup_location/phpfina
fi
if [ -d $backup_location/phpfiwa ]
then
sudo rm -r $backup_location/phpfiwa
fi
if [ -d $backup_location/phptimeseries ]
then
sudo rm -r $backup_location/phptimeseries
fi
#----------------------------------------------------------------------------------------
# Check emonPi / emonBase image version
#----------------------------------------------------------------------------------------
image_version=$(ls /boot | grep emonSD)
# Check first 16 characters of filename
image_date=${image_version:0:16}
if [[ "${image_version:0:6}" == "emonSD" ]]
then
echo "Image version: $image_version"
fi
# Detect if SD card image verion, used to restore the correct emonhub.conf
if [[ "$image_date" == "emonSD-17Jun2015" ]]
then
image="old"
else
image="new"
fi
#----------------------------------------------------------------------------------------
sudo service feedwriter stop
# Get MYSQL authentication details from settings.php
if [ -f $script_location/get_emoncms_mysql_auth.php ]; then
auth=$(echo $emoncms_location | php $script_location/get_emoncms_mysql_auth.php php)
IFS=":" read username password database <<< "$auth"
else
echo "Error: cannot read MYSQL authentication details from Emoncms $script_location/get_emoncms_mysql_auth.php php & settings.php"
echo "$PWD"
sudo service feedwriter start > /dev/null
exit 1
fi
# MYSQL Dump Emoncms database
if [ -n "$username" ]; then # if username string is not empty
mysqldump -u$username -p$password $database > $backup_location/emoncms.sql
if [ $? -ne 0 ]; then
echo "Error: failed to export mysql data"
echo "emoncms export failed"
sudo service feedwriter start > /dev/null
exit 1
fi
else
echo "Error: Cannot read MYSQL authentication details from Emoncms settings.php"
sudo service feedwriter start > /dev/null
exit 1
fi
if [ -f $emonhub_config_path/emonhub.conf ]
then
echo "-- copying $emonhub_config_path/emonhub.conf to $backup_location --"
cp --preserve=mode,timestamps $emonhub_config_path/emonhub.conf $backup_location
else
echo "no file $emonhub_config_path/emonhub.conf"
fi
if [ -f $emoncms_location/settings.ini ]
then
echo "-- copying $emoncms_location/settings.ini to $backup_location --"
cp --preserve=mode,timestamps $emoncms_location/settings.ini $backup_location
else
echo "no file $emoncms_location/settings.ini"
fi
if [ -f $emoncms_location/settings.php ]
then
echo "-- copying $emoncms_location/settings.php to $backup_location --"
cp --preserve=mode,timestamps $emoncms_location/settings.php $backup_location
else
echo "no file $emoncms_location/settings.php"
fi
# cp --preserve=mode,timestamps $emoncms_config_path/emoncms.conf $backup_location
# cp --preserve=mode,timestamps /home/pi/data/node-red/flows_emonpi.json $backup_location
# cp --preserve=mode,timestamps /home/pi/data/node-red/settings.js $backup_location
if [ -d $database_path/phpfina ]
then
echo "-- adding $database_path/phpfina to $backup_location --"
cp --verbose -r --preserve=mode,timestamps $database_path/phpfina/. $backup_location/phpfina
if [ $? -ne 0 ]; then
echo "Error: failed to copy phpfina"
fi
else
echo "no phpfina directory"
fi
if [ -d $database_path/phpfiwa ]
then
echo "-- adding $database_path/phpfiwa to $backup_location--"
cp --verbose -r --preserve=mode,timestamps $database_path/phpfiwa/. $backup_location/phpfiwa
if [ $? -ne 0 ]; then
echo "Error: failed to copy phpfiwa"
fi
else
echo "no phpfiwa directory"
fi
if [ -d $database_path/phptimeseries ]
then
echo "-- adding $database_path/phptimeseries to $backup_location --"
cp --verbose -r --preserve=mode,timestamps $database_path/phptimeseries/. $backup_location/phptimeseries
if [ $? -ne 0 ]; then
echo "Error: failed to copy phptimeseries"
fi
else
echo "no phptimeseries directory $database_path/phptimeseries"
fi
sudo service feedwriter start > /dev/null
echo "Backup saved: $backup_location"
date
echo "=== Emoncms export complete! ===" # This string is identified in the interface to stop ongoing AJAX calls, please ammend in interface if changed here
exit
I probably should have mentioned that I place this script in
/opt/emoncms/modules/backup/emoncms-copy.sh