@TrystanLea
As an aside I mentioned using USB 3 flash drives for speed but only the RPi 4 has a USB 3 bus which can take full advantage and it is only part of the story.
Speed is in 2 parts:
The time taken by the OEM script to create the backup on the SDHC (takes about 24 mins in my case of which 10 mins is compressing the archive: compressed size = 333Mbytes)
The time taken to copy this compressed backup to the USB flash drive (takes about 3 mins and I’m NOT running an RPi4)
And yes – my backups progressively increase in number but doing it weekly reduces the problem.
I’m certainly going to try @icenov’s suggestion posted above in this thread. And it does seem smart to retain the last few backups be they daily or weekly.
Re complete vs incremental backups – my instinct is to keep it simple – use complete backups.
In my opinion, the current OEM script is great – simple to use and serving many purposes – transferring to another RPi, after a major emon image update, disaster recovery (assuming the SDHC is still readable), etc.
Provided as an option - copying that compressed archive to a USB flash drive would additionally benefit many users but not those few who are more comfortable with NAS, cloud, etc solutions.
(I’m using +23 because I want to retain the last 3 of my weekly backups)
This command can readily be incorporated into a script and run as a cronjob.
And if you use a USB flash drive as a second backup location, this command can also readily be adapted to remove old backups from that location.
PS: On a separate matter - would greatly appreciate any suggestions re my problem with DemandShaper/openEVSE - see my separate Forum post.
Sorry @johnbanks, “It worked but I did have to add a backslash before the final semi-colon …”
Not sure how the backslash was lost, but yes it is needed…
Edit - looks like it was there in the original post, but is removed when rendered. I’ve edited the post and put in the backticks which quote it correctly.
Just curious since (after my recent sd card degradation problem) i also want some sort of backup: the export script stops the feedwriter , but are the measurements during the backup lost, or are they written after the feedwriter restarts?
No. the whole point of feedwriter is it transfers data held in Memory (REDIS) to disk in one go at regular intervals -redused disk wear. While the backup is running, the feed data is stored in memory and written once feedwriter is restarted.
Actually I think the backup process could start feedwriter while the compression is running as the data file access has finished at that point. @TrystanLea.
I’d like to offer the backup method I’ve been using for the last year.
It backs up directly to a USB flash drive. I then from one of my other machines (MacOS) do an scp of the contents of the flash drive. On MacOS I then perform the tar and compress of the files. I figure this method saves SD card writes from the tar and compress processes.
The script is based on the /opt/emoncms/modules/backup/emoncms-export.sh script so the resulting archive is compatible with the /opt/emoncms/modules/backup/emoncms-import.sh script which I believe is used by the GUI.
The mods I have made include
Updated error habndliing
Used variable names from /opt/emoncms/modules/backup/config.cfg vs hard coded paths
Wasn’t backing up database dump
Wasn’t backing up phpfiwa (at the time)
Added progress messages
Modified the copy’s to preserve the original file timestamps.
The script doesn’t backup the node-red configuration as I don’t use node-red but it shouldn’t be hard to add.
Hope this is useful to someone.
Peter
Script below …
#!/bin/bash
# This script is based off the emoncms-export.sh script and copies the same files to an attached USB Flash drive.
# It assumes the Flash drive is mounted at the /media/usbdisk/backup mount point.
#
# To automate the process do the following:
#
# Issue the following command
# sudo mkdir -p /media/usbdisk/backup
#
# Add the following line to the /etc/fstab
#
# LABEL="BACKUP" /media/usbdisk/backup vfat defaults 0 2
#
# A vfat (msdos) formated Flash drive will be auto mounted under /media/usbdisk/backup if it is formatted with a label of BACKUP
#
# A crontab entry
#
# 00 17 * * * sudo /opt/emoncms/modules/backup/emoncms-copy.sh >>/var/log/emoncms/usbcopy.log 2>&1
#
script_location="$(dirname "$0")"
date=$(date +"%Y-%m-%d")
echo "=== Emoncms export start ==="
date
echo "Backup module version:"
cat $script_location/backup-module/module.json | grep version
echo "EUID: $EUID"
echo "Reading $script_location/config.cfg...."
if [ -f "$script_location/config.cfg" ]
then
source "$script_location/config.cfg"
echo "Location of databases: $database_path"
echo "Location of emonhub.conf: $emonhub_config_path"
echo "Location of Emoncms: $emoncms_location"
echo "Backup destination: $backup_location"
echo "Backup source path: $backup_source_path"
else
echo "ERROR: Backup $script_location/config.cfg file does not exist"
exit 1
sudo service feedwriter start > /dev/null
fi
backup_location="/media/usbdisk/backup"
echo "Backup destination: $backup_location"
#----------------------------------------------------------------------------------------
# Remove Old backup files
#----------------------------------------------------------------------------------------
if [ -f $backup_location/emoncms.sql ]
then
sudo rm $backup_location/emoncms.sql
fi
if [ -f $backup_location/emonhub.conf ]
then
sudo rm $backup_location/emonhub.conf
fi
if [ -f $backup_location/settings.ini ]
then
sudo rm $backup_location/settings.ini
fi
if [ -f $backup_location/settings.php ]
then
sudo rm $backup_location/settings.php
fi
if [ -d $backup_location/phpfina ]
then
sudo rm -r $backup_location/phpfina
fi
if [ -d $backup_location/phpfiwa ]
then
sudo rm -r $backup_location/phpfiwa
fi
if [ -d $backup_location/phptimeseries ]
then
sudo rm -r $backup_location/phptimeseries
fi
#----------------------------------------------------------------------------------------
# Check emonPi / emonBase image version
#----------------------------------------------------------------------------------------
image_version=$(ls /boot | grep emonSD)
# Check first 16 characters of filename
image_date=${image_version:0:16}
if [[ "${image_version:0:6}" == "emonSD" ]]
then
echo "Image version: $image_version"
fi
# Detect if SD card image verion, used to restore the correct emonhub.conf
if [[ "$image_date" == "emonSD-17Jun2015" ]]
then
image="old"
else
image="new"
fi
#----------------------------------------------------------------------------------------
sudo service feedwriter stop
# Get MYSQL authentication details from settings.php
if [ -f $script_location/get_emoncms_mysql_auth.php ]; then
auth=$(echo $emoncms_location | php $script_location/get_emoncms_mysql_auth.php php)
IFS=":" read username password database <<< "$auth"
else
echo "Error: cannot read MYSQL authentication details from Emoncms $script_location/get_emoncms_mysql_auth.php php & settings.php"
echo "$PWD"
sudo service feedwriter start > /dev/null
exit 1
fi
# MYSQL Dump Emoncms database
if [ -n "$username" ]; then # if username string is not empty
mysqldump -u$username -p$password $database > $backup_location/emoncms.sql
if [ $? -ne 0 ]; then
echo "Error: failed to export mysql data"
echo "emoncms export failed"
sudo service feedwriter start > /dev/null
exit 1
fi
else
echo "Error: Cannot read MYSQL authentication details from Emoncms settings.php"
sudo service feedwriter start > /dev/null
exit 1
fi
if [ -f $emonhub_config_path/emonhub.conf ]
then
echo "-- copying $emonhub_config_path/emonhub.conf to $backup_location --"
cp --preserve=mode,timestamps $emonhub_config_path/emonhub.conf $backup_location
else
echo "no file $emonhub_config_path/emonhub.conf"
fi
if [ -f $emoncms_location/settings.ini ]
then
echo "-- copying $emoncms_location/settings.ini to $backup_location --"
cp --preserve=mode,timestamps $emoncms_location/settings.ini $backup_location
else
echo "no file $emoncms_location/settings.ini"
fi
if [ -f $emoncms_location/settings.php ]
then
echo "-- copying $emoncms_location/settings.php to $backup_location --"
cp --preserve=mode,timestamps $emoncms_location/settings.php $backup_location
else
echo "no file $emoncms_location/settings.php"
fi
# cp --preserve=mode,timestamps $emoncms_config_path/emoncms.conf $backup_location
# cp --preserve=mode,timestamps /home/pi/data/node-red/flows_emonpi.json $backup_location
# cp --preserve=mode,timestamps /home/pi/data/node-red/settings.js $backup_location
if [ -d $database_path/phpfina ]
then
echo "-- adding $database_path/phpfina to $backup_location --"
cp --verbose -r --preserve=mode,timestamps $database_path/phpfina/. $backup_location/phpfina
if [ $? -ne 0 ]; then
echo "Error: failed to copy phpfina"
fi
else
echo "no phpfina directory"
fi
if [ -d $database_path/phpfiwa ]
then
echo "-- adding $database_path/phpfiwa to $backup_location--"
cp --verbose -r --preserve=mode,timestamps $database_path/phpfiwa/. $backup_location/phpfiwa
if [ $? -ne 0 ]; then
echo "Error: failed to copy phpfiwa"
fi
else
echo "no phpfiwa directory"
fi
if [ -d $database_path/phptimeseries ]
then
echo "-- adding $database_path/phptimeseries to $backup_location --"
cp --verbose -r --preserve=mode,timestamps $database_path/phptimeseries/. $backup_location/phptimeseries
if [ $? -ne 0 ]; then
echo "Error: failed to copy phptimeseries"
fi
else
echo "no phptimeseries directory $database_path/phptimeseries"
fi
sudo service feedwriter start > /dev/null
echo "Backup saved: $backup_location"
date
echo "=== Emoncms export complete! ===" # This string is identified in the interface to stop ongoing AJAX calls, please ammend in interface if changed here
exit
I know I’m late to the party but I also wanted to have a scheduled backup for my data. So I have tried to make it use friendly.
I have created a fork of the backup module where I have added the ability to mount a disk partition or NFS share over the backup folder and send backups there instead. It includes the command to cleanup files older than 8 days at the end of the export script.
I have also added the ability to enable or disable a daily cron a backup.
It doesn’t appear to have any conflicts with the current master branch, so I suggest you create a PR for it.
It will probably prevent an Emoncms Update of the backup module, (as Git will see the folder is different) but I can’t be sure without digging through the update script.
To use it you need to change the git origin to point to a different (this) repo.
I took multiple things from this topic and sync / backup to my one drive account using rclone.
The main part you need to get backups into the cloud is rclone it supports a wide variety of remotes
I installed rclone
sudo apt-get install rclone
and configured my onedrive using (the reason i’m using sudo is so that the config is saved for the root user as the script i’ll make later is ran as root as well from cron)
sudo rclone config
Setting up the onedrive part is a bit of a pain as you need to select rights and so on and you also need another pc with gui that can load the browser. i used a vmware ubuntu image where i manually installed rclone and provided client secret and client values when requested the same ones as used on emonpi but i selected there the option that i’m using a headless server (so no gui) and then you basically need to to run a certain command on another pc with rclone and a gui / brower installed to login to the new app you created for onedrive for rclone. But once you get passed that it does work correctly ( see rclone Microsoft OneDrive for more information)
then i created a backup.sh that first runs the command to cleanup left over backup files (from 3 days old), run the normal backup export script and then run rclone to sync the backup folder to my onedrive in emoncms folder. The sync option will not upload the same files again and will also delete backups you had deleted in the local backup folder. If you want to keep all backups (in onedrive, not locally) you could do an rclone copy instead of rclone sync as it won’t delete files then. the onedrive: part is the name i gave my onedrive remote in the config part
then i ran sudo crontab -e selected nano as editor and added the following to run the script at midnight + 1 minute my local time (which is 23:01 on emonpi i’m using)
it just ran for the 1st time today and seemed to have gone fine but i’ll need to check tommorow and day after tommorow to be sure its working correctly (including deleting older backups etc).
The “Create backup schedule” button doesn’t do anything. A subheading shows “(daily 6:25)” but it doesn’t run then and “Delete backup schedule” doesn’t clear it and/or allow me to use the “Create backup schedule” button. “Create backup” still works but has to be launched manually.
