Is there an easy way to do an automatic daily backup? I have seen some topics trying to do it, but I haven’t seen an easy solution. I’m looking for a backup of the emoncms data and the node-red configuration.
This is not completely what you are after, but might get you on the way, if you are running a local version of emoncms and using a browser to interact. I’m not sure if you are familiar with linux command line - but it’s quite straightforward. You need to be able to ssh into your server to do it.
There is an emoncms-export.sh script which can be manually run from the Backup tab on the emoncms. On my sytstem (low-write 10.2.6) it is located in /opt/emoncms/modules/backup. This exports all the data as described in the Export tab of the Backup page. You can change the backup locations etc if you want by editing this file.
You can run an automated backup using cron (type crontab -e on the command line). For example, I run this script at 11:15 pm each night by adding this line into my crontab:
15 23 * * * /opt/emoncms/modules/backup/emoncms-export.sh
Of course this will accumulate backups unless you manually delete them, but it is easy to setup another cron job to just delete old backups. This command will find and delete backups (actually any files with a .gz extension in this case) that are older than 8 days (the +8)
To run this in cron, put it into a text file named like delete_backups.sh and make give it permission to run (chmod a+x delete_backups.sh).
If you aren’t familiar with cron check out https://crontab.guru/, where you can easily confirm if your schedule entries do what you expect.
This approach only backs up locally, so if you lose your SD drive you will also lose the backups. I use a remote unix backup server called rsync.net (about $US10/month for 400 GB storage - for all my backups, not just emoncms). This uses a similar cron approach, and backs up from my emon rpi every day. If you are interested I can post the rsync command - it’s very simple actually.
I run node-red on a different rpi, and backup this up using the same approach. All auto, just an occasional check to make sure all is being backed up as expected.
Yes, USB backup is also good!
I have the rsync cloud account so can take advantage of it for no extra cost.
my cron entry is:
rsync -avz /var/opt/emoncms/backup [email protected]:backup_emonpi
This copies everything in my local emoncms backup directory to a remote directory called emonpi.
If you want completely automatic remote backups, you need to upload your local public ssh-key to your account so it doesn’t ask for a password.
why can we not have native automatic backups to a USB drive? A 64GB is < £10
Data loss is a massive problem and not having a built in process has caused me to lose data. we should not need to write custom cron jobs. it sould be as simple as plugging a USB stick in going to the admin area and selecting the USB and a time/date to run the backup.
Whilst it might appear trivial, when you get into the guts of designing something for the myriad of different combinations in use, it gets quite a bit more complicated. Remounting USB drives on reboot correctly is, for instance, not easy.
I have to admit I’ve not had a lot of success with mounted drives, either usb (stick) or usb mechanical drives attached to a pi. Even network connected drives have not been reliable for me - largely due to a rogue router deciding to change IP addresses even though they were fixed. Inevitably there are also system updates etc that often override settings or move files that tend to break backups.
Rsync seems very robust and as long as there is a wifi connection it seems to work.
Thinking out aloud - I wonder if there might be a revenue stream here for Megni/OEM to offer a remote backup service?
I think keeping it simple to just USB would be a start.
As for finding the USB drive. when the backup is setup just look for a file on the root of the USB or even get the end user to create a file called “backup” on the USB much like the ssh file needed on the root of the SD card.
I do like the idea of a nice automated backup to USB drive, that would definitely be a nice feature.
I currently use the sync module to backup to an emoncms installation that I have installed on my laptop (ubuntu OS) I appreciate this is not exactly a simple solution… I haven’t needed to use it to restore a system yet, the USB importer tool has worked a treat alongside the fsck commands when something does go wrong. Though Im aware of @promy’s recent experience with a couple of feeds that tripped up restoring using this approach Running but corrupted emonpi - #6 by promy, perhaps the importer script could be more robust and skip the corrupted feeds somehow
Whilst this is not an automated backup solution to USB, it may be a starting point?
I’m running the July 2020 emon image updated to v10.2.6 on a Raspberry Pi.
Thanks to you OEM guys there is an export backup script that can be run from the Admin webpage or from an SSH terminal with the command …
This saves the backup to the SDHC.
I’ve taken it a bit further by then copying the backup to a USB flash drive. Excuse any verbosity as I’ve just copied & pasted from my Build Notes
A 32Gb or 64Gb flash drive is suitable but it should be USB 3.0 spec for the write speed.
The first task is to prepare the USB flash drive and to ensure it is correctly remounted at every reboot …
Do sudo mkdir /mnt/BACKUP
Do sudo chmod -R 777 /mnt/BACKUP … to add full write permissions
Plug in the USB Flash Drive
Do sudo blkid … which will identify the USB Flash Drive as being /dev/sda1
It is necessary to use the UUID=xxxxxxxxx form of drive identification and so make a copy note of UUID=xxxxxxxxx
Now create a filesystem on the USB Flash Drive …
Do sudo mkfs -t ext4 /dev/sda1
Then check it with sudo fsck /dev/sda1
Now make changes to /etc/fstab so the USB Flash Drive filesystem is mounted at start up/reboot … must use the UUID identification …
Copy the original fstab into the first (/boot) partition of the SDHC (as a precaution) …
sudo cp /etc/fstab /boot/fstab.ORIGINAL then …
sudo cp /boot/fstab.ORIGINAL /boot/fstab.NEW ready for some changes …
Do sudo nano /boot/fstab.NEW and add the following line …
UUID=xxxxxxxxx /mnt/BACKUP ext4 defaults,noatime,nodiratime 0 2
Save & exit
Do sudo cp /boot/fstab.NEW /etc/fstab
And then sudo reboot
After the reboot, df -hT should reveal the USB Flash Drive is mounted
Admittedly that’s all a bit of a pain but you only have to do it once.
Finally, create the enhanced backup script which makes an actual Backup on the SDHC and then copies it to the USB Flash Drive
Do sudo nano /home/pi/regular.backup.sh
Then enter the following and save/exit …
sudo rsync -axv /var/opt/emoncms/ /mnt/BACKUP
Make the script executable with …
sudo chmod a+x /home/pi/regular.backup.sh
You can now do a backup at any time with …
Or the script can be incorporated into a cronjob.
To set up a cronjob which runs at 1 minute after midnight each Sunday:
Do crontab -e and add the following and then save/exit …
1 0 * * sun /home/pi/regular.backup.sh
The script will not survive a major emon image update.
It’s been running successfully for me for months together with an enhancement – sends an email to say each backup successfully happened.
And I’ve never yet needed to use the backup because of an SDHC failure – famous last words?
This may be of interest.