Is there an easy way to do an automatic daily backup? I have seen some topics trying to do it, but I haven’t seen an easy solution. I’m looking for a backup of the emoncms data and the node-red configuration.
Maybe the easiest way to do it is the dropbox-archive script (GitHub - Paul-Reed/dropbox-archive: Upload emoncms data to Dropbox) but it’s not exactly what I’m looking for.
Thanks
Hello @poldom there isn’t an integrated way of doing this, but I think there are members on the forum here doing this @borpin?
No, sorry, my main system is on Proxmox so I use that to backup the container nightly.
This is not completely what you are after, but might get you on the way, if you are running a local version of emoncms and using a browser to interact. I’m not sure if you are familiar with linux command line - but it’s quite straightforward. You need to be able to ssh into your server to do it.
There is an emoncms-export.sh script which can be manually run from the Backup tab on the emoncms. On my sytstem (low-write 10.2.6) it is located in /opt/emoncms/modules/backup. This exports all the data as described in the Export tab of the Backup page. You can change the backup locations etc if you want by editing this file.
You can run an automated backup using cron (type crontab -e on the command line). For example, I run this script at 11:15 pm each night by adding this line into my crontab:
15 23 * * * /opt/emoncms/modules/backup/emoncms-export.sh
Of course this will accumulate backups unless you manually delete them, but it is easy to setup another cron job to just delete old backups. This command will find and delete backups (actually any files with a .gz extension in this case) that are older than 8 days (the +8)
find /var/opt/emoncms/backup/ -name '*.gz' -type f -mtime +8 -exec rm -f {} \;
To run this in cron, put it into a text file named like delete_backups.sh and make give it permission to run (chmod a+x delete_backups.sh).
If you aren’t familiar with cron check out https://crontab.guru/, where you can easily confirm if your schedule entries do what you expect.
This approach only backs up locally, so if you lose your SD drive you will also lose the backups. I use a remote unix backup server called rsync.net (about $US10/month for 400 GB storage - for all my backups, not just emoncms). This uses a similar cron approach, and backs up from my emon rpi every day. If you are interested I can post the rsync command - it’s very simple actually.
I run node-red on a different rpi, and backup this up using the same approach. All auto, just an occasional check to make sure all is being backed up as expected.
Thanks for all your answers.
@icenov, I’m used to linux commands, and I use rsync for other servers I have. Thanks for all your detailed explanation, it’ll be really helpfull, I didn’t know there was the emoncms-export script.
I would be interested in the rsync process if you would provide details.
@ian
If you wish to avoid cloud storage …
Alt 1 - Insert a USB flash drive into the Rasberry Pi and copy the daily export to it. This will be usable if the SD card fails.
Alt 2 - Have a second Raspberry Pi on the local network also running the emon software. Post data to it from the first Raspberry Pi using the HTPPInterfacer.
Yes, USB backup is also good!
I have the rsync cloud account so can take advantage of it for no extra cost.
my cron entry is:
rsync -avz /var/opt/emoncms/backup [email protected]:backup_emonpi
This copies everything in my local emoncms backup directory to a remote directory called emonpi.
If you want completely automatic remote backups, you need to upload your local public ssh-key to your account so it doesn’t ask for a password.
why can we not have native automatic backups to a USB drive? A 64GB is < £10
Data loss is a massive problem and not having a built in process has caused me to lose data. we should not need to write custom cron jobs. it sould be as simple as plugging a USB stick in going to the admin area and selecting the USB and a time/date to run the backup.
Contributions are always welcome…
Whilst it might appear trivial, when you get into the guts of designing something for the myriad of different combinations in use, it gets quite a bit more complicated. Remounting USB drives on reboot correctly is, for instance, not easy.
Here’s one way to solve that problem.
NOT saying implementing a backup solution is easy, just addressing the USB drive part of the problem
as that info in and of itself might prove useful for other projects.
But how do you deploy that on the SD Image for some people to setup is my point?
I didn’t say it couldn’t be done, I said it wasn’t easy to deploy for all.
Some might want to go to USB, some a NAS some Google Drive etc etc etc.
Um, I believe I said just that. 
Rather that the
I have to admit I’ve not had a lot of success with mounted drives, either usb (stick) or usb mechanical drives attached to a pi. Even network connected drives have not been reliable for me - largely due to a rogue router deciding to change IP addresses even though they were fixed. Inevitably there are also system updates etc that often override settings or move files that tend to break backups.
Rsync seems very robust and as long as there is a wifi connection it seems to work.
Thinking out aloud - I wonder if there might be a revenue stream here for Megni/OEM to offer a remote backup service?
Unfortunally my programming skills are 0
I think keeping it simple to just USB would be a start.
As for finding the USB drive. when the backup is setup just look for a file on the root of the USB or even get the end user to create a file called “backup” on the USB much like the ssh file needed on the root of the SD card.
I do like the idea of a nice automated backup to USB drive, that would definitely be a nice feature.
I currently use the sync module to backup to an emoncms installation that I have installed on my laptop (ubuntu OS) I appreciate this is not exactly a simple solution… I haven’t needed to use it to restore a system yet, the USB importer tool has worked a treat alongside the fsck commands when something does go wrong. Though Im aware of @promy’s recent experience with a couple of feeds that tripped up restoring using this approach Running but corrupted emonpi - #6 by promy, perhaps the importer script could be more robust and skip the corrupted feeds somehow
Whilst this is not an automated backup solution to USB, it may be a starting point?
I’m running the July 2020 emon image updated to v10.2.6 on a Raspberry Pi.
Thanks to you OEM guys there is an export backup script that can be run from the Admin webpage or from an SSH terminal with the command …
./opt/emoncms/modules/backup/emoncms-export.sh
This saves the backup to the SDHC.
I’ve taken it a bit further by then copying the backup to a USB flash drive. Excuse any verbosity as I’ve just copied & pasted from my Build Notes
A 32Gb or 64Gb flash drive is suitable but it should be USB 3.0 spec for the write speed.
The first task is to prepare the USB flash drive and to ensure it is correctly remounted at every reboot …
Do sudo mkdir /mnt/BACKUP
Do sudo chmod -R 777 /mnt/BACKUP … to add full write permissions
Plug in the USB Flash Drive
Do sudo blkid … which will identify the USB Flash Drive as being /dev/sda1
It is necessary to use the UUID=xxxxxxxxx form of drive identification and so make a copy note of UUID=xxxxxxxxx
Now create a filesystem on the USB Flash Drive …
Do sudo mkfs -t ext4 /dev/sda1
Then check it with sudo fsck /dev/sda1
Now make changes to /etc/fstab so the USB Flash Drive filesystem is mounted at start up/reboot … must use the UUID identification …
Copy the original fstab into the first (/boot) partition of the SDHC (as a precaution) …
sudo cp /etc/fstab /boot/fstab.ORIGINAL then …
sudo cp /boot/fstab.ORIGINAL /boot/fstab.NEW ready for some changes …
Do sudo nano /boot/fstab.NEW and add the following line …
UUID=xxxxxxxxx /mnt/BACKUP ext4 defaults,noatime,nodiratime 0 2
Save & exit
Finally …
Do sudo cp /boot/fstab.NEW /etc/fstab
And then sudo reboot
After the reboot, df -hT should reveal the USB Flash Drive is mounted
Admittedly that’s all a bit of a pain but you only have to do it once.
Finally, create the enhanced backup script which makes an actual Backup on the SDHC and then copies it to the USB Flash Drive
Do sudo nano /home/pi/regular.backup.sh
Then enter the following and save/exit …
#!/bin/bash
cd /opt/emoncms/modules/backup
./emoncms-export.sh
sleep 5
sudo rsync -axv /var/opt/emoncms/ /mnt/BACKUP
exit
Make the script executable with …
sudo chmod a+x /home/pi/regular.backup.sh
You can now do a backup at any time with …
./home/pi/regular.backup.sh
Or the script can be incorporated into a cronjob.
To set up a cronjob which runs at 1 minute after midnight each Sunday:
Do crontab -e and add the following and then save/exit …
#
1 0 * * sun /home/pi/regular.backup.sh
The script will not survive a major emon image update.
It’s been running successfully for me for months together with an enhancement – sends an email to say each backup successfully happened.
And I’ve never yet needed to use the backup because of an SDHC failure – famous last words?
This may be of interest.
Is there any reason you wouldn’t want to keep it on, and run it from, the USB drive?
@Bill.Thomson
No reason at all. That’s a good idea.
Copy the script to the USB flash drive and run it from there - and change the crontab entry accordngly.
Thanks for sharing @johnbanks looks good!
How often do you remove old backups, looks like it adds a new backup file every day?
If we were to include something like your solution as standard, how important is it to have a complete backup in a compressed archive vs an incremental backup that just transfers over the latest data?