Community
OpenEnergyMonitor

Community

emonCMS backup example to NAS network drive

Tags: #<Tag:0x00007f6e059b52b0>

I got a local network drive (NAS) backup method working, thought to share.

1. Using emoncms-export.sh

This is the script that runs when using the backup module in the emoncms browser interface.

Set up.
$ sudo nano /etc/fstab
modifies linux mount points, I added a line to mount the network drive:

192.168.1.235:/media/astralship/nas/emoncms-backup /mnt/nas nfs async,rw,proto=udp 0 0

This can look very different depending on the system!

I then remounted the drives listed in fstab:
$ sudo mount -a
I had an issue that the drive was not mounted on rPi reboot. This was a problem with systemd doing things in the wrong order. I fixed it following a guide, but ended up installing a sudo cronjob anyway:

$ sudo crontab -e
I added the line:
0 23 * * * mount -a making sure to mount all drives at 11pm each day.

Running backup
I wanted to have a minimal backup situation, where only the most recent backup file was copied to the network drive, IF it was newer than the NAS backup.

First, I test the backup shell script emoncms-export.sh

$ /home/pi/backup/emoncms-export.sh

It’s not backing up to a tidy folder called ‘backups’, so I change ~/backup/config.cfg finding the lines:

Destination location of exported backup .tar.gz - Must be writable
backup_location="/home/pi/data"

simply creating a new folder and changing it to:

backup_location="/home/pi/data/backups"

At this point I consider pointing this directly to the NAS, but I want a local AND remote backup.

I add the backup script to cron:

$ crontab -e

adding line

0 0 * * * /home/pi/backup/emoncms-export.sh

Running the script at midnight.

Then after a good hour of bashing away and searching the internet I came up with this script:

if [ $(ls -t ~/data/backups/emoncms-backup-* | head -1) -nt $(ls -t /mnt/nas/backups/backup* | head -1) ];
then
   cp $(ls -t ~/data/backups/emoncms-backup-* | head -1) /mnt/nas/backups/backup.tar.gz;
fi;

Here I have the conditional if, the operator newer than -nt, then it’s copied to the NAS file called backup.tar.gz, of course, overwritten if it already exists.

I install the cronjob to run one hour after the local backup, midnight again.

$ crontab -e

adding another line after the previous:

0 0 * * * if [ $(ls -t ~/data/backups/emoncms-backup-* | head -1) -nt $(ls -t /mnt/nas/backups/backup* | head -1) ]; then cp $(ls -t ~/data/backups/emoncms-backup-* | head -1) /mnt/nas/backups/backup.tar.gz; fi

The alternative method is rsync, which only copies files if modified. rsync is a useful tool, but I was getting a lot of permission errors and it looked ugly. Hence making the above script.
Here’s an rsync example crontab:

0 0 * * * rsync -azu /home/pi/data/backups /mnt/nas/backups

Cleanup

I delete old local backups with:

rm $(find /home/pi/data/backups/emoncms* -mtime +7)

Removing backups older than 7 days… and I add it to crontab at midnight.

0 0 * * * rm $(find /home/pi/data/backups/emoncms* -mtime +7)


Comments welcome.

P.S. Thank you stackoverflow.com

2. Running sudo php backup.php from usefulscripts.

todo:

  • Creating the dependent folders in the target backup directory.
  • Editing the backup.php file.
  • Observing the incremental nature of the backup, saving disk space and disk activity.

backup.php.zip (1.6 KB)

2 Likes

Thanks @danbates. I was looking for a way to auto run and store backups on a network drive and your approach helped sort it! I have a linux NFS shared drive and changed the emoncms backup destination to it. I decided against the conditional IF script, but now run a cron job on the server that deletes any backups older than a setting (eg 2 days). The only thing I need to keep an eye on is that the shared drive properly remounts on the rPi after a reboot/power fail!

I’m glad it helped.
If the the drive doesn’t mount on boot blame systemd :wink:
I did actually have that issue, using fstab, and had to mangle systemd in a way I can’t remember, somehow forcing it to wait until DHCP had resolved, before loading futher network related stuff.

Having the mount done by systemd is the other way.

I’m interested to know how you went about remounting after a reboot. How did you get on?

As a matter of interest, why not just rsync directly rather than mount? I found a way here https://tech.borpin.co.uk/2019/10/03/readynasduo-v1-rsync/

rsync -v <filename> <user>@x.x.x.x::<share>
1 Like

That’s useful, I’ll be testing that.

1 Like

The shared drive has remounted OK - after some fiddling with fstab entry and LAN router to use fixed IP addresses! Still having problems with deleting old backups though, as cron is very sensitive to paths, environments and permissions! (cron is running on the PC not on rPi). I have been using rsync for non OEM backups and it has been faultless, so maybe time to get is working on all the machines!

1 Like