Community
OpenEnergyMonitor

OpenEnergyMonitor Community

emonCMS backup example to NAS network drive

Tags: #<Tag:0x00007f1be25b8538>

I got a local network drive (NAS) backup method working, thought to share.

Using emoncms-export.sh

This is the script that runs when using the backup module in the emoncms browser interface.

Set up.
$ sudo nano /etc/fstab
modifies linux mount points, I added a line to mount the network drive:

192.168.1.235:/media/astralship/nas/emoncms-backup /mnt/nas nfs async,rw,proto=udp 0 0

This can look very different depending on the system!

I then remounted the drives listed in fstab:
$ sudo mount -a
I had an issue that the drive was not mounted on rPi reboot. This was a problem with systemd doing things in the wrong order. I fixed it following a guide, but ended up installing a sudo cronjob anyway:

$ sudo crontab -e
I added the line:
0 23 * * * mount -a making sure to mount all drives at 11pm each day.

Running backup
I wanted to have a minimal backup situation, where only the most recent backup file was copied to the network drive, IF it was newer than the NAS backup.

First, I test the backup shell script emoncms-export.sh

$ /opt/emoncms/modules/backup/emoncms-export.sh

The default setting place the archived backup files in /var/opt/emoncms/backup/

… backup/config.cfg defines the destination of the backup if one wants to place it elsewhere.

At this point I consider pointing this directly to the NAS, but I want a local AND remote backup. @Ryan_S reminds me avoiding excessive SD card writes is a good idea and used a memory stick for a local backup, which in itself can meet all requirements no doubt. He also brings attention to an emonCMS update causing config.cfg to be overwritten. If this file is modified for a custom backup script action consider backing up config.cfg or making it unwritable.

I add the backup script to cron:

$ crontab -e

adding line

0 0 * * * /opt/emoncms/modules/backup/emoncms-export.sh

Running the script at midnight.

Then after a good hour of bashing away and searching the internet I came up with this script:

if [ $(ls -t /var/opt/emoncms/backup/emoncms-backup-* | head -1) -nt $(ls -t /mnt/nas/backups/backup* | head -1) ];
then
   cp $(ls -t /var/opt/emoncms/backup/emoncms-backup-* | head -1) /mnt/nas/backups/backup.tar.gz;
fi;

Here I have the conditional if, the operator newer than -nt, then it’s copied to the NAS file called backup.tar.gz, of course, overwritten if it already exists.

I install the cronjob to run one hour after the local backup, midnight again.

$ crontab -e

adding another line after the previous:

0 1 * * * if [ $(ls -t /var/opt/emoncms/backup/emoncms-backup-* | head -1) -nt $(ls -t /mnt/nas/backups/backup* | head -1) ]; then cp $(ls -t /var/opt/emoncms/backup/emoncms-backup-* | head -1) /mnt/nas/backups/backup.tar.gz; fi

The alternative method is rsync. rsync is a useful tool with option to mitigate from excessive writing, but I was getting a lot of permission errors and it looked ugly. Hence making the above script.
Here’s an rsync example crontab to sync the local directory to the remote:

0 0 * * * rsync -azu /var/opt/emoncms/backup /mnt/nas/backups

Cleanup

I delete old local backups with:

rm $(find /home/pi/data/backups/emoncms* -mtime +7)

Removing backups older than 7 days… and I add it to crontab at midnight.

0 0 * * * rm $(find /home/pi/data/backups/emoncms* -mtime +7)


Comments welcome.

P.S. Thank you stackoverflow.com

2 Likes

Thanks @danbates. I was looking for a way to auto run and store backups on a network drive and your approach helped sort it! I have a linux NFS shared drive and changed the emoncms backup destination to it. I decided against the conditional IF script, but now run a cron job on the server that deletes any backups older than a setting (eg 2 days). The only thing I need to keep an eye on is that the shared drive properly remounts on the rPi after a reboot/power fail!

I’m glad it helped.
If the the drive doesn’t mount on boot blame systemd :wink:
I did actually have that issue, using fstab, and had to mangle systemd in a way I can’t remember, somehow forcing it to wait until DHCP had resolved, before loading futher network related stuff.

Having the mount done by systemd is the other way.

I’m interested to know how you went about remounting after a reboot. How did you get on?

As a matter of interest, why not just rsync directly rather than mount? I found a way here https://tech.borpin.co.uk/2019/10/03/readynasduo-v1-rsync/

rsync -v <filename> <user>@x.x.x.x::<share>
1 Like

That’s useful, I’ll be testing that.

1 Like

The shared drive has remounted OK - after some fiddling with fstab entry and LAN router to use fixed IP addresses! Still having problems with deleting old backups though, as cron is very sensitive to paths, environments and permissions! (cron is running on the PC not on rPi). I have been using rsync for non OEM backups and it has been faultless, so maybe time to get is working on all the machines!

1 Like

Edit: Scratch that. Dug up the main one and it works executing it within the ssh terminal. My mistake.

So I came across this and figure it would work well, modified slightly, for my needs. My only issue right now is test running the emoncms-export.sh script downloaded from the git site. I keep getting syntax errors.

[email protected]:~/backup $ sh emoncms-export.sh
emoncms-export.sh: 7: emoncms-export.sh: Syntax error: newline unexpected
[email protected]:~/backup $ bash emoncms-export.sh
emoncms-export.sh: line 6: syntax error near unexpected token newline' emoncms-export.sh: line 6:

Sure I’m missing something here.

You don’t need to get it from GitHub. It is already installed with the backup Module (/opt/emoncms/modules/backup/).

If you run it from there the config file will be correctly used.

bash /opt/emoncms/modules/backup/emoncms-export.sh

Your error is probably because it is a bash script not a sh script.

Yup, noticed that as I was digging into the errors and found the script where you mentioned. So, I test ran the script and volla, it worked.
Just for others wanted to maybe just put in a USB Flash stick or 2nd SD Card to locally store backups and automate it.

I took an old 2gb card I had lying around and plugged it into my Raspberry Pi 3. One it mounted I located the mount location, in my case this was /dev/sda1.

Now, I wanted it to mount to the same location anytime it might be removed or the unit rebooted. So, I found it’s UUID by running sudo blkid and location /dev/sda1 from the list which listed it’s UUID.
Note: UUID’s can change. If you reformat and change it’s file system the UUID can change.

I took the UUID and created and entry in /etc/fstab which allows the USB Flash stick to always mount to the location I created which was /media/backup. I tested by rebooting the device and pulling the stick. All works good.

Moving forward. After testing the script I needed it to point towards the USB Flash stick.
So I modified the config.cfg located in /opt/emoncms/modules/backup the same way the OP above noted, the backup_location. Mine now reads backup_location="/media/backup".

I test ran the emoncms-export.sh script and it created an output file at /media/backup which is what I wanted.

So, now to automate it. Same idea as the OP above, I modified the crontab using sudo crontab -e. There are guides from what the entries read, but I wanted mine to run at 2300 hrs on the 15th day of the month. So I placed an entry in there 00 23 15 * * /opt/emoncms/modules/backup/emoncms-export.sh

I tested it by modifing the time to just 2 minutes ahead of current time (ensure your Pi timezone is correct or run date to see what system time is) and * the day. It ran as expected.

I still need to test the remove of older files in my case greater than 60 days, but that will have to wait until I have enough in there to test. So have an entry commented out for now. #30 00 16 * * rm $(find /media/backup/emoncms* -mtime +65)

So I noticed something. It looks like an update overwrote the backup location change I made to the backup config file.

Checking /var/opt/emoncms/backup and sure enough there is the 11-15 file that was created last night.

Thinking I might just need to create an entry to move the file or just copy it (to cut down on unnecessary extra writes to the sd card) over to the memory stick.

Good to know.
I’ll add a note in the first post… Perhaps making the file unwritable would be a good way to stop an update messing it up?

I’ve updated the directory locations in the OP to match up with the recent emonSD images.

Good point on SD cards writes, 'tis a common failure mode of a system, excessive writing. I’ll suggest direct writing to a NAS or mem stick to minimise disk writes :slight_smile:

Git shouldn’t do that!

@TrystanLea - perhaps this needs a user config file?

1 Like

I would agree, but my modification is not there and it’s back to default. The only things I’ve done since my post about setting this up back on Oct 31 was run the Full Update in admin. Here is what the config.cfg looks like today.

My guess right now is update must have overwrote the config.cfg file with a new one putting it back to defaults. I would prefer the location to just write the backup directly to the mem stick I put in to cut off extra auto backup writes to the main card.

I show my auto backup did happen last night, but it’s location is the default location shown in the config.cfg file.

The cron job I put in was supposed to run at 23:00hrs local on the 15th. The 11-15 file confirms that occurred, but it should be in my custom location /media/backup, not /var/opt/emoncms/backup.