Automated backing up of EmonCMS

@TrystanLea

Thx for the suggestion re importing.

I would also welcome yr thoughts on regular backups and exporting these so as to avoid filling up the 4Gb / partition.

Thx

Good question, I’ve wondered whether to use a second SD card with card reader might be a good approach for an automated backup, the backup would not need to be compressed, just a simple file copy or even just a copy over of the most recent data… I’ve been doing a manual export here from time to time but Im relying on the usb-importer for my crash recovery in order to minimize data loss due to only having an older backup…

Others here have more automated solutions, using dropbox or syncing to storage boxes on the same network, @borpin @danbates ?

@TrystanLea

If the automated backup is compressed then it would be immediately usable in the event of a crash/problem.

Is there any advantage to simple file copying?
Which does beg the question - which files?

And a working script already exists for making backups. It can be leveraged. In fact I already use it in a cronjob to make weekly backups.

But that power cut really screwed up the USB HDD. Afterwards when I powered it up it didn’t even seem to be spinning so no chance of recovering those weekly backups.

The alternative to mechanical spinning devices? - SDHC cards or USB 3.0 Flash Drives?
Is a SDHC inherently better than a USB 3.0 Flash Drive which certainly avoids the somewhat Heath Robinson need for a USB card reader?
And storage size need not be an issue if a simple script solution can be devised for deleting backups more than x weeks old.

I’m a Linux neophyte - could you suggest a few loop script lines to achieve this?
As you well know, the existing export script outputs in the format: emoncms-backup-2019-08-03.tar.gz

My lessons learned:

  • Avoid mechanical spinning devices

  • Have backups on two separate solid state devices

Many thx and for the whole OEM concept & infrastructure
Happy Christmas!

2 Likes

Sounds like you’ve almost got it covered @johnbanks .
Here’s a link to a NAS (Network Attached Storage) implementation I took on if it helps. emonCMS backup example to NAS network drive - #4 by danbates .
@borpin’s rsync suggestion on post 5 is interesting, I’ve not had a go at it yet.

There are cloud options too, I’ve heard dropbox is a good one on the rPi, not used it though.

All the best.

@danbates
Many thx for yr response.
I’ve taken a quick look at the link you provided and the work you’ve done.
That is going to be super helpful esp yr method for selecting older backup files.
Happy Christmas!
PS: yr method will need quite a lot of mods to be compatible with the new 10.1.9 SD image

It is but Dropbox now restricts the number of machines it allows you to sync without paying.

@borpin
That being the case then just install an RPi to receive data via the HPPTInterfacer from several emonTx/RPi instances (as I have done) or create a local NAS. That would mean just one (free) connection to Dropbox is required, I think?

Assumes you don’t use Dropbox for anything else!

Oops.
Anyone get this one for Xmas?

1 Like

No but I’d like one.

But punctuation - surely…

There is no cloud; it’s just someone else’s computer

I’ll get my hat…

Haha.

I question the idea that it’s just someone’s, and not someThing else’s.

1 Like

I have a question about the emoncms-copy.sh script.

I’ve been using a cron job to execute the emoncms-copy.sh script to do an automated backup to a USB Flash drive mounted at the /home/pi/usbdisk/backup mount point.
I then, from a mac, scp the uncompressed files from the USB Flash drive, archiving them on the mac.
my question is … is there a reason the cp commands in the emoncms-copy.sh script don’t use the -p option to preserve timestamps? The ommission means the backed up files all have creation dates set to the time the copy was performed.

Thanks for any insights

I guess not. Sounds like a simple yet useful modification.
I think the best way to make changes to the source is a pull request on github, someone will get around to looking at it eventually.

Is this the version in the backup module?

The version in /home/pi/backup.

I’ve made local changes as below on each of the cp commands and works well.

cp --preserve=mode,timestamps $emonhub_config_path/emonhub.conf $backup_location

I’m a git novice so I’ll look into it when I get the time.

1 Like

Reasonably sure this is part of the backup module (in the old install location) so the source file is here GitHub - emoncms/backup: emoncms backup module

For a simple change like this, just edit the file in the browser, when you save, it will create a new Branch, then complete the text to create a Pull Request (PR) - git speak for “please can my changes be pulled into the main repository”.

1 Like