Regular reminder: Backups are a good idea

Emoncms Installation
1

TL;DR: Remember to keep multiple backups too.

In order to save space I deleted my MariaDB database files from under the running server yesterday.

Linux can continue using files even if they’ve been deleted so of course everything carried on working just fine…until I rebooted.

Seeing as this is me, of course it was complicated to recover. I backup every hour, but it backed up the empty database and so my most recent backup got wiped out.

I copied my archive of the backup from last year into mariadb database folder then did:

$ kubectl exec -it deployment/emoncms-mariadb -- bash

$ mysql -u root -p emoncms -e "SELECT COUNT(*) FROM users;"
+----------+
| COUNT(*) |
+----------+
|        0 |
+----------+

So that was a bit sad.

To recover:

$ mysql -u root -p emoncms < most_recent_2022-12-31

All is now well.

It’s time to get my backup rotation working properly that I threw out in the great rebuild of Christmas 2022.

Due to the architecture of EmonCMS I didn’t actually lose anything because the actual data is stored in PHPFina and PHPTimeseries (which I cleverly did not delete.)

(hint: remember to remove the backup from amidst the database files else the database will get confused next time it boots up)

2

To which I’d add:

I’ll never happen to me - until it does.

I learned that the easy way - back in the early 1980s and it involved a PDP11 and a 15" hard disc platter that someone (a manager, not my manager) who ought to have known better never got round to making a backup to. When there was a massive system crash and all his work got wiped out, it was a bit sobering when everyone realised that “0 blocks” on the slip of paper in the box really meant it was completely empty of data.

1 Like
3

Paraphrasing Robert…

To which I’ll add, *make at least two backups.

The last thing you want to discover is the backup you made is no good. At least with two (or more, depending on the value you place on your data) you have a somewhat better chance for a succesful restore.

Keeping at least one backup off-site is never a bad idea.

1 Like
4

Ironically I was trying to make my data set small enough to fit on a portable drive so I could have a secondary off-site backup :slight_smile:

5

Murphy strikes again! :frowning_face:

1 Like
6

My Proxmox VE setup makes this so easy. Just schedule a nightly snapshot and it is all backed up together on a NAS.

1 Like
7

Two! I have sixty…no, really, here’s my logrotate config:

"./database_backup" {
	dateext
	daily
	rotate 60
	missingok
	copy
	olddir daily
}

I have that same config for daily, weekly and monthly, so eventually I’ll have 180 backups.

This helps protect me from failures. On the odd occasion something goes wrong so the backup doesn’t happen, or I break the database and it takes me a week to notice.

Hmm, perhaps I don’t need the monthly to be 60, I should notice a problem in less than five years :slight_smile:

2 Likes
8

Testing that your backup will restore as you expect it to is worth doing too!

1 Like
9

But don’t restore a bad backup over good data. :cry:

1 Like
10

Yes, I was going to add that it’s worth doing very soon after a new install so you don’t lose much data if it doesn’t work, or restore to a spare device if you have it but decided to keep the post short.

11

I don’t think you need to keep your posts short, this is very much just a “chatter” topic. We’d love to hear what you have to say @christian .

12

Aw shucks :smiling_face:

When I led a development team on unix, we had backups on the hour, every hour.
System admin were responsible for the backups.
When we first needed to use a backup, it turned out the branch with our working files was only done once daily :flushed:

1 Like
13

Storage is cheap. Keep everything, never overwrite anything. I even keep my old hard drives, with the hardware to read them, just in case. Nothing beats making a copy.

Over 4 decades in the industry, I have lost a lot of data but backups have saved some of it.

1 Like