Has your backup ever been restored?
A backup file that exists is not a backup. We had that demonstrated on our own system, and the failure would only have surfaced on the day of an incident.
Published
We changed the database behind our internal CRM. Clean migration, data verified row by row, nothing lost. Then we looked at what the nightly automatic backup was doing.
It was still backing up the old database. The file existed, the job finished without error, the log said “OK” — and the contents no longer had anything to do with the system in service. It would have carried on like that every night, indefinitely.
Three traps we found while fixing it
A scheduled job doesn't see the same environment you do
A script launched automatically at night doesn't have the same settings as one launched by hand. Ours could no longer find the backup tool: command not found, every night, with nobody watching. An automatic job must be tested in ITS conditions, not yours.
Our first check was decorative
To avoid empty backups, we added a check counting the tables in the file. It proudly announced “31 tables” on a database holding zero rows: a file's index lists every table, full or empty. The check measured nothing. We replaced it with a count of the rows actually present.
A failed attempt erased the previous good backup
The script deleted the file when it judged it invalid — but two runs close together targeted the same name. So a failed attempt destroyed a correct backup. A safety mechanism that destroys what it protects is worse than having none.
The only test that counts
Take last night's backup, put it back into an empty system, and count. In our case: 15 contacts, 15 files, 20 log entries, 34 notifications — the same as in service. Until that has been done at least once, a backup is a hypothesis.
- How often should a restore be tested?
- Every time something around it changes: database, server, host, format. Those changes are precisely when a backup starts pointing at nothing — while never ceasing to succeed.
- My host says they back up. Is that enough?
- It's a good start, not a guarantee. Two questions to ask: how long does a full restore take, and can you take your data elsewhere in an open format? A backup you can neither test nor take with you isn't really yours.