Backup is the part of IT with the widest gap between how good it looks and how good it is. The failure modes are not exotic. They are ordinary and they are all invisible from the console: a database that was copied while open and will not start cleanly, a service account whose password changed after the backup job was configured, a retention policy that quietly expired the only copy anyone needed, an encryption key stored somewhere nobody has written down.
Every one of those is discovered by attempting a restore. There is no other way to find them.
Three copies, two media, one offsite — and one of them immutable
The classic shape still holds: three copies of the data, on two different kinds of media, with one copy offsite. What has changed is the third requirement. Modern ransomware does not only encrypt your servers — it goes looking for the backup system first, because a business with restorable backups can refuse to pay. So at least one copy must be immutable for a defined period: write-once storage that no compromised credential can delete or alter, held long enough that you notice the incident before the window closes.
That immutability period is a real design decision, not a checkbox. It should be longer than the time it would take you to discover a compromise, which is usually longer than people assume.
What a restore test actually involves
"We tested the backups" can mean anything from a file dragged out of a snapshot to a full rehearsal on isolated hardware. The useful distinction is what you proved.
- A file-level restore confirms the backup is readable and the file is intact. It is quick, it is worth doing regularly, and it proves the least.
- An application-level restore confirms the data is consistent: the database starts, the mail store mounts, the line-of-business application opens and finds its records. This is where most backup problems actually live, and it is the test most often skipped.
- A bare-metal or full-system restore confirms you can rebuild a server from nothing — operating system, configuration, and data — without the original hardware. It is the only test that reproduces a real loss.
- A documented recovery time turns all of the above into a number you can plan against. How long did it take, start to finish, including the time spent finding the credentials and reading your own instructions?
Restore to somewhere that is not production. An isolated network, or a sandbox, where the restore cannot reach live systems — and where a restored server cannot rejoin the domain under its old identity and start answering requests. This is the detail that turns a test into an incident.
How often, and what to write down
The cadence should follow how much you would lose. If a restore takes four hours, testing annually means you are trusting a procedure you have practised once. A workable arrangement for most small and mid-size environments is a file-level restore monthly, an application-level restore quarterly, and a full-system rehearsal annually — with the whole schedule reviewed whenever the environment changes materially.
Then write it down, because an undocumented successful test is worth surprisingly little six months later. For each test, record what was restored, where it was restored to, who performed it, when it started and finished, and what went wrong. That record has two uses. It is the working document your team reads at 2am instead of improvising, and it is the evidence that your recovery control operates — which is precisely what a SOC 2 observation window asks you to demonstrate rather than assert.
The last part is the one people skip: fix what the test found, and re-test the fix. A restore test that produces a finding and no change is a rehearsal for the same failure.