Backups Are Not a Disaster Recovery Plan — Until You've Tested a Restore

"We have backups."

These three words are the most dangerous sentence in operations. Because the question isn't whether you have backups. The question is whether you can restore from them, under pressure, with production down, at 3am, with your boss watching.

An untested backup is a prayer, not a plan.

The restore drill

A real disaster recovery test answers these questions:

1. Can you actually restore? The backup file exists. Does it load? Is it corrupt? Is the format even readable by your current software version?
2. How long does it take? Your RTO (Recovery Time Objective) is the measured restore time, not the number you wrote in a doc. If your RTO is 1 hour and restore takes 4, you don't have an RTO of 1 hour.
3. How much data did you lose? Your RPO (Recovery Point Objective) is the measured data loss, not the backup frequency. If you back up hourly but the last good backup is from 3 hours ago because the last two failed silently, your RPO is 3 hours.
4. Does the restored system actually work? A database that restores but has orphaned foreign keys, broken indexes, or missing migrations is not a working restore.
5. Can the on-call person do it alone? At 3am, with you on vacation, can the person who got paged actually perform the restore from runbook alone?

If you can't answer all five with measured numbers from a real drill within the last quarter, you don't have a DR plan. You have a backup script and hope.

How to actually test restores

Scheduled restore drills

Once a quarter, restore your latest backup into a fresh, isolated environment. Verify:

• The restore completes without errors
• A smoke test of critical paths passes (can a user log in? can they read/write the main entity?)
• Row counts match production within expected delta
• Application starts cleanly against the restored DB

Document the time-to-restore. Compare to last quarter. Trend it.

Chaos-style: pick a random Tuesday

Don't always test on the scheduled day. Occasionally pick a random Wednesday, restore yesterday's backup, and check. Backups that fail on Wednesdays but pass on Mondays indicate time-dependent bugs (cron issues, log-rotation collisions, weekend-only traffic patterns).

Test partial restores

Full DR is rare. The more common scenario: "a user deleted a row they shouldn't have, please get it back." Can you restore a single table or row without nuking production? Point-in-time recovery (PITR) is the feature that makes this possible — make sure your backup strategy includes it.

Common backup failure modes

Drills catch these. Untested backups eventually surface them in production:

• The backup writes to a full disk. It "succeeds" writing 0 bytes. Alerting never fires because the exit code is 0.
• The backup format upgraded and the old restore tooling doesn't read it.
• The backup is encrypted and the key is in the same AWS account that just got phished.
• The backup is in a different region that you can't reach because your network config drifted.
• The restore requires a step nobody documented — "oh yeah, you have to run migrate-rollback.sh first." The person who knew that left six months ago.
• The backup was always succeeding on the replica, but the replica was 8 hours behind and nobody noticed.

What "tested restore" looks like in a runbook

A real runbook entry:

PROD DB RESTORE (target RTO: 1h, target RPO: 15min)

1. From the on-call laptop, run: ./scripts/restore-prod.sh --point-in-time "YYYY-MM-DD HH:MM:SS UTC"
2. Expected duration: 35-50 min for current DB size (~120GB)
3. Verify: curl https://staging.internal/health after restore -> expect {"db":"ok"}
4. If step 1 fails: see "Restore failure" section below. Page DBA-on-call.
5. Last drill: 2025-01-15, actual RTO 42min, actual RPO 8min. Passed.

That last line — "Last drill: date, actual numbers" — is the difference between a runbook and fiction.

Run the checklist

The ShipReady Reliability section includes "Disaster recovery plan (tested, not just written)" and "Database backups + tested restore drills" as distinct items. They're separate for a reason.

Run the checklist, and if those two boxes aren't checked with a real date in the last 90 days — schedule a drill this week. Before the universe schedules one for you.