Backups and Disaster Recovery: The Plan You Hope You Never Need

Every web app has a backup story. Some have a script, some have a half-remembered cron job on a server that was decommissioned, and some have a customer calling to ask why their data is gone. The uncomfortable truth: a backup that has never been restored is not a backup — it's a hope. Here's how we think about backups and disaster recovery in production.

The 3-2-1 rule still holds

The classic principle is still the best starting point: keep 3 copies of your data, on 2 different media types, with 1 copy offsite.

3 copies — your live data, plus two backups (e.g., today's and last week's).
2 media — local disk and object storage (S3-compatible, or another server in a different location).
1 offsite — if your server's disk dies, a fire happens, or the hosting provider has a bad day, the offsite copy is what saves you.

A backup stored on the same machine it protects is a convenience, not a backup.

What to back up (and what not to)

The biggest mistake is backing up the wrong things. Your code is in git — it doesn't need to be in the backup. What you actually can't rebuild is your data:

The database — the single most important artifact. Everything user-generated lives here.
Uploads and user files — images, documents, exports. Often bigger than the database, often forgotten.
Environment and configuration — .env files, secrets, nginx configs. These are small but unrecoverable if lost. (Keep secrets in a vault or a separately-protected copy — never in the same git repo the rest of the code lives in.)

What you can skip: dependencies, caches, compiled assets, and anything a fresh deploy rebuilds. Including them just makes restores slower and backups bigger.

Define RPO and RTO before you need them

Two numbers that sound like consultant jargon but actually drive every decision:

RPO (Recovery Point Objective) — how much data loss is acceptable. Hourly database dumps mean you lose at most an hour of writes.
RTO (Recovery Time Objective) — how fast you need to be back online. If the app can be down for four hours, don't burn money on instant failover.

A hobby project can get away with nightly backups and an RTO of a day. A SaaS with paying customers usually wants hourly database dumps and a documented restore procedure measured in minutes. Be honest about where your project sits.

Automate it, and make restores boring

Manual backups are backups that stop happening after the third week. Automate everything, and make the restore path a documented runbook, not a Google search at 2 a.m.

# Daily database dump, retained for 14 days, shipped offsite
today=$(date +%Y-%m-%d)

mysqldump --single-transaction \
  -u backup -p"$DB_PASSWORD" "$DB_NAME" \
  | gzip > "backups/db-$today.sql.gz"

# Keep 14 daily copies locally
find backups -name 'db-*.sql.gz' -mtime +14 -delete

# Ship to remote object storage
rclone copy "backups/db-$today.sql.gz" s3:myapp-prod-backups/

The two lines that matter most: the -mtime +14 retention (storage grows without a cleanup policy) and the rclone upload (offsite copy). If the job isn't shipped offsite, it's not a disaster-recovery backup.

Test the restore — on a schedule

Here's the part almost everyone skips. Restore the backup into a fresh environment on purpose, at least quarterly, and time it. You will find the same issues every time:

The dump uses a MySQL flag your restore script doesn't know.
The uploads archive is corrupted but nobody scanned it.
The backup ran, but the database was being written to mid-dump, so it's internally inconsistent.
The person who wrote the restore script left the company, and nobody else has run it.

A restore test is the only way to discover these problems while they're cheap. If you can't restore, the backup is just disk space with a nice name.

The failure ladder

Disasters come in escalating sizes, and each rung needs a different response:

Accidental delete — a developer drops a table or a customer removes their data. Solution: point-in-time recovery or a recent dump.
Server failure — hardware dies, hosting provider loses the machine. Solution: the offsite copy and a fresh server provisioned from your deployment scripts.
Ransomware or a compromised server — an attacker encrypts what they can reach. Solution: immutable offsite backups and secrets not stored on the web server. This is the one case where a daily backup that's also encrypted by the attacker is worse than none.

The ladder matters because it decides where backups live. Data that must survive a compromised server needs to be in storage the server can't write to.

The payoff

Backups are unglamorous until the exact moment they're the most important thing in the company. The discipline is small: follow 3-2-1, back up the data you can't rebuild, automate it, and prove the restore works on a schedule. Then, when the worst day comes, it's a script you've already run a dozen times — not a miracle you're hoping for.

If your current project's backup story is "I think there's a cron job somewhere," today is a good day to write the first restore test.