Database Migrations in Production: Shipping Schema Changes Without Downtime

A two-minute deploy can turn into an hour of downtime over one line of schema. In the projects we take over, migrations cause more incidents than the application code does — because code rolls back, but a half-finished ALTER TABLE on a live table just sits there, holding a lock, while request queues fill up.

The fix is not better migrations. It's a different sequence: separate the schema change from the code change, and make sure the code works with both schemas at every step.

Deploy the code before the schema it needs

The naive order — ship the migration and the feature in the same release — has a gap of a few seconds where the new code runs against the old schema. Usually it survives. On a busy day it doesn't.

What we do instead is the expand/contract split. Renaming a column, for example, takes three deploys instead of one:

Expand. Add the new column next to the old one. The migration is instant and safe. Meanwhile, the code writes to both columns and reads from the old one.
Migrate. Backfill the new column, flip reads over to it. Still one release, nothing breaks.
Contract. Once logs show nothing touches the old column, remove it and the double-write.

The extra two deploys are the price of never having a moment where code and schema disagree. For anything on a table with real traffic, that price is low.

Mind the locks

Schema changes take locks, and locks on a hot table stack up fast. A few habits keep them short:

Never run DDL inside a transaction that also does data work. The transaction holds its locks until commit; a ten-minute backfill means ten minutes of blocked writes.
Set a lock timeout. In PostgreSQL, SET lock_timeout = '5s' before a migration makes it fail fast and retry later instead of silently freezing every query behind it.
Know your database's escape hatches. CREATE INDEX CONCURRENTLY in PostgreSQL, or MySQL 8's online DDL with ALGORITHM=INPLACE, turn the worst operations into near-instant ones. They're also the operations people forget exist until the table is already too big to touch casually.
Rehearse on a production-sized copy. The migration that takes 40ms on your dev seed of 50 rows can take 40 minutes on a million rows. Ten-row tables hide every problem, and DDL hides twice as many as queries.

Backfills are jobs, not migrations

The most common mistake we see: a migration that updates every existing row in one go. On a small table, fine. On a large one, it holds locks, blows the transaction log, and can time out halfway through, leaving rows in two states.

A backfill belongs in a queued job that processes the table in chunks:

// Chunked backfill — resumable, doesn't hold a transaction open
Order::whereNull('shipped_at_legacy')->chunkById(500, function ($orders) {
    foreach ($orders as $order) {
        $order->update(['shipped_at_legacy' => $order->created_at]);
    }
});

Chunked by ID, it's resumable: if the worker dies at row 300,000, the next run picks up where it left off. And while it runs, the application keeps serving requests like nothing is happening, because nothing is blocking.

Write the rollback before you need it

Most migrations are one-way. Laravel's down() methods sit empty in half the codebases we see, which means the rollback plan is "restore from backup" — the slowest possible option, discussed at length in our piece on disaster recovery.

The realistic rollback for a schema change is almost never "revert the migration." It's "the code still works with the old schema, so redeploy the previous release and figure it out calmly." That only works if you've kept the old path alive, which is exactly what expand/contract gives you. The two practices are the same practice.

The checklist

Code and schema never ship in the same release when a hot table is involved
Expand first, migrate second, contract last
Lock timeout set before any DDL; long transactions banned in migrations
Backfills run as chunked, resumable queue jobs
Every risky migration rehearsed against a production-sized copy
The rollback is a redeploy, not a restore

None of this is clever, and none of it is new. It's just the difference between a schema change that ships quietly and one that becomes the story you tell for the next year.