Debugging a Production Issue: A Method That Works at 9am
A client writes: "the checkout is broken". No screenshot, no error message, no timestamp. This message arrives at the worst possible moment in every project, and the teams that handle it well aren't smarter than the ones that flounder. They follow a method, and the method is the same whether the app is Laravel, Go, or WordPress. Debugging production issues fails when people skip the boring steps and jump straight to reading code.
Stabilize before you investigate
The first question is never "why". It's "how bad, and getting worse?"
- Is it affecting one user or all users? One page or the whole app?
- Did it start in the last few minutes (something just broke) or has it been broken for days (something drifted)?
- Can you make it worse by restarting anything? Then don't restart anything yet.
If users are actively losing orders or data, the correct move is often rollback, not diagnosis. Deploy the previous release, restore the calm, then investigate on your own time with no clock ticking. A rollback that turns out to have been unnecessary costs ten minutes. Debugging under pressure for three hours costs the afternoon and usually the data anyway.
Gather evidence before theories
Once nothing is actively on fire, collect facts in this order:
- What changed recently? Deploys, config edits, DNS changes, certificate renewals, cron jobs that ran for the first time. In our experience the recent-change list finds the culprit half the time before any code is read. Git log and deploy logs answer this in two minutes.
- What do the logs say? Not the code. Logs. The error message that actually reached the log file is worth more than an hour of speculation about likely causes.
- Can you reproduce it? If yes, everything gets easier. If no, look for what differs: user role, browser, data volume, time of day. Bugs you can't reproduce are usually bugs in an environment you haven't looked at (staging differs from production in exactly one load-bearing way).
This is why structured logs with request IDs matter before the incident, not after. You cannot add useful logging during a fire; you can only read what you bothered to write.
Form one hypothesis, test it cheaply
The failure mode we see most is shotgun debugging: change three things at once, see improvement, learn nothing, ship the fix for the wrong problem. One hypothesis at a time, and pick the test that costs the least. "Is the payment provider returning errors?" is answered by one curl command. "Is the database connection pool exhausted?" is answered by one metrics query. Save the code reading for after the cheap tests have narrowed the search.
Write the hypothesis down, even as a one-line note. It sounds unnecessary until you're forty minutes in and can't remember which theories you already ruled out.
Fix the cause, not the symptom
A retry loop around a failing call is a symptom fix. The failing call is the cause. Retry loops, try/catch blocks that swallow errors, and "we increased the timeout" all make the error quieter without making it rarer. Sometimes a symptom fix is the pragmatic choice for today, and that's legitimate. Write it down as debt, with the cause next to it, so the debt has an address.
Then do the part everyone skips: ask why the bug reached production. Missing test case? No monitoring on that path? A deploy process that let an unreviewed change through? The bug itself is worth an hour; the hole it came through is worth fixing every time, because the same hole produces the next bug too.
Afterwards
Post-mortems have a bad reputation because most are written as blame-avoidance theatre. A useful one fits in half a page: what happened, how long until detection, how long until fix, and what one change prevents a repeat. At SrcTeam we write these for our own incidents the same day, while the details are fresh. The one-change constraint matters. A post-mortem with five action items gets zero done; one with a single concrete item actually ships it.
Debugging is not a talent you're born with. It's a procedure: stabilize, gather evidence, hypothesize cheaply, fix causes, close the gap that let it through. Teams that follow it in order look calm during incidents, and calm is what clients remember.