Integrating third-party APIs: plan for the day they go down
Any real product calls APIs it does not own. Payment provider, transactional email, maps, SMS, cloud storage, some government portal. The integration works great the day you build it, with the vendor's docs open and their sandbox replying in 200 milliseconds. The actual job is every other day: the vendor is slow, returns 500s, rate-limits you, or deprecates an endpoint you relied on. Building for those days comes down to a handful of decisions, and they are cheaper to make up front.
One module per vendor
Wrap each external service in your own class. A PaymentGateway service that the rest of the app calls, with the vendor SDK or HTTP calls confined inside it. Controllers and jobs should never scatter raw vendor calls across the codebase.
The payoff shows up when you swap vendors or add a second one, when you write tests, and when something breaks, because timeouts, logging, and metrics all live in one place. If the vendor's ID format leaks into twenty files, you have married them, and divorce later costs a rewrite.
Set timeouts, real ones
Many HTTP clients will wait far longer than your users will. A vendor that normally answers in 300ms and today hangs for 45 seconds will take your checkout page down with it, queue workers included.
Set two limits on every call: a short connect timeout and a total timeout. For anything on an interactive path we keep the total around five to ten seconds. Background jobs can afford more, but they need a timeout too. A request without a timeout is a promise the vendor did not make.
Only retry what is safe to repeat
A timeout does not tell you whether the vendor processed the request. It might have. Retry blindly and you can charge a card twice.
The rules we follow:
- Retry network errors, timeouts, 5xx responses, and 429s, with exponential backoff and jitter. Never retry other 4xx responses; if the request is wrong, retrying just repeats it.
- For anything that creates money movement or records, send an idempotency key: a stable UUID generated once per logical operation. Vendors that support it (most payment providers do) will return the first result instead of executing the operation twice. If a vendor does not support it, generate your own uniqueness on their side, like a deterministic reference number.
- Cap total attempts. Eight retries spread over an hour handles a vendor restart; eight hundred retries handles nothing except filling your queue.
Respect their rate limits
When you get a 429, read the Retry-After header and honor it. Better, throttle on your side against the documented limits so you rarely see the 429 at all. Bulk imports hammering an API in a tight loop are the usual offenders; a token bucket in your vendor wrapper fixes most of it.
Degrade instead of failing
Decide up front what the feature does when the vendor is down, because "the whole page 500s" is a choice, not an accident:
- Serve the last known good data if stale content is acceptable. Show it with its age.
- Queue the call instead of making it inline. The user's action succeeds; the sync catches up when the vendor recovers.
- After several consecutive failures, stop calling for a while (a circuit breaker) so you are not generating load against a service that is already struggling.
- Put a config flag on the integration so you can turn one vendor off without a deploy. This is the flag you will actually use at 2am.
Log enough to debug it later
For each call, log the endpoint, a correlation ID you generated, the status code, and the duration. When the vendor's support asks for "the timestamp and request ID of the failing call", that log line is the difference between a five-minute answer and a week of guessing. Scrub secrets and personal data from what you keep; the log should help you debug, not become a liability.
Watch the vendor like a dependency
Track the error rate and latency per integration as metrics, and alert on them. The alternative is finding out from a customer. Vendors publish status pages, but status pages are written by the vendor, and they update them when they feel like it.
None of this is exotic. It is one wrapper class per vendor, timeouts, idempotency keys on the money paths, a degradation plan per feature, and metrics on the calls. The initial integration is a day of work; this is what makes it a year of quiet operation.