Designing Integrations That Survive Third-Party API Failures

Every third-party API integration fails sooner or later. The payment provider has a slow afternoon, the shipping API returns 503 during a sale, a CRM starts rate limiting you because another team added a sync. None of that is under your control. What your own application does during those minutes is, and it decides whether the result is a delayed email or a checkout that hangs for every customer.

These are the patterns we build in so that someone else's outage stays someone else's. The examples are in PHP and Laravel, and the ideas carry to any stack.

Expect slow, wrong and absent

Before writing code, list how the call can fail, because each failure needs a different response:

• Slow. The provider answers, eventually. Without a timeout your web workers wait with it.
• Down. Connection refused, a DNS failure or a 5xx status. Usually temporary.
• Throttled. A 429, often with a Retry-After header saying when to try again.
• Half done. The provider processed your request and the response never reached you. This is the dangerous one, because a retry may do the work twice.
• Rejected. A 4xx for bad input or expired credentials. Retrying will not help.
• Changed. A field is renamed or an API version is retired.

Timeouts first, then careful retries

An HTTP call with no sensible timeout is the usual way one provider's outage takes a whole site down: every PHP worker ends up waiting on the same dead connection. Set two limits on every outbound call, a short one for connecting and a longer one for the full request. Laravel's HTTP client waits up to 30 seconds by default, which is far too long when a customer is watching a spinner.

use Illuminate\Http\Client\ConnectionException;
use Illuminate\Support\Facades\Http;

$response = Http::connectTimeout(3)
    ->timeout(8)
    ->withHeaders(['Idempotency-Key' => $shipment->idempotency_key])
    ->retry(2, 300, function (Throwable $exception) {
        return $exception instanceof ConnectionException;
    })
    ->post('https://api.example.com/v1/shipments', $payload);

The first argument to retry is the total number of attempts, and the third decides which failures deserve another one. Here only connection problems and timeouts do. The options are described in the Laravel HTTP client documentation.

Rules that keep retries from making things worse:

• Retry timeouts, connection errors, 502, 503, 504 and 429. Never retry 400, 401, 403, 404 or 422. The request is wrong and will be wrong again.
• Wait longer after each failure (half a second, then two, then eight) and add some randomness, so a thousand clients do not all come back in the same instant.
• When a Retry-After header is present, obey it.
• Cap the attempts. Inside a web request, one quick retry is the most a user should wait for.
• Retry a write only if the operation is idempotent, which is why the example sends a key.

Idempotency keys prevent the double charge

You send "create payment" and the connection drops. Retry blindly and the customer may pay twice. Do nothing and the order may sit unpaid. An idempotency key removes the guess: you attach a unique value to the request, and the provider returns the original result when it sees that value again instead of acting twice.

1. Generate the key once per business operation (a UUID is fine) and save it with your record before the first attempt. A key created inside the retry loop is new each time and protects nothing.
2. Send the same key on every retry of that operation, and a different key for a different operation.
3. Check how long the provider remembers keys. Stripe documents its Idempotency-Key header and says keys may be removed once they are at least 24 hours old, so a retry several days later can be treated as a new request.

When a provider offers no idempotency support, send your own reference (an order number) and, before retrying, look up whether a record with that reference already exists on their side. Apply the rule to your own code too: a unique index on the external ID means a repeated message cannot create a second row.

Take the call out of the request with a queue

If the customer does not need the provider's answer to see the next page, do not make them wait for it. Save what you intend to do in your own database, push a job onto a queue and respond. A worker makes the API call, with a retry schedule measured in minutes and hours.

An outage then looks different. Orders are still accepted, CRM updates and shipping labels wait in the queue, and the backlog drains once the provider recovers. To make that safe:

• dispatch the job only after the database transaction commits, or write it to an outbox table that a scheduled task reads, so the record and the job cannot get separated;
• show an honest status ("label pending") instead of pretending the work is done;
• send jobs that run out of attempts to a failed-jobs list that someone reviews and can replay;
• keep the calls that must be synchronous, such as authorizing a card at checkout, on the short timeout path above.

Most frameworks include a queue that runs on the database you already have. If queues are new to your team, Laravel queues explained covers jobs, workers and failure handling.

A circuit breaker for the bad hour

Retries help with a blip. During a real outage they hurt: every request still waits for its timeout, and your retries add load to a provider that is trying to recover. A circuit breaker counts recent failures per provider. Past a threshold it opens, and for a minute or so calls fail immediately without touching the network. Then it lets one test request through and closes again if that succeeds.

Keep the counter in a shared store such as Redis, so every web server and worker sees the same state. For queued work, Laravel includes a ThrottlesExceptions job middleware that delays further attempts once a job has thrown a set number of exceptions, which has a similar effect for jobs that talk to an unstable service.

An open circuit needs a fallback, decided per feature: show the last cached shipping rates, hide the live-stock badge, accept the form and sync it later, or say plainly that card payments are unavailable and offer another method.

Verify webhooks and expect each one twice

A webhook endpoint is a public URL, so the first job is proving the request came from the provider. Most providers sign the raw request body with a shared secret. Compute the same signature and compare the two in constant time:

$payload   = file_get_contents('php://input');
$signature = $_SERVER['HTTP_X_SIGNATURE'] ?? '';
$expected  = hash_hmac('sha256', $payload, $secret);

if (! hash_equals($expected, $signature)) {
    http_response_code(401);
    exit;
}

The header name and the exact string that is signed differ per provider, and many include a timestamp to block replays, so follow their documentation or use their SDK. Always verify against the raw body, before a framework has parsed and re-encoded the JSON.

Then store the event, return 2xx at once and process it in a job. Providers retry after slow or failed responses, so the same event can arrive several times and out of order. Save each event ID under a unique index and skip duplicates. When order matters, fetch the current state from the API instead of trusting the payload. A nightly reconciliation job that compares their records with yours catches anything that never arrived.

Monitor the integration, not only your server

An uptime check stays green while an integration quietly stops working. Track these per provider:

• error rate and response time of outbound calls, with an alert when either leaves its normal range;
• the count of 429 responses, which climbs before you hit a hard limit;
• queue depth and the age of the oldest job;
• the failed-jobs count, with a notification whenever it is above zero;
• time since the last webhook arrived, for providers that normally send them all day.

Log the provider's request ID with every call, because their support team will ask for it. Subscribe to their status page and changelog, and put credential expiry dates in a shared calendar.

How much of this a third-party API integration needs

Not every integration needs every pattern. A contact form that posts to a CRM needs a timeout, a queue with retries and a failed-jobs alert, which is a small job for a developer who knows the framework. Anything that moves money, stock or enrollments needs idempotency keys, verified webhooks and reconciliation from day one, because failure there means a customer charged twice or an order that never ships.

Get help when several systems must stay consistent with each other, when a provider's documentation and its real behavior disagree, or when an existing integration fails in ways nobody can reproduce. Designing for those cases is a large part of our API development and integration work, and keeping integrations healthy afterwards belongs to maintenance and support. For store and ERP connections in particular, see integrating an ERP with your online store.

Frequently asked questions

How many times should a failed API call be retried?

Inside a web request, once, quickly, and only for connection errors or timeouts. In a background job, several attempts spread over minutes to hours with growing delays is normal. After the last attempt, move the job to a failed list and alert a person.

What is the difference between a timeout and a circuit breaker?

A timeout limits how long one call may take. A circuit breaker looks at many calls and, when most are failing, stops making them for a short period. You need the timeout first, because the breaker relies on calls failing quickly enough to be counted.

How do we test what happens when the provider is down?

Fake the failures. In automated tests, make the HTTP client return timeouts, 500 and 429 responses and duplicate webhooks, then assert that nothing is charged or created twice. Before launch, point staging at an address that never answers and watch what happens.