Building Reliable Webhooks with Circuit Breakers and Retry Logic

Your affiliate program does not run on one screen anymore. Every approved commission, qualified lead, paid payout, and fraud flag needs to travel somewhere else in real time — a partner's internal system, a CRM, a Slack channel, a lead-distribution pipeline. Webhooks are the nervous system that carries those signals. And like any nervous system, the failures are rarely dramatic. They are quiet. An endpoint starts responding in eight seconds instead of eighty milliseconds. A partner deploys a bad release and their receiver returns 500 for an hour. A DNS change points a URL at nothing. If your webhook layer is naive, one sick endpoint does not just miss its own messages — it backs up the queue, burns worker capacity retrying a corpse, and slows delivery for every healthy integration behind it.

That is the problem this article is about: how to keep a webhook pipeline healthy when the endpoints on the other end are not. Everything below is how TrackingMD actually delivers outbound events, not a whiteboard ideal.

The failure modes nobody warns you about

Outbound webhooks look trivial until you run them at volume. Then four distinct illnesses show up:

• The slow endpoint. It answers, but slowly. Left unbounded, a single laggard ties up a worker for the length of its timeout, over and over, on every retry.
• The dead endpoint. It refuses connections or 500s on everything. Retrying it immediately and forever is pure waste — you are pouring compute into a wall.
• Duplicate delivery. Retries mean the same event can arrive more than once. If the receiver treats "commission approved" as an increment, a retry double-pays. Ordering compounds this: events do not always arrive in the sequence they were emitted.
• Silent data loss. After the last retry, where does the event go? If the answer is "nowhere," you have lost a financial event and nobody knows.

Solving these is not one feature. It is five patterns working together: bounded timeouts with exponential backoff, idempotency keys, a circuit breaker, a dead-letter queue, and delivery analytics to see all of it. Take them in order.

Bounded timeouts and exponential backoff

Every delivery runs as a queued job on a dedicated webhooks queue, isolated from the rest of the platform's work. The HTTP request itself carries a hard timeout — 30 seconds by default, configurable per endpoint — so a slow receiver can stall its own delivery but never a worker indefinitely.

When a delivery fails, it does not retry instantly and it does not retry on a flat interval. It backs off on a widening curve: roughly one minute, then five minutes, then thirty minutes, then two hours. A delivery gets up to five attempts, and the whole retry window is capped at 24 hours. The logic behind exponential backoff is simple and humane: most endpoint outages are transient. A partner's deploy finishes in ten minutes; a network blip clears in seconds. Widening the gap between attempts gives the far side room to recover instead of hammering it while it is already down — the difference between checking on a patient every few hours and shaking them awake every thirty seconds.

There is also a rate limiter in front of delivery. If an endpoint is configured with a throughput ceiling, throttled deliveries are simply re-released with a delay. Crucially, being rate-limited is not counted as a failure — a throttled webhook stays pending and never counts against the endpoint's health. Slow-because-busy and broken are different diagnoses, and the system treats them differently.

Idempotency keys: making retries safe

Retries are only safe if the receiver can recognize a repeat. Every delivery is stamped with a deterministic idempotency key — a SHA-256 hash of the event ID combined with the endpoint ID. The important word is deterministic: the same event to the same endpoint always produces the same key, and that key is stable across every retry of that delivery. It rides along in the request as an X-Idempotency-Key header.

That single guarantee is what lets a receiver dedupe. Store the keys you have already processed, and when a retry arrives after your 200 was lost on the wire, you recognize it and no-op instead of double-applying. We hand you the key; you get exactly-once effects on top of at-least-once delivery.

Each request also carries an HMAC signature so the receiver can verify authenticity. One detail that has bitten integrators before, worth stating plainly: the signature is computed over the timestamp joined to the payload — timestamp-dot-payload — not the payload alone. Signed requests also carry that timestamp, and a tolerance window rejects anything too old, which closes the door on replay attacks.

The circuit breaker: stop kicking a dead endpoint

This is the pattern that protects everything else. Every endpoint has a health record with a circuit that lives in one of three states, borrowed straight from electrical engineering:

The transitions are driven by consecutive failures. Cross an alert threshold — three failures in a row — and the endpoint's owners get a notification, once, so a sustained outage does not spam them. Cross the circuit-breaker threshold — five consecutive failures — and the circuit trips open. While it is open, the dispatcher stops even attempting delivery to that endpoint. New events are skipped at the source rather than queued up to fail.

Open is not permanent. After a cooldown, the next dispatch flips the circuit to half-open and lets one probe through. If that probe succeeds, the circuit closes, the failure counter resets, and normal flow resumes. If the probe fails, the circuit snaps back to open and waits out another cooldown. A single healthy response is enough to bring an endpoint back; a single failure while convalescing sends it back to bed.

The payoff is systemic. A dead endpoint stops consuming worker cycles the moment it is diagnosed. Its illness is quarantined to its own integration and cannot bleed capacity away from the healthy endpoints sharing the queue.

The dead-letter queue: nothing gets silently dropped

Backoff and circuit breaking buy time, but some endpoints stay broken past the last retry. What happens to the event then matters, because in an affiliate platform these are financial signals — a commission was approved, a lead qualified, a payout completed.

When a delivery exhausts its retries, it is not discarded. It is moved into a dead-letter queue with everything you need for a post-mortem: the full original payload, the last response status and body, the attempt count, the failure reason, and the timestamp. It is quarantined, not deleted.

From there it can be replayed. Replay mints a fresh delivery from the stored payload — reusing the same deterministic idempotency key, so a receiver that already saw the event still deduplicates it — and dispatches it again. So when a partner tells you their endpoint was down all morning and asks you to resend, you do not reconstruct events by hand. You replay the dead-lettered batch once the endpoint is back, and idempotency keeps the replay safe even for anything that half-landed the first time.

Delivery analytics: you cannot fix what you cannot see

Reliability you cannot observe is just luck. Every delivery records its duration, so the analytics layer can compute true latency percentiles — P50, P95, and P99 — over any window, per endpoint or across the whole organization. Averages lie about tail behavior; percentiles are how you actually see the slow-endpoint failure mode before it becomes an outage.

Alongside percentiles, the dashboard surfaces:

• Daily success rates trended over time, so a slow degradation is visible as a slope, not a surprise.
• Most-failed event types, which points you at whether it is your fraud events or your commission events that are struggling.
• Latency warnings that flag any endpoint whose P95 has crept past 80 percent of its own configured timeout — an early symptom that it is about to start timing out under load.
• Retry health — first-attempt success rate, how many deliveries were rescued by a retry, average attempts, mean time to deliver, and the live backlog waiting on a backoff right now.

Together these turn webhook reliability from a black box into a chart you can read at a glance, the way a chart tells a doctor more than a patient's own account does.

The whole system, working as one

None of these five patterns is sufficient alone. Backoff without a circuit breaker still wastes cycles on a dead endpoint. A circuit breaker without a dead-letter queue still loses the events it refused to send. Idempotency without retries is a solution to a problem you never created. Analytics without any of it is just watching the fire. Put together, they form a pipeline that degrades gracefully: it slows down for the sick, quarantines the dead, never loses a signal, and tells you exactly what is happening the whole time. As integration graphs grow denser — more partners, more event types, more downstream systems depending on your data arriving on time — that resilience stops being a nice-to-have and becomes the thing that lets you add the next integration without fear that it will take down the last one.