How to Monitor Webhook Delivery Success Rate in Production

A webhook can fire correctly, get captured correctly, and still never reach the system it was relayed to. The sender sees a 200 and moves on. Nothing in your logs looks wrong unless you are specifically watching the leg between your relay and the downstream destination. By the time a customer notices their order sync stopped working three days ago, you are reconstructing a timeline from memory instead of a delivery log.

Webhook delivery success rate is the share of outbound relay attempts to a destination that finish with a 2xx response, counting both first tries and retries. It is the number that tells you whether "the webhook fired" actually meant anything downstream. A source-side 200 only proves the payload was captured; it says nothing about whether your relay logic delivered it.

What Counts as a Failed Webhook Delivery?

A delivery attempt fails when the destination times out, refuses the connection, or answers with anything outside the 2xx range — most commonly a 5xx from an overloaded receiver, a 401/403 from an expired auth header, or a 3xx the relay does not treat as success. A single failure is normal; every outbound HTTP call fails sometimes. What matters is the trend: is this destination failing 1% of attempts, or 40%?

Separate transient failures (one bad attempt that a retry clears) from structural ones (every attempt to this destination fails the same way). A success rate that recovers on retry within minutes is a blip. A success rate that stays at 0% for an hour is a broken integration, and no amount of retrying fixes a destination that is down or misconfigured. The distinction matters because the fix is different: a transient failure needs nothing from you, while a structural one needs a human to rotate a credential, fix a URL, or find out why the destination stopped accepting traffic altogether.

What to Log for Every Delivery Attempt

You cannot alert on data you never captured. At minimum, record these fields per attempt, not per webhook — a single inbound webhook can generate several delivery rows if it is relayed to multiple destinations or retried after a failure:

1. Destination URL and the relay rule or endpoint it belongs to
2. Attempt number (1, 2, 3…) and the timestamp of the attempt
3. Outcome: success, failed, or retrying, plus the next scheduled attempt time if retrying
4. The destination's response status code, or the error if the connection itself failed
5. A short snippet of the response body (enough to diagnose, not the full payload) and the request duration in milliseconds

A delivery log with just these five fields answers almost every incident question without re-sending anything: which destination is degraded, since when, and whether it is a timeout, an auth failure, or the destination actively rejecting the payload.

How to Set an Alert Threshold Before Customers Notice

A flat "alert on any failure" rule pages you constantly and gets muted within a week. A rolling success-rate threshold, evaluated over a fixed window per destination, catches real degradation without noise from the occasional dropped connection:

Set the window and thresholds per destination, not globally. A low-volume destination with three requests an hour will swing between 0% and 100% on tiny sample sizes, so give it a longer window or a minimum-attempt floor before the threshold applies.

Build It Yourself or Use a Hosted Delivery Log?

Rolling your own version of this means a table for delivery attempts, a cron job to evaluate the rolling window per destination, and a place to send the alert — all before you have relayed a single real webhook. That is a reasonable afternoon of work for one destination, and a maintenance burden once you have a dozen.

CanHook keeps a per-destination delivery log for every relay rule automatically — attempt number, status, response snippet, and duration for each try, with no separate logging code required. Retries run on exponential backoff and the log shows exactly which attempt succeeded or why the last one failed, which is the same information the table above assumes you already have.

If you are already relaying webhooks to more than one downstream service, see how the delivery log and mock responses work before you build the same thing from scratch. CanHook keeps delivery logs for 30 days regardless of plan, which is separate from — and usually longer than — the request-capture retention window itself, so the delivery history for an incident tends to outlast the raw captured payload that triggered it.

What Causes a Retry Storm and How Do You Stop One?

A retry storm happens when a destination goes down and every queued and newly failing delivery starts retrying at the same cadence, hammering it right as it is trying to recover. Fixed-interval retries make this worse; exponential backoff makes it self-limiting.

CanHook's relay worker retries a failed delivery at 60 seconds, then 5 minutes, then 25 minutes, then 2 hours, up to a per-rule attempt limit (default 3, configurable 0–10). After the last configured attempt fails, the delivery is marked failed and stops retrying — it does not retry forever, and it does not silently disappear; it stays in the log as a terminal failure you can inspect or manually replay. A minute-cadence worker processes due deliveries in priority order, so paying accounts do not sit behind a backlog caused by someone else's broken endpoint.

If you are writing the receiving side, the same principle applies in reverse: respond with a 200 as fast as possible and do slow work after, so a temporarily slow handler does not look like a failed delivery and trigger a retry you didn't need:

app.post('/webhooks/inbound', (req, res) => {
  // Acknowledge immediately, before any real work.
  res.status(200).send('ok');

  // Do the slow part after the response is already sent.
  queue.enqueue('process-webhook', {
    id: req.body.id,
    receivedAt: Date.now(),
  });
});

Is Monitoring Overkill for a Handful of Webhooks?

If you have one destination and it either works or you'd notice within the hour anyway, a full alerting pipeline is more infrastructure than the problem deserves — a delivery log you can glance at is enough. The point where this stops being optional is the moment a webhook feeds something a customer would notice breaking: billing sync, order fulfillment, account provisioning. At that point the cost of finding out from a support ticket is higher than the cost of a five-minute alert setup. A support ticket also arrives late by definition — the customer only opens it after the failure has already cost them something, while a threshold alert fires while the destination is still failing and before anyone downstream has had to notice.

The honest tradeoff is not "build monitoring" versus "skip it" — it's deciding how many destinations justify per-destination logging before you need it, and picking that point before an outage picks it for you.

Create a free CanHook endpoint, point one of your existing webhook sources at it, and send a real test payload through a relay rule. You will see the delivery attempt, its status, and its response in the log within seconds, which is the fastest way to know whether this level of visibility is worth adding to the rest of your webhooks.