What are webhooks, and what happens when the delivery fails

A webhook is an HTTP request another system sends to a URL you control, the moment something happens there, so you find out without having to ask. Your server receives a POST. That is the whole idea, and every definition on the first page of results says roughly that.

Then they stop, and that is where the trouble starts. A webhook is not a notification. It is a delivery, and a delivery has a schedule, an expiry and a point where the sender gives up on you. Those three things decide whether the integration survives contact with production. None of them are in the definition.

Why they exist

Webhooks exist because polling wastes nearly everything it does.

Say you want to know when a customer's subscription changes. Without a webhook you ask on a timer. Every thirty seconds is 2,880 requests a day for that one resource, and if the thing you are watching changes twenty times a day, then 2,860 of those requests exist to be told nothing happened. You pay for all of them. So does the system you are asking. And you still find out up to thirty seconds late.

The webhook inverts it. The system that already knows makes one request, when there is something to say.

That inversion is obviously correct, which is why the comparison is where most explainers both start and finish. It is also the least interesting thing about webhooks, because it describes the good case. The design questions all live in the bad one.

A delivery, not a ping

Here is the shift that matters: the sender is not notifying you, it is trying to hand you something, and it will keep trying.

What that means in practice is a set of numbers you should be able to name for any sender you depend on. Ours look like this. An event is posted to your HTTPS endpoint, and if that request fails or times out, it is tried again up to five times in total, backing off at roughly 10, 20, 40 and 80 seconds. That is about 150 seconds of trying before the delivery is written off. Each individual attempt gets 15 seconds to come back before we stop waiting.

Two and a half minutes is a deliberate amount of generosity. It covers a deploy and a restart. It does not cover a bad afternoon, and no retry ladder should, because at some point the honest thing is to stop and write the failure down where you can find it. Every attempt lands in a delivery log with its status, duration and attempt number.

Worth being blunt about the limit there, because it is the question you should ask every sender and the answer is not always the one you want: that log is a record, not a replay button. There is no console action today that re-drives a delivery we already gave up on. If your endpoint was down for an hour, the events from that hour are yours to backfill from your own side, which is a good reason to make sure the state a webhook tells you about is also reachable by an ordinary API call.

There is a second clock running alongside that one, and it catches people. The signature covers a timestamp that is stamped once, when the event is first dispatched, and then carried unchanged on every retry. So the fifth attempt arrives signed with a timestamp about two and a half minutes older than the request itself, against a window that closes at five minutes. The margin is real but it is thinner than it looks, and the ways it gets eaten are worth knowing: we wrote up why a delivery gets rejected after five minutes separately, because the usual cause turns out to be queueing inside the receiver rather than the clock drift everyone suspects first.

The signature, and why it expires

Every delivery is signed, because a URL that accepts POSTs is a URL anybody can POST to. Ours uses HMAC-SHA256 over the timestamp, a dot, and the body exactly as sent, with a secret shown once when you create the endpoint and rotatable from the console. On your side parseWebhookEvent() checks it and hands back the event.

The part worth understanding is why there is a time limit at all, since the signature is already cryptographic. An HMAC has no expiry of its own. A correctly signed request stays correctly signed forever, so anyone who ever captured one delivery could replay it at you a year later and the maths would still check out. The timestamp is what turns a signature into a signature and a freshness check, and five minutes is the window.

This is also the place where a working integration breaks for a reason that looks like a crypto bug and is not. The hash is over the raw bytes. Parse the body to JSON and re-serialize it before hashing and you have hashed something subtly different, which fails identically to a wrong secret. If you have hit that against Stripe it is the same shape of mistake, and the five causes of a failed signature check in Next.js are ranked by how often each one is actually it.

Fan-out is the part that bites

One event does not mean one request. It means one request per endpoint subscribed to it, and that multiplication is where webhook systems hurt themselves.

We learned this from our own. There are 129 events you can subscribe to across users, workspaces, billing, credits, workflows, audience and email, and an endpoint can subscribe to all of them with a wildcard. Early on, one event with N matching endpoints simply enqueued N jobs and nothing looked at the total. Then a campaign went out to 50,000 recipients. email.sent fires once per recipient, so a single wildcard endpoint turned that into 50,000 delivery jobs, and with the receiver down and five attempts each, up to 250,000 HTTP requests and 250,000 rows in the delivery log.

Nothing was broken, exactly. Every one of those deliveries was behaving as designed. The design just had no opinion about the total.

The ceilings that came out of it are the ones worth copying if you are building this yourself: a cap on endpoints per organization (100 today, above the most any plan sells), because every wildcard endpoint multiplies everything else; a budget of 240 deliveries a minute per organization, set deliberately below the rate of a large campaign and deliberately above normal traffic; and a limit of 3 deliveries in flight at once per organization, so one slow receiver cannot hold every worker while everybody else waits.

The last one is the fairness lever and it is the one that needed the most care. A delivery over the cap is delayed, never dropped. A delivery that is throttled gets a row in the log saying so, rather than vanishing, because a webhook you cannot account for is worse than one that did not arrive.

When a webhook is the wrong choice

Required reading before you wire your architecture to one, and the section the generic explainers skip entirely.

When you need ordering. Deliveries carry no ordering guarantee and retries break order on their own. If event B must be processed after event A, the sequence has to be in the payload and your handler has to enforce it.

When you need exactly-once. Retries mean duplicates. A receiver that timed out after doing the work still gets the event again, and from the sender's side those two cases are indistinguishable. Your handler has to be safe to run twice on the same event, which is the same problem and the same fix as idempotency in billing: key the work on something stable from the payload and make the second run a no-op.

When the receiver cannot be reached. Only HTTPS addresses are accepted, and the sender has to be able to route to you. A service behind a VPN or a laptop in dev does not qualify without a tunnel.

When you need an answer. A webhook is one-way. Your status code is the whole reply. Anything that needs a value back is a request you make, not an event you receive.

When the payload would be huge, or stale by arrival. Send an identifier and let the receiver fetch the current state. Our events carry the workspace, user, subscription and plan precisely so most handlers do not need a round trip, but that is a convenience for the common case, not a reason to treat the payload as your copy of the record.

What your handler owes the sender

Verify before you parse, and verify against the raw bytes.

Return a 2xx quickly and do the work afterwards. Fifteen seconds sounds generous until the handler is writing to three systems, and a slow handler also burns your own in-flight budget, so the sender ends up throttling you for being slow at the exact moment you are busiest.

Deduplicate on the event id rather than trusting that you will see each one once. Ours arrives as x-buildbase-event-id and is the same value for every endpoint subscribed to that event and on every retry of it, which is exactly what you want as a key: store it, and make a second sighting a no-op. Sort by something in the payload rather than trusting arrival order.

And log the deliveries you reject. The failure mode that costs a whole afternoon is not a webhook that never arrives, it is a webhook that arrives, gets refused by your own verification for a reason you did not record, and is counted as delivered by nobody.

Where this goes next

Webhooks are one of the ways a backend tells you something happened. The others are a queue you drain yourself and a stream you hold a connection to, and the choice between them is mostly a question of who owns the retry. If you are sizing up what a backend hands you before you build on it, what a SaaS backend actually includes covers the rest of the surface this one sits in.