Self-Healing CI: Recovering a Delayed Webhook Delivery
By Daniel Zoghalchali·Latchkey
A step that times out waiting for a webhook usually saw the callback arrive late, not never -- waiting a little longer or re-checking clears it.
The problem
A step that waits for an inbound webhook/callback (a deploy confirmation, a build-trigger ack, an external job-complete signal) times out because the delivery was delayed. The event was sent and is valid; the upstream simply delivered it late. A human re-runs the step or extends the wait and the callback is there.
Typical symptom
Error: timed out after 120s waiting for deploy webhook
# the webhook arrived shortly after the wait window closed
Why it happens
Webhook senders queue and retry deliveries, so a callback can arrive seconds to minutes after the triggering event under load or upstream backpressure. A fixed wait window sized for the common case can close just before a delayed delivery lands.
The delivery is delayed, not lost: re-checking or waiting a bit longer finds the event, which is why the same flow succeeds on a retry with no change.
The manual fix
Manual handling for a delayed webhook:
Re-run the step or re-check for the event after a short wait.
Increase the wait window to absorb normal delivery variance.
Poll the source of truth (status API) as a fallback instead of relying solely on the push.
How this gets automated
A delayed webhook is distinguishable from a missing one by whether the event eventually arrives. A self-healing CI pipeline detects the wait-timeout, extends the wait or re-checks for the delivery, and only escalates if the event never arrives, which is the real signal of a dropped delivery rather than a slow one.
Frequently asked questions
What causes Self-Healing CI: recovering a delayed webhook delivery?
A step that waits for an inbound webhook/callback (a deploy confirmation, a build-trigger ack, an external job-complete signal) times out because the delivery was delayed. The event was sent and is valid; the upstream simply delivered it late. A human re-runs the step or extends the wait and the callback is there.
How do I fix Self-Healing CI: recovering a delayed webhook delivery manually?
[object Object]
Can Self-Healing CI: recovering a delayed webhook delivery be fixed automatically?
A delayed webhook is distinguishable from a missing one by whether the event eventually arrives. A self-healing CI pipeline detects the wait-timeout, extends the wait or re-checks for the delivery, and only escalates if the event never arrives, which is the real signal of a dropped delivery rather than a slow one.
