How Do You Recover Missed Webhook Events After an Outage?

Recover missed webhook work by defining the affected population, checking what evidence still exists, and assigning each obligation to replay, current-state reconciliation or explicit exception handling. Restarting the endpoint and emptying a queue do not establish that recovery is complete. You need evidence of what the receiving application actually did, including work acknowledged before it was safely processed and work that expired before anyone investigated.

I would start with the business record that may now be wrong. A missed inventory notification, a delayed access change and an uncertain dispatch instruction need different repairs. The endpoint's availability is only the beginning of recovery; the consequential question is whether the business can again rely on the integration's result.

This paper develops an operational recovery method from twelve authoritative product documents. It compares documented mechanisms without treating them as interchangeable guarantees. The analysis and worked example are original synthesis, not an empirical outage study or a claim about a deployed customer's systems. Its research question is how to establish a defensible recovery boundary when normal event delivery has failed and automatic retries may no longer be available.

Define what must be recovered before choosing replay

Three different objects can require repair: a transport delivery, a historical business transition and a current-state projection. Redelivering a retained event addresses the first. Applying a missed cancellation or recording an approval may address the second. Refreshing an inventory balance from the source may address the third. A recovery procedure that calls all three “replay” makes its success criteria ambiguous.

Suppose the local record should show whether an account is currently active. A trustworthy current-state query may repair it without replaying every intermediate notification. If the obligation is to preserve each approval decision and its justification, the same query cannot reconstruct the missing history. If a notification previously initiated an external shipment, current order state alone may not establish whether the shipment was already requested.

For each affected workflow, state the required result and what constitutes evidence of it. Identify the authoritative source, local record, permitted side effects and person responsible for exceptions. Distinguish a business obligation that is still required from an event whose original action is now obsolete. Recovering yesterday's notification should not automatically instruct the system to perform an action that today's policy forbids.

This scope separates the present question from whether a signed webhook may update a business record. That earlier analysis establishes the trust and authorization conditions for processing. Here I assume those conditions must remain in force and examine how to identify and repair a population of missing or uncertain work after an outage.

Establish an affected interval without pretending clocks are complete evidence

An incident timestamp is a useful search boundary, but rarely a complete definition of the affected population. The first alert may follow the first missed event. Work already accepted before the outage may remain unprocessed afterward. A deployment may have acknowledged requests successfully while silently rejecting their contents. The relevant interval is therefore connected to the failure mechanism, not merely to the period when a health check was red.

I would record at least the last justified application checkpoint, the first known incorrect result, the restoration time and the cutoff chosen for the recovery run. State which clocks those times use. Event occurrence, provider receipt, attempted delivery, queue insertion and local commit are different observations. A selection based on one must not be described as a complete selection based on another.

Pub/Sub's seek documentation illustrates the problem: timestamp seek uses publication time assigned by Pub/Sub, and the documentation notes both possible server clock skew and the difference between publish-request arrival and the time the source event occurred. It also describes delivery consistency after seek as eventual rather than instantaneous (Google Cloud, n.d.). A time-based recovery boundary consequently needs interpretation, even when the platform offers a convenient rewind operation.

Choose and explain any overlap around the suspected interval. The purpose is to include uncertain boundary cases, not to invent a universal safety margin. Duplicate-safe processing can make a deliberate overlap acceptable, but it cannot recover an event that was never generated or retained. Where the start boundary remains unknown, label the resulting population as a candidate set and keep that completeness limitation visible.

Compare the provider's recovery surfaces

Do not assume a webhook provider will retry until your service returns. GitHub explicitly states that failed webhook deliveries are not automatically redelivered; recovery can be manual or implemented through a scheduled retrieval-and-redelivery process (GitHub, n.d.-a). Its redelivery documentation limits the available deliveries to the past three days and specifies who may initiate redelivery (GitHub, n.d.-b). Those facts make detection time and authorized access part of the recovery design.

Stripe exposes a different combination. Its recovery guidance describes automatic retry for up to three days and an Events API recovery procedure limited to events created within the last thirty days. The documented delivery_success=false filter selects failures at at least one endpoint, not necessarily the particular endpoint being repaired (Stripe, n.d.). A returned event is therefore a candidate requiring local context, not proof that this application missed it.

EventBridge rule-target delivery has another boundary. AWS documents a default retry policy of up to twenty-four hours and 185 attempts for retriable errors, followed by dropping the event when exhausted unless an appropriate dead-letter path retains it (Amazon Web Services, n.d.-c). These are documented defaults for that mechanism, not an assertion about a particular account's configuration or other EventBridge products.

Build a recovery matrix for the actual integration: available delivery history, event retrieval, archive, dead-letter storage, authoritative object query and change feed. Record scope, retention, permissions, filtering and the meaning of each timestamp. A provider's thirty-day event listing does not imply thirty days of delivery evidence, and an event archive does not necessarily contain every event the business intended to produce.

Check whether the recovery evidence survived

Retention is an active constraint during an incident. Older records can expire while the team diagnoses the primary failure. I would identify the earliest evidence at risk and preserve the minimum authorized recovery material before spending hours on a general replay script. That preservation should retain provenance and access controls, not create an uncontrolled dump of customer payloads on an engineer's laptop.

The age of an item in a dead-letter queue can be misleading. For SQS standard queues, AWS documents expiration based on the original enqueue timestamp, while the dead-letter queue's oldest-message age indicates time since the move. FIFO queues have different timestamp behavior on that move (Amazon Web Services, n.d.-e). Operationally, apparent time available in quarantine can differ from the retention budget that actually remains.

Record original event time, observed receipt time, first failure, move time and applicable expiry where available. If only some are known, do not synthesize the others from a dashboard label. Prioritize evidence capture by actual expiration risk and business consequence. A high-priority historical obligation with an expiring payload may require attention before a larger but easily reconstructible current-state backlog.

The record also needs to distinguish “no events existed” from “no events remain.” An empty archive created after the outage cannot establish the first claim. A filtered archive may legitimately exclude the event types under investigation. Ask whether collection was active, which population it covered, and whether the necessary decryption and read permissions still work. The recovery source itself is a dependency to verify.

Treat dead-lettering as another fallible delivery path

A dead-letter queue is useful only if events reached it and remained available. EventBridge documents both successful sends to a DLQ and failures to send there, along with event attributes describing the rule, target, error and exhausted retry condition. Certain errors, including missing permissions or unavailable targets, can bypass ordinary retries and go directly to the configured DLQ (Amazon Web Services, n.d.-f).

That means the recovery inventory should include failure-path evidence, not just the current queue contents. A storage permission change can damage the primary target and its fallback. A queue with zero messages might reflect successful processing, missing capture, expired messages or a mistaken destination. Distinguishing those possibilities requires configuration and delivery evidence from the relevant period.

Azure Event Grid's documented delivery-and-retry behavior likewise distinguishes retryable failures from errors that are dead-lettered or dropped. Dead-lettering is not enabled by default, and successful delivery status is based on the endpoint's HTTP response (Microsoft, 2026). The document describes a specific Event Grid delivery model; it should not be silently substituted for namespace-topic or other messaging behavior.

I would keep two failure classifications: why delivery failed, and why the business operation remains unresolved. A corrected DNS address may remove the first while leaving uncertainty about a previous partial effect. A schema correction may allow parsing but expose a now-invalid business action. Fixing the technical cause earns a controlled retest; it does not automatically authorize replaying every quarantined record.

Build a disposition ledger before moving messages

The recovery ledger should identify the logical obligation independently of its latest transport container. Retain the configured integration and tenant, event identity where supported, business object, required effect, original failure evidence and current disposition. Link each recovery attempt to that record. This lets multiple retained deliveries, retries and observations describe one obligation without inflating the number of distinct tasks.

My proposed initial dispositions are: already completed, eligible for controlled replay, repair through current-state reconciliation, intentionally superseded, and unresolved. Each needs a reason and evidence reference. “Already completed” requires a trustworthy effect or commit record; a previous HTTP success is inadequate when the failure involved acknowledging before processing. “Superseded” requires a business rule, not a convenient way to reduce the queue count.

Selection also requires an eligibility check against the intended endpoint, account and event types. An event retrieved because another endpoint failed may already have completed here. Conversely, a locally acknowledged event may never appear in a provider's failed-delivery list even though the application lost it. Join provider evidence to the receiver's durable records and, where possible, the authoritative business population rather than treating either log as exhaustive.

The distinction between uncertainty and non-execution matters when the receiver invokes another service. If a retained event triggered a remote operation whose answer was lost, resubmitting the event may recreate that operation. Preserve the earlier operation identity and reconcile it before initiating a replacement. A recovery ledger should not erase an uncertain effect merely because the same event is available to send again.

Choose the narrowest replay mechanism that answers the problem

Replay should have an explicit selection and destination boundary. EventBridge's replay creation procedure lets the operator choose a time frame and either all rules or specified rules on the original event bus (Amazon Web Services, n.d.-b). That choice affects which consumers see historical events. Replaying broadly to repair one consumer can also invoke consumers that already completed their work.

Before execution, capture the intended event types, tenant or account scope, interval, destination and processing version. Identify whether current routing rules and transformations will interpret old events differently from the original configuration. If the repair depends on historical behavior, prove that the relevant handler and schema remain available. If it intentionally uses new behavior, describe the resulting action as a new interpretation of retained evidence.

Do not assume the recovery tool can perform every desired selection. SQS redrive does not support filtering or modifying messages during the redrive operation, according to its documentation (Amazon Web Services, n.d.-d). A requirement to exclude completed effects or transform an old schema therefore needs a compatible processing design or another controlled recovery method. A selection requirement cannot be satisfied by a checkbox that the service does not provide.

I would separate a read-only classification pass from the effect-producing pass where the system supports it. The first estimates the eligible population and records exclusions. The second runs a bounded cohort against an explicit destination. This is a proposed operational control, not a claim that every provider offers a native dry run. If the platform cannot isolate a cohort safely, resolve that limitation before authorizing broad replay.

Preserve identity and ordering across the repair path

A transport identifier is not always stable through recovery. AWS states that SQS redrive assigns new message IDs and enqueue times and resets message retention; redriven messages can interweave with new producer messages at the destination (Amazon Web Services, n.d.-d). Deduplicating solely by the newly assigned transport ID can therefore misclassify old business work as new.

Carry the authenticated application event identity and intended effect identity through any recovery envelope, while preserving the original payload rather than rewriting its history. The envelope can add recovery-run metadata without claiming that a newly created message is the original delivery. Ensure that the receiving code actually reads the stable identity, and that records used to suppress completed effects still cover the age of the replayed material.

Ordering needs a separate decision. EventBridge documents that archived events are not necessarily replayed in their original archive order and that archive arrival can lag receipt on the bus (Amazon Web Services, n.d.-a). Those properties constrain what replay completion proves. A finished replay can cover the selected retained population without establishing that the consumer observed every historical transition in its original sequence.

For a state projection, an old notification may appropriately trigger a current-state refresh. For a transition-sensitive workflow, use the producer's documented sequence or revision contract where available and quarantine ambiguous cases. Do not infer total order from convenient IDs or timestamps. If newer live work continues while old events return, the handler must prevent historical state from overwriting a justified newer state while still preserving history the business needs.

Coordinate replay with automatic retries and live work

Manual recovery and normal delivery are not necessarily mutually exclusive. Stripe warns that automatic retries can overlap a manual recovery script, and that manually processing an event does not itself end the provider's retry schedule (Stripe, n.d.). The local effect boundary must consequently recognize the same work across both routes. Pausing only the recovery script does not necessarily stop all sources of repeated delivery.

Map every active origin of work: provider retries, live events, queue redelivery, scheduled reconciliation and the operator's repair run. Decide which remain enabled and how they share the same completion record. Avoid a separate “recovery” handler that bypasses normal tenant checks or effect suppression merely because its caller is an administrator. Administrative authorization to run a repair is not evidence that each event should produce a fresh action.

A pause policy needs equally careful interpretation. Stopping dispatch prevents additional admissions through that control, but work already accepted downstream may continue. Mark what has been submitted, what is in flight and what is confirmed complete. The SQS redrive documentation notes that cancellation leaves already moved messages at their destination (Amazon Web Services, n.d.-d). A cancelled control-plane task is not a rollback of business effects.

Preserve a recovery-run identifier that connects selection, operator authority, processing version and observed outcomes. Keep it distinct from event identity so another recovery run can revisit unresolved work without making completed effects new. This record makes interruption and restart reviewable: an operator should resume from known dispositions rather than repeat the entire incident interval because the previous console session disappeared.

Use state reconciliation when retained history is insufficient

When the obligation is current-state correctness, a supported change feed or authoritative query can be more appropriate than event replay. Microsoft Graph delta query uses opaque continuation and state tokens, warns against assuming item order, and documents synchronization resets that can require a full synchronization (Microsoft, 2025). These properties require a deliberate merge process rather than treating a cursor as an editable timestamp.

A proposed reconciliation process should preserve the query scope, finish the required page sequence, merge repeated observations safely and commit the resulting checkpoint consistently with the local updates. If a process crashes halfway through, it should not advertise a checkpoint beyond the work it durably applied. The exact transaction and checkpoint design depends on the resource and storage system; the documentation does not provide a universal implementation for every integration.

Keep synchronization deletion semantics resource-specific. A record absent from one response page is not automatically deleted at the source. A full refresh may need an explicit completeness boundary before removing local records, while local-only fields and business decisions may need preservation. Replacing a whole table with a partial export can turn recovery into a second incident even when every returned object is valid.

Most importantly, state convergence does not restore missing historical evidence. An account that is active now might have been cancelled and reinstated during the outage. Updating its current status can be correct without establishing when those transitions occurred or whether a one-time action happened. Report the repaired projection and the unresolved history separately. If historical reconstruction is required but no authoritative record survives, that is a real limitation to disclose.

Budget recovery against live demand

Recovery capacity is the capacity left after essential live work and shared dependencies have been accounted for. Sending an entire backlog as fast as an API accepts it can merely transfer the bottleneck to the database, downstream service or operator exception queue. Measure useful completed obligations per unit time, not only events submitted to the replay tool.

For an explicitly hypothetical calculation, assume a system can sustain 200 comparable operations per second and live traffic consumes 150. Its ideal spare rate is 50 per second. A backlog of 6,000 such operations would then require at least 120 seconds at that constant spare rate. These assumptions exclude retries, variable work, contention and failures; the calculation is a lower-bound planning example, not an observed throughput result or completion promise.

A controlled cohort should test the expensive path, not only the easiest records. Observe completion latency, new failures, duplicate suppression, dependency saturation and unresolved age. Increase admission only while the selected limits remain satisfied. If retries or downstream costs rise, reduce the recovery rate or stop admission while preserving already submitted work. The objective is net reduction in unresolved obligations without degrading the live service beyond its accepted policy.

Priorities may legitimately differ. A stale access decision can be more urgent than a delayed analytical update, while an expiring historical record can need preservation before either is processed. Record the rule rather than allowing queue order to make the business decision accidentally. Recovery should also have a bounded operator workload; creating thousands of exceptions without ownership does not make the integration safer.

Work through a recovery that an empty queue cannot close

Consider the fictional Northline Instruments and a deliberately bounded set of 120 candidate obligations identified from retained source records and local processing evidence. The example assumes those records cover the selected interval; it makes no claim about any real provider or incident. Seventy obligations have trustworthy completion evidence. Thirty are eligible for replay after a parser repair. Twelve require only current-state correction. Eight have uncertain prior external effects.

The first decision is to exclude the seventy completed effects from new execution, while retaining their evidence in the recovery ledger. The thirty replay candidates pass a controlled test and then complete under the assumed design. The twelve projection corrections are independently compared with the authoritative current state. At that point, 112 obligations have supported dispositions: seventy already complete, thirty newly completed and twelve reconciled.

The eight uncertain effects remain unresolved even if all transport queues are empty. Replaying their source events would be unsafe under the example's assumptions because earlier external actions might already have occurred. They require outcome evidence or an explicit business decision about replacement and possible duplication. A progress report that calls the whole recovery complete because it submitted thirty messages hides those eight obligations.

Now remove the example's coverage assumption. Suppose the provider's retained history begins after the suspected start of the outage. The same 112 supported dispositions remain useful, but they no longer establish coverage of the entire incident. The final report must disclose that earlier gap. Finishing every item you can see is different from proving that you found every item you were responsible for recovering.

Close recovery with evidence of coverage and disposition

I would require three separate claims before closing the incident: the affected population has a justified boundary, each identified obligation has an evidence-backed disposition, and normal processing has resumed under the intended controls. None can substitute for another. A perfect backlog repair does not fix a still-broken subscription, and a healthy subscription does not resolve expired historical work.

The acceptance exercise should include an acknowledged-but-unprocessed event, a completed effect with a lost acknowledgement, a record that expires during investigation, overlapping live and recovery deliveries, a new transport ID, an old schema and a crash during checkpoint advancement. These are proposed tests, not executions performed for this paper. Use controlled resources and observe the actual business state and external effect count where the test environment permits it.

Retain a concise incident evidence package: selection scope and limitations, source retention facts, initial and final disposition counts, handler version, recovery authority, failed-path observations, exceptions and confirmation of normal processing. Sensitive payloads should be minimized and access-controlled. Another engineer should be able to explain why a record was replayed, reconciled, excluded or left unresolved without reconstructing the incident from a console screenshot.

The approach has limits. Documentation does not prove that a particular provider instance, SDK, queue policy or database behaved as intended. A quiet source may offer no authoritative completeness signal. A missing event may be unrecoverable, and a completed effect may lack enough evidence for a confident determination. The method makes those limits explicit; it cannot create records that were never captured.

Keep the recovery design proportionate to the consequence

A disposable dashboard notification may need only a reliable refresh, while a workflow that changes access or initiates external work needs stronger evidence. The case for a detailed recovery ledger is strongest when omission and duplication have material consequences or when an incident spans multiple processing systems. Applying the same machinery everywhere can add cost and complexity without improving the decision.

There is also a genuine tradeoff between preserving history and minimizing retained data. Keeping every event forever is not the proposed answer. Choose retention and minimal evidence according to the business obligation, provider limits and the supported repair path. Where current-state refresh is enough, preserving a full sensitive event history may be unnecessary. Where history is essential, acknowledge the required storage and governance rather than pretending a short-lived delivery log supplies it.

For questions about a provider's capabilities and evidence, the supplier assurance review addresses that adjacent decision. The recovery question remains narrower: what can this integration still prove, what can it safely repair, and which obligations remain unresolved? Those answers should survive a change of operator and a restart of the recovery tool.

Request an integration recovery assessment when the current runbook ends at “restart the endpoint” or “redrive the queue.” Bring a sanitized event contract, retention settings, processing diagram and an example failure trace. I can help turn those into a bounded recovery procedure, evidence-based closure criteria and a prioritized list of gaps before the next outage tests them.