What should your order system do when a webhook arrives twice, late or not at all?

Treat the event as a signal, not a complete ledger

A webhook is a provider calling your endpoint after something changes. It can be a fast way to update an order screen, but the request itself is not proof that every earlier change was received or that a later change will arrive next. Shopify says its webhook delivery is not always guaranteed and does not guarantee ordering within or across topics. Stripe likewise says events may arrive out of order and may be delivered more than once. These are documented delivery properties, not rare theoretical exceptions. A system that assumes one neat sequence can mark a paid order unpaid or trigger dispatch twice after a retry.

Separate three questions in the design: did we receive a genuine request, did we record it, and did we apply the right business effect? An HTTP success response answers only the sender's delivery attempt. It does not by itself prove that stock moved, an invoice was issued, or a staff member saw an exception. The business should keep a durable event inbox with provider, account, event identifier, event type, object identifier, receipt time, processing state and a safe reference to the raw payload. The order ledger remains the place to see accepted business transitions. Staff should see both the current order state and any unresolved event, rather than a single reassuring badge.

Verify, store and acknowledge the delivery

The receiving endpoint should verify the provider's signature using the provider's documented method and the unmodified request body before it trusts the content. Reject requests that fail verification. After verification, write the event identifier and minimum processing record to durable storage. A uniqueness rule on provider, account and event ID prevents the same delivery from creating a second inbox row. Only then acknowledge receipt according to the provider's timing requirements. The handler should avoid a long chain of remote calls while the provider waits for its HTTP response. Queue the business work, expose failures to operators and use controlled retries. Store secrets outside the draft or logs, rotate them through the normal operations process, and keep access to raw payloads narrow because they may contain personal information.

An event identifier handles repeat delivery of the same event. It does not solve every duplicate business effect. Stripe notes that two separate Event objects can sometimes concern the same object and event type. Therefore the dispatch action needs its own uniqueness rule, such as one fulfilment instruction for an order and fulfilment stage. A payment provider may send different event types that both imply the same customer payment outcome. Make the business transition conditional on the current state, not on the number of notifications. Do not mark every unexpected event as a harmless duplicate: a second event can also contain a real later change. The inbox lets a reviewer tell the difference without replaying blindly.

Apply state changes without assuming arrival order

Suppose a payment succeeds at 10:02, an order update is created at 10:03, and the update notification reaches your system first. A naive handler might write “pending” from the older payment event when it finally arrives, overwriting the newer order view. Do not solve this by sorting solely on the timestamp in the webhook. Stripe specifically warns that distinct snapshot events can share a created timestamp and that created is not a reliable event order or deduplication key. Shopify recommends using timestamps to help organize its webhooks, yet also recommends fetching authoritative records for reconciliation. The safe approach depends on the provider and object: fetch the current object when the transition matters, compare its identifiers and state with the local record, then apply an allowed transition.

Define the allowed transitions with the business owner. “Payment pending” may become “paid”, “failed” or “needs review”; “paid” should not become “pending” merely because an older notification arrived. A refund is a separate financial action, and a cancelled order is not identical to a returned parcel. Link each change to the source event or an explicit operator action. If a provider's object cannot be fetched, put the order in review rather than guessing. For high-impact actions such as dispatch or entitlement activation, require the relevant confirmed state and make that action independently repeat-safe. This gives staff a traceable explanation when a customer asks why a parcel was held.

Find what never reached your endpoint

A perfect event handler cannot process a delivery it never sees. Shopify explicitly recommends reconciliation jobs because webhook delivery can be missed or mishandled. For each provider and relevant object type, define a regular query for records changed since a saved checkpoint. Use a deliberate overlap window and stable pagination so a change near the edge of one run is not skipped; deduplicate the retrieved records when they reappear in that overlap. Compare authoritative provider state with the local order, payment and fulfilment records. A mismatch goes to an exception queue with the provider reference, observed states and proposed repair, not straight to an invisible overwrite. A manual “sync this order” action can help support staff investigate a single case, provided it uses the same safe transition rules.

Keep a checkpoint only after a page or batch has been processed durably. If a job crashes midway, rerun the uncommitted portion. Monitor last successful run, count of unresolved mismatches, oldest unresolved case and whether the provider API is unavailable. These are operational signals, not proof of zero missing orders. The query design depends on what that provider actually exposes; some APIs filter by update time, others require a different cursor or object route. Test access scopes and rate behaviour with the real integration. When the provider is unreachable, say so in the staff view and postpone risky downstream actions rather than silently showing a stale state as current.

Rehearse a damaged delivery path

Use a non-production order to rehearse four cases: the same signed event delivered twice; a later state delivered before an earlier state; a handler failure after inbox storage but before the business update; and an event absent from the inbox but visible through the provider API. For each, inspect the inbox, order history, downstream dispatch instructions and reconciliation report. The expected result is one valid business action, a clear unresolved exception where facts are incomplete, and a safe route to retry. Also test a bad signature, which should create no trusted business effect. Never paste real secrets or customer payloads into the test report.

Give the operations team an incident card. It should say who pauses dispatch, how to find the provider object, whether the event is already recorded, which state transition is being proposed, and who approves a manual repair. Preserve the old and new values when a correction is made. Review the queue after a deployment or provider change, because a new event shape can fail while the HTTP endpoint still appears healthy. The goal is a system that can explain and repair divergence. A green webhook delivery chart alone tells the owner little about whether the correct customer received the correct order exactly once.

Agree a recovery threshold before an incident. A missing marketing preference update, a missing payment confirmation and a missing dispatch instruction have different consequences; they should not share one silent retry policy. Define which discrepancies stop fulfilment immediately, which can wait for the next reconciliation run, and which need a customer-facing correction. Keep a small set of known test objects so operators can verify the integration after changing its endpoint or permissions. A provider dashboard may report delivery attempts, while your own inbox reports accepted events; compare both sides when investigating. If a manual replay is needed, record who initiated it and which provider object was checked first. The replay should use the normal verification and transition path. A special administrative shortcut that writes “paid” directly into the order table removes the very evidence needed to understand the failure. After the repair, confirm the downstream result as well as the screen state: one dispatch instruction, the right customer message and no second financial action.