Webhook Idempotency in Production

The bug report said a customer was charged twice for the same course enrollment. Our code looked correct - we checked payment status before granting access. The problem was two identical Razorpay webhook deliveries arrived 400 milliseconds apart, both passed the status check before either committed, and both wrote an enrollment record.

Payment webhooks are not fire-and-forget notifications. They are unreliable-delivery guarantees: the provider will retry on non-2xx responses, network timeouts cause duplicate delivery, and some providers send the same event to multiple endpoints during migration. Idempotency is not optional.

The Idempotency Pattern for Webhooks

Every webhook handler follows the same four steps: verify signature, extract idempotency key, check-if-processed, process-and-record. The idempotency key comes from the provider - Razorpay sends x-razorpay-event-id, Stripe sends event.id, PayU sends mihpayid + status combination.

Webhook POST arrives
       │
       ▼
  Verify signature (HMAC/RSA)
       │ fail → 401, stop
       ▼
  Extract idempotency key (event_id)
       │
       ▼
  INSERT INTO webhook_events (event_id, ...) 
  ON DUPLICATE KEY → return 200, stop
       │ new row
       ▼
  Process business logic (enroll, fulfill, notify)
       │
       ▼
  UPDATE webhook_events SET status = 'processed'
  Return 200

The critical insight: the idempotency check and the processing must be atomic or sequenced so two concurrent deliveries cannot both pass the check.

Database-Backed Idempotency (MySQL)

MySQL is the idempotency store, not Redis. Payment events must survive Redis restarts and cluster failovers. A simple webhook_events table with a unique constraint on event_id is the foundation.

CREATE TABLE webhook_events (
  id            BIGINT AUTO_INCREMENT PRIMARY KEY,
  event_id      VARCHAR(255) NOT NULL,
  provider      ENUM('razorpay', 'stripe', 'payu') NOT NULL,
  event_type    VARCHAR(100) NOT NULL,
  payload       JSON NOT NULL,
  status        ENUM('received', 'processing', 'processed', 'failed') DEFAULT 'received',
  processed_at  DATETIME NULL,
  created_at    DATETIME DEFAULT CURRENT_TIMESTAMP,
  UNIQUE KEY uk_event_id (event_id)
);

The handler inserts a row with status 'received'. If the unique constraint fires, return 200 immediately - the event was already handled. If insert succeeds, proceed to business logic.

Handling Concurrent Duplicate Delivery

The insert-or-detect pattern handles sequential retries. Concurrent duplicates - two requests hitting at the same millisecond - need additional protection. I use a two-phase approach: insert with status 'processing', do the work, update to 'processed'.

If a row exists with status 'processing' and created_at is less than 30 seconds ago, the concurrent request waits briefly and re-checks. If still 'processing' after 30 seconds, assume the first handler crashed and take over.

async function handleWebhook(eventId, payload) {
  const inserted = await db.query(
    "INSERT IGNORE INTO webhook_events (event_id, provider, event_type, payload, status) VALUES (?, ?, ?, ?, 'processing')",
    [eventId, payload.provider, payload.type, JSON.stringify(payload)],
  );
  if (inserted.affectedRows === 0) {
    const existing = await db.query("SELECT status FROM webhook_events WHERE event_id = ?", [eventId]);
    if (existing[0]?.status === "processed") return { status: 200, message: "already processed" };
    if (existing[0]?.status === "processing") return await waitAndRetry(eventId);
  }
  try {
    await processPayment(payload);
    await db.query("UPDATE webhook_events SET status = 'processed', processed_at = NOW() WHERE event_id = ?", [eventId]);
    return { status: 200 };
  } catch (err) {
    await db.query("UPDATE webhook_events SET status = 'failed' WHERE event_id = ?", [eventId]);
    throw err;
  }
}

Provider-Specific Considerations

Each payment provider has quirks that affect idempotency design.

Stripe includes a timestamp in its signature. Reject events older than 5 minutes to prevent replay attacks. Razorpay does not timestamp - rely on event_id uniqueness alone. PayU sends form-encoded payloads, not JSON - normalize before hashing.

Always verify signatures before touching the idempotency table. An unsigned or invalid request should get 401, never 200. Returning 200 to invalid requests teaches attackers your endpoint exists.

Business Logic Idempotency

The webhook_events table prevents duplicate processing of the same event. But business logic must also be idempotent independently. If enrollment creation uses INSERT without a unique constraint on (user_id, course_id), two different event IDs for the same payment (provider bug or manual retry) could still double-enroll.

I add unique constraints at the business level: one enrollment per (user_id, course_id), one order per payment_id. The webhook handler checks these before creating records.

For side effects that cannot be deduplicated by constraint (sending email, triggering fulfillment), use the event_id as an idempotency key in those downstream calls too.

Engineering Takeaways

• Use the provider's event ID as the idempotency key - never generate your own.
• Store idempotency records in MySQL with a unique constraint, not Redis.
• Verify webhook signatures before any processing or idempotency check.
• Handle concurrent duplicates with a processing status and timeout-based takeover.
• Add business-level unique constraints (user + course, order + payment) as a second defense.
• Return 200 for already-processed events - providers interpret non-2xx as failure and retry.

Trade-offs I Would Consider

Option: MySQL unique constraint (insert-or-detect)

• Pros: Durable, survives restarts, audit trail built in, familiar ops.
• Cons: Adds DB write latency (~2-5ms), table grows over time (needs retention policy).

Option: Redis SET NX with TTL

• Pros: Sub-millisecond check, automatic expiry, no table growth.
• Cons: Not durable (数据 loss on restart), no audit trail, TTL expiry allows reprocessing.

What I Would Do in Production

Create the webhook_events table before going live - not after the first duplicate incident.

Return 200 fast. Offload heavy processing (email, analytics) to a queue after recording the event. Providers timeout at 10-30 seconds.

Set up a daily job to alert on webhook_events stuck in 'processing' or 'failed' status.

Archive webhook_events older than 90 days to cold storage. You need them for dispute resolution, not forever in hot MySQL.

Test with provider sandbox tools: Stripe CLI (stripe trigger), Razorpay test webhooks, PayU test transactions. Send the same event twice and verify single processing.

FAQ

Q: Should I process webhooks synchronously or async?

Record the event and return 200 synchronously (under 500ms). Process business logic async via Redis Streams or SQS if it takes longer. Providers retry on timeout - async prevents retry storms.

Q: What HTTP status should I return for duplicate events?

200 OK. The provider's goal is confirmed delivery. A duplicate is successfully handled - return 200 so they stop retrying.

Q: How do I handle webhook events out of order?

Payment events can arrive out of order (authorized before created). Store the event regardless, and use state machine logic: only transition forward (created → authorized → captured). Ignore backward transitions unless they represent refunds.

Q: What if my handler crashes mid-processing?

The 'processing' status with timeout takeover handles this. After 30 seconds, the next delivery attempt takes over. Add a manual replay CLI for ops to re-process 'failed' events.

Q: Do I need separate endpoints per provider?

Separate endpoints (POST /webhooks/razorpay, /webhooks/stripe) with shared idempotency logic. Each provider has different signature verification and payload formats.