Example: a webhook with an idempotency key

A payment provider posts deposit events to a Worker, which credits an account in a D1 ledger. The provider resends any event it got no 2xx for, and every event carries an id that stays the same across resends, so the Worker does what the provider's documentation asks: it keeps the ids of the events it has processed and skips any it has seen. The check and the two writes are each correct, and every ordinary test passes.

Run the check

From examples/webhook-idempotency in the repository:

pnpm install
pnpm check

The check runs the scenario in deposits.sim.ts inside workerd: two deposits, a provider that resends up to three times, and a fault (a crash, an error or a lost answer) at every effect the Worker performs. It reports:

VIOLATED  deposits  (seed deposits)

Invariant credited-once is violated: 2 rows of ledger in D1:DB share event_id = "evt-1"

Schedule (run seed ...), events kept: req-1
  ambiguous at req-1#1 effect #1 (D1:DB run)

The credit committed but its answer was lost on the way back from D1, so the handler threw before it recorded the event as processed and answered 500. The provider resent the event, the lookup found nothing, and the account was credited twice. Any failure between the two writes does the same; the trace under the schedule shows every effect of both attempts, and deposits.regression.test.ts, written next to the scenario, replays it until the bug is fixed.

The fix

The lookup and the record of processed events are two statements, so something can always come between them. worker.fixed.ts drops both and makes the credit itself carry the key: schema.fixed.sql makes the ledger's event_id unique and the insert says ON CONFLICT (event_id) DO NOTHING, so a resend changes nothing and still gets its 200. pnpm check:fixed runs the same scenario against it and reports BOUNDED: no violation within the stated bounds, which the report says plainly is not a proof.

Using it in your own project

Copy deposits.sim.ts next to your Worker, replace the requests and the invariants with yours, and point worker at your entry and wrangler config. unique reads the table at the end of every run, so it counts the rows of one table however many statements carry the same key; atMostOncePerKey over d1Write counts committed writes instead, and fits when the write you care about is the only one keyed by that value.

The code

The complete project is examples/webhook-idempotency in the repository.

worker.ts

/**
 * A payment provider's deposit webhook. Each event credits an account in a
 * D1 ledger. The provider resends any event it got no 2xx for, so the
 * handler keeps the ids of the events it has processed and skips a resend
 * it has seen. Run `eutaxy check ./deposits.sim.ts` to see why that is not
 * enough.
 */

export interface Env {
  readonly DB: D1Database;
}

/** The event the provider posts; `id` is the provider's idempotency key, the same on every resend. */
export interface DepositEvent {
  readonly id: string;
  readonly account: string;
  readonly amountCents: number;
}

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);
    if (request.method !== "POST" || url.pathname !== "/webhooks/deposits") {
      return new Response("not found", { status: 404 });
    }
    const event = (await request.json()) as DepositEvent;
    const seen = await env.DB.prepare("SELECT event_id FROM processed_events WHERE event_id = ?1")
      .bind(event.id)
      .first();
    if (seen) return Response.json({ duplicate: event.id });
    await env.DB.prepare("INSERT INTO ledger (event_id, account, amount_cents) VALUES (?1, ?2, ?3)")
      .bind(event.id, event.account, event.amountCents)
      .run();
    await env.DB.prepare("INSERT INTO processed_events (event_id) VALUES (?1)")
      .bind(event.id)
      .run();
    return Response.json({ credited: event.id });
  },
} satisfies ExportedHandler<Env>;

deposits.sim.ts

import {
  atLeastOncePerMessage,
  d1Param,
  d1Rows,
  d1Schema,
  d1Write,
  defineScenario,
  http,
  unique,
} from "eutaxy/cloudflare";

// The provider posts two deposits and resends any it got no 2xx for, up to three times.
const deposits = http(
  [
    { path: "/webhooks/deposits", body: { id: "evt-1", account: "acct-1", amountCents: 1200 } },
    { path: "/webhooks/deposits", body: { id: "evt-2", account: "acct-1", amountCents: 500 } },
  ],
  { retries: 3 },
);

export default defineScenario({
  name: "deposits",
  worker: { entry: "./worker.ts", config: "./wrangler.jsonc" },
  setup: [d1Schema("DB", "./schema.sql")],
  drive: deposits,
  invariants: [
    // No event may credit the account twice: one ledger row per event id.
    unique("credited-once", d1Rows("DB", "ledger"), "event_id"),
    // And no event may be lost: its id is the first parameter of the committed credit.
    atLeastOncePerMessage("every-deposit-credited", deposits, d1Write("DB"), d1Param(0), "id"),
  ],
  bounds: { maxFaultsPerRun: 1, maxRedeliveries: 3, maxConcurrentInvocations: 1, budgetMs: 2000 },
});

worker.fixed.ts

/**
 * The same webhook, made safe to resend: the credit itself is keyed by the
 * event id, so the check and the write are one statement and a resend
 * changes nothing. `eutaxy check ./deposits.fixed.sim.ts` reports BOUNDED.
 */

import type { DepositEvent, Env } from "./worker.ts";

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const url = new URL(request.url);
    if (request.method !== "POST" || url.pathname !== "/webhooks/deposits") {
      return new Response("not found", { status: 404 });
    }
    const event = (await request.json()) as DepositEvent;
    const { meta } = await env.DB.prepare(
      "INSERT INTO ledger (event_id, account, amount_cents) VALUES (?1, ?2, ?3) ON CONFLICT (event_id) DO NOTHING",
    )
      .bind(event.id, event.account, event.amountCents)
      .run();
    return Response.json(meta.changes === 0 ? { duplicate: event.id } : { credited: event.id });
  },
} satisfies ExportedHandler<Env>;