Sagas and the outbox pattern: transactions without distributed transactions

Placing an order means creating it, reserving stock and charging a card, in three services with three databases. No single transaction covers them. Something still has to keep them consistent when a step fails, a message is lost or a process crashes halfway.

The answer is a saga, plus two building blocks every saga needs: the transactional outbox and idempotent consumers.

Why not a distributed transaction?

Two-phase commit makes several databases commit atomically: everyone prepares and votes, then everyone commits. It works inside one database. Between services it blocks:

Brokers, NoSQL stores and third-party APIs don’t speak prepare/commit anyway. Helland’s “Life beyond Distributed Transactions” draws the conclusion: at scale, coordinate entities with messages instead (see also the consensus article).

Sagas

A saga is a sequence of local transactions T₁…T_n, each committed in one service. Each step that might need undoing has a compensating transaction C_i. If T_k fails, C_k−1…C₁ run in reverse.

A compensation is not a rollback: it undoes the business effect and the original step stays in history. A charge is refunded, not un-charged.

Designing compensations

Order the steps by what can still be undone:

• Compensatable steps come first: create a pending order, reserve stock.
• The pivot is the point of no return, usually the charge. Once it commits, the saga completes.
• Retriable steps come after: they can fail transiently but never for a business reason, so they are retried until they succeed. Anything irreversible, like the confirmation email, goes here.

Choreography or orchestration

Choreography suits two or three loosely related steps. Once there are compensations to get right, orchestration is easier to reason about, test and monitor.

What sagas give up: isolation

A saga is atomic, consistent and durable, but not isolated: its intermediate states are committed, so others can read stock that is about to be released. The fixes are per step:

• Semantic locks: mark records PENDING and have others refuse, wait or show it.
• Commutative updates: debit and credit instead of set-balance.
• Reread before writing, like optimistic locking.
• Order by risk: steps whose half-done state does most harm go after the pivot.

The dual-write problem

Every step has the same shape: change my database, then tell someone. Two independent writes always leave a crash window between them, and the process that would retry is the one that crashed.

The transactional outbox

Write the event to an outbox table in the same transaction as the state change. A relay publishes it: by polling the table (simple, adds load and latency) or by change data capture from the WAL or binlog, e.g. Debezium (low latency, commit-ordered). Delivery is at-least-once. Partition by aggregate id to keep per-entity order (see the Kafka article).

Idempotent consumers

At-least-once delivery means every consumer sees duplicates. “Exactly once” in practice is at-least-once delivery plus idempotent processing.

Durable execution

The orchestrator must survive its own crashes. Durable execution engines (Temporal, Step Functions, Durable Functions) record each step’s result, so you write the saga as plain sequential code and a crashed run replays to where it stopped. Steps can still run twice, so they stay idempotent.

A saga and its outbox in code

A minimal orchestrator: steps with compensations, a durable log to resume from, and business failures (compensate) told apart from transient ones (retry).

// An orchestrated saga. Each step is a local transaction in one service; each has a
// compensation that semantically undoes it. Progress is persisted, so a crashed
// orchestrator resumes where it stopped instead of starting over.
type Step = {
  name: string;
  run: (ctx: OrderCtx) => Promise<void>;        // must be idempotent: it may run twice after a crash
  compensate?: (ctx: OrderCtx) => Promise<void>; // must be idempotent and must not fail permanently
};

const placeOrder: Step[] = [
  { name: "createOrder",  run: (c) => orders.create(c.orderId, "PENDING"), compensate: (c) => orders.cancel(c.orderId) },
  { name: "reserveStock", run: (c) => inventory.reserve(c.orderId, c.items), compensate: (c) => inventory.release(c.orderId) },
  { name: "chargeCard",   run: (c) => payments.charge(c.orderId, c.total) }, // the pivot: after this we only go forward
  { name: "approveOrder", run: (c) => orders.approve(c.orderId) },           // retriable, can't fail for business reasons
];

export async function runSaga(sagaId: string, steps: Step[], ctx: OrderCtx) {
  const log = await sagaLog.load(sagaId); // { completed: string[], state: "running" | "compensating" | ... }
  if (log.state === "running") {
    for (const step of steps) {
      if (log.completed.includes(step.name)) continue; // resuming after a crash
      try {
        await step.run(ctx);
        await sagaLog.markCompleted(sagaId, step.name);
      } catch (err) {
        if (!isBusinessFailure(err)) throw err; // transient: retry the saga later, from here
        await sagaLog.setState(sagaId, "compensating");
        log.state = "compensating";
        break;
      }
    }
  }
  if (log.state === "compensating") {
    // Undo completed steps in reverse order. Compensations are retried until they succeed.
    for (const step of [...steps].reverse()) {
      if (!log.completed.includes(step.name) || !step.compensate) continue;
      await retryForever(() => step.compensate!(ctx));
      await sagaLog.markCompensated(sagaId, step.name);
    }
    return sagaLog.setState(sagaId, "failed");
  }
  return sagaLog.setState(sagaId, "succeeded");
}

The outbox write on the producer, and the dedupe insert on the consumer:

-- Producer side: the state change and the event commit in the same local transaction.
BEGIN;
  UPDATE orders SET status = 'APPROVED' WHERE id = $1;
  INSERT INTO outbox (id, aggregate_id, type, payload)
  VALUES (gen_random_uuid(), $1, 'OrderApproved', $2);
COMMIT;
-- A relay (a poller, or CDC reading the write-ahead log) publishes outbox rows in order,
-- then marks or deletes them. If it crashes in between, it publishes again.

-- Consumer side: record the event id in the same transaction as the effect.
BEGIN;
  INSERT INTO processed_events (event_id) VALUES ($event_id);  -- primary key: a duplicate fails here
  UPDATE shipments SET status = 'READY' WHERE order_id = $order_id;
COMMIT;
-- On a unique-violation error, the event was already handled: acknowledge it and move on.

References

1. Garcia-Molina, H. & Salem, K. (1987). Sagas. SIGMOD. doi:10.1145/38713.38742
2. Helland, P. (2007). Life beyond Distributed Transactions: an Apostate’s Opinion. CIDR. cidrdb.org
3. Gray, J. & Reuter, A. (1992). Transaction Processing: Concepts and Techniques. Morgan Kaufmann. sciencedirect.com
4. Richardson, C. (2018). Microservices Patterns, ch. 4: Managing transactions with sagas. Manning. manning.com
5. Kleppmann, M. (2017). Designing Data-Intensive Applications, ch. 9 and 11. O’Reilly. dataintensive.net
6. Debezium. Outbox Event Router. debezium.io
7. Leach, B. (2017). Implementing Stripe-like Idempotency Keys in Postgres. brandur.org. brandur.org/idempotency-keys