Event sourcing, sagas, and the transactional outbox

What it is

Three patterns that appear together because they solve consecutive problems in the same design: how do you store state as a history, how do you coordinate a transaction across services, and how do you get an event out of a database reliably.

Event sourcing stores state as an append-only sequence of events rather than as a current snapshot. The current state is a fold over the events.

Traditional:  accounts table, balance column: 400
Event sourced: AccountOpened, Deposited(500), Withdrawn(100),
               -> balance = fold(events) = 400

Sagas coordinate a business transaction across services that have no shared transaction. Each step commits locally, and failure triggers compensating actions rather than a rollback.

The transactional outbox solves the dual-write problem: you cannot atomically write to a database and publish to a broker, so you write the event to an outbox table in the same transaction and a separate process publishes it.

What these are confused with: event sourcing being required for event-driven architecture. Publishing events is not event sourcing. Event sourcing means the events are the state, with no separate current-state store as the source of truth. Most systems that say "we do event sourcing" publish events from a state-based store, which is a normal and often better design.

The problem it solves

Event sourcing solves the loss of history. A traditional store overwrites, so "why is this balance 400" and "what was it on Tuesday" and "who changed it" are unanswerable unless you built an audit log, which is an event log with less rigour.

Traditional:  UPDATE accounts SET balance = 400 WHERE id = 1
              -> the previous value, the reason, and the actor are gone
              -> a bug that computed the wrong balance is UNDIAGNOSABLE
                 and UNFIXABLE (you cannot recompute)
Event sourced: the events remain. Fix the fold, replay, get the right answer.

Being able to fix a bug retroactively by replaying is event sourcing's strongest argument, and it is the one that justifies the cost in domains where correctness of history matters: finance, healthcare, anything audited.

Sagas solve the absence of distributed transactions. Two-phase commit exists and is avoided because it holds locks across services for the duration and blocks on coordinator failure (see sagas vs 2PC). A saga trades atomicity for availability: each step commits, and failure is compensated rather than rolled back.

The outbox solves the dual-write problem, which is covered in full on the CDC page: you cannot make a database write and a broker publish atomic, so you make them one write.

Mechanics

Event sourcing: the store schema

CREATE TABLE events (
    stream_id     UUID    NOT NULL,        -- the aggregate: account:4471
    version       INT     NOT NULL,        -- position WITHIN the stream
    event_type    TEXT    NOT NULL,
    payload       JSONB   NOT NULL,
    metadata      JSONB   NOT NULL,        -- causation, correlation, actor
    global_pos    BIGSERIAL,               -- total order, for projections
    occurred_at   TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (stream_id, version)       -- <- the concurrency control
);

The (stream_id, version) primary key is the optimistic concurrency mechanism, and it is the single most important line:

def append(stream_id, expected_version, events):
    # If another writer appended since we read, this INSERT violates the
    # primary key and we know to retry with fresh state.
    for i, e in enumerate(events, start=expected_version + 1):
        db.execute("INSERT INTO events (stream_id, version, ...) VALUES (?, ?, ...)",
                   stream_id, i, ...)
    # A unique violation here = a concurrent write. Reload and retry.

Without it, two concurrent commands both read balance 500, both append a Withdrawn(400), and the balance is -300. The version check is what makes the aggregate a consistency boundary (see DDD tactical).

Snapshots

Replaying 400,000 events to load one aggregate is untenable:

def load(stream_id):
    snap = snapshots.latest(stream_id)           # {state, version}
    state = snap.state if snap else initial_state()
    from_version = snap.version if snap else 0
    for e in events.read(stream_id, after=from_version):
        state = apply(state, e)                   # fold the remainder
    return state

Snapshot every N events (100-1000 is typical):
  load cost: O(N) instead of O(total events)

Snapshots are an optimisation and must never be the source of truth. A corrupt snapshot must be discardable and regenerable from the events, which means the events are authoritative and the snapshot is a cache. If you cannot delete every snapshot and rebuild, you have a state store with an event log attached, not event sourcing.

Upcasting: the schema-evolution problem

Events are immutable and live forever, so a v1 event written in 2019 must still be readable by 2026 code.

# The event changed shape. You cannot rewrite history, so you TRANSFORM
# on read.
def upcast(event):
    if event.type == "OrderPlaced" and event.version == 1:
        # v1 had `total`; v2 splits it into subtotal and tax.
        p = event.payload
        return Event("OrderPlaced", 2, {
            "subtotal_cents": p["total_cents"],   # best-effort migration
            "tax_cents": 0,                        # unknown for old events
            **{k: v for k, v in p.items() if k != "total_cents"}})
    return event

Upcasting is where event sourcing's cost concentrates. Every schema change adds an upcaster that must be maintained forever, and the chain grows: a v1 event may pass through four upcasters to reach v4. The discipline that keeps it manageable is treating events as a published API: additive changes only, never remove or repurpose a field, and version explicitly. That is the same rule as Schema Registry compatibility, applied to a store you can never re-publish.

Event stores are append-only and GDPR requires erasure, which is a direct conflict.

# Encrypt personal data per subject, with a key you CAN delete.
def write_event(stream_id, subject_id, payload):
    key = keystore.get_or_create(subject_id)
    return store.append(stream_id, {
        "non_pii": payload["non_pii"],
        "pii_encrypted": encrypt(key, payload["pii"]),
        "subject_id": subject_id})

# Erasure request: delete the KEY. The events remain, the PII is
# cryptographically unrecoverable.
def erase(subject_id):
    keystore.delete(subject_id)          # the events are now undecryptable

Crypto-shredding is the standard reconciliation: the event stream stays immutable and intact (so replays and projections still work structurally), and the personal data becomes unrecoverable. The caveats are real: the key must never have been backed up somewhere you cannot delete, projections that cached the decrypted value must also be purged, and a regulator may or may not accept cryptographic erasure as erasure. It is the best available answer and it is not unambiguously compliant, which is worth stating rather than presenting it as solved.

Why Kafka is not an event store

This is a common design error and the reasons are specific:

1. NO per-stream optimistic concurrency. Kafka has no "append at version
   N or fail," so you cannot enforce an aggregate's invariant on write.
   You would need an external lock, which defeats the point.

2. Reading ONE aggregate's history means scanning a partition. An event
   store indexes by stream_id; Kafka indexes by offset within a partition.
   Loading account:4471 means reading every event in its partition.

3. Retention. Kafka's model is a retention window; compaction keeps the
   LAST value per key, which is the opposite of what an event store needs
   (all events, forever). Infinite retention is possible and it is not
   what the storage layout is optimised for.

4. No transactional read-modify-write against the log.

Kafka is an excellent event bus and a poor event store. The correct architecture is an event store as the source of truth (EventStoreDB, Marten on Postgres, or a well-designed Postgres table) with Kafka as the transport for downstream consumers, fed by an outbox.

Sagas: orchestration versus choreography

Choreography: each service reacts to events; there is no coordinator.

OrderCreated -> [Payment] PaymentTaken -> [Inventory] StockReserved
             -> [Shipping] ShipmentBooked
Failure: StockReservationFailed -> [Payment] refunds

+ no single point of failure, services are decoupled
- the flow exists NOWHERE: to understand it you read every service
- cyclic dependencies are easy to create accidentally
- debugging "where did this order stop" requires correlating across N services

Orchestration: a coordinator holds the process.

class OrderSaga:
    def handle(self, state, event):
        match (state.step, event):
            case ("started", OrderCreated()):
                return Command(PaymentService, "take_payment", ...), "awaiting_payment"
            case ("awaiting_payment", PaymentTaken()):
                return Command(InventoryService, "reserve", ...), "awaiting_stock"
            case ("awaiting_stock", StockReservationFailed()):
                # COMPENSATE, in reverse order
                return Command(PaymentService, "refund", ...), "compensating"
            ...

+ the flow is IN ONE PLACE, readable and testable
+ the saga's state answers "where is this order"
- the orchestrator is a dependency and can become a bottleneck
- risk of it accumulating business logic that belongs in services

My position: orchestration for anything with more than about three steps or any compensation logic, because the ability to read the flow in one file and query "where did this order stop" is worth the coordinator. Choreography is right for simple fan-out where no coordination is needed.

Compensations are not rollbacks

Rollback:      the transaction never happened. No trace.
Compensation:  a NEW action that semantically undoes the first, and
               BOTH are in the history.

Payment taken, then stock reservation fails:
  compensation = a REFUND, not "un-taking" the payment.
  The customer sees a charge and a refund on their statement.

Three properties compensations must have:

IDEMPOTENT:   a compensation may be retried; refunding twice is a bug.
              Use an idempotency key.
COMMUTATIVE
  where possible: compensations may arrive out of order.
ALWAYS POSSIBLE: this is the design constraint. Some actions cannot be
              compensated (an email sent, a physical shipment). Order the
              saga so IRREVERSIBLE steps come LAST.

"Order the saga so irreversible steps come last" is the design rule that most saga discussions omit, and it is the one that determines whether the pattern works: if sending the email is step 2 of 5, a failure at step 4 leaves a customer told about an order that will not happen.

The outbox, briefly

BEGIN;
  UPDATE orders SET status = 'confirmed' WHERE id = 4471;
  INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
    VALUES ('Order', '4471', 'OrderConfirmed', '{...}');
COMMIT;   -- ONE transaction: the state change and the event are atomic

A relay (Debezium reading the WAL, or a poller) publishes from the outbox. The event is guaranteed to be published if and only if the state change committed, which is the property the dual write cannot provide. Full treatment on the CDC page.

In an event-sourced system the outbox is often unnecessary, because the event store is the log: a relay reads the event stream directly. That is one of event sourcing's genuine simplifications.

A worked example: event sourcing adopted for one aggregate

A payments platform. A recurring class of incident: balance discrepancies that could not be explained or corrected because the current-state store had been overwritten.

The problem:

incident pattern:  a customer reports a balance that does not match their
                   transaction history
investigation:     the balance column is 400; the transactions sum to 450
                   -> WHY? the code that computed it has changed since;
                      there is no record of what it computed or when
resolution:        manually correct the balance, no root cause
frequency:         ~6/month
unresolved discrepancies (cumulative): 1,840 accounts

The balance was a column, so a bug that computed it wrongly was permanent: you could not recompute because the inputs had not been kept.

The decision: event-source the ledger only. Not the whole system.

event sourced:      the account ledger (balances, transactions)
NOT event sourced:  customer profiles, product catalogue, sessions,
                    notifications, everything else

Scoping event sourcing to the aggregate where history is the requirement is the decision that made it tractable. Event-sourcing the whole system is where teams get into trouble, because they pay the upcasting and projection cost on aggregates whose history nobody needs.

The implementation:

CREATE TABLE ledger_events (
    stream_id UUID NOT NULL,                    -- account id
    version   INT  NOT NULL,
    event_type TEXT NOT NULL,
    payload    JSONB NOT NULL,
    metadata   JSONB NOT NULL,                  -- actor, correlation, causation
    global_pos BIGSERIAL,
    occurred_at TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (stream_id, version)
);
CREATE INDEX ON ledger_events (global_pos);     -- for projections

# Balance is a PROJECTION, rebuilt from events.
def balance(account_id):
    snap = snapshots.get(account_id)             # every 500 events
    state = snap.balance if snap else 0
    for e in events.read(account_id, after=snap.version if snap else 0):
        state = apply(state, e)
    return state

Results over the first year:

                              before      after
unexplained discrepancies     ~6/month    0
time to explain a balance     hours-never ~2 min (read the stream)
bugs fixable retroactively    no          yes (fix the fold, replay)

The retroactive fix happened twice and justified the project both times:

Incident: a fee calculation applied a rounding error to ~14,000 accounts
          over 3 months.
  before event sourcing: correct each balance manually; the ORIGINAL
                         amounts are unknown, so the correction is an
                         estimate.
  after:  fix the projection's fold, replay the 3 months of events,
          every balance is recomputed correctly from the original inputs.
  time:   4 hours, exact.

The costs, stated honestly:

upcasters written in year 1:      11
  of which still needed:          11 (they never go away)
projection rebuild time (full):   ~40 min for 180M events
snapshot storage:                 ~2 GB
developer onboarding:             noticeably harder; "where is the
                                  balance stored" has a non-obvious answer

Eleven upcasters in the first year, all permanent, is the cost that compounds, and it is why scoping to one aggregate mattered: eleven upcasters for the ledger is manageable, and eleven per aggregate across forty aggregates would not be.

Then the saga, for the order-to-payment-to-shipping flow:

First attempt: choreography.

OrderCreated -> Payment reacts -> PaymentTaken -> Inventory reacts ->
StockReserved -> Shipping reacts -> ShipmentBooked

problem 1:  "where did order 4471 stop?" required checking 4 services
problem 2:  a new engineer added a handler creating a cycle
            (Shipping emitted an event Payment reacted to, which
            re-triggered Inventory) -> an infinite loop in production
problem 3:  the compensation logic was spread across 4 services and
            was inconsistent: two services compensated, two did not

The cycle is the choreography failure mode: no single place describes the flow, so nothing prevents a new handler closing a loop.

Second attempt: orchestration.

class OrderSaga:
    steps = [
        Step("reserve_stock",  compensate="release_stock"),
        Step("take_payment",   compensate="refund_payment"),
        Step("book_shipment",  compensate="cancel_shipment"),
        Step("send_confirmation", compensate=None),   # IRREVERSIBLE: LAST
    ]

The step ordering was the design decision: the confirmation email is irreversible, so it is last, after every compensable step has succeeded. In the choreographed version the email had been sent on PaymentTaken, which meant customers received confirmations for orders that then failed stock reservation.

                              choreography   orchestration
"where did this order stop"   4 services     1 query
cycles possible               yes            no (the flow is a list)
compensation consistency      4 impls        1
confirmations for failed
  orders                      ~40/month      0

Final:

                              before      after
unexplained balance
  discrepancies               ~6/month    0
retroactive corrections       impossible  2 done, exact
saga visibility               4 services  1 query
premature confirmations       ~40/month   0
event sourcing scope          n/a         1 aggregate of ~40
upcasters maintained          n/a         11 (permanent)

The two decisions that made it work were both about scope: event-source only the aggregate whose history is the requirement, and orchestrate rather than choreograph so the flow lives in one place. Both are refusals to apply the pattern everywhere, which is the recurring judgement with these patterns.

Production evidence

Greg Young's work established event sourcing's vocabulary and the CQRS pairing, and EventStoreDB is the reference purpose-built store. Marten (Postgres) and Axon (JVM) are the widely-used alternatives, and the fact that most production event sourcing runs on Postgres rather than a specialised store is worth knowing.

The (stream_id, version) optimistic concurrency check is universal across implementations, because it is the mechanism that makes an aggregate a consistency boundary. Any event store lacking it cannot enforce invariants.

Kafka's unsuitability as an event store is acknowledged by Confluent's own material: Kafka lacks per-key optimistic concurrency and its partition-based layout makes per-aggregate reads expensive. The recommended architecture is an event store plus Kafka as transport.

Sagas were introduced by Garcia-Molina and Salem (1987) for long-lived database transactions, and the microservices adaptation is Chris Richardson's, whose pattern catalogue documents both orchestration and choreography with the trade-offs above.

Crypto-shredding for GDPR is the documented approach in the event-sourcing community (Michiel Rook's writing and the EventStoreDB guidance), and the caveat about regulator acceptance is equally documented: it is the best available reconciliation of immutability with erasure, not a settled compliance answer.

The transactional outbox appears in Richardson's catalogue and is implemented as a first-class Debezium feature (the outbox event router), which is the strongest signal that it is the standard solution to dual-write.

The debate

Should you event-source? Rarely, and for specific aggregates. The requirement is that the history is the business value: ledgers, audit-critical domains, anything where "how did we get here" is a question users or regulators ask. For a product catalogue or a user profile, the cost (upcasters forever, projection rebuilds, a harder mental model) buys nothing. Event-source the one aggregate that needs it and leave the rest alone, which is what made the worked example tractable.

What is event sourcing's real cost? Upcasters, which are permanent and accumulate. Eleven in the first year for one aggregate, none of which can ever be deleted, because a 2019 event must still be readable. The second cost is cognitive: "where is the balance" has a non-obvious answer, and every new engineer pays that.

Orchestration or choreography? Orchestration for anything with compensation or more than about three steps, because the flow existing in one place is worth the coordinator. The choreography failure mode is the cycle: no single place describes the flow, so nothing prevents a new handler closing a loop, and one did in the worked example. Choreography is right for simple fan-out with no coordination.

How do you handle irreversible steps in a saga? Order them last, which is the design rule most discussions omit. If the confirmation email is step 2 of 5, a failure at step 4 means a customer was told about an order that will not happen, and no compensation un-sends an email. The saga's step order is a design decision driven by reversibility, not by business sequence.

Is Kafka an event store? No, for four specific reasons: no per-stream optimistic concurrency (so you cannot enforce an aggregate invariant on write), per-aggregate reads scan a partition, the retention model is a window rather than forever, and there is no transactional read-modify-write. It is an excellent event bus, and the correct architecture pairs a real event store with Kafka as transport.

Does event sourcing require CQRS? Not formally, and in practice yes: reading an event stream to answer a query is impractical, so you build projections, and a projection is a read model. The pairing is natural rather than mandatory, and the CQRS ladder page treats where to stop on that path.

Follow-up Q&A

"When should you event-source?"

When the history is the business value: ledgers, audit-critical domains, anything where "how did we get here" is a question users or regulators ask. And you scope it to the specific aggregate that needs it rather than the system. In one case a payments platform event-sourced only the ledger, one aggregate of about forty, which made the upcasting cost manageable at eleven upcasters in the first year; eleven per aggregate across forty would not have been.

"What is event sourcing's strongest argument?"

Retroactive correction. A traditional store overwrites, so a bug that computed a balance wrongly is permanent because the inputs are gone. With events you fix the fold and replay: in one case a rounding error affecting 14,000 accounts over three months was corrected exactly in four hours, where the before-state answer would have been a manual estimate per account.

"What is the real cost?"

Upcasters, and they are permanent. Events are immutable and live forever, so a v1 event from 2019 must still be readable by today's code, which means every schema change adds a transform that can never be deleted, and the chain grows as a v1 event passes through four upcasters to reach v4. The discipline that keeps it manageable is treating events as a published API: additive only, never remove or repurpose a field. The second cost is cognitive, because "where is the balance stored" has a non-obvious answer.

"Why is Kafka not an event store?"

Four reasons. No per-stream optimistic concurrency, so you cannot express "append at version N or fail," which is the mechanism that enforces an aggregate's invariant. Reading one aggregate's history means scanning a partition, because Kafka indexes by offset rather than by stream. Retention is a window and compaction keeps the last value per key, which is the opposite of what an event store needs. And no transactional read-modify-write against the log. It is an excellent event bus paired with a real event store.

"Orchestration or choreography for sagas?"

Orchestration for anything with compensation or more than about three steps, because the flow lives in one file, is testable, and answers "where did this order stop" with one query. Choreography's failure mode is the cycle: no single place describes the flow, so a new handler can close a loop, and one did in production in a case I would point to. Choreography is right for simple fan-out where no coordination is needed.

"How do compensations differ from rollbacks?"

A rollback means the transaction never happened; a compensation is a new action that semantically undoes the first, and both are in the history: the customer sees a charge and a refund. Compensations must be idempotent (retried compensations must not double-refund) and, critically, must be possible, which means irreversible steps like sending an email or shipping goods must be ordered LAST in the saga. If the confirmation email is step 2 of 5, a failure at step 4 tells a customer about an order that will not happen.

Is there a large production system genuinely built on event sourcing, or is it all conference talks? Nubank is the strongest public example: the Brazilian digital bank runs its core banking ledger on Datomic, whose data model is an immutable, append-only set of facts with time as a first-class dimension, and reads are queries against a value of the database at a point in time. It is not event sourcing in the "bespoke event store plus projections" sense, which is the point worth making: the database provides the immutable log, so the team did not have to build and operate one, and that removes most of what makes event sourcing expensive. Nubank later acquired Cognitect, the company behind Datomic and Clojure, which is about as strong a commitment to an architectural bet as exists.

The transferable lesson for an interview is the one about scope. Financial ledgers are the domain where event sourcing's costs are cheapest to justify, because immutability and a full audit trail are regulatory requirements you would have to build anyway, and because a balance genuinely is a fold over transactions rather than a value that happens to be stored. The argument that does not transfer is "Nubank does it, so we should" applied to a CRUD service whose audit requirement is a updated_by column.

Common misconceptions

"Publishing events means we do event sourcing." Event sourcing means the events are the state, with no separate current-state source of truth. Publishing events from a state-based store is a normal and often better design, and it is not event sourcing.

"Snapshots are the state." They are a cache. If you cannot delete every snapshot and rebuild from events, the events are not authoritative and you do not have event sourcing.

"Kafka is an event store." No per-stream concurrency control, expensive per-aggregate reads, a retention model designed for a window, and no transactional read-modify-write.

"A compensation is a rollback." It is a new, visible action. Both the original and the compensation are in the history, and some actions cannot be compensated at all, which is why step order matters.

"Event sourcing gives you GDPR compliance problems with no answer." Crypto-shredding (encrypt PII per subject, delete the key) is the standard reconciliation. It has real caveats around key backups, cached projections and regulator acceptance, and it is the best available answer.

Interview delivery note

Say this verbatim: "Event-source the one aggregate whose history is the business value, not the system. The strongest argument is retroactive correction: a rounding error affecting 14,000 accounts over three months was fixed exactly by correcting the fold and replaying, where a state-based store would have made it a manual estimate. The cost is upcasters, which are permanent and accumulate forever." Scope, the argument, and the honest cost.

The senior-versus-staff separator is ordering irreversible saga steps last. A senior engineer explains orchestration versus choreography and compensations correctly. A staff engineer notes that some actions cannot be compensated at all (an email, a shipment), so the saga's step order is a design decision driven by reversibility rather than business sequence, and that putting the confirmation email at step 2 means customers are told about orders that then fail. That rule determines whether the pattern works.

The second signal is the four specific reasons Kafka is not an event store, particularly the absence of per-stream optimistic concurrency. Knowing that "append at version N or fail" is the mechanism enforcing an aggregate's invariant, and that Kafka cannot express it, shows you understand what an event store is for rather than what it stores.

The Staff & Team Lead Interview Reference