CDC with Debezium, and the dual-write problem

What it is

The dual-write problem is what happens when a single logical operation must update two systems and there is no transaction spanning both. You write to the database, then publish to Kafka. Two writes, no atomicity, and four possible outcomes where only two are correct:

DB write    Kafka publish    Result
────────────────────────────────────────────────────────────
success     success          correct
failure     not attempted    correct (nothing happened)
success     FAILURE          DB has the order, no event. Downstream never learns.
failure     success          Event for an order that does not exist. Phantom.

The third row is the common one, because the database write usually succeeds and the publish is the flakier of the two. Its effect is silent divergence: the search index misses a product, the analytics pipeline misses revenue, the notification never fires, and nothing errors anywhere.

Change Data Capture (CDC) solves this by removing the second write entirely. Instead of your application publishing an event, a separate process reads the database's own replication log (Postgres WAL, MySQL binlog, MongoDB oplog) and publishes an event for every committed change. The database transaction is the only write your application makes, and the event is derived from it rather than sent alongside it.

Debezium is the standard implementation: a set of Kafka Connect source connectors that read replication logs and produce change events to Kafka topics.

The confusion to clear up: CDC is not polling a table for changed rows. A WHERE updated_at > ? poll is a different technique with different failure modes: it misses deletes entirely, it misses intermediate values between polls, it depends on the application maintaining updated_at correctly, and it can miss rows entirely because of the interaction between transaction commit order and timestamp assignment. Log-based CDC reads the same stream the database's own replicas read, so it sees every change, in commit order, including deletes.

The problem it solves

Beyond the four-row table above, there are three specific failures worth naming because they are what actually gets reported.

The retry that makes it worse. A team notices the publish can fail and wraps it in a retry. Now the failure mode changes: the publish succeeds on retry 3, but the process crashed between retries, so nothing published. Or the publish actually succeeded and the acknowledgement was lost, so the retry publishes a duplicate. You have converted a lost-message problem into a lost-or-duplicated-message problem.

The transaction that makes it worse still. A team puts the publish inside the database transaction:

@Transactional
public void placeOrder(Order order) {
    orderRepository.save(order);
    kafkaTemplate.send("orders", order.getId(), toEvent(order));  // WRONG
}

This looks safer and is not. The Kafka send is asynchronous and not transactional, so it can complete before the database commits, meaning a consumer can read the event and query the database for an order that is not yet visible. Worse, if the transaction rolls back after the send, you have published an event for an order that will never exist. Putting a non-transactional call inside a transaction does not make it transactional; it makes the failure window harder to reason about.

Ordering. Even when both writes succeed, two concurrent updates to the same row can publish their events in the opposite order to the order they committed in. Downstream applies the older value last, and the state is permanently wrong until the next update. Log-based CDC cannot have this problem because the log is the commit order.

Mechanics

Reading the log

Every database that supports replication maintains a durable, ordered log of committed changes, because that is how replicas stay in sync. CDC attaches to that mechanism.

Database	Mechanism	What you must enable
PostgreSQL	Logical replication slot	`wal_level = logical`, a publication, `pgoutput` plugin
MySQL	Binary log	`binlog_format = ROW`, `binlog_row_image = FULL`, GTIDs recommended
MongoDB	Change streams (oplog)	Replica set (not standalone)
SQL Server	CDC tables	`sys.sp_cdc_enable_db`
Oracle	LogMiner or XStream	Supplemental logging

The MySQL detail matters: binlog_format = ROW (not STATEMENT) is required because statement-based replication logs the SQL, not the resulting row values, and CDC needs values. binlog_row_image = FULL is required for the connector to produce a before image on updates and deletes; with MINIMAL you get only changed columns and the primary key, which breaks any consumer needing the old value.

A Debezium change event:

{
  "before": {"id": 1042, "status": "PENDING", "amount": 4500},
  "after":  {"id": 1042, "status": "SHIPPED", "amount": 4500},
  "source": {
    "db": "shop", "table": "orders",
    "lsn": 24589103, "txId": 8841,
    "ts_ms": 1717430400123,
    "snapshot": "false"
  },
  "op": "u",
  "ts_ms": 1717430400456
}

op is c (create), u (update), d (delete) or r (read, from the initial snapshot). Having both before and after is what lets a consumer compute a delta rather than just observe a new state, which matters for aggregations.

The two phases: snapshot then stream

A connector starting on an existing database must first capture what is already there. Debezium's default (snapshot.mode: initial) takes a consistent snapshot, emitting every row as an op: r event, then switches to streaming from the log position recorded at snapshot time.

The classic problem with this is that a snapshot of a large table blocks or takes hours, and if the connector restarts mid-snapshot it starts over. Incremental snapshotting (the DDD-3 algorithm, Debezium 1.6+) fixes it: the snapshot proceeds in chunks, interleaved with live streaming, resumable after a restart, and triggerable at any time via a signal table for a specific table. This is the feature that makes CDC operationally viable on large databases and it is worth knowing by name, because "how do you add a new table to CDC without stopping the world" is a natural follow-up.

-- Trigger an ad-hoc incremental snapshot of one table, no restart.
INSERT INTO debezium_signal (id, type, data)
VALUES ('adhoc-1', 'execute-snapshot',
        '{"data-collections": ["shop.public.products"], "type": "INCREMENTAL"}');

The transactional outbox: CDC on your terms

Raw CDC on business tables has a real problem: it publishes your schema. Every consumer becomes coupled to your table structure, and a column rename becomes a breaking change for six teams. It also publishes rows rather than events, so a consumer must infer "order was shipped" from a status column changing, which is implicit and fragile.

The transactional outbox pattern fixes both while keeping the atomicity:

CREATE TABLE outbox (
    id            UUID PRIMARY KEY,
    aggregate_type VARCHAR(255) NOT NULL,   -- "Order"        -> routes to a topic
    aggregate_id   VARCHAR(255) NOT NULL,   -- "1042"         -> becomes the message key
    event_type     VARCHAR(255) NOT NULL,   -- "OrderShipped" -> a header
    payload        JSONB        NOT NULL,   -- the event you designed
    created_at     TIMESTAMPTZ  NOT NULL DEFAULT now()
);

@Transactional
public void shipOrder(long orderId) {
    Order order = orderRepository.findById(orderId).orElseThrow();
    order.setStatus(SHIPPED);
    orderRepository.save(order);

    // Same transaction, same database. Atomic with the state change.
    outboxRepository.save(new OutboxEvent(
        UUID.randomUUID(), "Order", String.valueOf(orderId),
        "OrderShipped", toJson(new OrderShippedEvent(order))));
}

One transaction, one database, so the state change and the event are atomic by construction. Debezium then captures the outbox table, and its EventRouter transformation unwraps the row into a proper event: routing to a topic by aggregate_type, using aggregate_id as the message key (which gives you per-order ordering), and putting payload in the message body.

transforms=outbox
transforms.outbox.type=io.debezium.transforms.outbox.EventRouter
transforms.outbox.route.by.field=aggregate_type
transforms.outbox.table.field.event.payload=payload
transforms.outbox.table.field.event.key=aggregate_id

The outbox table grows and must be pruned. Delete rows after they are captured, not before, and note that in Postgres the delete itself produces a CDC event, so configure the connector to drop tombstones for the outbox topic or filter them downstream. A common approach is deleting in the same transaction that inserts (yes, really: the row exists long enough for the WAL to record both the insert and the delete, and Debezium captures the insert), which keeps the table empty. That trick is clever and confusing; a scheduled DELETE ... WHERE created_at < now() - interval '7 days' is easier to explain and I would default to it.

The other half: the inbox

CDC and the outbox guarantee at-least-once delivery. A connector restart replays from the last committed offset, so consumers see duplicates. That is not a defect to be engineered away; it is the guarantee. The consumer side must be idempotent, and the standard mechanism is an inbox:

CREATE TABLE processed_events (
    event_id    UUID PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

@Transactional
public void handle(OrderShippedEvent event) {
    // Insert first: the PK constraint is the dedup mechanism.
    int inserted = jdbc.update(
        "INSERT INTO processed_events (event_id) VALUES (?) ON CONFLICT DO NOTHING",
        event.getEventId());
    if (inserted == 0) return;              // already handled

    applyBusinessLogic(event);              // same transaction as the marker
}

The marker insert and the business logic must be in the same transaction, or you can mark an event processed and then fail to process it. See idempotency for the general treatment.

A worked example: 0.3 percent of orders missing from search

An e-commerce platform. Orders written to Postgres; an OrderCreated event published to Kafka from application code immediately after the commit; a downstream service indexes orders into OpenSearch for the internal order-lookup tool.

Support reported that "some orders don't show up in search." A reconciliation job counted the gap:

orders in Postgres (30 days):     4,218,904
orders in OpenSearch:             4,206,371
missing:                             12,533   (0.297%)

Sampling the missing orders showed no pattern by merchant, amount or region, but a strong pattern in time: they clustered in windows of 30 to 90 seconds, roughly twice a week.

Those windows correlated with Kafka broker restarts and with two brief network incidents. The application code:

@Transactional
public Order placeOrder(OrderRequest req) {
    Order order = orderRepository.save(Order.from(req));
    try {
        kafka.send("orders", order.getId(), toEvent(order)).get(2, SECONDS);
    } catch (Exception e) {
        log.error("failed to publish order event", e);   // and then... nothing
    }
    return order;
}

The catch block logged and swallowed. Every time Kafka was briefly unavailable, the orders committed and the events did not. The system had been reporting its own data loss to a log file for two years, at a rate of about 400 messages a week, in a log nobody alerted on.

Their first attempted fix, which is the natural one, was to not swallow: rethrow and fail the request. This was rejected in review, correctly, because it makes order placement unavailable whenever Kafka is unavailable, coupling the customer-facing write path to a downstream system's availability. That is a worse trade than losing search-index freshness.

The fix: transactional outbox plus Debezium.

Phase 1 (2 weeks)  Add the outbox table. Dual-publish: write to outbox AND
                   keep the direct send. Deploy Debezium reading the outbox.
                   Consumers dedupe on event_id, so duplicates are harmless.
                   This is the safety property that made the migration boring.

Phase 2 (1 week)   Compare: events from the direct path vs events from CDC.
                   Confirmed CDC was a strict superset, by 411 events over
                   the observation window, which is exactly the loss rate.

Phase 3            Remove the direct send. Outbox is now the only path.

Phase 4 (1 week)   Backfill the 12,533 missing orders via an incremental
                   snapshot signal on the orders table.

Measured after three months:

                                  before          after
missing orders (30d)              12,533          0
event publish p99                 41ms (in txn)   n/a (removed from request path)
order placement p99               187ms           149ms   (-20%)
Kafka outage impact on orders     silent loss     CDC lag only, zero loss
outbox table steady size          n/a             ~8,000 rows (7-day prune)
CDC end-to-end lag p99            n/a             340ms

Order placement got faster, by 38 milliseconds at p99, because the synchronous Kafka send with its 2-second timeout was removed from the request path. That is the result worth remembering: the outbox pattern is usually presented as a correctness fix that costs latency, and here it improved both, because a synchronous publish inside a transaction was holding a database connection open for a network round trip.

The failure mode did not disappear, it moved and became visible. A Kafka outage now means CDC lag rather than data loss: the WAL retains the changes, Debezium resumes from its last offset, and events flow when Kafka returns. The one new risk is that an unconsumed replication slot holds WAL indefinitely and can fill the database's disk, which is covered below and is the thing to alert on.

Production evidence

Debezium is a Red Hat project used widely enough that its connector behaviours are the de-facto reference for CDC semantics. Its outbox EventRouter transformation exists as a first-class feature specifically because the outbox pattern is the recommended way to use CDC for event publishing rather than schema publishing.

Netflix's DBLog paper describes a CDC framework built for the same reasons, and its watermark-based approach to interleaving snapshot and streaming (avoiding locks entirely) is the direct ancestor of Debezium's incremental snapshotting. The paper is the clearest published explanation of why the snapshot phase is the hard part.

Airbnb's SpinalTap and LinkedIn's Databus are earlier in-house CDC systems, both built on the same insight that the replication log is the correct source of change events. Databus's writing (from around 2012) predates the term CDC becoming common and makes the argument from first principles.

Shopify has published on using Debezium for their data platform, and their operational writing emphasises replication slot monitoring as the primary risk, which matches the failure mode described below.

The outbox pattern is documented in Chris Richardson's microservices pattern catalog and in Debezium's own blog, and it is the standard recommendation from both. Its presence in a vendor's own documentation as "do it this way rather than the obvious way" is a useful signal.

The debate

CDC on business tables versus the outbox. Raw CDC is less code: point a connector at your tables and events appear. Its cost is that your database schema becomes a published API. Every consumer couples to your column names and types, a rename is a breaking change for everyone, and consumers must infer domain events from row diffs. The outbox costs a table, a transaction participant, and an explicit event design, and buys you a real contract.

My position: outbox for events other teams consume, raw CDC for data pipelines you own. Feeding your own data warehouse or search index from business tables is fine, because you control both ends and a schema change is a coordinated change within one team. Publishing to other teams should always go through an outbox, because the whole value of an event is that it is a stable contract, and a table is not one.

CDC versus application-level events, when neither is broken. The argument for application events is that they carry domain meaning the database does not have: "the customer cancelled" versus "status changed to 4." The outbox gets you that meaning and the atomicity, which is why it is the synthesis rather than a compromise. The remaining argument for direct publishing is one less moving part, and it is only tenable if you genuinely accept the loss rate, which most teams have not measured.

Is CDC's coupling to the database acceptable? A real objection: the connector depends on replication configuration, on the database version, and on schema DDL that the application team changes without telling you. Debezium handles schema changes (it tracks DDL and emits a schema-change topic), but a DROP COLUMN on a column a downstream consumer needs is still a broken pipeline. The mitigation is the outbox again, because the outbox table's schema is stable by design and changes to business tables do not propagate.

The operational risk nobody plans for. A Postgres replication slot that is not being consumed retains WAL indefinitely, and it will fill the disk and take the database down. This is the single most likely way CDC causes an outage, and it is worse than it sounds because it is triggered by the connector stopping, which is exactly what happens during an incident. Alert on pg_replication_slots.confirmed_flush_lsn lag and set max_slot_wal_keep_size (Postgres 13+) to bound the damage, accepting that the slot becomes invalidated and the connector needs a re-snapshot. That trade is right: a broken CDC pipeline is recoverable, a full database disk during an incident is not.

Follow-up Q&A

"Why not just publish the event inside the transaction?"

Because the Kafka send is not part of the database transaction, so putting it inside @Transactional changes nothing about atomicity and makes the failure window harder to reason about. Specifically: the send can complete before the commit, so a consumer reads the event and queries for a row that is not yet visible, and if the transaction then rolls back you have published an event for something that never happened. There is no ordering of the two writes that fixes it, which is the point: you need one write, not two writes in a clever order.

"How does CDC differ from polling updated_at?"

Polling misses deletes entirely (the row is gone, there is nothing to select), misses intermediate values between polls, and depends on every write path maintaining updated_at, which one code path always forgets. It also has a subtle correctness bug: a transaction that starts at T1 and commits at T3 writes updated_at = T1, so a poll at T2 does not see it (not committed) and a poll at T4 with WHERE updated_at > T2 does not see it either, because T1 < T2. Rows are silently skipped, and the rate depends on transaction duration. Log-based CDC reads commit order and cannot have this problem.

"CDC delivers duplicates. How do you handle that?"

You do not eliminate them, you make the consumer idempotent. The standard mechanism is an inbox table: insert the event ID with ON CONFLICT DO NOTHING and skip if the insert affected zero rows, with the marker insert and the business logic in the same transaction. Debezium provides a stable event ID via the source LSN and transaction ID, or you generate one in the outbox row. The important detail is that the dedup marker and the effect must commit together, or you can mark an event handled and then fail to handle it.

"What happens when Debezium is down for an hour?"

Nothing is lost, and that is the whole point: the database's WAL retains the changes, and the connector resumes from its last committed offset when it returns. What you get is lag, which is a visible, alertable, recoverable condition rather than silent loss. The danger is the opposite one: if the connector is down long enough, the replication slot's retained WAL fills the database disk. So the alert is on replication slot lag, and the bound is max_slot_wal_keep_size, and the accepted consequence of hitting that bound is re-snapshotting.

"How do you add a new table to an existing CDC pipeline?"

Incremental snapshot via a signal table: insert an execute-snapshot signal naming the table, and Debezium chunks through it while continuing to stream everything else. No restart, no downtime, resumable. Before this existed (Debezium 1.6, based on Netflix's DBLog watermark approach) the answer was to restart the connector with a new table list and take a fresh blocking snapshot, which is why "can you add a table" used to be a genuinely hard question.

"The outbox table is a hot spot. Does that matter?"

It is an insert-only table with no reads from the application, so it is close to the cheapest thing you can add to a transaction: one sequential insert, no index maintenance beyond the primary key. The real costs are WAL volume, which roughly doubles for the affected transactions, and table growth, which needs a prune job. If insert volume is genuinely extreme, partition the outbox by day and drop old partitions rather than deleting rows, because a DELETE of millions of rows in Postgres creates bloat and vacuum work that an untracked DROP PARTITION does not.

Common misconceptions

"Wrapping both writes in a transaction makes them atomic." Only writes to the same transactional resource are atomic. A Kafka send inside a database transaction is still two independent operations, and the transaction boundary gives you no guarantee about the send at all.

"CDC gives exactly-once." At-least-once. A connector restart replays from the last committed offset, which is a Kafka Connect offset committed periodically, so events after that offset are re-emitted. Consumers must be idempotent. Anyone claiming exactly-once from CDC has either not restarted a connector or has an idempotent consumer and is crediting the wrong component.

"CDC means my consumers see domain events." They see row changes. "Status changed from 3 to 4" is not "the order shipped," and making consumers translate is how you get six different, divergent interpretations of your schema. The outbox is what turns row changes into domain events.

"CDC is asynchronous, so it is eventually consistent and that is a downside." The application-level publish was also asynchronous from the consumer's perspective, and it was additionally lossy. CDC does not add eventual consistency; it removes the loss from a system that was already eventually consistent.

"The replication slot is Debezium's problem." It is the database's disk. An unconsumed slot retains WAL until the disk fills, and the database goes down, not the connector. This is a shared operational concern and it needs an alert owned by whoever gets paged for the database.

Interview delivery note

Say this verbatim: "You cannot make two writes atomic without a transaction spanning both, so the fix is not a better retry, it is having only one write. The outbox pattern puts the event in the same database transaction as the state change, and CDC turns that row into a message." That is the whole idea, and stating it as "one write, not two" is what makes it click.

The senior-versus-staff separator is the replication slot risk. A senior engineer explains the outbox and Debezium correctly. A staff engineer adds that an unconsumed Postgres replication slot retains WAL until the database's disk fills, that this triggers precisely when the connector is down during an incident, and that the mitigation is alerting on slot lag plus max_slot_wal_keep_size with an accepted consequence of re-snapshotting. Knowing how the fix fails is the difference between having read about CDC and having run it.

The second signal is distinguishing where raw CDC is fine from where the outbox is required: your own pipelines versus other teams' contracts. Saying "I would not publish my table schema to six teams" shows you are thinking about coupling rather than mechanism.

The Staff & Team Lead Interview Reference