Log compaction vs retention, and compacted topics as state

What it is

Kafka topics have a cleanup.policy, and it has two possible values that mean entirely different things:

delete (the default) is time or size based retention. Segments older than retention.ms (default 7 days) or beyond retention.bytes are deleted whole. This treats the topic as a buffer: data flows through and ages out.

compact is key-based retention. Kafka guarantees to retain at least the last known value for every message key in the partition, forever, and it deletes superseded values for keys that have been overwritten. This treats the topic as a changelog whose replay reconstructs a table.

The distinction that matters and is routinely missed: compaction is not compression, and it does not reduce the size of a topic with unique keys. A topic where every message has a distinct key compacts to exactly its original size, because nothing is superseded. Compaction only reclaims space where a key was written more than once. If you set cleanup.policy=compact hoping to save disk on an event stream with UUID keys, you will save nothing and lose your retention policy at the same time.

The second thing it is confused with is a database. A compacted topic gives you the latest value per key, which sounds like a key-value store, but you cannot query it by key. To read a compacted topic you consume it from the beginning and build the state yourself, which is why compacted topics almost always appear underneath something that materialises them (a Kafka Streams state store, a Connect sink, a service's in-memory cache).

The problem it solves

You have a service that needs the current configuration for every one of two million tenants. The options without compaction:

Query a database on every request. Adds a network hop and a dependency to the hot path, and the database becomes a scaling bottleneck shared by every instance.
Cache with a TTL. Now you have staleness bounded by the TTL and a thundering herd when entries expire together.
Publish changes to a normal Kafka topic with 7-day retention. Works for updates, fails on startup: a new instance replaying 7 days of changes only learns about tenants that changed in the last 7 days. The other 1.9 million are unknown.

That last failure is the specific problem compaction solves. With cleanup.policy=compact, replaying from offset 0 gives you at least one message for every key that has ever existed, so a cold start reconstructs complete state. The topic is a durable, replicated, ordered snapshot that also happens to be a live change feed, and those two things being the same object is the useful part.

This is the same insight as event sourcing and as a database's write-ahead log: the log of changes is the primary artifact, and the current state is a projection of it. Kafka is unusual in exposing that log as a first-class product surface.

Mechanics

The cleaner

A partition's log is split into segments. The active segment (the one currently being written) is never compacted. Behind it, a pool of cleaner threads (log.cleaner.threads, default 1) works on the "dirty" portion of the log.

The algorithm, per partition:

Build an offset map: a hash map from key to the highest offset at which that key appears in the dirty section. This lives in memory, sized by log.cleaner.dedupe.buffer.size (default 128 MB, shared across cleaner threads).
Scan the log from the beginning. For each message, if the offset map says a higher offset exists for that key, drop it. Otherwise keep it.
Write the survivors into new segments and swap them in atomically.

Two properties follow from this and they are what make compaction usable:

Offsets are preserved, not renumbered. After compaction, offsets have gaps (offset 500 may be followed by offset 517), but any offset that still exists means the same message it always did. A consumer's committed offset stays valid.
Order is preserved for the messages that survive. Compaction never reorders.

The cleaner selects the partition with the highest dirty ratio (bytes in the dirty section over total bytes) and only starts if that ratio exceeds min.cleanable.dirty.ratio, default 0.5. That default means a compacted topic can be up to twice its compacted size on disk, which surprises people sizing disks. Lowering it to 0.1 compacts more aggressively at the cost of more I/O.

Before compaction (offsets 0-11):

 off:  0    1    2    3    4    5    6    7    8    9   10   11
 key:  A    B    A    C    B    A    D    C    B    A    E    D
 val: a1   b1   a2   c1   b2   a3   d1   c2   b3   a4   e1   d2
                                                  └──── active ────┘

After compaction of the dirty section (offsets 0-8):

 off:  5    7    8    9   10   11
 key:  A    C    B    A    E    D
 val: a3   c2   b3   a4   e1   d2

Every key that ever appeared (A, B, C, D, E) still has a value.
Offsets are gapped but stable. The active segment was untouched.

Tombstones and delete.retention.ms

To delete a key you write a message with that key and a null value. This is a tombstone. The cleaner treats it specially: it removes all earlier values for the key, keeps the tombstone for delete.retention.ms (default 24 hours), and then removes the tombstone too.

The retention window exists for consumers. A consumer that is behind must see the tombstone in order to remove the key from its own materialised state. If tombstones were deleted immediately, a slow consumer would replay the log, never see the deletion, and hold a key forever that no longer exists.

This gives you a hard operational constraint: a consumer that lags more than delete.retention.ms can end up with permanently incorrect state, and there is no error to tell you. It is the compaction equivalent of an offset falling out of retention, but silent instead of loud. If your consumers can lag for days, raise delete.retention.ms accordingly and alert on lag against that threshold rather than an arbitrary one.

// Delete a key: same key, null value.
producer.send(new ProducerRecord<>("tenant-config", tenantId, null));

There is also min.compaction.lag.ms (a message must be at least this old before it can be compacted away, useful to guarantee consumers see every intermediate value for some window) and max.compaction.lag.ms (force compaction even if the dirty ratio is not met, which matters for GDPR: without it, a tombstone might sit uncompacted indefinitely on a low-traffic partition, so the old value is still on disk).

The combination policy

cleanup.policy=compact,delete applies both: compact by key, and delete segments older than retention.ms. This is the right setting when you want a keyed changelog that does not grow forever and you can accept that very old keys eventually vanish. It is what Kafka Streams uses for windowed state stores, where a window older than the retention period is genuinely dead and keeping its key forever would be a leak.

Partitioning is load-bearing

Compaction is per partition, and it retains the last value per key within a partition. If the same key were written to two partitions, you would get two "last values" and no ordering between them. So a compacted topic requires that all messages for a key always land in the same partition, which means the default hash partitioner on the key, and it means you cannot change the partition count of a compacted topic without breaking the mapping. Adding partitions rehashes keys, so a key's new partition has no history and its old partition still holds a value that will never be superseded.

This is a real operational trap: adding partitions to a compacted topic silently corrupts the state it represents. The recovery is to create a new topic with the desired partition count and replay, which for a large state topic is a project. Size the partition count of a compacted topic for its lifetime.

A worked example: a feature-flag service that could not cold start

A platform team ran feature flags for about 340,000 flag-and-segment combinations, published to a Kafka topic consumed by every service instance. Topic settings:

cleanup.policy=delete
retention.ms=604800000        # 7 days
partitions=12

Steady state was fine: flags changed a few hundred times a day, consumers stayed current. The failure appeared during a deployment. A newly started pod consumed from auto.offset.reset=earliest, read 7 days of changes, and had state for the roughly 2,400 flags that had changed that week. For the other 337,600 it had nothing, and its code treated "no entry" as "flag off." A deploy therefore turned off almost every feature flag for the fraction of traffic hitting new pods, ramping as the rollout progressed.

Their interim workaround had been to load a snapshot from S3 at startup and then apply the Kafka topic on top, which worked and introduced a second source of truth that drifted.

The fix:

cleanup.policy=compact
min.cleanable.dirty.ratio=0.1     # tighter than default: this topic is small
delete.retention.ms=86400000      # 24h, and consumers never lag close to that
min.compaction.lag.ms=0
partitions=12                     # unchanged, and now frozen for the topic's life

Migration was not a config flip, because the existing topic had already discarded the history for unchanged flags. They created feature-flags-v2 as compacted, published the full current state from the source of truth (340,000 messages, about 90 seconds), switched producers to dual-write, moved consumers over, then retired the old topic.

Results:

                              before          after
cold-start completeness       0.7% of flags   100% of flags
cold-start time               3.1s            8.4s
topic size on disk            varies, ~7d     412 MB steady
S3 snapshot dependency        required        removed
consumer memory (state map)   ~40 MB          ~40 MB

Cold start got slower, from 3.1 to 8.4 seconds, because the pod now reads 340,000 messages instead of 2,400. That was the trade and it was worth stating: compaction moves work from "maintaining a separate snapshot" to "reading a longer log at startup." For a topic with tens of millions of keys, that startup cost becomes the dominant concern and the answer is a Kafka Streams state store with a local RocksDB checkpoint, so a restart replays only the changelog tail rather than the whole topic.

The disk figure is worth noting too: 412 MB for 340,000 keys is roughly 1.2 KB per key, and with min.cleanable.dirty.ratio=0.1 the on-disk size stays close to the compacted size. At the default 0.5 it would have hovered nearer 600 MB, which is the concrete cost of the default.

Production evidence

Kafka's own __consumer_offsets topic is compacted, with 50 partitions by default. The key is (group, topic, partition) and the value is the committed offset. This is the reference implementation of the pattern: a compacted topic that is a table, replayed by the group coordinator on startup to rebuild the offset state. The same is true of __transaction_state for the transaction coordinator.

Kafka Streams uses compacted topics for every state store's changelog. A store named counts gets a topic <app-id>-counts-changelog with cleanup.policy=compact, and a task restarted on a different instance restores its RocksDB store by replaying that changelog. Windowed stores use cleanup.policy=compact,delete with retention matched to the window plus grace, which is the canonical use of the combined policy.

Kafka Connect stores connector configuration and offsets in compacted topics (connect-configs, connect-offsets, connect-status), and the documentation is explicit that connect-configs must have exactly one partition, because configuration must be totally ordered.

Confluent's Schema Registry stores every schema in a compacted topic (_schemas) with a single partition, and rebuilds its in-memory index by replaying it. This is the clearest example of "a compacted topic as the system of record for a service," and it explains a Schema Registry behaviour that confuses operators: the registry cannot serve until it has consumed to the end of _schemas.

Uber and LinkedIn have both described compacted topics as the transport for reference data (geofence definitions, member attributes) precisely because the cold-start property means a new consumer needs no bootstrapping path separate from the streaming path.

The debate

Compacted topic versus a database with change data capture. Both give you current state plus a change stream. The compacted topic's advantages: one system instead of two, the cold start and the live feed are the same code path, and consumers get local state with no query latency. Its disadvantages are real and should be stated: no point queries against the topic, no secondary indexes, no transactions across keys, every consumer must hold the full state (or a partitioned share of it) in memory or local disk, and the whole-topic replay cost grows with the key count.

My position: compacted topics for reference data that is small enough that every consumer can hold it, and that consumers need with zero read latency. Feature flags, configuration, geofences, currency rates, entity metadata under roughly a few million keys. Above that, or when consumers only need a small slice, put the data in a database and use change data capture, because forcing every consumer to materialise a hundred-million-key topic is a memory tax paid by every service.

Is a compacted topic a database? No, and the sharpest way to see it is that it has no read path. You cannot ask it a question; you can only replay it. Everything that looks like a query is happening in the consumer's materialised view. Treating it as a database leads to designs where someone wants a point lookup and adds a KTable and an interactive-query REST endpoint, at which point you have built a distributed database with none of the operational tooling of one.

Should compacted topics be the system of record? Confluent's Schema Registry says yes and it works. My caution is the operational surface: a compacted topic as system of record means the partition count is frozen forever, tombstone retention becomes a correctness parameter, GDPR erasure depends on max.compaction.lag.ms actually firing, and there is no backup story other than another Kafka cluster. For a component you own end to end, like Schema Registry, that is manageable. For business data with compliance obligations, a database with an audit log is less clever and easier to defend.

Follow-up Q&A

"Does compaction guarantee I see every value for a key?"

No, and this is the guarantee people most often overstate. It guarantees a consumer reading from the beginning sees at least the last value for every key. A consumer reading the tail in real time sees every value as it is written, because the active segment is never compacted, but a consumer that starts later or falls behind may find intermediate values already removed. If you need every intermediate value, either use cleanup.policy=delete with adequate retention, or set min.compaction.lag.ms to a window inside which you guarantee consumers will have read.

"How do you delete a key, and what can go wrong?"

Produce a record with the key and a null value: a tombstone. What goes wrong is delete.retention.ms, default 24 hours. The tombstone is itself removed after that window, so a consumer lagging by more than 24 hours replays the log, never sees the tombstone, and keeps a key that has been deleted, permanently and silently. The second thing that goes wrong is compliance: on a low-traffic partition the dirty ratio may never reach min.cleanable.dirty.ratio, so the old value is never actually removed from disk. max.compaction.lag.ms forces the issue and is the setting to reach for when a deletion has a legal deadline.

"Why can't you add partitions to a compacted topic?"

Because compaction retains the last value per key per partition, and correctness depends on every message for a key landing in the same partition. Adding partitions changes hash(key) % numPartitions, so a key starts going to a new partition where it has no history, while its old partition retains a stale value that is now permanently the "last value" there. A consumer building state from all partitions sees both and the ordering between them is undefined. The recovery is a new topic plus a full replay, so partition count on a compacted topic is a one-time decision.

"A compacted topic is growing without bound. Why?"

Four candidates, in the order I would check them. (1) Unique keys: nothing is being superseded, so compaction has nothing to reclaim, and the topic should not have been compacted. (2) Dirty ratio never met: at the default 0.5 the topic can sit at double its compacted size; check min.cleanable.dirty.ratio and the cleaner metrics. (3) The cleaner is dead: log.cleaner.enable off, or a cleaner thread that hit an exception and stopped, which is a known failure where one bad partition stalls the thread for everything it owns. Check kafka.log:type=LogCleanerManager,name=time-since-last-run-ms. (4) The dedupe buffer is too small for the number of distinct keys in the dirty section, so the cleaner can only compact part of the log per pass.

"How does Kafka Streams use this?"

Every state store gets a compacted changelog topic. Writes to the store are also written to the changelog. If the instance dies, another instance takes over the partition and rebuilds the store by replaying the changelog from the beginning, which is complete precisely because compaction guarantees the last value per key. With standby replicas or a persisted RocksDB directory plus a checkpoint file, the restore reads only the tail rather than the whole changelog, which is the difference between a 20-second failover and a 20-minute one on a large store.

"Compacted or CDC from Postgres, for a 50-million-row reference table?"

CDC. Fifty million keys means every consumer materialising the whole compacted topic holds fifty million entries, and the initial replay is minutes to tens of minutes per consumer per restart. The compacted-topic pattern earns its keep when the state is small enough to be free at every consumer. At 50 million rows, keep the table in Postgres, expose a query path, and stream changes for the consumers that genuinely need to react. The exception is if every consumer truly needs a full local copy for latency reasons, in which case use Kafka Streams with RocksDB rather than an in-memory map, so the state lives on local disk and restores incrementally.

Common misconceptions

"Compaction saves disk space." Only for repeated keys. Compaction on a stream of unique keys reclaims nothing and removes your time-based retention at the same time, so the topic grows forever. Compaction is a semantic choice about what the topic means, not a storage optimisation.

"Compacted means only one message per key." It means at least the last one. The active segment is never compacted, the dirty ratio gates when compaction runs, and min.compaction.lag.ms can hold messages deliberately. At any moment a compacted topic typically contains several values for recently-updated keys.

"Compaction renumbers offsets." It does not. Offsets become sparse but never change meaning, which is what keeps consumer offsets valid across compaction. A consumer that seeks to offset 500 and finds the next available record is at 517 is seeing normal behaviour.

"A tombstone deletes the key immediately." It marks the key for deletion. The cleaner removes prior values on its next pass, and removes the tombstone itself after delete.retention.ms. On a partition whose dirty ratio stays below the threshold, the old value can persist for a long time, which matters if the deletion was a compliance request.

"I can switch a topic from delete to compact and get history back." Switching the policy changes future cleaning behaviour. Data already aged out under the delete policy is gone. Getting a complete compacted topic from an incomplete one requires republishing full state from wherever the truth actually lives.

Interview delivery note

The line to say verbatim: "A compacted topic is a table shipped as a log: replaying it from zero reconstructs current state, which is why a new consumer needs no bootstrap path separate from the streaming path. The price is that partition count is frozen forever and tombstone retention becomes a correctness parameter." That states the value and the two operational costs in one breath, and the second half is what most candidates leave out.

The senior-versus-staff separator is delete.retention.ms as a correctness parameter. Anyone can explain that compaction keeps the last value per key. The staff-level observation is that a consumer lagging beyond the tombstone retention window silently retains deleted keys, with no error, and that this makes consumer lag alerting a correctness control rather than a performance one. The second signal is knowing that adding partitions corrupts a compacted topic, because that is a one-way door that teams walk through by accident.

If asked to choose between a compacted topic and CDC, commit and give the variable: compacted topic when every consumer needs the full state and the key count is small enough to be free at each consumer; CDC when consumers need a slice or the state is large. Then name the number where you would switch, which for an in-memory materialisation is around a few million keys.

The Staff & Team Lead Interview Reference