ISR, min.insync.replicas, acks and unclean leader election

What it is

Kafka's durability guarantee is not a single setting. It is the intersection of four independent knobs, three of which live in different config files owned by different teams, and the guarantee you actually get is the weakest of the four. This page is about what each one does and how they combine, because the common failure is a team that set acks=all, believed they had durability, and lost data anyway.

The four:

Knob	Set by	Controls
`acks`	Producer	How many replicas must acknowledge before the producer call returns
`min.insync.replicas`	Topic (or broker default)	How many replicas must be in sync for a write to be allowed
`replication.factor`	Topic	How many copies exist at all
`unclean.leader.election.enable`	Broker/topic	Whether an out-of-sync replica may become leader

The ISR (in-sync replica set) is the connective tissue. It is the set of replicas that the leader considers caught up: they have fetched from the leader within replica.lag.time.max.ms (default 30 seconds). The ISR is dynamic; replicas join and leave it as they keep up or fall behind, and the controller records membership in cluster metadata.

The critical definition, which is where most confusion lives: a message is "committed" when every replica in the current ISR has it. Not when a majority has it. Not when all replicas have it. When the current ISR has it. Consumers can only read committed messages, so the ISR is simultaneously the durability boundary and the visibility boundary.

This is a different design from quorum systems like Raft, where a majority commits. Kafka's ISR approach tolerates more failures for a given replication factor (with RF=3 and min.insync.replicas=2 you can lose one replica and keep writing, and you can lose two and keep reading), at the cost of the ISR itself being a piece of shared state that must be maintained correctly.

The problem it solves

You have a payments event stream. A broker's disk fails. What happens to the messages that broker had acknowledged?

Without replication: they are gone. With replication but acks=1: the producer got a success response as soon as the leader wrote to its page cache, so if the leader dies before followers fetch, those acknowledged messages are gone and the producer has no idea. It moved on. The upstream system recorded the payment as published.

The deeper problem is that the failure is silent and delayed. A producer with acks=1 works perfectly for months and then loses 40,000 messages during a five-minute broker failure, and the loss is discovered days later by a reconciliation job. There is no error, no exception, no retry. The producer was told the write succeeded, and it had. It just was not durable.

Mechanics

acks: what the producer waits for

acks=0    fire and forget. The producer does not wait at all.
          Loses data on any failure, including a full producer buffer.
          Throughput: highest. Use for metrics you can afford to lose.

acks=1    leader acknowledges after writing to its local log (page cache,
          not necessarily fsynced). Followers may not have it yet.
          Loses data if the leader fails before followers replicate.

acks=all  (or acks=-1) the leader waits until every replica in the ISR
          has fetched the message. THEN it acknowledges.

The subtlety in acks=all: it waits for the current ISR, not for replication.factor. If two of three replicas have fallen out of the ISR, the ISR is {leader}, and acks=all means "wait for the leader," which is acks=1 wearing a different name. This is precisely the hole that min.insync.replicas closes.

min.insync.replicas: the floor

min.insync.replicas=2 means: if the ISR has fewer than 2 members, reject the write with NotEnoughReplicasException rather than accepting it with weaker durability.

This is a deliberate availability sacrifice, and it is the right one for data you cannot lose. The producer gets a retriable exception and can back off or buffer rather than silently downgrading its guarantee.

Note that min.insync.replicas is only consulted when acks=all. With acks=1 it does nothing at all, which is one of the sharper edges in Kafka's configuration surface: you can set min.insync.replicas=3 on a topic, feel safe, and have a producer with acks=1 writing to it with zero durability guarantee.

The combination that works

# Topic
replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false

# Producer
acks=all
enable.idempotence=true          # also sets retries=MAX, max.in.flight<=5
max.in.flight.requests.per.connection=5
delivery.timeout.ms=120000

RF=3 with min.insync.replicas=2 is the standard configuration and the reasoning is arithmetic: you tolerate one broker failure with full write availability (ISR drops to 2, which still meets the floor), and you refuse writes on the second failure rather than accepting undurable ones. Setting min.insync.replicas=3 with RF=3 means any single broker restart, including a rolling upgrade, stops writes, which is why it is almost always wrong.

enable.idempotence=true deserves its own note. Without it, a retry after a network timeout can duplicate a message: the broker wrote it, the ack was lost, the producer retried. With idempotence the producer attaches a producer ID and per-partition sequence number, and the broker deduplicates. Since Kafka 3.0 it defaults to true, and it is nearly free (the sequence number is a few bytes). It also prevents reordering on retry, which the old retries setting did not.

Unclean leader election: the one that silently loses data

Every replica tracks its log end offset (LEO), and the leader tracks the high watermark (HW), the offset up to which all ISR members have replicated. Consumers read up to the HW.

Now suppose the ISR shrinks to just the leader (the other two brokers are down), and then the leader dies. There are two out-of-sync replicas available and no in-sync ones. Two choices:

unclean.leader.election.enable=false (the default since Kafka 0.11): the partition goes offline. No leader, no reads, no writes, until an in-sync replica returns. Availability sacrificed, no data lost.
unclean.leader.election.enable=true: an out-of-sync replica becomes leader. It is missing whatever the old leader had past its own LEO. Those messages are gone, and worse, the log truncates: consumers that had already read past that offset will see the offset reused for different data. Availability preserved, committed data lost.

That second failure mode is worth dwelling on because it breaks an assumption consumers make. A consumer that committed offset 5,000,000 and comes back to find the partition's log end offset is 4,800,000 has an offset out of range, and its auto.offset.reset policy decides whether it re-reads 200,000 messages (earliest) or skips ahead (latest). Neither is correct. This is why the setting defaults to false and why turning it on should require a written justification.

A protocol trace

Three replicas: leader L (broker 1), followers F1 (broker 2), F2 (broker 3). acks=all, min.insync.replicas=2, RF=3.

t0  ISR = {L, F1, F2}.  HW = 100.  All LEOs = 100.

t1  Producer sends batch (offsets 101-110).
    L appends to its log.  L.LEO = 110.  HW still 100.
    L does NOT acknowledge yet.

t2  F1 fetches from offset 101, gets 101-110, appends.  F1.LEO = 110.
    F1's next fetch request carries "I am at 110".

t3  F2 is on a slow disk. F2.LEO = 100 still.
    HW = min(LEO over ISR) = 100.  Still no ack to the producer.

t4  F2 catches up.  F2.LEO = 110.
    HW = 110.  L acknowledges the producer.  Consumers can now read 101-110.

t5  F2's broker dies.  After replica.lag.time.max.ms, L shrinks ISR to {L, F1}.
    ISR size 2 >= min.insync.replicas 2, so writes continue.

t6  F1's broker dies.  ISR shrinks to {L}.
    ISR size 1 < min.insync.replicas 2.
    Next produce request -> NotEnoughReplicasException. Writes STOP.
    Reads continue: everything up to HW is on L.

t7  L dies.  No in-sync replica exists.
    unclean.leader.election.enable=false -> partition OFFLINE.
    F1 or F2 returning triggers a clean election and service resumes.

Step t6 is the design working as intended. The cluster chose to stop accepting writes rather than accept them with one copy. A team seeing NotEnoughReplicasException in production is seeing durability protection, not a bug, and the correct response is to fix the brokers rather than lower the floor.

Where fsync fits, and why Kafka does not do it per message

Kafka acknowledges when the message is in the page cache of the ISR replicas, not when it is fsynced to disk. flush.messages and flush.ms can force fsync but are almost never used, because per-message fsync costs an order of magnitude in throughput.

The reasoning: with RF=3 across three brokers, ideally in three availability zones, losing the data requires three simultaneous OS-level crashes (not process crashes, a process crash leaves the page cache intact for the OS to flush). Kafka trades fsync durability for replication durability, which is a defensible position and one worth being able to state, because the interview question "does acks=all mean it is on disk" is testing exactly this. The honest answer is no, it means it is in the page cache of N machines, and the risk is correlated power loss across a rack or an AZ, which is why rack awareness matters.

Set broker.rack and let Kafka spread replicas across racks or AZs. Without it, Kafka's default assignment could place all three replicas of a partition in the same AZ, and then a single AZ failure takes the partition offline despite RF=3.

A worked example: 40,000 messages, and the config that lost them

An order events pipeline. Three brokers, RF=3, and the following state discovered after an incident:

# topic config
replication.factor=3
min.insync.replicas=1            # <- set during an outage 8 months earlier
unclean.leader.election.enable=false

# producer (in a service the streaming team did not own)
acks=1
enable.idempotence=false

Broker 2 was taken down for a kernel patch at 14:02. Broker 3 had been running with a degraded disk for two days and was flapping in and out of the ISR. At 14:07 the ISR for the busiest partition was {broker1}.

Broker 1's host hit a hardware fault at 14:11.

What was lost. Between 14:07 and 14:11 the producer wrote roughly 40,000 messages with acks=1, all acknowledged by broker 1 alone. Broker 3 was out of the ISR and had not fetched them. Broker 1's disk was unrecoverable.

Which knob would have prevented it, individually:

acks=all alone: would not have helped. With min.insync.replicas=1, the ISR was {broker1} and acks=all means "wait for the ISR," which was broker 1 alone. Identical outcome.
min.insync.replicas=2 alone: would not have helped. It is only enforced when acks=all, and the producer was on acks=1.
Both together: writes would have failed with NotEnoughReplicasException from 14:07 onward. The producer would have retried and then buffered or errored upstream, which is a visible, actionable failure. Zero messages lost.

That is the load-bearing observation on this page: the two settings only work as a pair, and either one alone is a false sense of security. They also live in different places (producer code versus topic config), owned by different teams, and neither team can see the other's value at runtime without going looking.

The fix that stuck was not a config change, it was a guardrail: a broker-side default of min.insync.replicas=2, plus a CI check on producer configuration for any service publishing to a topic tagged as durable, plus an alert on UnderMinIsrPartitionCount > 0. The last one is the metric that would have paged someone at 14:07.

Production evidence

Confluent's documented recommendation for durable topics is exactly RF=3, min.insync.replicas=2, acks=all, and their docs are explicit that min.insync.replicas has no effect unless acks=all. This is a case where the vendor guidance and the field practice agree, which is not always true.

Kafka changed the default of unclean.leader.election.enable from true to false in 0.11 (KIP-106), which is a rare instance of a project deciding that its default had been on the wrong side of a correctness-versus-availability trade and changing it. The rationale in the KIP is that users were losing data without understanding they had opted in.

LinkedIn, where Kafka originated, has published on running it at multi-trillion message per day scale, and their operational writing emphasises rack-aware replica placement and the UnderMinIsr metric as the primary durability alert. The partition count per broker and ISR shrink rate are the operational signals they watch.

Cloudflare's log pipeline post-mortems and engineering posts describe using acks=all with min.insync.replicas=2 for logs that feed billing, and lower settings for logs that feed dashboards, which is the right pattern: durability is a per-topic decision driven by what the data is for, not a cluster-wide policy.

Kafka 4.0 removed ZooKeeper entirely in favour of KRaft, where cluster metadata including ISR membership lives in an internal Raft-replicated log rather than in ZooKeeper. The durability semantics for topic data did not change, but the metadata plane now uses quorum consensus while the data plane still uses ISR, which is a useful illustration that the two approaches suit different jobs: consensus for low-volume metadata, ISR for high-volume data.

The debate

ISR versus quorum replication. Kafka's ISR approach commits when all in-sync replicas have the data; Raft commits when a majority does. The trade:

ISR with RF=3, min.insync.replicas=2 tolerates 1 failure for writes and gives you 3 copies of committed data. A quorum system with 3 nodes also tolerates 1 failure and gives you 2 copies of committed data.
ISR's latency is bounded by the slowest in-sync replica; quorum latency is bounded by the median. That makes Kafka more sensitive to one slow broker, and it is why a degraded disk on one broker raises produce latency across every partition it leads.
ISR requires maintaining the ISR set itself, which is metadata that must be correct. Historically this lived in ZooKeeper and ISR churn was a known source of cluster instability.

My read: Kafka's design is right for its workload. For a high-throughput log where you want more copies per unit of tolerated failure and you can accept occasional latency spikes from a slow replica, ISR is a better fit than quorum. For low-volume metadata where you need consistently low latency and a clean failure model, quorum wins, which is exactly the split KRaft adopted.

Should you ever enable unclean leader election? There is one defensible case: a topic whose data is genuinely disposable and whose availability matters more, such as a metrics or click-stream feed where a gap is acceptable and an offline partition backs up producers into an outage. Even then, be aware that the log truncation confuses consumers, so pair it with auto.offset.reset=latest and an alert. For anything transactional, financial, or feeding a system of record, it should be off and there is no argument. My position is that the default should stay false and enabling it should be a documented per-topic exception with a named owner.

Is min.insync.replicas=2 with RF=3 always right? For durable data, yes, and I would treat deviation as a smell. RF=2 with min.insync.replicas=2 means any single broker restart stops writes, which is unacceptable for rolling upgrades. RF=3 with min.insync.replicas=3 has the same problem. RF=5 with min.insync.replicas=3 exists and is used for the highest-value topics, tolerating two failures while writing, at 5x storage. Pick RF from your storage budget and set min.insync.replicas = RF - 1 for RF=3, or RF - 2 for RF=5.

Follow-up Q&A

"acks=all is set. Am I safe?"

Not necessarily, and this is the question I would ask back: what is min.insync.replicas on that topic? acks=all waits for the current ISR, and if the ISR has shrunk to one replica then acks=all is acks=1. The pair is what gives the guarantee. Second thing I would check: is enable.idempotence on, because without it a retry after a lost acknowledgement duplicates the message, so you have durability without exactly-once. Third: is unclean.leader.election.enable false, because if it is true, committed data can still be truncated away by a leader election.

"What does it mean for a message to be committed in Kafka?"

Every replica in the current ISR has it. That is a moving target, because ISR membership changes as replicas keep up or fall behind, which is why min.insync.replicas exists: it puts a floor under how small the ISR can get before writes are refused. Consumers can only read up to the high watermark, which is the minimum log end offset across the ISR, so committed and consumer-visible are the same boundary.

"Does acks=all mean the data is on disk?"

No. It means it is in the page cache of every in-sync replica. Kafka does not fsync per message because it would cost roughly an order of magnitude in throughput. The durability argument is replication rather than fsync: three machines, ideally in three availability zones, would all have to lose power (not merely crash, since a process crash leaves the page cache for the OS to flush) before the data is gone. That argument depends entirely on the replicas not sharing a failure domain, which is why broker.rack is not optional in a multi-AZ deployment.

"Your producers are getting NotEnoughReplicasException. What do you do?"

Not lower min.insync.replicas, which is the tempting move and converts a visible failure into silent data loss. The exception means fewer than the required replicas are in sync, so the diagnosis is why: a broker down, a broker with a slow or failing disk falling behind, network saturation between brokers, or too many partitions per broker so replication fetches are queueing. Check UnderMinIsrPartitionCount and IsrShrinksPerSec. Meanwhile the producer should be buffering and retrying, because the exception is retriable, so the immediate customer impact depends on how long your producer buffer holds out, which is buffer.memory divided by your produce rate.

"How does this interact with exactly-once semantics?"

Durability is a prerequisite, not a substitute. Kafka's transactional producer (covered in Kafka exactly-once) gives you atomic writes across partitions and a read-committed isolation level for consumers. It builds on the idempotent producer, and it assumes the underlying writes are durable. Running transactions with acks=1 gives you atomic loss: the transaction commits and then the data disappears. The transaction coordinator's own log is a Kafka topic with its own replication settings, and transaction.state.log.min.isr defaults to 2 for exactly this reason.

"Three brokers, RF=3, min.insync=2. Two brokers die. What can you still do?"

Read, not write. The ISR is now {leader}, size 1, below the floor, so produce requests are rejected. But everything up to the high watermark is on the surviving leader, and consumers keep reading normally. This is the intended degradation: Kafka preserves the read path and refuses to compromise the write path. If the surviving broker then also dies, the partition goes offline entirely (with unclean election disabled) and waits for a replica with the committed data to return.

Common misconceptions

"acks=all means all replicas." It means all replicas in the ISR, which can be one. This is the single most consequential misreading of Kafka's configuration and it is what makes min.insync.replicas necessary.

"min.insync.replicas protects me." Only in combination with acks=all. With acks=1 it is inert, and Kafka will not warn you about the mismatch because the two settings live in different places and are validated independently.

"Replication factor 3 means I can lose 2 brokers." For reading, yes. For writing with min.insync.replicas=2, no: the second failure stops writes by design. Conflating read and write availability is common and leads to surprise during the second failure.

"Unclean leader election just means a slightly stale replica takes over." It means the log truncates, so offsets that consumers already read and committed can be reused for different messages. It is not staleness, it is a rewritten history, and downstream systems that keyed on (topic, partition, offset) will be wrong.

"Kafka is a database, so it fsyncs." It does not, by default, and the design document says so. Its durability model is N replicas in page cache across failure domains. That is a different guarantee from a database's WAL fsync and it is worth being precise about, especially when someone asks whether Kafka can be the system of record.

Interview delivery note

Say this verbatim: "acks=all waits for the current ISR, and the ISR can shrink to one, so acks=all without min.insync.replicas=2 is acks=1 with extra latency. The two settings only work as a pair, and they are owned by different teams, which is why this fails in production." That last clause is what makes it a staff answer rather than a documentation recital: it names the organisational reason the misconfiguration survives.

The senior-versus-staff separator is knowing that min.insync.replicas is inert under acks=1, and following it with the guardrail rather than the config value. A senior engineer gives you the correct settings. A staff engineer says "the settings are RF=3, min.insync=2, acks=all, and I would enforce it with a broker-side default plus a CI check on producer config plus an alert on UnderMinIsrPartitionCount, because the settings drift and nobody notices until the second broker dies."

If asked about fsync, do not overclaim. "Kafka acknowledges from page cache, not disk; the durability comes from replication across failure domains, which is why rack awareness is load-bearing" is a precise answer and demonstrates you know what the guarantee actually is.

The Staff & Team Lead Interview Reference