Tunable consistency, repair mechanisms and gc_grace_seconds

What it is

Cassandra replicates each partition to RF nodes and lets you choose, per query, how many replicas must respond before the coordinator answers. That per-query choice is tunable consistency, and it is what distinguishes Cassandra from systems where the consistency model is fixed by the engine.

The consistency levels that matter:

Level	Replicas that must respond	Meaning
`ONE`	1	Fastest, may read stale data
`QUORUM`	`floor(RF/2) + 1`	Majority across the whole cluster
`LOCAL_QUORUM`	Majority within the local datacenter	The production default
`EACH_QUORUM`	A quorum in every datacenter (writes only)	Strong multi-DC, expensive
`ALL`	All `RF`	Any replica down means the query fails
`LOCAL_ONE`	1, in the local DC	Analytics, tolerant reads

The rule that makes this useful is arithmetic:

$$R + W > RF \implies \text{the read set and write set overlap}$$

With RF=3, W=QUORUM (2) and R=QUORUM (2), we have 2 + 2 > 3, so any read contacts at least one replica that saw the write. That overlap is the entire guarantee, and it is the same quorum-intersection argument as Paxos, applied per query rather than per protocol.

What it is confused with: QUORUM does not give you linearizability. It gives you read-your-writes and monotonic reads when R + W > RF holds, and it does not order concurrent writes: two clients writing the same cell at QUORUM resolve by timestamp (last-write-wins), and the loser's write is silently discarded. For genuine linearizability you need lightweight transactions (IF NOT EXISTS, SERIAL consistency), which run Paxos and cost roughly four round trips.

The problem it solves

Cassandra chose AP in CAP terms: it stays available under partition and gives up strong consistency. Tunable consistency is the recognition that "give up consistency" is not one decision, it is a decision per query, and different queries in the same application genuinely want different points.

A concrete example from one system:

Writing an audit record:         W = QUORUM   (must not be lost)
Reading a user's profile:        R = LOCAL_QUORUM (must be current after their edit)
Reading a recommendation list:   R = ONE      (stale by seconds is fine, latency matters)
Reserving a unique username:     LWT with SERIAL  (needs real linearizability)
Bulk analytics scan:             R = LOCAL_ONE (never contend with production traffic)

Fixing one level for the whole application means either paying quorum latency for the recommendation read or risking a lost audit record. Per-query choice is what lets both be right.

The second problem, and the one the repair machinery addresses: replicas drift. A write at QUORUM succeeds with 2 of 3 replicas, so the third is stale, permanently, unless something fixes it. A node down for an hour misses every write in that hour. Without an anti-entropy mechanism, a R=ONE read has an unbounded chance of returning data from an arbitrarily stale replica.

Mechanics

The three repair mechanisms

They operate at different times and cover different gaps, and only one of them is complete.

1. Hinted handoff. When a replica is down, the coordinator stores a hint (the write, plus its intended destination) locally and replays it when the node returns.

max_hint_window_in_ms: 10800000    # 3 hours, default

Coverage: nodes that are briefly down. Beyond the hint window, hints stop being stored, so a node down for four hours has three hours of hints and one hour of permanent gap. Hints are also lost if the coordinator itself dies. This is a latency optimisation for common transient failures, not a durability mechanism.

2. Read repair. On a read, the coordinator compares digests from the replicas it contacted. If they disagree, it fetches full data, resolves by timestamp, returns the answer, and writes the correct value back to the stale replicas.

-- Cassandra 4.0+: blocking read repair happens automatically at QUORUM and above.
-- The old read_repair_chance / dclocal_read_repair_chance settings were REMOVED in 4.0.

Coverage: only data that is read. Cold data is never repaired by this mechanism, and cold data is exactly the data most likely to be stale, because it has had the longest to drift.

3. Anti-entropy repair (nodetool repair). Replicas build Merkle trees over their data, compare them, and stream the differing ranges. This is the only mechanism that covers everything.

# Full repair of one keyspace, primary ranges only, one node at a time.
nodetool repair -pr my_keyspace

# Incremental: only data not marked repaired since the last run.
nodetool repair my_keyspace

nodetool repair is not optional maintenance. It is the only complete anti-entropy mechanism, and the interval at which it must run is not a preference. It is set by gc_grace_seconds, for the reason below.

gc_grace_seconds and zombie data

A delete in Cassandra writes a tombstone. The tombstone must persist long enough that every replica has learned about the delete. gc_grace_seconds (default 864,000, or 10 days) is how long a tombstone survives before compaction may discard it.

If a replica misses a delete and the tombstone is garbage-collected before repair propagates it, the deleted data comes back. The sequence:

Day 0:  DELETE row X at QUORUM. Replicas A and B write tombstones. C is down.
Day 1:  C returns. It still holds the live value of X; hints expired 3h after day 0.
Day 10: gc_grace_seconds passes. Compaction on A and B discards the tombstones.
Day 11: Read at QUORUM contacts A and C.
          A: no data (tombstone gone, row gone)
          C: X exists, timestamp day -5
        Resolution by timestamp: C's value is the only one. X IS RESURRECTED.

That is a zombie row, and it is the reason for the hard operational rule:

You must run a full repair on every node within gc_grace_seconds.

With the default of 10 days, repair must complete cluster-wide at least every 10 days, with margin. A cluster where repair takes 3 days and is run monthly is producing zombies silently, and nothing reports it.

The two ways to fix a cluster that cannot repair in time:

-- Option A: extend the window (more tombstone accumulation, slower reads)
ALTER TABLE events WITH gc_grace_seconds = 1728000;   -- 20 days

-- Option B: make repair faster (subrange repair, incremental, more parallelism)

Option B is the right one in almost every case, because extending gc_grace_seconds means tombstones live longer, which makes range scans slower (see Cassandra tombstones) and is treating the symptom.

The special case worth knowing: a table with a uniform TTL and no explicit deletes can set gc_grace_seconds = 0. TTL expiry produces tombstones whose timestamps every replica agrees on, because they are derived from the write's own timestamp rather than from a client-issued delete, so there is no resurrection risk. Time-series tables under TWCS routinely do this and it removes a large source of tombstone accumulation.

Multi-datacenter: LOCAL_QUORUM is the default for a reason

CREATE KEYSPACE app WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'us_east': 3,
  'eu_west': 3
};

With RF=3 in each of two datacenters, RF total is 6, so:

QUORUM = 4 replicas, which must span datacenters, so every query pays cross-Atlantic latency (roughly 80 ms). It is also unavailable if one DC is partitioned, because 3 remaining replicas is fewer than 4.
LOCAL_QUORUM = 2 replicas within the local DC. Local latency, survives losing the other DC entirely.

LOCAL_QUORUM is what nearly every multi-DC deployment uses, and the honest statement of what it gives up is important: R + W > RF holds within a datacenter, so read-your-writes works locally and not across datacenters. A user who writes in us_east and immediately reads from eu_west may see stale data, bounded by replication latency (typically tens to hundreds of milliseconds, unbounded during a partition).

If cross-DC read-your-writes matters, the options are: pin a user's session to one datacenter (the usual answer), write at EACH_QUORUM (expensive, and unavailable when a DC is down), or accept the staleness with a UI that hides it.

Lightweight transactions: when quorum is not enough

-- Linearizable: runs Paxos across the replicas.
INSERT INTO users (username, user_id) VALUES ('alice', ?) IF NOT EXISTS;

UPDATE accounts SET balance = 400 WHERE id = ? IF balance = 500;

LWTs use Paxos and cost roughly four round trips instead of one, so they are typically 4 to 10x slower than a normal write. They are correct for genuine uniqueness and compare-and-set requirements and they are not a general-purpose consistency upgrade.

The trap: LWT and non-LWT writes to the same partition do not mix safely. A normal write does not participate in the Paxos round, so it can be lost or can violate the condition another client is testing. If a partition uses LWT, every write to it should use LWT, and SERIAL reads should be used to read it linearizably.

A worked example: deleted accounts that came back

A SaaS platform, RF=3 in a single datacenter, 40 nodes, using LOCAL_QUORUM for reads and writes. GDPR deletion requests removed a customer's records across several tables.

The report: a customer who had been deleted six weeks earlier reappeared in the application, with their old data. Then two more.

Investigation. Repair had been configured as a weekly cron running nodetool repair -pr on each node in sequence. Over eighteen months the data had grown from 4 TB to 31 TB, and repair duration had grown with it:

repair duration per node (measured):     4.5 hours
nodes:                                    40
sequential repair cycle:                  40 x 4.5 = 180 hours = 7.5 days
cron interval:                            weekly (168 hours)
gc_grace_seconds:                         864000 (10 days)

The cycle took longer than the interval. Each week's run was still going when the next fired, the cron used a lock file and skipped, so in practice a full cycle completed roughly every 15 days. Against a 10-day gc_grace_seconds, that meant a window in which tombstones were collected before every replica had seen them.

Actual full-repair cycle:   ~15 days
gc_grace_seconds:            10 days
                             ─────────
Zombie window:               ~5 days of exposure per cycle

Three deleted customers had reappeared. The team could not determine how many other rows had resurrected, because there was no record of what had been deleted, which is the part that made this a compliance problem rather than a bug.

What they changed.

# 1. Subrange repair: split each node's token ranges and repair in parallel,
#    with checkpointing so a failure resumes rather than restarting.
#    (Reaper does this; it is the standard tool.)

They deployed Cassandra Reaper, which segments the token ring, repairs segments in parallel with configurable concurrency, tracks progress, retries failed segments, and schedules around a target cycle time rather than a fixed cron.

                                before      after
full repair cycle               ~15 days    3.2 days
repair failures per cycle       6-9         0 (retried per segment)
gc_grace_seconds                10 days     10 days (unchanged)
margin                          NEGATIVE    3x

-- 2. gc_grace_seconds = 0 on the pure time-series tables.
ALTER TABLE usage_events WITH gc_grace_seconds = 0
  AND compaction = {'class': 'TimeWindowCompactionStrategy', ...}
  AND default_time_to_live = 7776000;

Those tables had TTL expiry and no client deletes, so the resurrection risk did not apply, and removing the grace period cut tombstone accumulation on the largest tables in the cluster. That alone reduced total data by about 18 percent, which fed back into repair duration.

-- 3. An alert that would have caught this at any point in eighteen months.

- alert: RepairCycleExceedsGCGrace
  expr: cassandra_repair_cycle_days > (cassandra_gc_grace_seconds / 86400) * 0.5
  for: 1h
  annotations:
    summary: "Repair cycle is more than half of gc_grace_seconds. Zombie risk."

The alert is the part worth copying. The relationship between repair duration and gc_grace_seconds is a correctness invariant, it degrades gradually as data grows, and nothing in Cassandra reports it. The team had two numbers that were each individually reasonable and whose relationship was the bug.

Final:

                                  before       after
full repair cycle                 ~15 days     3.2 days
zombie exposure window            ~5 days      none
total data                        31 TB        25.4 TB
p99 read latency                  180ms        94ms   (fewer tombstones)
alert on the invariant            none         yes

Read latency improved as a side effect: fewer accumulated tombstones meant fewer tombstones read during range scans.

Production evidence

Cassandra's documentation states the repair requirement explicitly: a full repair must run within gc_grace_seconds or deleted data can resurrect. That it is stated as a requirement rather than a recommendation reflects that it is a correctness invariant.

Cassandra Reaper (originally Spotify, then developed by The Last Pickle and DataStax) exists because nodetool repair at scale is operationally difficult: it is long-running, it fails partway, and it has no scheduling or progress tracking. Reaper's segmented, resumable, scheduled model is the de-facto standard, and its existence is evidence that the built-in tool is insufficient for large clusters.

Cassandra 4.0 removed read_repair_chance and dclocal_read_repair_chance, making blocking read repair automatic at QUORUM and above. The reasoning in the change was that probabilistic read repair gave weak guarantees that users overestimated. Knowing these settings were removed matters because a great deal of published Cassandra tuning advice still references them.

Netflix, Apple and Discord all publish on Cassandra operations, and the common thread is that repair scheduling is the primary operational burden. Netflix's Priam and later tooling automate exactly this.

Incremental repair had known correctness issues before Cassandra 4.0 (over-streaming and repaired/unrepaired data mixing), and the standard advice for a long time was to use full repair only. Cassandra 4.0 fixed the underlying problems, and many operators remained on full or subrange repair out of caution. If asked, the safe answer is subrange full repair via Reaper.

The debate

Which consistency level should be the default? LOCAL_QUORUM for both reads and writes, in single-DC and multi-DC alike. It gives R + W > RF within a datacenter, so read-your-writes holds; it survives one replica being down at RF=3; and it does not pay cross-DC latency. QUORUM in a multi-DC cluster is usually a mistake made by copying single-DC configuration: it forces cross-DC round trips on every query and makes the cluster unavailable when a datacenter is partitioned.

Is ONE ever right? Yes, and it should be a deliberate per-query decision. For a recommendation list, a feed, a cache-like read where staleness of seconds is invisible, LOCAL_ONE gives the lowest latency and the least load. The rule I use: ONE when a stale answer is merely suboptimal, LOCAL_QUORUM when a stale answer is wrong. What you must not do is set ONE globally to improve latency numbers, because it silently downgrades the queries where correctness mattered.

Should you ever use ALL? Almost never. It requires every replica up, so it turns any single node failure into a query failure, in a database whose entire premise is surviving node failure. The legitimate uses are one-off administrative reads where you need to see every replica's state.

Full, incremental, or subrange repair? Incremental repair is faster because it skips data already marked repaired, and it had genuine correctness problems before 4.0 that made many operators avoid it. Subrange full repair via Reaper is the answer I would give: it is correct, it is resumable, it parallelises, and it can be scheduled against a target cycle time. On 4.0+ incremental is viable and I would still want the cycle-time alert regardless of which you choose.

Should you lower gc_grace_seconds? Only for tables with uniform TTL and no client deletes, where resurrection is impossible and setting it to 0 removes a real source of tombstone accumulation. Lowering it on a table with deletes, to make tombstones clear faster, trades a visible performance problem for an invisible correctness one, which is the wrong direction. If repair cannot complete within the grace period, fix repair.

The uncomfortable truth about tunable consistency. It is per-query, so it is only as good as the discipline applied to every query in the codebase. One ONE read on a path that needed LOCAL_QUORUM, added by someone chasing a latency number, silently breaks read-your-writes for that path. The mitigation is to set the level in a shared data access layer rather than per call site, with deviations requiring justification. A per-query knob with no policy is a per-query bug surface.

Follow-up Q&A

"Explain R + W > RF."

If the number of replicas you read from plus the number you wrote to exceeds the replication factor, the two sets must overlap by at least one replica, so any read sees at least one replica that has the latest write. With RF=3, W=2, R=2: 4 > 3, so overlap is guaranteed. That gives read-your-writes and monotonic reads. It does not give linearizability, because concurrent writes to the same cell resolve by timestamp and one is silently discarded.

"What is gc_grace_seconds for, and what happens if repair is slower than it?"

It is how long a tombstone survives before compaction may discard it, and its purpose is to give repair time to propagate the delete to every replica. If a replica misses the delete and the tombstone is collected before repair reaches that replica, a subsequent read finds the old live value on that replica and no tombstone anywhere, so the row resurrects. That is a zombie, and it is silent: nothing logs it, nothing alerts, and you find out when a deleted customer reappears. The invariant is full repair cycle time must be comfortably under gc_grace_seconds, and it should be a monitored alert because it degrades as data grows.

"What are the three repair mechanisms and what does each miss?"

Hinted handoff replays writes to a node that was briefly down, and misses anything beyond the hint window (3 hours by default) or lost when the coordinator dies. Read repair fixes inconsistency discovered during a read, and misses all data that is never read, which is exactly the data most likely to have drifted. nodetool repair compares Merkle trees and streams differences, which is complete, and it is the only complete one, which is why the schedule is a correctness requirement rather than housekeeping.

"In a two-DC cluster with RF=3 each, what does QUORUM mean?"

RF is 6 total, so QUORUM is 4, which cannot be satisfied within one datacenter. Every query pays cross-DC latency, and losing one datacenter makes the cluster unavailable for QUORUM queries because only 3 replicas remain. LOCAL_QUORUM is 2 within the local DC: local latency, survives losing the other DC, and gives read-your-writes locally but not across datacenters. The usual answer to that gap is pinning a session to one DC.

"When would you use a lightweight transaction?"

Genuine uniqueness (claiming a username) or compare-and-set (state machine transitions where a concurrent update must not be lost). They run Paxos and cost roughly four round trips, so they are 4 to 10x slower than a normal write. The important constraint is that LWT and non-LWT writes to the same partition do not mix safely: a plain write does not participate in the Paxos round and can be lost or can invalidate a condition another client is testing. If a partition uses LWT, every write to it should.

"Reads are slow after a bulk delete. What is happening?"

Tombstones. Every deleted row wrote a marker, and a range scan over that range must read every tombstone to return nothing, so a scan over a million deleted rows reads a million tombstones. They will not be collected until gc_grace_seconds has passed and a compaction processes them, so a table on size-tiered compaction whose large files are not being merged can hold them far longer. The fixes are on the tombstones page; the point here is that the delete made reads slower, not faster.

Common misconceptions

"QUORUM gives strong consistency." It gives read-your-writes and monotonic reads when R + W > RF. It does not order concurrent writes: two writers to the same cell resolve by timestamp and one is discarded silently. Linearizability needs LWTs.

"Repair is maintenance you should do when convenient." It is the only complete anti-entropy mechanism and its interval is bounded by gc_grace_seconds as a correctness requirement. Skipping it produces zombie data.

"Read repair keeps replicas in sync." Only for data that is read. Cold data is never touched by it, and cold data has had the longest to drift.

"Hinted handoff means a node that was down catches up." Only within max_hint_window_in_ms, 3 hours by default, and only if the coordinator survived to replay them. Beyond that the gap is permanent until repair.

"QUORUM is a safe default in multi-DC." With RF=3 per DC in two DCs, QUORUM is 4 replicas spanning datacenters: cross-DC latency on every query and unavailability when one DC is partitioned. LOCAL_QUORUM is the multi-DC default.

Interview delivery note

Say this verbatim: "R + W > RF guarantees the read and write sets overlap, so LOCAL_QUORUM on both sides at RF=3 gives read-your-writes within a datacenter. And the operational half is that a full repair must complete within gc_grace_seconds, or tombstones are collected before every replica has seen the delete and deleted rows come back." The arithmetic and the invariant, which is the whole topic in two sentences.

The senior-versus-staff separator is the repair-cycle-versus-gc_grace_seconds invariant as a monitored alert. A senior engineer explains consistency levels and knows repair exists. A staff engineer notices that repair duration grows with the dataset while gc_grace_seconds is a fixed constant, so a cluster silently crosses from safe to producing zombies as it grows, with no signal. Two individually reasonable numbers whose relationship is the bug is a recognisable class of problem, and alerting on the relationship rather than either number is the fix.

The second signal is knowing that tunable consistency is only as good as the discipline applied to every call site, and putting the level in a shared data access layer rather than per query. A per-query knob with no policy is a per-query bug surface.

The Staff & Team Lead Interview Reference