Active-active conflict resolution
What it is
The mechanisms for reconciling concurrent writes to the same data in different regions, and the operational reality of running them. The decision about whether to be in this situation at all is the write path; this page is what you do once you are.
FOUR MECHANISMS, in increasing order of what they preserve
DISCARD Last-writer-wins. One write survives, the
other is gone.
Preserves: nothing beyond the winner.
DETECT Version vectors. Concurrent writes are
identified as concurrent and returned as
siblings for the application to resolve.
Preserves: both, and pushes the decision up.
MERGE CRDTs. The data type's merge function is
commutative, associative and idempotent, so
replicas converge with no lost update.
Preserves: everything, by construction.
AVOID Single-writer per entity. There is no
conflict to resolve.
Preserves: everything, by construction, and
costs cross-region latency for writes away
from home.
Commonly confused as a spectrum of quality. They are different guarantees, and the right one is per data type, not per system. A like counter and a bank balance need different mechanisms and a single choice is wrong for one of them.
The problem it solves
Two regions accept a write to the same key at the same time. There is no global clock, no coordination, and both writes returned success.
t=0.000 region A: set cart = [apple]
t=0.010 region B: set cart = [banana]
t=0.150 replication crosses
Neither region saw the other's write. Both told a user their
write succeeded. Something has to happen when they meet, and
"the later timestamp wins" is a decision with a cost, not a
neutral default.
The failure this page is really about: the mechanism is chosen once, applied everywhere, and the losses are silent. Nobody reports "my edit disappeared" as a bug because they cannot tell the difference between their edit vanishing and their edit never having been saved.
Mechanics
Last-writer-wins, and its two failure modes
resolve(a, b) = a if a.timestamp > b.timestamp else b
Failure mode 1: the discarded write. By definition, one concurrent write is destroyed. Whether that matters is a per-data-type question:
session last-active timestamp fine, genuinely
user's display name tolerable, rare, and the
user notices and re-types
a shopping cart NOT fine: the customer added
an item and it silently
vanished
a document edit unforgivable
Failure mode 2, and this is the one to name: with wall clocks, clock skew decides the winner.
Region A's NTP is 180 ms fast.
Every conflict involving region A is won by region A,
permanently and invisibly, regardless of which write was
actually later.
That is not a probabilistic bias, it is a systematic one,
and it is invisible in every metric.
The fix is hybrid logical clocks, which make the ordering monotonic and consistent with causality regardless of skew. See time and clocks.
What HLC does not fix: the discarded write. LWW with HLC still destroys one of two concurrent writes; it just stops the destruction being determined by whose NTP is worse. Stating that distinction is the signal, because teams adopt HLC and believe the problem is solved.
Version vectors: detect rather than resolve
# One counter per REGION, not per client. Bounded by the
# replica count, which is why Dynamo-style systems version
# by node rather than by caller.
def compare(a: dict, b: dict) -> str:
a_gt = any(a.get(k, 0) > b.get(k, 0) for k in a | b)
b_gt = any(b.get(k, 0) > a.get(k, 0) for k in a | b)
if a_gt and b_gt: return "CONCURRENT"
if a_gt: return "A_DOMINATES"
if b_gt: return "B_DOMINATES"
return "EQUAL"
On CONCURRENT, the store returns both siblings and the application decides. That is honest:
the writes really were concurrent and no timestamp can say which "should" win.
The operational costs, which are the reason this is less popular than it deserves:
SIBLING EXPLOSION
A key written concurrently in several regions repeatedly
accumulates siblings. Each read returns all of them, each
write must resolve them, and a hot key can reach hundreds.
-> Cap the sibling count and merge or discard the oldest,
accepting some loss to bound the metadata.
APPLICATION BURDEN
Every read path must handle "you got three versions of
this". Most code does not, and the common shortcut is to
pick one arbitrarily in a helper, which is LWW with extra
steps and none of its honesty.
VECTOR GROWTH
Bounded if you version per region. UNBOUNDED if you
version per client, which is the documented failure that
makes people think vector clocks do not work.
The pattern to recommend: version vectors for detection, plus a type-specific merge, so the application is not asked to resolve arbitrary siblings. Detection without a merge strategy is work pushed to a place that will not do it well.
CRDTs: no conflict by construction
The strongest option where the data type allows, and the catalogue matters because the constraint is real.
COUNTERS
G-Counter grow-only. Per-region counts, merge is
per-region max, value is the sum.
PN-Counter two G-Counters (increments, decrements).
Supports decrement, and CANNOT enforce a
floor: it cannot prevent going negative.
SETS
G-Set add-only. Trivial.
2P-Set add and remove, and a removed element can
never be re-added.
OR-Set add and remove with unique tags per add, so
re-adding works. THE practical set type, and
the metadata grows with the number of
add/remove operations unless tombstones are
collected.
LWW-Set add and remove with timestamps. Simpler,
and it inherits LWW's discard problem.
REGISTERS
LWW-Register a single value with a timestamp. This is
just LWW, framed as a CRDT.
MV-Register multi-value: keeps all concurrent writes,
which is version vectors framed as a CRDT.
SEQUENCES
RGA, LSEQ, ordered lists for collaborative text.
Logoot, Genuinely convergent, and the metadata
Yjs/Automerge overhead per character is the practical
constraint.
MAPS
OR-Map keys with CRDT values, composing the above.
The constraint that decides applicability: a CRDT can express "combine these" and cannot express "only if".
CAN CANNOT
increment a counter keep a balance above zero
add to a set enforce uniqueness
set a field (LWW semantics) "set to X only if it is
merge two documents currently Y"
So inventory is the instructive case: a PN-Counter tracks stock perfectly and will happily go negative, because the invariant "never below zero" is not expressible in a merge function. Overselling is the CRDT's correct behaviour, and if that is unacceptable the data needs a single writer.
class GCounter:
"""Merge is per-region max. Commutative, associative and
idempotent, so replicas converge regardless of message
order or duplication, which is the whole CRDT property."""
def __init__(self): self.c: dict[str, int] = {}
def incr(self, region: str, n: int = 1):
self.c[region] = self.c.get(region, 0) + n
def value(self) -> int: return sum(self.c.values())
def merge(self, other):
m = GCounter()
m.c = {r: max(self.c.get(r, 0), other.c.get(r, 0))
for r in self.c | other.c}
return m
The cost people underestimate is metadata. An OR-Set's tags and tombstones grow with operation count, not with element count, so a set that is added to and removed from repeatedly grows without bound unless tombstones are garbage-collected, and collecting them safely requires knowing every replica has seen the removal.
Application-level merge: the pragmatic middle
Between "discard one" and "the type merges itself" is a type-specific merge function you write.
def merge_cart(a: Cart, b: Cart) -> Cart:
"""Union by item, quantity is the max, removals are
tracked explicitly so a removal is not undone by a stale
replica that still has the item."""
items = {}
for item in a.items + b.items:
prev = items.get(item.sku)
items[item.sku] = item if prev is None else \
(item if item.qty > prev.qty else prev)
removed = a.removed | b.removed # union of removals
return Cart(items={k: v for k, v in items.items()
if k not in removed},
removed=removed)
That is a hand-rolled OR-Set, and writing it explicitly rather than reaching for a library is often right, because the semantics ("max quantity wins", "a removal is permanent") are business decisions rather than mathematical ones.
The property to verify: the merge must be commutative, associative and idempotent, or replicas do not converge and the system is silently wrong. Property-based testing is the right tool here, and it is one of the few places where it is clearly worth the setup:
@given(carts(), carts(), carts())
def test_merge_is_a_semilattice(a, b, c):
assert merge(a, b) == merge(b, a) # commutative
assert merge(merge(a, b), c) == merge(a, merge(b, c)) # associative
assert merge(a, a) == a # idempotent
The operational half, which is usually missing
Conflict resolution is not a design decision you make once; it is a thing you run.
MEASURE THE CONFLICT RATE, per key class.
If it is zero, you are paying for machinery you do not
need and could use single-writer instead.
If it is high on one key class, that class probably needs
home-region ownership rather than a better merge.
MEASURE REPLICATION LAG, per region pair.
Divergence is bounded by lag times write rate, so lag is
the leading indicator of conflict volume.
ALERT ON SIBLING COUNT.
A key accumulating siblings is a key whose merge is not
running or not converging.
LOG DISCARDED WRITES.
With LWW, the discarded write should be logged with both
values. It is the only way to know what the mechanism is
costing, and it converts a silent loss into a
measurable one.
*** Almost nobody does this, and it is the single most
useful thing on this list. ***
TEST CONVERGENCE.
Partition two regions in a game day, write conflicting
values, heal, and assert the replicas agree. A merge
function that is not commutative fails here and nowhere
else until production.
Logging discarded writes is the recommendation I would make first, because it costs almost nothing and it is the only way to answer "is last-writer-wins actually acceptable for this data", which is otherwise argued from intuition.
A worked example: choosing per data type
A COLLABORATION PRODUCT going active-active across three
regions.
data conflict rate mechanism
---------------------------------------------------------
presence / last-seen very high LWW + HLC. A
discarded write is
meaningless here.
view counts very high G-Counter. Merges,
no loss, no
coordination.
reactions (emoji) high OR-Set. Add and
remove both work,
re-adding works.
user preferences low LWW + HLC, and LOG
the discarded write
so we can see what
it costs.
document content moderate Sequence CRDT (Yjs).
A discarded edit is
unacceptable, and
this is the case the
type was designed
for.
workspace membership very low SINGLE WRITER, home
region. Uniqueness
on email is an
invariant no merge
function can express.
billing ~zero SINGLE WRITER plus a
consensus group.
WHAT THE TABLE SHOWS
Five different mechanisms across seven data types, and the
two that use single-writer are the two with genuine
invariants.
WHAT A SINGLE CHOICE WOULD HAVE COST
LWW everywhere: document edits silently lost, and two
users able to register the same email.
CRDTs everywhere: membership uniqueness unenforceable,
and billing able to go negative,
because those invariants are not
expressible in a merge function.
Single-writer
everywhere: every presence update and view count
pays a cross-region round trip, which
is most of the traffic.
THE OPERATIONAL FINDING, six months in
The discarded-write log on user preferences showed a
conflict rate of 0.02%, almost all from a single user
editing on two devices simultaneously. That was low enough
to confirm LWW was right there, and the log is what turned
that from an assumption into a measurement.
Production evidence
Shapiro, Preguiça, Baquero and Zawirski, "A comprehensive study of Convergent and Commutative Replicated Data Types" (INRIA 2011) is the CRDT reference, including the proofs that merge must be commutative, associative and idempotent for convergence, and the catalogue of types.
Amazon's Dynamo paper (SOSP 2007) introduced version vectors with sibling reconciliation returned to the application, and it is explicit that the application has the semantics needed to merge, which the store does not.
Riak implemented Dynamo's sibling model and later added CRDTs (Riak DT) specifically because application-level sibling reconciliation proved to be the hardest operational burden, which is strong evidence for preferring a merging type where the data allows.
Automerge and Yjs are the production sequence CRDTs behind collaborative editors, and their documented metadata overhead per character is the practical constraint on that class.
DynamoDB Global Tables uses last-writer-wins across regions and documents it plainly, which is the honest version: a large managed service choosing the simple mechanism and saying what it loses.
Kulkarni et al. on hybrid logical clocks (2014) is the basis for LWW that is not decided by NTP quality, and CockroachDB, YugabyteDB and MongoDB's adoption of HLC is the production evidence.
The debate
The case for last-writer-wins: it is simple, it has no metadata growth, it needs no application changes, and for a large share of data a discarded concurrent write genuinely does not matter. DynamoDB Global Tables ships it.
The case for version vectors: it is honest. Concurrent writes were concurrent, and surfacing that lets the application decide with semantics the store does not have. Discarding silently is a decision made by infrastructure that should be made by the product.
The case for CRDTs: no lost updates at all, by construction, with no coordination. Where the data type fits, it is strictly better than resolving conflicts well.
The case for avoiding conflicts entirely: single-writer per entity means there is nothing to resolve, no metadata, no merge function to test, and no silent loss. The cost is cross-region write latency for a minority of writes.
My position: choose per data type, use CRDTs where the type fits, single-writer where an invariant exists, and LWW with hybrid logical clocks plus a discarded-write log for the rest.
The per-data-type framing is the whole answer, and the worked example is why: five mechanisms across seven data types, and any single choice is catastrophically wrong for at least one of them. LWW everywhere loses document edits and permits duplicate emails. CRDTs everywhere cannot express uniqueness or a balance floor. Single-writer everywhere puts a cross-region round trip on presence updates, which are most of the traffic.
The distinction I would insist on is that hybrid logical clocks fix the wrong-winner problem and not the lost-write problem. HLC stops clock skew systematically deciding conflicts, which is a real and invisible bias where a region 180 milliseconds fast wins every time. It does not stop one of two concurrent writes being destroyed, because that is what last-writer-wins means. Teams adopt HLC and believe the problem is solved.
The CRDT boundary I would state as a rule: a merge function can express "combine these" and cannot express "only if". Inventory is the instructive case, because a PN-Counter tracks stock perfectly and will happily go negative, and overselling is its correct behaviour rather than a bug. If the invariant matters, the data needs a single writer, and no better CRDT exists.
And the operational recommendation I would make first, because it costs almost nothing: log the discarded write, with both values. It is the only way to turn "is LWW acceptable for this data" from an argument into a measurement, and in the worked example it showed a 0.02 percent conflict rate concentrated in one behaviour, which confirmed the choice with evidence. Almost nobody does it.
Where I would push back on a design: if the conflict rate for a key class is measured at zero, the machinery is unnecessary and single-writer is simpler. Teams build conflict resolution for data that never conflicts, and the merge function then goes untested until the day it runs.
Follow-up Q&A
"How do you resolve conflicts in an active-active system?" Per data type, not per system. Four mechanisms: discard one write (last-writer-wins), detect concurrency and return siblings (version vectors), merge by construction (CRDTs), or avoid the conflict entirely (single writer per entity). In a collaboration product I worked on, that produced five different mechanisms across seven data types, and any single choice would have been catastrophically wrong for at least one of them.
"What's wrong with last-writer-wins?" Two things, and only one is fixable. With wall clocks, clock skew decides the winner: a region whose NTP is 180 milliseconds fast wins every conflict, permanently and invisibly, which is a systematic bias rather than a probabilistic one. Hybrid logical clocks fix that. What HLC does not fix is that one of two concurrent writes is destroyed, because that is what LWW means, and teams adopt HLC and believe the problem is solved.
"When are CRDTs the answer, and when are they not?" A merge function can express "combine these" and cannot express "only if". So counters, sets, registers and sequences work, and uniqueness constraints and balance floors do not. Inventory is the instructive case: a PN-Counter tracks stock perfectly and will happily go negative, because "never below zero" is not expressible in a merge. Overselling is the CRDT's correct behaviour, and if that is unacceptable the data needs a single writer.
"What do CRDTs cost?" Metadata, and it is usually underestimated. An OR-Set's tags and tombstones grow with the number of operations rather than the number of elements, so a set that is added to and removed from repeatedly grows without bound unless tombstones are collected, and collecting them safely requires knowing every replica has seen the removal. Sequence CRDTs for text have a per-character overhead that is the practical constraint on document size.
"Why aren't version vectors more popular?" Sibling explosion and application burden. A key written concurrently in several regions accumulates siblings, each read returns all of them, and a hot key can reach hundreds, so you cap the count and lose something. And every read path has to handle "you got three versions", which most code does not, so the common shortcut is a helper that picks one arbitrarily, which is last-writer-wins with extra steps and none of its honesty. Riak added CRDTs specifically because application-level sibling reconciliation was the hardest operational burden.
"How would you verify a merge function is correct?" Property-based testing, which is one of the few places it is clearly worth the setup. Assert commutativity, associativity and idempotence directly, because those three properties are exactly what guarantees convergence, and a merge that fails any of them produces replicas that never agree. Then a game day: partition two regions, write conflicting values, heal, and assert the replicas converge. A non-commutative merge fails there and nowhere else until production.
"What would you measure in production?" Four things, and the first is the one nobody does. Log the discarded write with both values, because it is the only way to turn "is LWW acceptable here" from an argument into a measurement. Then conflict rate per key class, because zero means you are paying for machinery you do not need. Then replication lag per region pair, since divergence is bounded by lag times write rate. And alert on sibling count, because a key accumulating siblings has a merge that is not running or not converging.
"You measure a zero conflict rate on some key class. What does that tell you?" That single-writer would be simpler and lose nothing. Teams build conflict resolution for data that never conflicts, and the merge function then goes untested until the day it finally runs, which is the worst possible time to discover it is not commutative. A measured zero is a signal to remove machinery, not to congratulate the machinery.
"How does this relate to choosing active-active in the first place?" It is downstream of it. The decision about where writes are authoritative comes first, and for data with real invariants the answer is that active-active is not available and the entity needs a single writer. This page is what you do for the data where active-active is appropriate, and the two most common mistakes are treating conflict resolution as a substitute for that decision, and choosing one mechanism for everything.
Common misconceptions
"Hybrid logical clocks fix last-writer-wins." They fix the wrong-winner problem caused by clock skew. They do not stop a concurrent write being discarded, which is what LWW means.
"CRDTs eliminate conflicts." They eliminate lost updates for types whose semantics are a merge. They cannot express an invariant, so a counter will go negative.
"Version vectors resolve conflicts." They detect them. Resolution is an application decision, and a helper that picks one arbitrarily is LWW without the honesty.
"Pick a conflict resolution strategy." Pick several, per data type. Any single choice is wrong for some of your data.
"Conflict resolution is a design decision." It is a thing you operate: conflict rate, replication lag, sibling counts, discarded-write logs and convergence testing.
Interview delivery note
Refuse the single-mechanism framing immediately, because it is the whole answer: "I'd choose per data type rather than per system. In a collaboration product that meant five mechanisms across seven data types, and any single choice would have been badly wrong for at least one: LWW everywhere loses document edits and lets two users register the same email; CRDTs everywhere can't express uniqueness or keep a balance above zero; single-writer everywhere puts a cross-region round trip on presence updates, which are most of the traffic."
Make the HLC distinction precisely, because it is the one people get wrong: "Hybrid logical clocks are worth using and they fix a different problem than people think. With wall clocks, a region whose NTP is 180 milliseconds fast wins every conflict, permanently and invisibly, and that's a systematic bias. HLC fixes that. It does not stop one of two concurrent writes being destroyed, because that's what last-writer-wins means."
Give the CRDT boundary as a rule: "A merge function can express 'combine these' and can't express 'only if'. Which is why a PN-Counter tracks inventory perfectly and will happily go negative: overselling is its correct behaviour, not a bug. If that invariant matters, the data needs a single writer and no better CRDT exists."
Then the operational half, which is what separates a design answer from an experienced one: "And I'd treat this as something you run rather than decide. The recommendation I'd make first is logging the discarded write with both values, because it's the only way to turn 'is LWW acceptable here' from an argument into a measurement. In one case it showed a 0.02 percent conflict rate concentrated in a single user editing on two devices, which confirmed the choice with evidence. Almost nobody does it."
Close with the convergence test: "and I'd verify the merge functions with property-based tests for commutativity, associativity and idempotence, plus a game day that partitions two regions, writes conflicting values and asserts convergence on heal. A non-commutative merge fails there and nowhere else until production."
Further reading
- Shapiro et al., "A comprehensive study of Convergent and Commutative Replicated Data Types" (INRIA 2011), for the catalogue and the convergence proofs.
- DeCandia et al., "Dynamo" (SOSP 2007), for version vectors with application-level sibling reconciliation.
- Kulkarni et al., "Logical Physical Clocks..." (2014), for HLC.
- The Automerge and Yjs documentation, for production sequence CRDTs and their metadata costs.
- The multi-region write path, for the decision that precedes this one.