Entity resolution

What it is

Deciding which records refer to the same real-world thing. Acme Corp, ACME Corporation, Acme Corp. and acme are four strings and, usually, one company.

The vocabulary varies by field and means roughly the same operation: record linkage (statistics), deduplication (data engineering), identity resolution (marketing), coreference resolution (NLP, within a document). This page uses entity resolution for all of it.

The three-stage shape that every implementation has, whatever it is called:

1. BLOCKING     reduce O(n²) comparisons to something tractable
2. SCORING      compute a similarity for each surviving candidate pair
3. CLUSTERING   turn pairwise decisions into groups, transitively

The arithmetic that makes stage 1 non-optional:

100,000 records, all-pairs:  n(n-1)/2 = 5 x 10^9 comparisons
at 10 microseconds each:     ~14 hours
at 1 million records:        5 x 10^11 comparisons, ~58 days

Blocking is the entire reason this is an engineering problem rather than a string comparison problem.

What it is confused with: fuzzy string matching. Levenshtein distance on names is one signal among several, and on its own it is both too permissive (Acme Corp and Acne Corp are one edit apart and different companies) and too restrictive (IBM and International Business Machines share almost no characters and are the same company). Resolution is a classification problem over multiple features, not a distance threshold.

The problem it solves

Unresolved entities break systems in ways that produce no errors, which is what makes this worth a page.

In a knowledge graph (see GraphRAG): five variants of one company means five nodes, each holding a fifth of the edges. A traversal from any one of them finds a fifth of what is known, and reports it confidently.

In a customer database: the same person as four records means four partial histories, a support agent seeing one of them, and a "we have no record of that" for a conversation that happened.

In analytics: counting distinct customers over-counts by the duplication rate, and every per-customer metric is diluted by it. A 30 percent duplication rate makes average revenue per customer 30 percent too low, consistently, in a way that looks like a business problem rather than a data problem.

In compliance: a deletion request that matches one of four records leaves three, and "we deleted your data" is false.

The costs of getting it wrong run in both directions and they are not symmetric:

FALSE MERGE (two different entities combined):
  Two customers' data merged. A support agent sees the wrong person's history.
  Under GDPR, a disclosure. Usually the more damaging error.

FALSE SPLIT (one entity left as two):
  Fragmented history, over-counted customers, incomplete deletion.
  Usually recoverable, and often invisible.

False merges are harder to detect and harder to undo, because once two records are merged the evidence that they were distinct is gone unless you kept it. That asymmetry should drive the threshold.

Mechanics

Blocking

Only compare records that share a cheap key.

def blocking_keys(record) -> set[str]:
    """A record joins several blocks. More keys = higher recall, more comparisons."""
    keys = set()
    if record.email:
        keys.add(f"email_domain:{record.email.split('@')[1]}")
    if record.postcode:
        keys.add(f"postcode:{record.postcode[:4]}")
    if record.name:
        norm = normalise(record.name)                 # lowercase, strip legal suffixes
        keys.add(f"name_prefix:{norm[:4]}")
        keys.add(f"name_sorted:{''.join(sorted(norm.split()))[:8]}")
        keys.add(f"soundex:{soundex(norm)}")
    return keys

100,000 records:
  no blocking:            5.0 x 10^9 pairs
  name_prefix only:       ~2.1 x 10^6 pairs      (2,400x reduction)
  5 keys, union:          ~8.4 x 10^6 pairs      (600x reduction, higher recall)

The trade is explicit: more blocking keys means more candidate pairs and higher recall. A pair that shares no blocking key is never compared and can never be merged, so blocking sets a hard ceiling on recall that no amount of scoring quality can recover. That is the step to over-invest in.

Sorted neighbourhood is the other classic approach: sort records by a key, slide a window of size w, compare within the window. Recall depends on the sort key placing duplicates near each other, and it degrades gracefully rather than cliff-edging.

Embedding-based blocking is the modern option and is often the best: embed a concatenation of the record's fields, index with HNSW, and take the top-k neighbours as candidates. It handles IBM versus International Business Machines where every string-based key fails, and it costs an embedding per record plus an index.

def embedding_blocks(records, k=20):
    texts = [f"{r.name} {r.address} {r.industry}" for r in records]
    index = hnswlib_index(embed(texts))
    return [(i, j) for i, r in enumerate(records)
                   for j in index.knn(i, k=k) if j > i]

Scoring: features, not a distance

@dataclass
class PairFeatures:
    name_jaro_winkler: float        # good for typos and prefixes
    name_token_jaccard: float       # good for word reordering
    name_soundex_match: bool        # phonetic
    email_exact: bool
    email_local_match: bool         # same local part, different domain
    phone_normalised_match: bool
    address_similarity: float
    postcode_exact: bool
    industry_match: bool
    founded_year_delta: int | None
    embedding_cosine: float

Jaro-Winkler for names specifically, because it weights matching prefixes, which suits the way names are truncated and abbreviated. Token Jaccard catches Smith John versus John Smith, which edit distance handles badly.

Two scoring approaches:

Fellegi-Sunter (the classical probabilistic model) computes per-feature weights from the probability a feature agrees given a match (m) versus given a non-match (u):

$$w_i = \log_2 \frac{m_i}{u_i} \text{ if agreeing}, \quad \log_2 \frac{1-m_i}{1-u_i} \text{ if not}$$

The total is the sum, compared against two thresholds: above upper, auto-match; below lower, auto-reject; between, human review. The two-threshold design with a review band is the part worth carrying, because it is what makes the accuracy tunable against review capacity.

A trained classifier (gradient-boosted trees on the feature vector) is the modern default and typically outperforms Fellegi-Sunter given labelled pairs. Both need labels; Fellegi-Sunter can estimate m and u with expectation-maximisation from unlabelled data, which is its remaining advantage.

def score(a, b) -> float:
    f = extract_features(a, b)
    if f.email_exact:
        return 1.0                       # a deterministic rule beats a model here
    if f.phone_normalised_match and f.name_jaro_winkler > 0.85:
        return 0.98
    return classifier.predict_proba(f.as_vector())[1]

Deterministic rules before the model, for high-precision signals. An exact match on a verified email or a government ID is stronger evidence than any learned combination of fuzzy features, and hard-coding it is both faster and more explainable.

Clustering: pairwise decisions are not transitive

A ~ B  (score 0.91)
B ~ C  (score 0.89)
A ~ C  (score 0.42)     <- the pairwise scorer says these are different

Something must decide whether {A, B, C} is one entity or two, and the choice of clustering algorithm is that decision:

Connected components:   A, B, C all merge. Simple, and it CHAINS: one bad
                        edge merges two unrelated clusters entirely.
Correlation clustering: minimise disagreement with the pairwise scores.
                        Better, NP-hard, approximated in practice.
Hierarchical + cut:     agglomerate, cut at a threshold. Common and workable.

Connected components is the default in most naive implementations and it is dangerous, because a single false-positive edge can merge two large clusters. On a customer database that is exactly the false-merge failure, at scale, from one bad pair.

The mitigation that works without a sophisticated algorithm: cap cluster size and flag anything over it for review. A "customer" cluster with 400 records is not a customer, it is a chained merge, and a size cap catches it.

Incremental resolution

Batch resolution is a full re-cluster; production usually needs "resolve this new record against what exists":

def resolve_incremental(new_record, store) -> EntityId:
    candidates = store.candidates(blocking_keys(new_record))     # blocking, as before
    scored = [(c, score(new_record, c)) for c in candidates]
    best, s = max(scored, key=lambda x: x[1], default=(None, 0))

    if s >= AUTO_MATCH:
        store.attach(new_record, best.entity_id)
        return best.entity_id
    if s >= REVIEW_LOWER:
        store.queue_for_review(new_record, best)
        return store.create_provisional(new_record)     # do NOT merge on a maybe
    return store.create(new_record)

Creating a provisional entity rather than merging on an uncertain match is the right default, given the asymmetry: a false split is recoverable and a false merge may not be.

The complication is that incremental resolution drifts from what batch resolution would produce, because early decisions constrain later ones. A record that would have merged given evidence that arrived later is already a separate entity. Periodic full re-resolution is the answer, and it needs a way to carry forward human review decisions so they are not re-asked.

Using an LLM

LLMs are good at the ambiguous middle and too expensive for the whole problem:

def llm_adjudicate(a, b) -> tuple[bool, str]:
    return model(f"""
Are these the same organisation? Consider name variants, abbreviations,
subsidiaries and rebrands. A subsidiary is NOT the same as its parent.

A: {a.name} | {a.address} | {a.industry} | founded {a.founded}
B: {b.name} | {b.address} | {b.industry} | founded {b.founded}

Answer same/different and give one sentence of reasoning.
""", schema=Adjudication)

Reserve it for the review band, which is typically 2 to 5 percent of pairs. Running it on every candidate pair costs orders of magnitude more than a classifier for a small accuracy gain, and it is slower.

The subsidiary instruction is doing real work. LLMs merge parent and subsidiary companies readily, because they are related in the model's world knowledge, and that is a false merge with legal consequences in a customer database.

A worked example: 30 percent duplication in a CRM

A B2B company with 340,000 company records accumulated from four sources: a CRM, self-service signups, a purchased data provider and imported event attendee lists.

Symptoms:

records:                        340,000
estimated true companies:       ~240,000   (~30% duplication)
sales complaints:               "I called them last week, why is there
                                 no record?"
account manager confusion:      two AMs owning what turned out to be
                                 one account
marketing:                      duplicate sends, unsubscribe honoured on
                                 one record and not others
reported revenue per account:   understated by roughly the duplication rate

The first attempt, by an analyst, was exact-normalised-name matching:

key = re.sub(r'\b(inc|ltd|llc|corp|corporation|limited|gmbh)\b', '',
             name.lower()).strip()

merged:                 41,000 records
remaining duplicates:   estimated 58,000 (most of them)
false merges:           ~900 (found later)

It caught the easy cases and made 900 false merges, because Smith Consulting in Toronto and Smith Consulting in Manchester normalised identically. A name is not an identifier, and any approach that treats it as one produces exactly this.

The rebuild.

Stage 1: blocking, tuned for recall.

def blocking_keys(r):
    keys = set()
    if r.website:      keys.add(f"domain:{registered_domain(r.website)}")
    if r.email_domain: keys.add(f"domain:{r.email_domain}")
    if r.phone:        keys.add(f"phone:{normalise_phone(r.phone)[-7:]}")
    if r.name:
        n = normalise(r.name)
        keys.add(f"np:{n[:5]}")
        keys.add(f"ns:{''.join(sorted(n.split()))[:8]}")
    keys.add(f"emb:{lsh_bucket(embed(f'{r.name} {r.address}'))}")
    return keys

candidate pairs:        4.1 million (from 5.8 x 10^10 all-pairs)
blocking recall
  (on a labelled set):  97.2%

The 2.8 percent blocking miss is a permanent ceiling, and measuring it against a hand- labelled sample was the step that made the rest of the tuning meaningful. Most implementations never measure blocking recall and therefore do not know their ceiling.

Stage 2: a classifier over features, plus deterministic rules.

# Deterministic, high-precision, checked first.
if same_registered_domain(a, b):          return 1.0
if a.duns_number and a.duns_number == b.duns_number:  return 1.0
if a.vat_number and a.vat_number == b.vat_number:     return 1.0
# Otherwise, the model.

Training data: 8,000 pairs hand-labelled by two people, disagreements adjudicated by a third (Cohen's kappa 0.83).

gradient-boosted trees, held out:
  precision @ 0.90 threshold:   0.994
  recall    @ 0.90 threshold:   0.871
  precision @ 0.75 threshold:   0.961
  recall    @ 0.75 threshold:   0.943

Two thresholds rather than one:

score >= 0.90    auto-merge          (precision 0.994)
0.60-0.90        human review queue  (~3.2% of pairs)
score < 0.60     auto-reject

The asymmetry of errors set the upper threshold. At 0.994 precision, auto-merging produces roughly 1 false merge per 170; at 0.961 it would be 1 in 26. Given that a false merge can be a data disclosure, the team took the recall loss and sent the middle to review.

Stage 3: clustering, with a cap.

clusters = hierarchical_agglomerative(pairs, linkage="average", cut=0.90)
for c in clusters:
    if len(c) > CLUSTER_SIZE_CAP:            # 25
        review_queue.add(c, reason="oversized_cluster")

The size cap caught 14 chained merges immediately, the largest of which had linked 1,100 records through a chain of consultancies sharing a serviced-office address.

Stage 4: LLM adjudication for the review band.

pairs in the review band:       131,000
LLM-adjudicated:                131,000 at ~$0.002 each = $262
agreement with human labels
  on a 500-pair audit:          91.4%
remaining for human review:     11,300  (LLM low-confidence)

$262 to adjudicate 131,000 pairs, against an estimated 900 hours of human review. The 8.6 percent disagreement rate is why the LLM's low-confidence cases still went to humans rather than being auto-applied.

The subsidiary problem showed up here. Before the explicit instruction, the LLM merged Contoso Ltd with Contoso Financial Services Ltd at a high rate. Adding "a subsidiary is NOT the same as its parent" to the prompt took that error class from 23 percent of a sample to 4 percent.

Results:

                              before      after
records                       340,000     340,000 (source records preserved)
resolved entities             340,000     239,400
duplication rate              ~30%        1.8% (audited on a sample)
false merges (audited)        ~900        11
blocking recall               n/a         97.2%
review queue                  n/a         11,300 one-off, ~180/week ongoing
resolution cost               n/a         $262 one-off + ~$40/month
revenue per account           understated corrected (+29%)

Two things worth separating.

The 29 percent revenue-per-account correction was not a business change, it was a data correction, and communicating that was harder than the engineering. Every per-customer metric moved at once and several dashboards had to be re-baselined.

Eleven false merges remained, and they were treated as incidents, each investigated and unmerged. That was possible only because the pipeline kept the source records and treated the entity as a cluster of records rather than overwriting them:

@dataclass
class ResolvedEntity:
    entity_id: str
    member_record_ids: list[str]      # sources preserved, ALWAYS
    canonical: dict                    # the merged view
    decisions: list[MergeDecision]     # what merged, why, at what score, by whom

Keeping the source records and the merge decisions is what makes a false merge recoverable. A pipeline that overwrites records with a merged view has no way back, and given that false merges are the more damaging error, that is the design decision that matters most.

Production evidence

Fellegi and Sunter's "A Theory for Record Linkage" (1969) is the foundational probabilistic model and its two-threshold design (auto-match, review band, auto-reject) remains the standard operational shape sixty years later.

Splink (UK Ministry of Justice, open source) implements Fellegi-Sunter with expectation-maximisation parameter estimation at scale on Spark and DuckDB, and is the most widely used open implementation. Its documentation on blocking rules and the recall ceiling they impose is the best practical treatment available.

Dedupe (Python) implements active learning: it asks a human to label the pairs it is most uncertain about, which is a much cheaper path to a training set than labelling random pairs.

Zingg and the commercial identity-resolution vendors (LiveRamp, Experian) exist because this is a large recurring problem, and their common architecture is the same three stages.

The Magellan project (Doan et al.) produced a body of work on entity matching as a supervised learning problem, and Ditto (Li et al., 2020) showed transformer-based matchers outperforming classical feature engineering on standard benchmarks, which is the research direction LLM adjudication descends from.

GraphRAG implementations all include a resolution step and Microsoft's documentation is explicit that resolution quality determines graph quality, which is the same finding this page reaches from the CRM direction.

The debate

How much should you invest in blocking? More than most teams do, because blocking sets a hard ceiling on recall that no scoring improvement can lift. A pair never compared can never be merged. The step almost nobody does is measuring blocking recall against a hand-labelled sample, and without it you do not know your ceiling and cannot tell whether a scoring improvement is worth pursuing.

Where should the threshold be? Driven by the asymmetry of errors, not by an F1 optimum. A false merge combines two entities and may be a data disclosure; a false split is fragmentation, which is recoverable and often invisible. Set the auto-merge threshold for high precision, accept the recall loss, and send the middle to review. F1 treats the two errors as equivalent and they are not.

Is a review queue realistic? It is the part that gets cut, and it is what makes the accuracy tunable. Three percent of pairs in a review band on a large corpus is a lot of human time, which is exactly where LLM adjudication earns its place: $262 for 131,000 pairs against roughly 900 human hours. The design that works is model, then LLM for the middle, then human for the LLM's low-confidence cases, and each stage handles an order of magnitude less volume than the last.

Should you use an LLM for the whole thing? No. It is orders of magnitude more expensive than a classifier per pair and slower, for a small accuracy gain on the easy cases where a classifier is already at 0.99 precision. Use it in the ambiguous band, and give it explicit instructions about the confusions it makes (subsidiaries, franchises, rebrands, same-name different-location), because its world knowledge causes false merges a feature-based classifier would not make.

Batch or incremental? Both. Incremental for new records so the system stays current, and periodic full re-resolution because incremental decisions drift: a record that would have merged given later evidence is already separate. The requirement people miss is carrying human review decisions forward through a re-resolution, so the same pairs are not re-adjudicated.

Should you ever destructively merge? No. Keep the source records and the merge decisions, and treat the resolved entity as a cluster with a canonical view. Given that false merges are the more damaging error and that they will happen at any threshold, an un-merge path is a requirement rather than a nicety. A pipeline that overwrites has no way back.

Follow-up Q&A

"Why is blocking necessary?"

All-pairs comparison is n(n-1)/2, so 100,000 records is 5 billion comparisons and a million records is 500 billion. Blocking restricts comparison to records sharing a cheap key, typically reducing pairs by two to three orders of magnitude. The critical property is that it sets a hard ceiling on recall: a pair that shares no blocking key is never compared and can never be merged, so blocking recall should be measured against a labelled sample before any effort goes into scoring.

"How do you score a pair?"

Features, not a distance. Jaro-Winkler on names for typos and prefixes, token Jaccard for reordering, exact matches on email, phone and any identifier, address similarity, embedding cosine. Then either Fellegi-Sunter weights (which can be estimated with EM from unlabelled data) or a trained classifier, which usually wins given labels. Deterministic high-precision rules go first: an exact registered-domain or VAT-number match is stronger evidence than any learned combination.

"Why is a single threshold wrong?"

Because the two errors are not symmetric. A false merge combines two real entities, which in a customer database can be a data disclosure and destroys the evidence that they were distinct. A false split fragments a history, which is recoverable. So set the auto-merge threshold for high precision (0.994 rather than 0.96 in one case), auto-reject at the bottom, and send the middle to a review band. Optimising F1 treats the errors as equal.

"What goes wrong with clustering?"

Pairwise decisions are not transitive: A matches B, B matches C, and A does not match C. Connected components merges all three and, worse, chains: one false-positive edge merges two large clusters entirely. In one case a chain through a shared serviced-office address linked 1,100 records. The cheap mitigation is a cluster size cap with review above it; the better one is correlation clustering or hierarchical with a cut.

"Where does an LLM fit?"

In the review band, roughly 2 to 5 percent of pairs, where a classifier is uncertain. Running it on every candidate is orders of magnitude more expensive for a small gain where the classifier is already at 0.99 precision. It needs explicit instructions about the confusions its world knowledge causes: it merges parents with subsidiaries readily, and adding "a subsidiary is not the same as its parent" took that error class from 23 percent to 4 percent in one sample.

"What is the most important design decision?"

Keeping the source records and the merge decisions, rather than overwriting with a merged view. False merges will happen at any threshold, they are the more damaging error, and an un-merge path only exists if the evidence was preserved. Model the resolved entity as a cluster of member records plus a canonical view plus the decision log, and un-merging becomes an operation rather than an archaeology project.

Common misconceptions

"It is fuzzy string matching." Edit distance is one feature. It is too permissive (Acme and Acne are one edit apart) and too restrictive (IBM and International Business Machines share almost nothing), and resolution is a classification problem over several features.

"Normalise the name and group." That is what produced 900 false merges in the worked example, because Smith Consulting in Toronto and in Manchester normalise identically. A name is not an identifier.

"Optimise F1." F1 weights false merges and false splits equally and they are not equally costly. Set the threshold from the asymmetry.

"Pairwise decisions cluster themselves." They are not transitive, and connected components chains: one bad edge merges two large clusters silently. A size cap catches the worst of it.

"Resolve once and you are done." New records arrive and incremental decisions drift from what batch resolution would produce, because early decisions constrain later ones. Periodic full re-resolution is needed, carrying forward human decisions so they are not re-asked.

Interview delivery note

Say this verbatim: "The two errors are not symmetric: a false merge combines two real entities and may be a data disclosure, and a false split is recoverable fragmentation. So I set the auto-merge threshold for precision rather than F1, send the ambiguous band to review, and keep the source records so a merge can be undone." The asymmetry drives the design and it is the thing most implementations get backwards.

The senior-versus-staff separator is blocking recall as a measured ceiling. A senior engineer describes blocking, scoring and clustering correctly. A staff engineer measures blocking recall against a hand-labelled sample first, because a pair never compared can never be merged and no scoring improvement recovers it, and therefore knows whether the 97 percent ceiling or the classifier is the binding constraint. Most implementations never measure it.

The second signal is preserving source records and merge decisions. Saying "false merges will happen at any threshold, they are the damaging direction, so the entity is a cluster of member records with a decision log rather than an overwritten row" shows you have had to undo one.

The Staff & Team Lead Interview Reference