Judgment collection and a training-data pipeline

What it is

A relevance judgment (also called a label, a grade, or in TREC vocabulary a qrel) is a recorded assessment of how well a document satisfies a query. Judgment collection is the process of producing those labels at sufficient volume and quality that you can (a) measure ranking quality offline and (b) train a ranking model.

It is worth being precise about the two consumers, because they have different requirements and teams routinely build for one and then discover it does not serve the other:

Consumer	Needs	Volume	Bias tolerance
Offline evaluation (NDCG, MRR)	Judgments on a fixed query set, stable over time	Hundreds of queries × ~50 docs	Low: bias here misleads every decision
Model training (LTR)	Judgments correlated with the target, on the live distribution	Millions of rows	Higher: the model can learn around some noise

The confusion to clear up: implicit feedback is not a cheaper version of explicit judgment. A click is evidence about the presented ranking under the conditions the user saw it. A human grade is an assessment of the document against the query. They measure different things, they have different biases, and the correct pipeline uses both for different purposes rather than substituting one for the other.

The problem it solves

Without judgments you cannot answer "did that change help," and every ranking decision becomes an argument between people with opinions. That is the visible problem. The invisible one is worse: without judgments you cannot detect a regression, so quality degrades in small increments that nobody attributes to any particular change, until someone runs a competitive comparison and finds the product is behind.

The second problem is that the naive substitute, "just use clicks," is systematically biased in ways that are large enough to invert conclusions:

Position bias. The item at rank 1 gets clicked far more than the same item at rank 3, independent of relevance. Published estimates of the examination propensity at rank 1 versus rank 5 commonly land in the 3x to 5x range depending on the interface, and it is larger on mobile where fewer results are visible.
Presentation bias. An item with a thumbnail, a price, or a badge gets clicked more. Your logs are measuring your UI as much as your ranker.
Selection bias (the big one). You only observe feedback on documents your current ranker showed. A model trained on that data learns to reproduce the current ranker, including its blind spots, and can never discover that document #200 was the right answer, because it was never shown. This is the feedback loop that produces the popularity bias spiral, and it compounds every retraining cycle.
Trust bias. Users click the top result partly because it is top, treating rank as an endorsement, which is position bias with an extra reinforcing mechanism.

Mechanics

The three sources, and what each is good for

1. Expert judgments. Trained annotators (often the search team, or a specialised vendor) grade query-document pairs against a written guideline. Highest quality, lowest volume, most expensive. Use these for the evaluation set, which must be trustworthy above all else.

2. Crowdsourced judgments. Platforms like Amazon Mechanical Turk or Appen, with redundancy: 3 to 5 workers per pair, aggregated. Ten to fifty times cheaper per label, noisier. Use these to extend the evaluation set and to cover long-tail queries.

3. Implicit feedback from logs. Free, enormous volume, biased. Use for training, after debiasing, and never as the sole basis for offline evaluation of a change to the ranker itself.

The grading scale

A four or five point graded scale is the workhorse, because NDCG needs graded relevance and because binary labels throw away the distinction between "this is the answer" and "this is related."

Grade	Gain (for NDCG)	Meaning
3 Perfect	7	This is precisely what the user asked for
2 Excellent	3	Fully satisfies the intent, not the single best item
1 Good	1	Related and plausibly useful
0 Irrelevant	0	Does not satisfy the query

Gain values of 2^grade - 1 are the standard NDCG formulation, and they matter: the exponential gain means a Perfect at rank 1 is worth more than two Excellents, which is usually the product intent.

The guideline document is where quality is actually won or lost. It must resolve the ambiguous cases in advance with examples: what grade does an out-of-stock exact match get, what about the right product in the wrong language, what about a category page when the query is a specific item. Every rule in the guideline should exist because two annotators disagreed once.

Measuring annotator agreement

Do not skip this. Agreement is the ceiling on what your labels can tell you.

Cohen's kappa for two annotators, Fleiss' kappa for more, corrects raw agreement for chance:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

where $p_o$ is observed agreement and $p_e$ is agreement expected by chance. For graded scales use weighted kappa (quadratic weights), because grading 3 versus 2 is a much smaller disagreement than 3 versus 0 and unweighted kappa treats them identically.

Practical thresholds: below 0.4 the guideline is broken and you must fix it before collecting more; 0.4 to 0.6 is workable for training data; above 0.6 is good for an evaluation set. If your model's measured improvement is smaller than your annotator disagreement, you have not measured an improvement.

from sklearn.metrics import cohen_kappa_score

# Two annotators, graded 0-3, quadratic weights so near-misses cost less.
kappa = cohen_kappa_score(annotator_a, annotator_b, weights="quadratic")

Debiasing implicit feedback

The standard treatment is inverse propensity scoring: weight each observed click by the inverse of the probability that the user examined that position.

$$\hat{R}(\pi) = \frac{1}{n}\sum_{i=1}^{n} \frac{c_i}{p(\text{examined at rank } k_i)}$$

A click at rank 8, where examination propensity is low, becomes strong evidence; a click at rank 1 becomes weak evidence. Estimating the propensities is the work, and there are two credible ways:

Result randomisation (RandPair / FairPairs): on a small fraction of traffic, swap two adjacent results at random. Since the assignment to positions is now random, the click rate difference between the positions estimates the propensity ratio directly. Costs a small amount of quality on the randomised slice, which is why teams do it on 1 to 5 percent of traffic.
Intervention harvesting: exploit the fact that you already ran ranking changes. The same document appeared at different positions across A/B variants and over time, and that natural variation identifies the propensity curve without deliberately degrading anything. Slower to accumulate, free.

The pipeline, end to end

                 ┌─────────────────────────────────────────────┐
  query logs ───▶│ 1. Query sampling                            │
                 │    stratified by head / torso / tail,        │
                 │    weighted by volume, refreshed quarterly   │
                 └────────────────┬────────────────────────────┘
                                  ▼
                 ┌─────────────────────────────────────────────┐
  candidate  ───▶│ 2. Pooling                                   │
  generation     │    top-k from EVERY system under comparison  │
                 │    plus a random sample from the long tail   │
                 └────────────────┬────────────────────────────┘
                                  ▼
                 ┌─────────────────────────────────────────────┐
                 │ 3. Annotation                                │
                 │    3 raters/pair, guideline v-pinned,        │
                 │    10% gold questions, kappa monitored       │
                 └────────────────┬────────────────────────────┘
                                  ▼
                 ┌─────────────────────────────────────────────┐
                 │ 4. Aggregation + QA                          │
                 │    majority / Dawid-Skene, drop raters below │
                 │    gold threshold, flag high-variance pairs  │
                 │    for expert adjudication                   │
                 └────────────────┬────────────────────────────┘
                                  ▼
                 ┌──────────────────────┐    ┌──────────────────┐
                 │ 5a. eval set (frozen)│    │ 5b. training set │
                 │     NDCG@10, MRR     │    │  + IPS-weighted  │
                 │     versioned        │    │    click data    │
                 └──────────────────────┘    └──────────────────┘

Step 2 (pooling) is the step teams skip and should not. If you judge only the top 10 of your current ranker, then a challenger that surfaces a genuinely better document at rank 3 gets no credit, because that document has no judgment and unjudged documents score 0. Your evaluation set will systematically favour the incumbent. TREC solved this in the 1990s with pooling: take the top-k from every participating system, judge the union. Do the same with every ranker variant you are seriously considering, plus a random sample so you have some signal about what you are missing entirely.

Gold questions (step 3) are pairs with a known correct grade, seeded invisibly into the work queue at around 10 percent. A rater whose gold accuracy falls below threshold is removed and their labels discarded. Without this, crowdsourced label quality decays because a fraction of workers optimise for throughput.

A worked example: an evaluation set that could not detect a real win

A marketplace search team had 500 judged queries with the top 10 results from production judged for each: 5,000 pairs, expert-graded, kappa of 0.71. Solid by every internal measure.

They built a semantic retrieval candidate generator and measured it offline:

                       NDCG@10
production (BM25)       0.412
+ semantic retrieval    0.389    (-5.6%)

Offline said it was worse. They ran it in an A/B test anyway, on a hunch:

                       CTR      add-to-cart    null-result rate
control                 --          --              8.1%
semantic                +2.4%      +1.9%            4.7%

Online said it was clearly better. The offline evaluation was wrong, and the reason was mechanical: the judgment pool contained only documents that BM25 had ranked in its top 10. The semantic system's contribution was surfacing documents BM25 never returned at all, and every one of those documents was unjudged, scoring 0 gain. The new system was being penalised precisely for the behaviour that made it valuable.

The fix was pooling. They re-pooled: top 20 from BM25, top 20 from semantic, top 20 from a hybrid, union deduplicated, which took the pool from 5,000 to about 14,000 pairs. After judging the additional 9,000:

                       NDCG@10 (old pool)   NDCG@10 (repooled)
production (BM25)          0.412                0.386
+ semantic retrieval       0.389                0.437

Note that production's score dropped on the repooled set, which is expected and correct: previously unjudged documents that turned out to be relevant now appear in the ideal ranking's denominator, so the achievable ceiling rose for everyone. Absolute NDCG values are only comparable within a pool version, which is why the pool version must be recorded alongside every reported number.

The cost was roughly 9,000 additional expert judgments. At around 30 seconds per judgment that is 75 hours of annotator time, which bought them a reliable offline signal for every subsequent retrieval change, and would have prevented shipping a regression as easily as it prevented killing a win.

Production evidence

TREC (the Text REtrieval Conference, NIST, since 1992) established pooling as the standard methodology precisely because judging every document in a collection is impossible. Their qrels files, the pooling depth conventions, and the analyses of pool bias are the reference material for anyone building an evaluation set, and the lesson that unjudged documents bias against novel systems was learned there first.

Bing published the position-bias problem and the randomisation solution through Joachims' and colleagues' line of work on unbiased learning to rank; the "Accurately Interpreting Clickthrough Data as Implicit Feedback" work established that raw click rates are not a valid relevance signal and that pairwise preferences derived from clicks (a clicked document is preferred to a skipped document ranked above it) are far more robust than absolute click rates.

Google's Search Quality Rater Guidelines are public, run to over 170 pages, and are the most detailed published example of what a judgment guideline looks like at scale. The E-E-A-T framework and the "Needs Met" scale are the operational definition of relevance for tens of thousands of raters. The document's length is the point: nearly all of it is worked examples resolving edge cases.

Amazon's and Yandex's published LTR work both describe using clicks for training with human judgments reserved for evaluation, which is the split recommended above. Yandex's released click datasets (for the Personalized Web Search Challenge) made the position-bias magnitude publicly measurable.

Airbnb's search ranking papers describe using booking outcomes rather than clicks as the training label, which is a specific instance of a general principle: the further down the funnel your label sits, the less biased and the sparser it is.

The debate

Should you train on human judgments or on clicks?

The case for judgments: they are unbiased with respect to position and presentation, they cover documents your ranker never showed, and they measure what you claim to care about.

The case for clicks: there are millions of them per day, they reflect real user intent rather than an annotator's model of user intent, and they update continuously as the catalog and the user population change. A judgment collected six months ago about a product that is now out of stock is worse than useless.

My position: human judgments for the evaluation set, debiased clicks for the training set, and never the reverse. The evaluation set is the thing you must be able to trust when it disagrees with your expectations, and clicks fail exactly there because they encode the current ranker's behaviour. Training on clicks is acceptable because scale beats bias for a model with millions of parameters, particularly with IPS weighting, and because the training distribution should match the serving distribution.

Should you use an LLM as an annotator? This is the live question. LLM judges correlate reasonably with human graders on straightforward relevance tasks, cost around two orders of magnitude less, and can be run on tens of thousands of pairs overnight. The concerns are real: they share failure modes with the retrieval model if both use related embeddings, they are sensitive to prompt phrasing in ways humans are not, and they cannot be audited the way a rater's gold accuracy can. My position is to use LLM judgment for pool expansion and triage (deciding which pairs are worth a human's time, filling in the obvious 0s), and to keep the frozen evaluation set human-graded, with a periodic measurement of LLM-to-human agreement using the same kappa you would use on any rater. The moment you evaluate a change using only LLM judgments, you have a metric whose relationship to users is unaudited.

Where I would spend less than the textbook says: the long tail. Judging tail queries is expensive per unit of traffic explained and annotator agreement is lower there because tail intent is genuinely ambiguous. Sample the tail for coverage (so you notice catastrophic failures like the German null-result problem in analyzers per language), but weight your evaluation set by traffic so the number tracks the business.

Follow-up Q&A

"Your offline NDCG says +3 percent and your A/B test says no change. What do you investigate?"

In order: (1) Pool bias, the case above, though it usually produces the opposite sign. (2) Query set drift: the evaluation queries were sampled a year ago and the traffic distribution has moved, so you improved on queries nobody runs now. (3) Metric-to-outcome mismatch: NDCG@10 improved by reordering positions 6 through 10, and users never scroll past 4, so the improvement is real and invisible. Check NDCG@3 and NDCG@5 separately. (4) Statistical power: a 3 percent NDCG change on 500 queries may not be significant; run a paired t-test or bootstrap over queries and report the confidence interval. (5) Segment cancellation: the change helps one segment and hurts another, netting to zero overall. That fourth point deserves emphasis, because reporting offline metrics without a confidence interval is extremely common and roughly half the small reported wins do not survive one.

"How many judged queries do you need?"

For detecting a 2 percent relative NDCG change with reasonable power, several hundred queries with per-query variance measured, and you should compute this rather than guess: run a bootstrap over your existing judged set to get the standard error of NDCG at your current set size, then scale by $1/\sqrt{n}$. In practice teams land between 300 and 2,000 queries for the frozen set. The depth matters as much as the count: judging 500 queries to depth 20 with pooling beats 2,000 queries to depth 5, because shallow pools reintroduce the bias problem.

"How do you keep an evaluation set from going stale?"

Version it and refresh a slice on a schedule rather than replacing it wholesale. Concretely: keep the query set fixed for a year so trends are comparable, but re-pool and re-judge whenever a new retrieval architecture enters serious consideration, and re-judge documents whose content changed (price, availability, title). Record the pool version with every number. When you do replace the query set, run both old and new for one cycle so you can translate historical numbers.

"An annotator disagrees with the guideline and thinks their grade is right. What do you do?"

Treat it as a guideline bug until proven otherwise. Collect the disputed cases, review them as a group with the search PM and an engineer, and either add a rule with an example or accept the annotator's reading and update the rule. The failure mode is silently overriding annotators, which produces raters who guess at what you want rather than applying a written standard, and destroys the reproducibility that makes the labels worth anything.

"How do clicks become training labels concretely?"

The most robust construction is pairwise preferences under a click model. Under the "skip-above" rule: if the user clicked the document at rank 5 and skipped ranks 1 through 4, generate preference pairs (doc5 > doc1), (doc5 > doc2), and so on. This is far more robust than absolute click rates because both documents in the pair were examined under similar conditions, so position bias largely cancels. Then weight by IPS for the residual. Feed those pairs to a pairwise objective like LambdaMART's, as described on the learning to rank page.

Common misconceptions

"More labels is always better." More labels from a broken guideline encode the confusion at higher volume. Fix kappa first. A 5,000-pair set at kappa 0.7 is worth more than a 50,000-pair set at kappa 0.35, because the second one cannot resolve the differences you care about.

"Unjudged means irrelevant." Every standard NDCG implementation treats unjudged as gain 0, which is a defensible default and a systematic bias against any system that retrieves differently from the one that built the pool. This is the single most consequential fact about offline evaluation and the worked example above is what it looks like in practice.

"Clicks measure relevance." Clicks measure examination times attractiveness times relevance, and the first two are properties of your ranking and your UI. A title with a number in it gets more clicks. That is not relevance and a model trained naively will learn to promote clickbait.

"We can skip judgments because we A/B test everything." A/B tests are the ground truth and they are slow (weeks per decision), expensive in traffic, and cannot be run on ideas you have not built. Offline judgments let you kill nine bad ideas in a day and A/B the tenth. Teams without an offline set ship fewer ranking changes per quarter, not more.

"The evaluation set should reflect our best guess at ideal results." It should reflect user satisfaction on the actual query distribution. An evaluation set built from the queries the team finds interesting will improve the search experience for the team.

Interview delivery note

The line worth saying verbatim: "Unjudged documents score zero, so an evaluation pool built from one ranker's results systematically punishes any challenger that retrieves differently. Pooling across every system under comparison is not optional." That single observation is the difference between an evaluation program that works and one that quietly blocks every retrieval improvement, and most candidates have never thought about it.

The senior-versus-staff separator here is treating judgments as a program with an operating cost rather than a one-off task. A senior engineer describes a grading scale and NDCG. A staff engineer talks about kappa as the ceiling on measurable improvement, gold questions and rater removal, pool versioning, refresh cadence, and the budget: "roughly 30 seconds per judgment, so a re-pool of 500 queries to depth 20 is about 75 annotator-hours, which we schedule once per architecture change." Naming the cost signals you have actually run one.

If asked to design this from nothing, commit to the split: human-graded frozen evaluation set, click-derived IPS-weighted training set, LLM judgment for triage only. Then name the first thing you would measure, which is annotator agreement, because everything downstream is bounded by it.

The Staff & Team Lead Interview Reference