Learning to rank: features and a worked feature set

What it is

Training a model to order results, rather than hand-tuning a scoring formula. Three formulations, distinguished by what the loss function sees:

POINTWISE     Predict a relevance score per (query, document).
              Loss: regression or classification, per item.
              Simple, and it optimises the wrong thing: getting
              every absolute score right is neither necessary
              nor sufficient for a good ORDERING.

PAIRWISE      Predict which of two documents is better.
              Loss: over pairs within a query.
              Directly optimises ordering. RankNet, LambdaRank,
              LambdaMART.

LISTWISE      Optimise a list-level metric directly.
              Loss: over the whole result list. ListNet,
              LambdaMART with NDCG-weighted gradients,
              SoftRank.

Commonly confused: LambdaMART is usually called listwise and is mechanically pairwise, with each pair's gradient scaled by the NDCG change that swapping that pair would produce. That scaling is what makes it optimise a list metric using pairwise machinery, and it is the reason it remains the strongest practical method on tabular features.

Also commonly confused with "use a neural ranker". Gradient-boosted trees still win on heterogeneous tabular features, which is what ranking features are; neural rankers win when the features are raw text and the model can learn representations.

The problem it solves

A hand-tuned formula stops scaling at about five signals.

score = 2.0*bm25_title + 1.0*bm25_body + 0.5*log(popularity)
      + 0.3*freshness - 0.8*log(price)

Problems:
  - The weights were chosen by someone's intuition and tested
    on a handful of queries.
  - Interactions are invisible: freshness matters enormously
    for news queries and not at all for reference queries, and
    a linear form cannot express that.
  - Every new signal requires re-tuning every weight.
  - Nobody can say whether a change helped, because there is
    no metric being optimised.

Learning to rank replaces the weights with a fit and the intuition with a metric. The model learns the interactions, and the question "did this help" becomes measurable.

What it does not solve, and the boundary to state: it reorders what retrieval returned. If the answer-bearing document was never a candidate, no ranker recovers it, which is why retrieval recall is measured separately and is the ceiling.

Mechanics

The worked feature set

This is the part that decides whether an LTR system works, and it is where most of the engineering time goes. Features in five families:

QUERY-ONLY (no document involved; the model uses these to
modulate everything else)
  query_length_tokens
  query_length_chars
  is_navigational          classifier: does it name one thing?
  has_exact_identifier     regex: SKU, model number, ISBN
  detected_language
  query_frequency_bucket   head / torso / tail
  intent_class             transactional / informational / nav

DOCUMENT-ONLY (precomputable, no query involved)
  quality_score            editorial or learned
  popularity_30d           log-scaled interactions
  freshness_days           age in days, and log(age)
  content_length
  has_image, has_price, in_stock
  seller_rating
  spam_score

QUERY-DOCUMENT (the ones that matter most, and the only ones
that must be computed at query time)
  bm25_title, bm25_body, bm25_tags     per-field lexical
  exact_phrase_match_title             boolean
  cosine_similarity_dense              vector score
  term_coverage                        fraction of query terms present
  proximity_min_window                 tightest span containing all terms
  field_with_best_match                categorical
  edit_distance_to_title               for near-miss typos

USER-DOCUMENT (personalisation; optional and expensive)
  user_category_affinity
  has_viewed_before
  days_since_last_interaction
  price_vs_user_typical_band

CONTEXT
  device_type, hour_of_day, day_of_week
  session_position         first query or a refinement?
  previous_query_in_session

Three design rules for this set, and each has a failure behind it:

1. Log-scale anything with a heavy tail. Popularity, view counts and prices span orders of magnitude, and a raw count means the model spends its capacity on the top 0.1 percent of items. log(1 + x) is almost always right.

2. Never feed a raw identifier as a numeric feature. seller_id = 88412 implies an ordering that does not exist. Trees will happily split on it and learn nothing generalisable. Use aggregate statistics of the entity instead: seller_rating, seller_return_rate.

3. Query-document features carry most of the signal, and they are the only ones that cannot be precomputed, so they set the query-time cost. Everything else comes from the feature store.

Why gradient-boosted trees, still

Ranking features are HETEROGENEOUS: counts, ratios, booleans,
categoricals, log-scaled scores, all on different scales with
different distributions.

TREES                              NEURAL NETS
+ Scale-invariant: no need to      - Need normalisation, and the
  normalise anything                 right normalisation per feature
+ Handle missing values natively   - Need imputation, which is a
                                     modelling decision
+ Capture interactions without     - Capture them, given enough data
  feature crosses
+ Fast inference: a few hundred    - Slower, and often needs a GPU
  tree traversals
+ Interpretable: feature           - Harder to attribute
  importance, SHAP
- Cannot learn representations     + Learn from raw text directly
  from raw text

The rule: trees for tabular features, neural for raw text. Which is why production systems usually run both, with a cross-encoder doing the text understanding on a small candidate set and a GBDT doing the feature-based ranking on a larger one. See the multi-stage funnel.

LambdaMART, and the idea worth knowing

The problem with a pairwise loss: swapping the documents at
positions 1 and 2 changes NDCG far more than swapping the
documents at positions 49 and 50, and a plain pairwise loss
treats those swaps identically.

LambdaRank's insight: don't define a loss function, define the
GRADIENT directly, scaled by the metric change.

  lambda_ij = |ΔNDCG_ij| * sigmoid_derivative(s_i - s_j)

  where ΔNDCG_ij is the NDCG change from swapping i and j.

Pairs whose swap barely moves NDCG get small gradients. Pairs
at the top of the list get large ones. So the model spends its
capacity where the metric is sensitive.

That is the whole idea and it is elegant: you cannot differentiate NDCG, so you weight the pairwise gradients by it instead. LambdaMART is LambdaRank's gradients inside gradient boosting, and it won the Yahoo Learning to Rank Challenge and remains the strong baseline.

import lightgbm as lgb

# The essentials for a ranking objective, with the parts that
# people get wrong called out.
ranker = lgb.LGBMRanker(
    objective="lambdarank",
    metric="ndcg",
    ndcg_eval_at=[5, 10],
    # Position discount truncation: pairs below this position
    # contribute nothing. Matching it to the page size focuses
    # the model where users look.
    lambdarank_truncation_level=20,
    n_estimators=500,
    learning_rate=0.05,
    num_leaves=63,
)

ranker.fit(
    X_train, y_train,
    # CRITICAL: group tells the ranker which rows belong to the
    # same query. Without it, it compares documents ACROSS
    # queries, which is meaningless and silently produces a
    # much worse model.
    group=group_sizes_train,
    eval_set=[(X_valid, y_valid)],
    eval_group=[group_sizes_valid],
)

The group parameter is the single most common implementation error. Ranking loss is defined within a query, and omitting the grouping makes the model learn to compare a document for one query against a document for another, which is meaningless. It trains without error and produces a much worse model.

Training data: the hard part

The model is the easy part. Labels are where LTR projects fail.

HUMAN JUDGEMENTS
  Raters score (query, document) pairs on a graded scale.
  + High quality, graded, no position bias.
  - $0.50 to $3 per judgement. 1,000 queries x 20 documents
    is 20,000 judgements.
  - Inter-rater agreement is typically 0.6-0.8, which is the
    noise floor of everything downstream and must be measured.
  - Raters judge TOPICAL relevance; users want usefulness.

CLICK LOGS
  + Free, abundant, at scale, reflects real intent.
  - POSITION BIAS dominates: position 1 gets several times the
    clicks of position 5 regardless of relevance.
  - PRESENTATION BIAS: the click reflects the snippet.
  - SELECTION BIAS: you only observe what you showed, so the
    log is a record of your current ranker's decisions.

DERIVED SIGNALS (better than raw clicks)
  click-through with a dwell threshold ("long click")
  add-to-cart, purchase, save          strong intent
  query reformulation after a click    a NEGATIVE signal: the
                                       result did not satisfy
  last click in a session              often the satisfying one

Deriving labels from clicks properly requires debiasing:

# Inverse propensity weighting. A click at a low-propensity
# position is much stronger evidence than one at position 1.
def label_from_clicks(impressions, propensity, clip=0.05):
    rel = defaultdict(float)
    for imp in impressions:
        p = max(propensity[imp.position], clip)   # clip bounds variance
        if imp.clicked and imp.dwell_ms > 30_000:
            rel[imp.doc_id] += 1.0 / p
        elif not imp.clicked:
            rel[imp.doc_id] -= 0.2 / p            # weak negative
    return rel

The clip matters and is not a detail: very low propensities produce enormous weights and the variance of the estimator explodes, so a handful of rare observations dominate training. Clipping trades a little bias for a large variance reduction.

And the practical recommendation: use both sources. Human judgements for a smaller, high-quality, position-unbiased evaluation set; debiased click data for the much larger training set. Judgements alone are too expensive to cover the tail; clicks alone bake in the current ranker's biases.

A worked example: from formula to model

STARTING POINT
  A hand-tuned formula, 6 weights, NDCG@10 = 0.62 on a
  1,000-query judged set.

STEP 1: instrument
  Log every (query, document, features, position, click, dwell)
  tuple at serving time. Without this there is no training
  data, and it takes a week of traffic to accumulate enough.

STEP 2: build the label set
  5,000 queries sampled stratified by frequency bucket, so the
  tail is represented rather than swamped by head queries.
  Human judgements on 1,000 of them (20 documents each,
  20,000 judgements) for evaluation.
  Debiased click labels on the remaining 4,000 for training.

STEP 3: features
  Start with the 12 query-document features, because they carry
  most of the signal and are the ones the current formula
  already approximates. Add document-only and query-only next.
  Personalisation last, because it is the most expensive and
  the least certain.

STEP 4: train
  LightGBM LambdaRank, group by query id, truncation at 20.
  Split by QUERY, never by row: rows from the same query in
  both train and test leaks badly and inflates the metric.

STEP 5: measure
  NDCG@10 = 0.71 offline, +14.5% over the formula.

STEP 6: DO NOT SHIP ON THAT
  Interleave against the formula. Offline gains routinely fail
  to replicate, and the most common cause is evaluation-set
  bias: the judged pool came from the OLD ranker's results, so
  the new ranker's genuinely new documents are unjudged and
  scored as irrelevant.
  Pool judgements over BOTH rankers before comparing.

STEP 7: feature importance, as a sanity check
  bm25_title              0.19
  cosine_dense            0.14
  popularity_30d_log      0.11
  exact_phrase_title      0.09
  freshness_log           0.08
  ...
  seller_id_numeric       0.06   <-- WRONG. A raw identifier
                                      should not be predictive;
                                      this is the model
                                      memorising sellers.

Step 7's finding is the kind of thing feature importance is for, and removing that feature and replacing it with seller_rating and seller_return_rate both generalises better and removes a fairness problem nobody had noticed.

Production evidence

Burges, "From RankNet to LambdaRank to LambdaMART: An Overview" (Microsoft Research, 2010) is the definitive account of the progression, including the insight that you define the gradient rather than the loss.

Chapelle and Chang, "Yahoo! Learning to Rank Challenge Overview" (2011) documents that gradient-boosted decision trees dominated the competition, and the datasets released are still standard benchmarks.

LightGBM's lambdarank objective and XGBoost's rank:ndcg are the production implementations, and LightGBM's documentation of lambdarank_truncation_level reflects that position truncation is a real tuning parameter.

Joachims et al., "Accurately Interpreting Clickthrough Data as Implicit Feedback" (SIGIR 2005) established position bias with eye-tracking, and Joachims, Swaminathan and Schnabel's "Unbiased Learning-to-Rank with Biased Feedback" (WSDM 2017) is the propensity-weighting reference.

Elasticsearch's Learning to Rank plugin and OpenSearch's equivalent implement the standard shape: features defined as queries, logged at serving time, and a model applied as a rescorer over the top N, which is exactly the funnel structure.

Airbnb's published search-ranking work documents the progression from GBDT to neural ranking and is unusually honest about the neural model initially performing worse, which is useful evidence against assuming neural is an upgrade.

The debate

The case for GBDT (LambdaMART): it wins on tabular features, needs no normalisation, handles missing values natively, trains in minutes on commodity hardware, is interpretable through feature importance and SHAP, and infers fast. Fifteen years after the Yahoo challenge it remains the strong baseline.

The case for neural ranking: it learns from raw text without hand-engineered lexical features, captures semantic matching that no BM25 feature expresses, and improves with data where trees plateau. A cross-encoder over the top 50 genuinely outperforms any feature-based model.

The case for keeping the hand-tuned formula: it is debuggable, it needs no training data pipeline, no labels, no retraining cadence, and for a small corpus with clear signals it can be close enough. The LTR machinery is a real ongoing cost.

My position: LambdaMART over a good feature set for the light ranker, a cross-encoder for the heavy ranker, and neither until there is a labelling pipeline.

The sequencing matters more than the model choice. A team that trains a ranker before it can measure whether the ranker helped has built something it cannot improve. So step one is serving-time feature logging plus a judged evaluation set, and step two is the model. Teams consistently do those in the opposite order and then cannot tell whether their model is good.

On the model, GBDT for feature-based ranking because ranking features are exactly the heterogeneous tabular data trees are best at, and because it trains in minutes, which means you can iterate. Neural rankers earn their place where the input is raw text, which is the cross-encoder stage over a small candidate set, not the feature-based stage over hundreds.

Two implementation details I would treat as non-negotiable. Group by query, because omitting it makes the model compare documents across queries, which is meaningless, trains without error and silently produces a much worse model. And split by query, never by row, because rows from the same query appearing in both train and test leaks badly and inflates the offline metric, which then fails to replicate online.

And the discipline that separates a working LTR system from a demo: pool judgements across both rankers before comparing offline. If the judged set came from the old ranker's results, the new ranker's genuinely new documents are unjudged and scored as irrelevant, so it measures worse precisely because it found something new. That is the most common cause of an offline gain failing to replicate, it is silent, and TREC solved it thirty years ago.

Where I would push back: feature engineering beats model choice at this stage. Going from six hand-tuned signals to a well-designed forty-feature set with proper log-scaling and no raw identifiers is worth far more than swapping LambdaMART for a neural ranker, and it is where I would spend the first month.

Follow-up Q&A

"Pointwise, pairwise or listwise?" Pairwise in practice, via LambdaMART, and it is worth knowing why. Pointwise predicts an absolute score per document, which optimises the wrong thing: getting every score right is neither necessary nor sufficient for a good ordering. Pairwise optimises which of two documents is better, which is what ranking is. LambdaMART is usually called listwise and is mechanically pairwise, with each pair's gradient scaled by the NDCG change that swapping it would cause, so it optimises a list metric using pairwise machinery.

"What is LambdaRank's actual insight?" That you cannot differentiate NDCG, so instead of defining a loss and deriving a gradient, you define the gradient directly and scale it by the metric change. Swapping positions 1 and 2 changes NDCG far more than swapping 49 and 50, and a plain pairwise loss treats those identically. Weighting each pair's gradient by its delta-NDCG makes the model spend capacity where the metric is sensitive.

"Why gradient-boosted trees rather than a neural network?" Because ranking features are heterogeneous tabular data: counts, ratios, booleans, categoricals and log-scaled scores on completely different scales. Trees are scale-invariant, handle missing values natively, capture interactions without explicit crosses, train in minutes, and are interpretable through feature importance. Neural nets need normalisation decisions per feature and imputation, and their advantage is learning representations from raw text, which is the cross-encoder stage over a small candidate set rather than the feature stage over hundreds.

"Walk me through your feature set." Five families. Query-only, like length, whether it is navigational, and whether it contains an exact identifier, which the model uses to modulate everything else. Document-only, precomputable: quality, log popularity, freshness, stock status. Query-document, which carry most of the signal and are the only ones computed at query time: per-field BM25, exact phrase match, dense cosine, term coverage and proximity. User-document for personalisation, which is expensive and optional. And context, like device and session position.

"What are the rules for building features?" Log-scale anything with a heavy tail, so popularity and price do not make the model spend its capacity on the top 0.1 percent of items. Never feed a raw identifier as a numeric feature, because seller_id = 88412 implies an ordering that does not exist and the model will memorise sellers rather than learn something generalisable; use aggregate statistics of that entity instead. And remember that query-document features are the only ones that cannot be precomputed, so they set the query-time cost.

"What is the most common implementation error?" Omitting the group parameter, which tells the ranker which rows belong to the same query. Ranking loss is defined within a query, so without grouping the model compares a document for one query against a document for another, which is meaningless. It trains without any error and produces a much worse model, which makes it hard to find. Close second: splitting train and test by row rather than by query, which leaks rows from the same query into both and inflates the offline metric.

"Where do the labels come from?" Both sources, deliberately. Human judgements for a smaller high-quality evaluation set, because they are graded and position-unbiased and they cost fifty cents to three dollars each, so twenty thousand judgements is a real budget. And debiased click data for the much larger training set, using inverse propensity weighting so a click at position five counts for more than one at position one, with clipping because low propensities otherwise produce enormous weights and the variance explodes. Judgements alone cannot cover the tail; clicks alone bake in the current ranker's biases.

"Your offline NDCG improved 14 percent. Do you ship?" No, interleave first. The most common cause of an offline gain failing online is evaluation-set bias: if the judged pool came from the old ranker's results, the new ranker's genuinely new documents are unjudged and default to irrelevant, so it measures worse precisely because it surfaced something new. Pooling judgements over both rankers before comparing is the TREC fix and it is thirty years old. Then interleaving decides, because it needs far less traffic than an A/B test and controls for position bias by construction.

"What would feature importance tell you?" Mostly whether the model is learning what you think. The finding worth looking for is a feature that should not be predictive being predictive: a raw seller identifier at six percent importance means the model is memorising sellers rather than learning generalisable signal, which both generalises worse and is a fairness problem. Replacing it with seller rating and return rate is strictly better. Feature importance is a debugging tool rather than an explanation.

Common misconceptions

"LambdaMART is listwise." It is mechanically pairwise with NDCG-weighted gradients, which is how it optimises a list metric with pairwise machinery.

"Neural rankers are an upgrade over GBDT." On tabular features they usually are not. Airbnb's published experience of an initial neural regression is a useful counter-example. They win on raw text.

"A better model fixes bad results." It reorders what retrieval returned. Recall lost at retrieval is unrecoverable.

"Clicks are relevance labels." Clicks are relevance times attractiveness times position bias, and the position term dominates. They need debiasing before they are labels.

"Feature importance explains the model." It is a debugging tool. Its most useful output is finding features that should not be predictive and are.

Interview delivery note

Give the three formulations and land on one with a reason: "Pointwise predicts an absolute score, which optimises the wrong thing, because getting every score right is neither necessary nor sufficient for a good ordering. Pairwise optimises which of two documents is better, which is what ranking actually is. In practice LambdaMART, which people call listwise and is mechanically pairwise with each pair's gradient scaled by the NDCG change from swapping it."

Explain LambdaRank's idea, because it is genuinely elegant and few candidates can: "The insight is that you can't differentiate NDCG, so instead of defining a loss and deriving a gradient you define the gradient directly and weight it by the metric change. Swapping positions one and two moves NDCG far more than swapping forty-nine and fifty, and a plain pairwise loss treats them the same."

Spend most of the answer on features, because that is where the work is: "The model is the easy part. Five feature families, and the query-document ones carry most of the signal and are the only ones you can't precompute. Two rules I'd hold: log-scale anything heavy-tailed, or the model spends its capacity on the top tenth of a percent of items; and never feed a raw identifier as a numeric feature, because it implies an ordering that doesn't exist and the model just memorises sellers."

Name the implementation trap: "The most common error is omitting the group parameter, which tells the ranker which rows are the same query. Without it the model compares documents across queries, which is meaningless, and it trains without any error and silently produces a much worse model."

Close on the sequencing, because it is the judgement: "and I wouldn't train anything until there's serving-time feature logging and a judged evaluation set. A team that trains a ranker before it can measure whether the ranker helped has built something it can't improve, and teams reliably do those in the opposite order."

The Staff & Team Lead Interview Reference