Classical ML you still need: calibration, leakage, drift

What it is

Three concepts from pre-LLM machine learning that did not stop mattering, and that interviewers use to distinguish people who have shipped models from people who have called APIs.

Calibration is whether a model's confidence means what it says. A well-calibrated model that says "0.8" is right 80 percent of the time. Accuracy and calibration are independent: a model can be accurate and badly calibrated, or poorly performing and well calibrated.

Leakage is information in your training data that will not be available at prediction time. It produces excellent offline metrics and a model that fails in production, and it is the single most common reason a model that validated at 94 percent performs at 61 percent live.

Drift is the world changing after you trained. Two kinds, and they need different responses: data drift (the inputs change distribution) and concept drift (the relationship between inputs and labels changes).

What this is confused with: all three apply to LLM systems, not just to gradient-boosted trees. An LLM-as-judge is a classifier whose calibration you should check. A RAG evaluation set built from documents your retriever indexed is leaking. A prompt tuned in March degrades by September because user behaviour drifted. The vocabulary is older than the technology and the failures are the same.

The problem it solves

Why calibration matters concretely: any system that acts differently based on confidence needs the confidence to mean something.

Fraud model outputs 0.85 for a transaction.
Policy: auto-block above 0.9, review 0.6-0.9, allow below 0.6.

If the model is calibrated: 0.85 means 85% of such transactions are fraud.
                            The review queue is correctly sized.

If it is overconfident (says 0.85, actually 0.55):
    You are reviewing transactions that are mostly legitimate,
    and the 0.9+ auto-block bucket contains many false positives,
    each of which is a blocked customer.

Neural networks are systematically overconfident, and modern ones more so than older ones. Guo et al. showed that as networks got deeper and more accurate through the 2010s, their calibration got worse, which is counterintuitive and well established.

Why leakage matters: it is invisible in every offline metric, by construction. The metric is computed on data containing the leak, so the metric is high, and the higher it is the more confident the team becomes. A suspiciously good result is the primary symptom of leakage, which makes it the one failure mode where good news should trigger investigation.

Why drift matters: models are trained on a snapshot and deployed into a moving world. Without monitoring, degradation is gradual and nothing alerts, which is the same shape as several failures in the storage chapter: an invariant between two numbers, degrading, with nobody watching the relationship.

Mechanics

Measuring calibration

Expected Calibration Error (ECE): bin predictions by confidence and compare average confidence with actual accuracy in each bin.

def expected_calibration_error(confidences, correct, n_bins=10):
    edges = np.linspace(0, 1, n_bins + 1)
    ece = 0.0
    for lo, hi in zip(edges[:-1], edges[1:]):
        in_bin = (confidences > lo) & (confidences <= hi)
        if in_bin.sum() == 0:
            continue
        bin_conf = confidences[in_bin].mean()
        bin_acc  = correct[in_bin].mean()
        ece += (in_bin.mean()) * abs(bin_conf - bin_acc)
    return ece

A reliability diagram, for a typical overconfident network:

confidence bin   predicted   actual accuracy   gap
  0.5-0.6          0.55          0.51         -0.04
  0.6-0.7          0.65          0.58         -0.07
  0.7-0.8          0.75          0.64         -0.11
  0.8-0.9          0.85          0.71         -0.14
  0.9-1.0          0.96          0.79         -0.17    <- worst where it matters most

ECE = 0.121

The gap widens with confidence, which is the worst possible shape: the model is least trustworthy exactly where you rely on it most.

Fixing it: temperature scaling. One parameter, fitted on a held-out validation set, dividing the logits before softmax:

def fit_temperature(val_logits, val_labels):
    T = torch.nn.Parameter(torch.ones(1) * 1.0)
    opt = torch.optim.LBFGS([T], lr=0.01, max_iter=100)
    def closure():
        opt.zero_grad()
        loss = F.cross_entropy(val_logits / T, val_labels)   # NLL
        loss.backward()
        return loss
    opt.step(closure)
    return T.item()          # typically 1.2-2.5 for an overconfident model

Temperature scaling does not change the model's ranking at all, so accuracy, AUC and every ranking metric are unchanged. It only rescales the probabilities. That is why it is the default: it is nearly free and it cannot make your accuracy worse.

Platt scaling (fit a logistic regression on the scores) and isotonic regression (fit a monotonic step function) are the alternatives. Isotonic is more flexible and needs more data (it overfits below roughly 1,000 validation examples); Platt assumes a sigmoid shape. Temperature scaling first, isotonic if you have plenty of validation data and temperature is not enough.

Leakage: the taxonomy

Target leakage. A feature that is a consequence of the label rather than a predictor.

Predicting: will this customer churn next month?
Feature:    number_of_support_tickets_last_30_days

If the label window overlaps the feature window, tickets filed BECAUSE the
customer was leaving are predicting that they left. AUC 0.96 offline,
0.61 in production.

The test: would this feature's value be known, in this form, at the moment I need the prediction? If it is populated by the event you are predicting, it leaks.

Train-test contamination. The same information in both splits.

Random split on a table with one row per transaction, where a customer has
many transactions -> the same customer appears in train and test. The model
memorises customers rather than learning the pattern.

Fix: split by ENTITY (customer), not by row.

Temporal leakage. Training on data from after the test period.

Random split of time-series data means training on the future and testing on
the past. In production you only ever have the past.

Fix: time-ordered split. Train on Jan-Sep, validate Oct, test Nov-Dec.

Preprocessing leakage. Fitting a transformation on all the data before splitting.

# WRONG: the scaler has seen the test set's distribution.
X = scaler.fit_transform(X_all)
X_train, X_test = train_test_split(X)

# RIGHT: fit on train only, apply to test.
X_train, X_test = train_test_split(X_all)
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)          # transform, NOT fit_transform

Same for imputation, feature selection, target encoding, and any resampling. Use a Pipeline so this cannot happen, because the manual version is right until someone adds a step.

Leakage in LLM systems, which is where this is now most commonly encountered:

Benchmark contamination: the eval set was in the model's training data.
    Symptom: the model does implausibly well on a public benchmark and
    poorly on a paraphrase of it.

RAG eval contamination: the evaluation questions were written by reading the
    documents the retriever indexes, so they use the documents' exact
    phrasing. Retrieval looks excellent and fails on real user phrasing.

Fine-tuning leakage: the same document appears in the fine-tuning set and the
    eval set under a different ID.

Drift: detecting and responding

Data drift (P(X) changes):

from scipy.stats import ks_2samp

def population_stability_index(expected, actual, bins=10):
    """PSI: the standard drift metric in industry."""
    edges = np.percentile(expected, np.linspace(0, 100, bins + 1))
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return np.sum((a - e) * np.log(a / e))

PSI < 0.1     no significant shift
PSI 0.1-0.25  moderate shift, investigate
PSI > 0.25    significant shift, likely retrain

Concept drift (P(y|X) changes) is harder because it requires labels, which usually arrive late. The proxies:

Prediction drift: the distribution of the model's outputs shifts, which is a leading indicator available immediately.
Delayed label monitoring: track accuracy on whatever labels do arrive, accepting the lag.
A holdout of human-labelled samples: expensive, and the only ground truth.

The response depends on which drift you have:

Data drift, concept stable:   retrain on recent data. The relationship holds.
Concept drift:                retraining may not be enough; features may need
                              rethinking, because what predicts the label changed.
Both:                         usually a real product or world change; investigate
                              before automating a response.

Automatic retraining on a drift alert is a trap. If the drift is caused by an upstream data bug (a field that started arriving null, a unit change from dollars to cents), you have just trained on corrupted data and deployed it. Alert, investigate, then retrain, and gate the retrained model behind the same evaluation as the original.

A worked example: 94 percent offline, 61 percent live

A lending platform building a default-risk model. Gradient-boosted trees on 340,000 historical loans, 47 features.

Offline results:

AUC:              0.94
precision @ 0.5:  0.89
recall @ 0.5:     0.86

The team shipped. Six weeks later:

production AUC:   0.61      <- barely better than random on the hard cases

Three separate problems, and finding them in order is the exercise.

Problem 1: target leakage. Feature importance was dominated by days_since_last_contact:

feature importance:
  days_since_last_contact       0.34     <- top feature by a wide margin
  debt_to_income                0.11
  credit_score                  0.09
  ...

days_since_last_contact was populated by the collections system. A defaulting borrower gets contacted by collections, so the feature was recording the consequence of the label. At prediction time (loan origination) it was always null or stale, so the model's best feature was unavailable exactly when it was needed.

Remove the feature and retrain:
  offline AUC:  0.94 -> 0.79      <- the honest number
  production:   0.61 -> 0.77

The offline metric got worse by 15 points and the production metric got better by 16. That is the signature of leakage removal and it is a hard conversation, because the dashboard now shows a worse model.

Problem 2: temporal leakage. The split had been random across three years of loans, so the model trained on 2024 loans and validated on 2022 ones.

Random split:        offline AUC 0.79
Time-ordered split:  offline AUC 0.74      <- the honest number again
                     production  0.76      (now MATCHES offline)

The random split had been hiding a regime change: underwriting standards had tightened in 2023, so the model was learning post-2023 patterns and being validated on pre-2023 loans where those patterns partially held.

Once the split was time-ordered, offline and production agreed within 2 points, which is the property you actually want from an evaluation set. An offline number that does not predict production is worse than no number.

Problem 3: calibration. The scores fed a policy with three tiers.

reliability, before calibration:
  bin        predicted   actual    gap
  0.0-0.2      0.11       0.09    -0.02
  0.2-0.4      0.31       0.24    -0.07
  0.4-0.6      0.50       0.37    -0.13
  0.6-0.8      0.71       0.49    -0.22
  0.8-1.0      0.89       0.61    -0.28     <- badly overconfident

ECE = 0.147

The auto-decline threshold was 0.8, and the model's 0.8-plus bucket was actually 61 percent default rather than 89 percent. They were declining about four applicants in ten in that bucket who would have repaid.

# Temperature scaling on a held-out validation set. Ranking unchanged.
T = fit_temperature(val_logits, val_labels)     # fitted T = 1.83
calibrated = softmax(logits / T)

after calibration:
  ECE:                  0.147 -> 0.021
  AUC:                  0.74 -> 0.74           (unchanged, as expected)
  approval rate:        61% -> 68%
  realised default rate on approved: 3.1% -> 3.2%   (essentially flat)

Seven percentage points more approvals at the same realised default rate, because the thresholds now meant what they said. The calibration fix changed no rankings and made the policy correct.

Problem 4, found later: drift monitoring, which did not exist.

# Weekly PSI per feature against the training distribution.
for feature in FEATURES:
    psi = population_stability_index(train[feature], last_week[feature])
    if psi > 0.25:
        alert(f"{feature}: PSI {psi:.3f}")

The first run flagged something immediately:

employment_length_months   PSI 0.41

An upstream change three months earlier had switched the field from months to years without a schema change, so 36 (three years) had become 3. The model had been reading three-year employment as three months for a quarter, and nothing had detected it because the field was still a valid number in a plausible range.

Final:

                    initial     after all fixes
offline AUC          0.94          0.74
production AUC       0.61          0.76
offline/production
  agreement          -33 pts       +2 pts
ECE                  0.147         0.021
approval rate        61%           68%
realised defaults    3.1%          3.2%
drift monitoring     none          weekly PSI, per feature

The headline offline metric fell from 0.94 to 0.74 and the system got substantially better. That is the lesson worth carrying: an offline number that does not predict production is not a measurement, and improving it is not progress. The team's hardest work was explaining to stakeholders why the model's advertised accuracy had dropped 20 points while its business outcome improved.

Production evidence

Guo et al., "On Calibration of Modern Neural Networks" (ICML 2017) established that modern networks are systematically overconfident and that the problem got worse as architectures got deeper, and demonstrated temperature scaling as a simple, effective fix that preserves accuracy. It is the reference for the whole area.

Kaggle's competition history is a catalogue of leakage. Several competitions have been invalidated or re-scored after leaks were found, and the community's "leakage" tag documents patterns including ID ordering that correlates with the target, timestamps in file metadata, and duplicate rows across splits. It is the best available corpus of real examples.

Evidently AI, Arize, WhyLabs and Fiddler are commercial ML monitoring products whose core feature is drift detection, with PSI and KS tests as the standard metrics. That a product category exists for this is evidence that the failure is common and consequential.

Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (Google, NIPS 2015) is the canonical paper on ML systems failing for non-modelling reasons, and its "entanglement" and "correction cascades" sections describe exactly the class of failure where a data pipeline change silently degrades a model.

Benchmark contamination in LLMs is now widely documented: models scoring highly on public benchmarks and much lower on held-out paraphrases, which is train-test contamination at internet scale. This is why serious evaluation increasingly uses private held-out sets and recency-filtered data.

The debate

Does calibration matter if you only use rankings? For a pure ranking application (ordering search results, sorting a queue), no: temperature scaling does not change the order, so it changes nothing. Calibration matters the moment a threshold, an expected value, or a human decision depends on the number. Since almost every deployed classifier eventually acquires a threshold, and often several, I would calibrate by default.

Is temperature scaling enough? For most neural classifiers, yes, and its advantages are that it is one parameter, it cannot hurt accuracy, and it needs little validation data. Isotonic regression is more flexible and overfits below roughly 1,000 validation examples. Temperature scaling first, and reach for isotonic only if you have plenty of validation data and a measured residual miscalibration. Per-class calibration matters when classes are very imbalanced.

How do you find leakage before it ships? Three habits, in order of value. Suspicion of good results: an AUC above about 0.95 on a genuinely hard problem should trigger an investigation rather than a celebration. Feature importance review: if one feature dominates, ask what populates it and when. The temporal test: build the feature set as of the prediction timestamp and confirm every feature would have had that value then. The third is the only one that finds it reliably and it is the most work.

Should retraining be automatic? No, and this is a real disagreement in the field. The argument for automation is that models degrade and manual retraining lags. The argument against, which I find stronger: if drift is caused by an upstream data bug, automatic retraining trains on corrupted data and deploys it, converting a data quality problem into a model quality problem that is much harder to diagnose. The employment-length unit change in the worked example would have been baked in. Alert, investigate, then retrain through the same evaluation gate as the original.

Does any of this apply to LLM systems? All of it, and the mapping is direct. An LLM-as-judge is a classifier and you should check whether its confidence is calibrated (they are typically overconfident, and their verbalised confidence is worse than their token probabilities). A RAG evaluation set written by reading the indexed documents is leaking, and it is the most common evaluation mistake in RAG work. A prompt tuned six months ago is subject to drift in user behaviour, and the same monitoring applies. The teams that skip this in LLM work are usually the ones who never did it in classical ML, and the failures are recognisably the same.

Follow-up Q&A

"What is calibration and how do you fix it?"

Whether the model's stated probability matches its empirical accuracy: a calibrated model saying 0.8 is right 80 percent of the time. Measure with expected calibration error by binning predictions and comparing average confidence with accuracy per bin, and plot a reliability diagram, because the shape matters. Fix with temperature scaling: one parameter fitted on held-out validation data, dividing the logits before softmax. It does not change the ranking at all, so accuracy and AUC are unchanged, which is why it is essentially free.

"Your model is 94 percent offline and 61 percent in production. What do you check?"

Leakage first, because that gap is its signature. Look at feature importance: if one feature dominates, ask what populates it and when, because a feature filled in by the process you are predicting is target leakage. Then the split: random splits leak when there are repeated entities (the same customer in train and test) or a time dimension (training on the future). Then preprocessing: a scaler or imputer fitted before the split has seen the test distribution. Then the serving path, where a feature computed differently online than offline gives the model different inputs than it trained on.

"How do you detect leakage before shipping?"

Be suspicious of good results, which is the uncomfortable one: an AUC above 0.95 on a genuinely hard problem is more likely a leak than a breakthrough. Review the top features and ask what writes them and at what time relative to the prediction. And do the temporal test: reconstruct the feature vector as of the prediction timestamp and confirm every value would have been available then. That last one finds leakage reliably and is the most work, which is why it is skipped.

"Data drift or concept drift, and how do you respond differently?"

Data drift is P(X) changing: the inputs shift but the relationship holds, so retraining on recent data works. Concept drift is P(y|X) changing: what predicts the label has changed, so retraining may not be enough and the features may need rethinking. Detect data drift with PSI or a KS test per feature, which needs no labels. Concept drift needs labels and they usually arrive late, so use prediction-distribution drift as a leading indicator and a small human-labelled holdout as ground truth.

"Should retraining be automatic on a drift alert?"

No. If the drift is an upstream data bug, automatic retraining trains on the corrupted data and deploys it, which turns a data problem into a model problem that is far harder to diagnose. In one case a field silently changed units from months to years and the model had been reading it wrong for a quarter; automatic retraining would have baked that in. Alert, investigate the cause, then retrain through the same evaluation gate as the original.

"How does this apply to LLM systems?"

Directly. LLM-as-judge is a classifier whose calibration you should measure, and they are typically overconfident, with verbalised confidence worse calibrated than token probabilities. RAG evaluation sets written by reading the indexed documents leak the documents' phrasing into the questions, so retrieval looks excellent and fails on real user language. Benchmark contamination is train-test contamination at internet scale. And prompts drift as user behaviour changes, which needs the same monitoring as any model.

Common misconceptions

"An accurate model is well calibrated." Independent properties. Modern networks are systematically overconfident and got worse at calibration as they got more accurate. You can have 95 percent accuracy and an ECE of 0.15.

"Calibration improves accuracy." Temperature scaling does not change the ranking at all, so accuracy and AUC are identical. It changes what the numbers mean, which is what thresholds and expected-value calculations depend on.

"A random train-test split is the safe default." It leaks whenever entities repeat across rows or the data has a time dimension. Split by entity and by time.

"Leakage shows up as poor validation performance." It shows up as excellent validation performance. That is what makes it dangerous: the metric confirms the mistake.

"Drift means retrain." It means investigate. The cause is as often an upstream data bug as a genuine world change, and retraining on a bug ships the bug.

Interview delivery note

Say this verbatim: "An offline metric that does not predict production is not a measurement. In one case removing target leakage took offline AUC from 0.94 to 0.79 and production AUC from 0.61 to 0.77, and then a time-ordered split took offline to 0.74 and brought the two within two points. The headline number fell twenty points and the system got much better." A concrete demonstration that the metric and the goal can point in opposite directions.

The senior-versus-staff separator is treating a suspiciously good result as a bug report. A senior engineer knows what leakage is and can list the types. A staff engineer says an AUC of 0.94 on a genuinely hard problem should trigger an investigation, reviews what populates the top feature and when relative to the prediction, and is prepared to argue for a model whose advertised accuracy is twenty points lower. Being willing to make the dashboard worse is the judgment being tested.

The second signal is refusing automatic retraining on drift alerts. Saying "if the drift is an upstream data bug, automatic retraining bakes the bug into the model" shows you have seen a unit change or a null-field regression propagate, and it connects drift monitoring to data quality rather than treating it as a modelling concern.

The Staff & Team Lead Interview Reference