Speaking in numbers: p90, Monte Carlo, and the language of estimates

What it is

The register that separates a lead who is trusted with a date from one who is not. It has two halves: the vocabulary (p90, order of magnitude, within noise, directionally correct, material, across the board) and the arithmetic underneath it, which is what stops the vocabulary from being decoration.

The central technical idea is that an estimate is a distribution, not a number, and almost every estimation failure in engineering follows from collapsing it to a number too early. Once you hold the distribution, three things become sayable that otherwise cannot be: how confident you are, what would change the answer, and what the cost of the confidence is. Monte Carlo is the cheapest way to get a distribution when you cannot derive one, and it takes about fifteen lines.

Don't be confused: p90 is a percentile of a distribution, not a safety margin you add. "Give me your p90" and "give me your estimate plus buffer" sound like the same request and are not. A p90 is the value below which 90 percent of outcomes fall, which is a statement about a distribution you have in mind. A buffer is a number you invented. The practical difference shows up immediately: p90s do not add, as the code below demonstrates, while buffers do, which is why an organisation that runs on buffers systematically over-quotes and then absorbs the slack invisibly.

Don't be confused: "we are not statistically significant yet" is usually the wrong sentence. Statistical significance answers "is this effect distinguishable from zero", which is rarely the decision you face. The decision is usually "is this effect big enough to be worth the cost", which is a question about the confidence interval and the minimum detectable effect, not the p-value. A result can be highly significant and commercially irrelevant, or non-significant and clearly worth shipping given the cost of waiting. Leads who say "significant" when they mean "big" get caught by anyone numerate in the room.

The problem it solves

A director asks when the migration will be done. There are three bad answers and each fails differently. A single number ("six weeks") is a promise you did not mean to make and will be held to. A refusal ("it depends") is read as evasion and is the answer that costs leads the most credibility. A padded number ("twelve weeks", privately meaning six) works once, and then the padding is discovered and every future estimate is discounted, which is strictly worse than where you started.

The fourth answer is a distribution with a commitment attached: "p50 is about eight weeks, p90 is about ten, and I will commit to ten. The spread is driven almost entirely by the backfill, so if you want the range tighter, the thing to fund is a week of prototyping the backfill." It is honest, it is actionable, it gives them a lever, and it is defensible when it goes wrong, because a p90 missed one time in ten is a p90 working correctly.

Mechanics

"""Monte Carlo estimation, and why you cannot add p90s.

Runnable: `python3 lead_estimation.py`. Deterministic: the seed is fixed, so the
numbers in the chapter are reproducible.

The scenario is one a lead actually faces. Six tasks, each with a three-point
estimate from the engineer who will do it. A director asks for a date they can
put in a plan. The naive answers are "add the optimistic numbers" and "add the
pessimistic numbers", and both are wrong in ways worth being able to name.
"""

import random
import statistics

# (task, optimistic, most likely, pessimistic) in engineer-days.
# These are the numbers you actually get when you ask, and the spread is wide
# because that is what honest estimates look like on unfamiliar work.
TASKS = [
    ("schema migration",      3,  5, 15),
    ("backfill job",          2,  4, 20),
    ("dual-write path",       4,  7, 14),
    ("read cutover",          1,  2,  6),
    ("delete old path",       1,  2,  4),
    ("load test and tune",    2,  6, 18),
]

TRIALS = 100_000


def triangular_sample(rng, lo, mode, hi):
    """A triangular distribution is the honest default for a three-point
    estimate: it uses exactly the three numbers you were given and assumes
    nothing else. PERT/beta weights the mode more heavily and is the other
    common choice; the point of this file is not the distribution, it is that
    ANY distribution beats adding single numbers."""
    return rng.triangular(lo, hi, mode)


def percentile(sorted_values, p):
    """Nearest-rank percentile. Written out rather than imported so the
    definition is visible: the p-th percentile is the smallest value below which
    p percent of observations fall."""
    if not sorted_values:
        raise ValueError("no data")
    k = max(0, min(len(sorted_values) - 1,
                   int(round(p / 100.0 * len(sorted_values) + 0.5)) - 1))
    return sorted_values[k]


def simulate(tasks, trials, rng):
    """Sum the tasks per trial, because they run in sequence. Each trial is one
    possible world; the distribution over trials is the answer."""
    totals = []
    for _ in range(trials):
        totals.append(sum(triangular_sample(rng, lo, mode, hi)
                          for _, lo, mode, hi in tasks))
    totals.sort()
    return totals


def per_task_percentiles(tasks, trials, rng, p):
    """The p-th percentile of each task INDEPENDENTLY, which is the number an
    engineer gives you when you ask 'what is your worst case'."""
    out = {}
    for name, lo, mode, hi in tasks:
        samples = sorted(triangular_sample(rng, lo, mode, hi) for _ in range(trials))
        out[name] = percentile(samples, p)
    return out


def with_one_parallel_track(tasks, trials, rng):
    """The same work, but the backfill and the load test run in parallel with
    the critical path instead of in series. The point: parallelism changes the
    shape of the distribution, not just its mean, because the finish time
    becomes a MAX over tracks and a max of two random variables is later than
    either one's typical value."""
    serial = [t for t in tasks if t[0] not in ("backfill job", "load test and tune")]
    side = [t for t in tasks if t[0] in ("backfill job", "load test and tune")]
    totals = []
    for _ in range(trials):
        a = sum(triangular_sample(rng, lo, mode, hi) for _, lo, mode, hi in serial)
        b = sum(triangular_sample(rng, lo, mode, hi) for _, lo, mode, hi in side)
        totals.append(max(a, b))
    totals.sort()
    return totals


if __name__ == "__main__":
    rng = random.Random(20260808)

    print("Three-point estimates, in engineer-days")
    print(f"  {'task':22} {'opt':>5} {'likely':>7} {'pess':>6}")
    for name, lo, mode, hi in TASKS:
        print(f"  {name:22} {lo:5} {mode:7} {hi:6}")

    sum_opt = sum(t[1] for t in TASKS)
    sum_mode = sum(t[2] for t in TASKS)
    sum_pess = sum(t[3] for t in TASKS)
    print()
    print(f"  naive sum of optimistic  : {sum_opt:6.1f} days")
    print(f"  naive sum of most likely : {sum_mode:6.1f} days")
    print(f"  naive sum of pessimistic : {sum_pess:6.1f} days")

    totals = simulate(TASKS, TRIALS, rng)
    print()
    print(f"Monte Carlo, {TRIALS:,} trials, tasks in series")
    for p in (10, 50, 80, 90, 95, 99):
        print(f"  p{p:<3}: {percentile(totals, p):6.1f} days")
    print(f"  mean : {statistics.mean(totals):6.1f} days")

    # The headline comparison: adding per-task p90s versus the p90 of the total.
    p90s = per_task_percentiles(TASKS, TRIALS, rng, 90)
    sum_of_p90s = sum(p90s.values())
    true_p90 = percentile(totals, 90)
    print()
    print("Why you cannot add p90s")
    for name, v in p90s.items():
        print(f"  p90 of {name:22}: {v:5.1f}")
    print(f"  SUM of the per-task p90s      : {sum_of_p90s:5.1f} days")
    print(f"  p90 of the TOTAL (simulated)  : {true_p90:5.1f} days")
    print(f"  overstatement                 : {sum_of_p90s - true_p90:5.1f} days "
          f"({100 * (sum_of_p90s / true_p90 - 1):.0f}% too pessimistic)")
    print(f"  and the naive most-likely sum : {sum_mode:5.1f} days, which lands at "
          f"p{100 * sum(1 for t in totals if t <= sum_mode) / len(totals):.0f} "
          f"of the real distribution")

    par = with_one_parallel_track(TASKS, TRIALS, rng)
    print()
    print("Same work, backfill and load test moved off the critical path")
    for p in (50, 90):
        print(f"  p{p:<3}: {percentile(par, p):6.1f} days "
              f"(was {percentile(totals, p):.1f})")
    saved_p50 = percentile(totals, 50) - percentile(par, 50)
    saved_p90 = percentile(totals, 90) - percentile(par, 90)
    print(f"  p50 improves by {saved_p50:.1f} days, p90 by {saved_p90:.1f} days")

The scenario is a six-task migration where each engineer gave a three-point estimate, which is what you actually get when you ask properly. Four details in the code matter.

The triangular distribution uses exactly the three numbers you were given and assumes nothing else, which makes it the honest default. PERT (a beta variant) weights the mode more heavily and is the other common choice. The point of the file is not the distribution: any distribution beats a point estimate, and arguing about which one is a good way to avoid doing the useful thing.

The tasks are summed inside each trial, then the trials are sorted. That ordering is the whole method. Each trial is one possible world in which each task took some particular time; the spread across trials is the answer. Summing the percentiles instead, which is the intuitive thing to do, produces the wrong number for a reason developed below.

The percentile function is written out rather than imported so the definition is visible, and because percentile conventions differ (nearest-rank, linear interpolation, several others) by enough to matter at the tails on small samples. With 100,000 trials the convention is irrelevant; with 40 measurements it is not, which is worth knowing when you are reading a latency dashboard.

The parallel variant takes a max rather than a sum. That is not a detail, it is the reason parallelising work helps less than people expect: the finish time of two parallel tracks is the maximum of two random variables, and the maximum of two draws is later than the typical value of either. Halving the critical path does not halve the schedule.

Worked example

Three-point estimates, in engineer-days
  task                     opt  likely   pess
  schema migration           3       5     15
  backfill job               2       4     20
  dual-write path            4       7     14
  read cutover               1       2      6
  delete old path            1       2      4
  load test and tune         2       6     18

  naive sum of optimistic  :   13.0 days
  naive sum of most likely :   26.0 days
  naive sum of pessimistic :   77.0 days

Monte Carlo, 100,000 trials, tasks in series
  p10 :   30.6 days
  p50 :   38.3 days
  p80 :   44.0 days
  p90 :   47.2 days
  p95 :   49.7 days
  p99 :   54.3 days
  mean :   38.7 days

Why you cannot add p90s
  p90 of schema migration      :  11.6
  p90 of backfill job          :  14.6
  p90 of dual-write path       :  11.4
  p90 of read cutover          :   4.6
  p90 of delete old path       :   3.2
  p90 of load test and tune    :  13.6
  SUM of the per-task p90s      :  59.0 days
  p90 of the TOTAL (simulated)  :  47.2 days
  overstatement                 :  11.8 days (25% too pessimistic)
  and the naive most-likely sum :  26.0 days, which lands at p1 of the real distribution

Same work, backfill and load test moved off the critical path
  p50 :   22.2 days (was 38.3)
  p90 :   27.2 days (was 47.2)
  p50 improves by 16.1 days, p90 by 20.0 days

The single most important number here is p1. The sum of the most-likely estimates is 26 days, and 26 days sits at the first percentile of the actual distribution. That plan has roughly a one in a hundred chance of being met. Every person who gave an estimate was being honest, every individual number was reasonable, and the addition of reasonable numbers produced a date that will essentially never happen. This is the arithmetic behind why software projects are late, and it does not require anyone to be optimistic or incompetent. The cause is that task durations are right-skewed (a task can take five times as long, it cannot take negative time) so the mode is well below the mean, and summing modes accumulates the gap six times over.

The second number is the 25 percent overstatement. Adding the per-task p90s gives 59 days against a true p90 of 47.2. The reason is that the sum's p90 does not require every task to hit its own p90; it requires the total to be high, and in most such worlds some tasks run long while others run short. Independent variation cancels. This is the same mathematics as "you cannot average percentiles" in the percentiles page, arriving from the other direction, and it has a direct organisational consequence: an organisation that asks everyone for a worst case and adds them up produces estimates roughly a quarter too large, gets treated as sandbagging, and then gets its numbers cut arbitrarily, which destroys the information content of the estimate entirely.

The third result is about parallelism. Moving the backfill and load test off the critical path improves p50 by 16.1 days and p90 by 20.0. Note that p90 improved more than p50, which is the non-obvious part and is the real argument for parallelising: taking work off the critical path does not merely shorten the schedule, it narrows the distribution, because the total no longer accumulates every task's variance. When you are asked to justify adding a person to a project, "it reduces the tail" is a much stronger claim than "it goes faster", and this is the demonstration.

Production evidence

Monte Carlo schedule estimation is standard practice in construction, aerospace and finance and is comparatively rare in software, which is the gap worth exploiting. The technique dates to Ulam and von Neumann's work at Los Alamos in the 1940s, and the name comes from the casino, via Ulam's uncle. In project management it appears as a standard extension to PERT and is built into mainstream scheduling tools.

Reference class forecasting is the same idea with empirical rather than simulated distributions, and it is mandated rather than merely recommended in some jurisdictions: the UK Treasury's Green Book requires optimism-bias adjustments derived from historical outturns for public projects, following Bent Flyvbjerg's work on megaproject cost overruns. The engineering translation is that your last five migrations are a better prior than your team's estimate of this one, and if you have that data you should use it in preference to any simulation.

Percentile-based commitments are how service levels already work, which is the argument for importing them into schedules. Nobody promises that every request will be under 200 ms; they promise a p99, and everyone understands that the remaining 1 percent is not a broken promise. Making a schedule commitment at p90 and saying so is exactly the same contract, and framing it that way to a non-engineering stakeholder usually lands, because they already accept the logic in the reliability context.

On the vocabulary side, "order of magnitude" reasoning is the documented interview norm at several of the companies in this book's loop formats, and the latency numbers table exists for exactly this purpose: not to be recited, but to make a factor-of-ten estimate available in ten seconds.

The debate

Is Monte Carlo worth it for a two-week project? No. The overhead is not the code, it is the three-point estimates, which take real time to collect honestly and which people resent providing. My position: use it when the decision is expensive and the spread is wide, which in practice means multi-month projects, anything with a hard external date, and anything where you are being asked to commit publicly. For a two-week project, ask for the most likely number, commit to roughly double it, and spend your effort elsewhere. Stating that threshold explicitly is better than either evangelising the technique or dismissing it.

Should you show the distribution to stakeholders, or just the commitment? Show the p50 and p90 and commit to one of them; do not show the full curve unless asked. The reason is not condescension, it is that a distribution invites negotiation over which percentile to plan against, and that negotiation is one you will lose, because the person with the budget will always prefer p50 and will remember it as the date. Give two numbers and a recommendation. The exception is when the spread is the message: if p50 is 8 weeks and p90 is 20, the shape is the finding, and the right conversation is about reducing uncertainty rather than about picking a date.

Are three-point estimates better than just asking for a number? Yes, and for a reason that has nothing to do with the arithmetic. Asking "optimistic, likely, pessimistic" makes people articulate what the pessimistic case actually is, which surfaces the risk. Half the value of the exercise is that someone says "20 days if the source data turns out to be inconsistent", and now you know the real question is the source data, which you can investigate this week instead of discovering in month two. The estimate is a by-product; the risk register is the product.

When is a point estimate the right answer? When you are being asked for a decision input rather than a commitment, and precision would be false. "Is this a week or a quarter" is a completely legitimate question, and answering with percentiles is unhelpful theatre. Match the precision of the answer to the precision of the decision.

Follow-up Q&A

Your tasks were sampled independently. Real project tasks are correlated. Does that break the model? It biases it, and in the dangerous direction, so this is the right challenge to raise about your own model before someone else does. If the underlying cause of the schema migration running long is the same as the cause of the backfill running long (the source data is worse than believed), those durations are positively correlated, and positive correlation increases the variance of the sum, which pushes the true p90 higher than the independent simulation says. Independent Monte Carlo is therefore optimistic about the tail. The practical fixes: model the shared driver explicitly as one variable that feeds several tasks, or apply a global multiplier drawn per trial, or simply say out loud that the p90 is a floor. Naming the assumption and its direction is worth more than fixing it.

Where does the mean sit relative to the p50, and why do you care? In this run the mean is 38.7 and the p50 is 38.3, so the mean sits slightly above the median, which is the signature of a right-skewed distribution: the long tail pulls the average up. That matters because the mean is the right number for aggregation and the median is the right number for a single commitment. If you are planning twenty such projects, use the mean, because the tails average out across a portfolio. If you are committing to one, use a percentile. Confusing them is how portfolio-level planning produces per-project promises nobody can keep.

How many trials do you need? Enough that the percentile you care about is stable, which you determine by running it twice with different seeds and seeing whether the number moves. For a p50, a few thousand trials is plenty. For a p99, you need considerably more, because the p99 is estimated from the top 1 percent of samples, so 1,000 trials gives you 10 observations to estimate it from and the answer will jump around. The code uses 100,000, which takes under a second and removes the question. The general principle worth stating: the further into the tail, the more data the estimate needs, which is also why p99.9 dashboards over five-minute windows are usually noise.

What does "within noise" actually mean and how do you avoid abusing it? It means the observed difference is smaller than the variation you would see between two measurements of the same condition. The abuse is invoking it without ever having measured that variation. The discipline: before comparing A to B, run an A/A comparison and see how different two identical things look. If your A/A test shows a 3 percent swing, then a 2 percent A/B result is within noise and you should say so with that number attached. "Within noise" with a measured noise floor is a strong statement; without one it is a way of dismissing data you dislike.

A stakeholder says "just give me a date". What do you say? Give one, then qualify in that order rather than the reverse. "Ten weeks. That is my p90, so I expect to make it nine times out of ten, and I would rather commit to that than to the eight-week median and miss it half the time. The one thing that could break it is the backfill, and I will know by the end of week two." Leading with the number respects the question; leading with the caveat sounds like evasion and gets you interrupted. The order of the sentences is doing as much work as their content.

How do you handle an estimate for work that has genuine unknown unknowns? You do not estimate it; you buy information. Time-box a spike with an explicit question and an explicit budget: "give me one week and I will tell you whether this is a two-week or a two-month problem." That is a commitment you can keep, and it converts an unanswerable question into a scheduled decision. Estimating through an unknown unknown is the single most common way leads lose credibility, because the number is not merely wrong, it is unfounded, and everyone finds out at the same time.

Common misconceptions

"The p90 means there is a 90 percent chance we finish on that day." It means a 90 percent chance of finishing on or before it. The distinction sounds pedantic and is not: percentiles are cumulative, and treating them as point probabilities leads people to think a p50 and a p90 are two competing predictions rather than two points on one curve.

"Adding the worst cases gives the worst case." It gives a number far beyond any realistic worst case, as the 59 versus 47.2 comparison shows. The genuine worst case is the tail of the total, and it is much closer to the p99 of the sum than to the sum of the p99s.

"Monte Carlo gives you accuracy." It gives you the consequences of your assumptions computed correctly. If the three-point estimates are wrong, the simulation is confidently wrong, and the false precision of "47.2 days" is actively dangerous. Round the output. Quoting a simulated schedule to one decimal place is a tell.

"p50 is the average." It is the median. They coincide only for symmetric distributions, and task durations are never symmetric. The gap between them is precisely the effect that makes projects late.

"Percentiles need a normal distribution." They need no distributional assumption at all, which is the main reason to prefer them over mean-and-standard-deviation for anything right-skewed, including both latency and task duration.

Interview delivery note

Two sentences are worth having word-perfect. For a date question: "My p50 is eight weeks and my p90 is ten. I will commit to ten, and the entire spread comes from the backfill, so if you want a tighter number the thing to fund is two days of prototyping it." For a metrics question: "That difference is within noise; our A/A variation is about three percent and this is two."

The senior-to-lead separator is whether the uncertainty is presented as information or as an excuse. A senior engineer says "it's hard to estimate, there are a lot of unknowns", which is true and is heard as hedging. A lead says "the uncertainty is concentrated in one place, here is what it would cost to remove it, and here is the number I will commit to meanwhile". Same underlying uncertainty; one version hands the decision-maker a lever and the other hands them a problem. Interviewers who have managed engineers have heard the first version many times and are listening specifically for the second.

The Staff & Team Lead Interview Reference