Trunk-based development and the merge queue

What it is

Trunk-based development is a branching policy: everyone integrates into one shared branch at least daily, branches live hours rather than weeks, and incomplete work is hidden behind flags rather than isolated on a branch.

A merge queue is the mechanism that keeps that shared branch green once more than a handful of people are merging into it. It serialises integration: before a pull request lands, CI runs against main plus every change ahead of it in the queue, so the state that gets tested is the state that will exist.

Without a queue                     With a queue
-------------------------------------------------------------
PR A: green against main@t0         queue: [A, B]
PR B: green against main@t0         test A on main+A       -> green
both merge                          test B on main+A+B     -> RED
main is broken                      B is rejected, A lands
                                    main is never broken

What this is confused with: "require branches to be up to date before merging." That setting forces each PR to rebase onto the current main and re-run CI before merging, which is a merge queue with a concurrency of one and a human doing the queueing. It is correct and it does not scale: with a 20-minute CI and enough merge volume, the branch you just updated is stale again before CI finishes.

Also confused: trunk-based development and "no code review." Trunk-based says branches are short and integration is frequent. It says nothing about skipping review, and short branches make review easier because the diffs are small.

The problem it solves

Integration pain grows superlinearly with branch age, for two separate reasons.

Textual conflicts scale with the number of edits made to the same files while you were away.

Semantic conflicts do not appear as conflicts at all, and they are the reason a merge queue exists:

PR A: renames `getUser(id)` to `fetchUser(id)`, updates all 40 call
      sites that exist on main.
PR B: adds a new module with 3 calls to `getUser(id)`.

Both are green against main@t0. Git merges both cleanly, because
they touch different files.

main is now broken: three calls to a function that no longer exists.
Neither author did anything wrong, and neither CI run was capable of
detecting it, because neither ever saw the other's change.

Both PRs were tested against a codebase state that ceased to exist, which is the general statement of the problem. Any check run against a base that is not the base you will merge onto is a check about a hypothetical.

And the cost of a broken main is collective:

main is red for 25 minutes.
30 engineers are working. Roughly 12 pull or rebase in that window.
Each spends 5 to 20 minutes determining that the failure is not
theirs.

Direct cost: ~2 engineer-hours.
Real cost: engineers learn to ignore a red main, which is the state
in which the next real break ships.

Mechanics

Trunk-based development, concretely

- Branches live hours to at most a day. If it cannot be finished in
  a day, split it or hide it behind a flag.
- Merge to main at least daily, per engineer.
- Main is always releasable. Not "usually". The release decision is
  which commit, not whether main works.
- Incomplete features live behind flags, not on branches.
  (See deploy is not release.)
- Large refactors use expand-contract in the codebase, the same
  shape as a schema migration: add the new thing, migrate callers
  incrementally, delete the old thing.
- Release branches, if any, are cut FROM main and only receive
  cherry-picks. Work never happens on them.

The prerequisite that makes or breaks it is CI speed. Trunk-based development with a 60-minute CI is worse than the alternative, because the feedback loop is longer than the branch's intended lifetime. Investing in test-suite speed is not adjacent to this policy, it is the entry fee.

Stacked pull requests are the answer for changes genuinely too large for one PR: a chain of small PRs each based on the previous, reviewed and merged bottom-up. Tooling (Graphite, git town, Phabricator's arc historically) exists because git does not model the chain natively.

Merge queue mechanics

Strictly serial is the correct-but-slow baseline:

for pr in queue:
    build = main + pr
    run CI on build
    if green: fast-forward main; else: reject pr, notify author

Throughput = 1 merge per CI run.

The arithmetic that forces something better:

CI duration        20 minutes
Merges per day     40
Serial capacity    3 merges/hour = 24 per 8-hour day

40 > 24, so the queue grows without bound. By mid-afternoon the
wait to merge exceeds the remaining working day.

Speculative (optimistic) batching is how real queues get past this:

Batch of N: build main + PR1 + PR2 + ... + PRN, run CI once.
  green -> all N land from one CI run.
  red   -> at least one is bad; bisect the batch.

With N=5 and a per-PR failure rate of 5%:
  P(batch green) = 0.95^5 = 77%
  Expected CI runs per batch = 1 + 0.23 * (bisect cost ~ log2(5) ~ 3)
                             ~= 1.7
  Effective throughput = 5 PRs / 1.7 runs = ~2.9 PRs per CI run,
  against 1.0 for strictly serial.

40 merges/day / 2.9 = ~14 CI runs = ~4.6 hours of queue time,
which fits a working day with headroom.

The batch size is a tuning parameter against your failure rate, and the relationship is intuitive: a low failure rate justifies larger batches, a high one makes bisection dominate. A queue whose batch size is not tuned against the measured failure rate is usually too small.

Speculation trees are the further refinement, and the published example is Uber's:

Instead of one batch, build a TREE of speculative states:
       main
        |- main+A            (assume A lands)
        |   |- main+A+B
        |   |   |- main+A+B+C
        |- main+B            (assume A does NOT land)
            |- main+B+C

Run CI on multiple branches of the tree concurrently. Whichever
branch reality takes, the result is already computed.

Cost: CI capacity, which is cheap relative to engineer wait time.
Uber published this as SubmitQueue (EuroSys 2019), including a
probabilistic model that predicts which changes are likely to pass
so that speculation effort goes to the likely branches.

The generalisable idea: spend machine time to buy human wait time, and predict where to spend it.

What a merge queue requires

1. DETERMINISTIC CI. A flaky test in a merge queue does not just
   annoy its author, it rejects innocent PRs and blocks the queue
   for everyone. Quarantine is not optional here, it is a
   precondition. (See the testing-ratio discussion of flake budgets.)

2. FAST CI, or a tiered approach: a fast required suite in the
   queue, and slower suites post-merge with a fast revert path.

3. AUTOMATIC REVERT or a clear revert culture. Something will land
   broken eventually; the fix is "revert first, diagnose after",
   and that must be socially normal, not a rebuke.

4. A LINEAR HISTORY policy (rebase or squash), because the queue's
   correctness argument assumes a well-defined "main plus these
   changes in this order".

5. CAPACITY. Speculation costs CI runs. If CI capacity is the
   constraint, the queue's throughput is capped by it rather than
   by the algorithm.

The failure modes of a merge queue

- FLAKE AMPLIFICATION. One test with a 2% flake rate, in a queue
  running 30 batches a day, fails roughly 0.6 batches a day and each
  failure triggers a bisect that re-runs it several more times.
  Measured flake rates matter far more here than in per-PR CI.

- BATCH POISONING. A PR that fails only in combination with another
  can bounce repeatedly. Queues need a retry limit and an eviction
  rule.

- THE LONG POLE. A single 90-minute integration test in the required
  set sets the queue's cycle time regardless of everything else.
  Move it out of the queue and run it post-merge with a revert path.

- QUEUE STARVATION under load: emergency fixes stuck behind 20
  routine PRs. Needs a priority lane, used sparingly and audited,
  because an unaudited priority lane becomes the default lane.

A worked example: a platform that could not keep main green

A 45-engineer product organisation, one large repository, roughly 55 merges a day.

Before:

Policy: "require branches to be up to date before merging".
CI: 26 minutes (unit 6, integration 14, E2E 6).
Measured over 30 days:
  main red                     18% of working hours
  average red duration         34 minutes
  reverts                      2.1 per day
  merges requiring 2+ rebase
    cycles before landing      41%
  engineer-hours lost to
    "is main broken or is it
     me?" (survey estimate)    ~14 per week

Why "up to date before merging" failed: with 55 merges a day, roughly one merge every 8 minutes during working hours, a PR that starts a 26-minute CI run is stale before it finishes about 96 percent of the time.

P(no merge lands during my CI run)
  merges arrive at ~7.5/hour during the working day
  CI takes 26 min = 0.43 h
  expected arrivals = 3.25
  P(zero) = e^-3.25 = 3.9%

So 96% of PRs were stale on completion and had to rebase and re-run,
which is where the 41% multi-cycle figure came from. The policy was
generating the churn it was meant to prevent.

That single Poisson line was what carried the decision internally, because it reframed the problem from "people are not rebasing promptly" to "the policy cannot succeed at this merge rate."

The changes, in the order they were made:

1. CI SPEED FIRST, because the queue's throughput is bounded by it.
     - test sharding across 8 workers:        26 min -> 11 min
     - moved 3 slow E2E suites out of the
       required set to post-merge:            11 min -> 8 min
     - flaky-test quarantine (14 tests
       quarantined, owners assigned)
   Measured per-PR failure rate attributable to flake: 9% -> 1.2%

2. MERGE QUEUE, batch size 1 initially, to build trust.
     throughput 1 PR / 8 min = 7.5/hour = 60/day. Just enough.
     main red: 18% of hours -> 0.4% (only post-merge suite
     failures, which now trigger auto-revert).

3. SPECULATIVE BATCHING, size tuned from the measured failure rate.
     failure rate 1.2% -> P(batch of 8 green) = 0.988^8 = 90.8%
     expected runs per batch ~ 1 + 0.092*3 = 1.28
     throughput = 8 / 1.28 = 6.25 PRs per run
     -> queue wait at peak: 45 min -> 7 min

4. PRIORITY LANE for reverts and incident fixes, with an audit
   trail. Used 0.7 times per week on average.

The batch-size tuning was mechanical once the flake rate was known, which is the argument for doing step 1 first: with a 9 percent failure rate, a batch of 8 would have been green only 0.91^8 = 47% of the time and bisection would have dominated.

Then trunk-based development became viable, which it had not been before:

                            before        after
median branch age            4.2 days      0.6 days
PRs > 400 lines changed          31%           9%
median review latency         9 hours     2.5 hours
merges per day                    55            71
main red (% of hours)             18%          0.4%
reverts per day                  2.1          0.9

Branch age fell because merging became cheap, not because of a policy announcement. The first attempt at trunk-based development, six months earlier, had been a policy announcement with no change to CI or merging, and it failed within three weeks because engineers correctly observed that merging small changes frequently meant paying the rebase-and-rerun tax more often.

That sequence is the lesson: the branching policy is downstream of the merge mechanics, and the merge mechanics are downstream of CI speed and flake rate. Announcing the policy first inverts the dependency and reliably fails.

One thing that did not improve, stated honestly:

Post-merge suite failures (the 3 E2E suites moved out of the
required set) now caused ~0.9 auto-reverts per day, up from 0.4.

That is the explicit trade: the queue is faster because it tests
less, and the difference lands as reverts. It was accepted because
a revert of a single small PR is cheap and diagnosable, whereas a
26-minute required suite made every merge expensive.

It is a real cost and pretending otherwise is how these decisions
get relitigated later.

Production evidence

Google runs a single large repository with a submit queue and automated large-scale change tooling (Rosie, TAP), and its published descriptions of the Test Automation Platform include running affected tests against candidate states and reverting breakages automatically. Google's own writing on this is the origin of much of the "keep the trunk green" practice.

Uber published SubmitQueue (Ananthanarayanan et al., EuroSys 2019, "Keeping Master Green at Scale"), which describes speculative execution over a tree of possible post-merge states plus a probabilistic model that predicts which changes will land, so speculation effort concentrates on the likely branches. It is the clearest published treatment of the throughput problem.

GitHub's merge queue is a productised version of the same mechanism, including grouping multiple pull requests into a single CI run and rejecting the offending change on failure.

The Rust project's bors/homu has enforced "test against main plus the queued change before merging" since well before merge queues were a mainstream feature, and Rust's never-broken-master property is the visible result.

The DORA research programme consistently identifies trunk-based development, specifically short branch lifetimes and daily integration, as a practice correlated with higher software delivery performance, and pairs it with test automation and continuous integration as the enabling capabilities.

Facebook's and Shopify's public engineering writing both describe moving from "rebase-and-rerun" branch protection to a queue for the same throughput reason, which is the pattern the worked example above follows.

The debate

Trunk-based or GitFlow? Trunk-based for products that deploy continuously. GitFlow's release and develop branches solve a real problem, supporting multiple released versions in the field, which is genuinely the situation for shipped software, libraries with long support windows, and firmware. For a service you deploy several times a day, the branches are pure integration cost. The position: trunk-based by default, and if you need release branches, cut them from main and only cherry-pick.

Is a merge queue worth it below some team size? The threshold is merge rate against CI duration, not team size. If your CI duration times your merge rate means most PRs are stale on completion, you need a queue, and that Poisson line is the calculation. Below that, "require up to date" is sufficient and simpler.

Should the queue's required suite be the full suite? No. The queue's cycle time sets everyone's wait, so the required set should be the fastest suite that catches the semantic-conflict class, with slower suites post-merge behind auto-revert. The honest cost, more reverts, must be stated up front rather than discovered.

Is speculative batching risky? It changes what was tested: a green batch of eight means the eight together are green, not that each is green alone. In practice that is the state you are shipping anyway, and bisection recovers per-PR attribution on failure. The genuine risk is CI capacity cost, which is the trade being made deliberately.

Do flaky tests just need retries? In a merge queue, no. Retries in a queue mask flake while still consuming the queue's throughput and rejecting innocent PRs during the failure, so the flake rate must be measured and quarantined rather than absorbed. This is the environment where "a flaky test is worse than no test" is most literally true, because the cost lands on everyone in the queue.

Should incomplete work go behind flags rather than on a branch? Yes, and the objection, that flags accumulate as debt, is correct and separately solvable with a flag lifecycle policy. The comparison is not "flag debt versus nothing", it is "flag debt versus long-branch integration risk", and long-branch risk is discovered at the worst moment while flag debt is a scheduled cleanup.

Follow-up Q&A

"What problem does a merge queue solve that branch protection does not?"

The semantic conflict. Two pull requests can each be green against main and merge cleanly in git while breaking main together, because neither CI run ever saw the other's change: one renames a function and updates its call sites, the other adds new calls to the old name in a different file. A merge queue tests main plus every change ahead of it in the queue, so the state that gets validated is the state that will exist. "Require branches to be up to date" approximates this with a concurrency of one and stops working once your merge rate makes PRs stale before CI finishes.

"When do you actually need a merge queue?"

When your CI duration times your merge arrival rate means most PRs are stale when CI completes. With 55 merges a day, about 7.5 an hour during working hours, and a 26-minute CI run, the expected number of merges landing during your run is 3.25, so the probability of none is e to the minus 3.25, about 4 percent. Ninety-six percent of PRs had to rebase and re-run, which is where a 41 percent multi-cycle merge rate came from. The policy was generating the churn it was meant to prevent, and one Poisson line makes that unarguable.

"How does speculative batching work and how do you size the batch?"

Build main plus N queued changes and run CI once: if green, all N land from one run; if red, bisect to find the offender. Size it from the measured per-PR failure rate. At a 1.2 percent failure rate a batch of 8 is green 0.988 to the eighth, about 91 percent of the time, giving roughly 1.28 CI runs per batch and an effective throughput of 6.25 PRs per run. At a 9 percent failure rate the same batch is green only 47 percent of the time and bisection dominates. So reducing flake is a prerequisite for large batches, not an independent improvement.

"What does a merge queue require to work?"

Deterministic CI above all, because a flaky test in a queue rejects innocent PRs and blocks everyone, so quarantine is a precondition rather than hygiene. Fast CI, or a tiered approach with a fast required suite in the queue and slower suites post-merge. A normalised revert culture, ideally automated, since something will eventually land broken. A linear history policy, because the queue's correctness argument assumes a defined ordering. And CI capacity, because speculation spends machine time to buy human wait time.

"Why did a trunk-based development policy fail before the queue was introduced?"

Because the policy is downstream of the merge mechanics, which are downstream of CI speed and flake rate. Announcing that engineers should merge small changes daily, while merging still cost a rebase-and-rerun cycle 96 percent of the time, means asking people to pay that tax more often. They correctly declined. After CI went from 26 to 8 minutes, flake-attributable failures from 9 percent to 1.2 percent, and a queue removed the rebase cycle, median branch age fell from 4.2 days to 0.6 without any further policy announcement.

"What is the honest cost of moving slow suites out of the required set?"

More reverts. In one case auto-reverts went from 0.4 to 0.9 per day when three E2E suites moved to post-merge. That is the explicit trade: the queue is faster because it validates less, and the difference shows up as breakages caught after merge. It was accepted because reverting one small PR is cheap and diagnosable while a 26-minute required suite made every merge expensive, but stating the cost up front is what keeps the decision from being relitigated after the first bad week.

Common misconceptions

"Require branches to be up to date is equivalent to a merge queue." It is a queue with concurrency one and a human doing the queueing, and it stops working once PRs go stale before CI finishes.

"Git merged cleanly, so the merge is safe." Git detects textual conflicts. Semantic conflicts merge cleanly and break the build.

"Trunk-based development means less testing or no review." It means short branches and frequent integration. Small diffs make review faster, not optional.

"A merge queue slows everyone down." Serial queueing does. Speculative batching moves throughput from one PR per CI run to several, at the cost of CI capacity.

"Flaky tests are an annoyance." In a merge queue they reject innocent changes and consume shared throughput, so a 2 percent flake rate becomes an organisation-wide tax.

"Adopt the branching policy first, then fix CI." The dependency runs the other way, and doing it in that order reliably fails within weeks.

Interview delivery note

Say this verbatim: "Two pull requests can each be green against main and still break it together, because neither CI run saw the other's change. That is a semantic conflict, git cannot detect it, and a merge queue is the mechanism that fixes it by testing main plus everything ahead of you in the queue." One sentence that names the problem precisely and rules out the cheaper-sounding alternatives.

The senior-versus-staff separator is the arrival-rate arithmetic. A senior engineer says branch protection creates rebase churn. A staff engineer computes it: at 7.5 merges an hour and a 26-minute CI run, the expected number of merges during your run is 3.25, so 96 percent of PRs are stale on completion, and "engineers should rebase promptly" is not a fixable behaviour problem, it is an arithmetic one. Turning a cultural complaint into a Poisson calculation is what moves the decision.

The second signal is stating the dependency order and the honest cost. Saying "CI speed and flake rate first, then the queue, then the branching policy, because announcing trunk-based development while merging is still expensive asks people to pay the tax more often" shows you have watched this fail. And adding "moving slow suites out of the required set took reverts from 0.4 to 0.9 a day, which we accepted deliberately" shows you price your own proposals.

The Staff & Team Lead Interview Reference