A mobile crash at 40 percent rollout

"Your mobile app is at 40 percent staged rollout and crash-free sessions have dropped from 99.8 to 97.9 percent. Walk me through what you do."

What it is

A staged rollout releases a mobile binary to a growing percentage of users over days. Both stores support it: Google Play staged rollout can be halted and the percentage changed, and App Store Connect phased release runs over seven days at a fixed schedule with a pause control.

The property that makes mobile different from every other deployment, and the one this question is testing:

You cannot roll back a mobile release. You can only stop it, and ship a new one.

Google Play does not allow reducing a staged rollout percentage below what has already been served, and users who have the update keep it. App Store phased release can be paused but not reversed. The users who already crashed still have the crashing binary, and the fastest fix path is a new build through review plus however long users take to update, which is days.

Commonly confused with a server rollback, where reverting the deploy resolves the incident in minutes. The entire discipline of mobile release engineering follows from this one asymmetry.

Also commonly confused with the crash-free users metric. Crash-free sessions and crash-free users move differently: a crash affecting 2 percent of sessions may affect 8 percent of users if it recurs, or 0.5 percent if one user is crash-looping. Knowing which you are looking at is the first thing to establish.

The problem it solves

Staged rollout exists because the alternative is shipping a crash to your entire install base at once, with no mechanism to stop it. The staging converts a total outage into a bounded one, and the percentage is your blast radius.

But staging alone is not enough, and the failures it does not prevent are the ones this scenario is about:

The crash affects a device, OS version or locale that your test devices did not cover, so it is invisible until real users hit it.
The crash is in a code path gated by a server response, so it is a server change that triggers it rather than the binary.
The metric is aggregate, so a crash affecting 100 percent of one Android OEM's devices looks like a small aggregate dip.

A kill switch is what turns "stop the rollout" into "stop the crash", and whether you have one is what determines how bad this incident is.

Mechanics

The first ten minutes

1. HALT the rollout. Both stores support this and it is reversible.
   Play Console -> Release -> Halt rollout
   App Store Connect -> Phased Release -> Pause
   Do this FIRST, before diagnosing. It costs nothing and it stops
   the exposure growing.

2. Establish the real blast radius:
     crash-free SESSIONS  99.8% -> 97.9%   (-1.9pp)
     crash-free USERS     ?                 (get this too)
     affected users = install base x rollout % x (1 - crash-free users)

3. Segment the crash. This is the diagnosis, and it is almost always
   decisive:
     by OS version      (Android 13 vs 14, iOS 17 vs 18)
     by device model    (one OEM? one chipset? low-memory devices?)
     by locale          (RTL, a specific date or number format)
     by app version     (is it only the new build?)
     by whether the user upgraded or installed fresh

That last dimension catches a whole class of bug: a migration that fails on data written by the previous version, which fresh installs never see and which therefore passes every internal test.

The decision, at ten minutes

Is there a server-side kill switch for the feature?
  YES -> Flip it. Crash stops for everyone including users already
         on the new build. Then decide about the rollout separately,
         with the pressure off.
  NO  -> The rollout stays halted and the only fix is a new build.
         Go to expedited review.

Can the crash be triggered by a server response you control?
  YES -> Change the server response. Same effect as a kill switch
         even though you did not build one. (Return the old payload
         shape, disable the endpoint, feature-off in the config.)
  NO  -> New build.

This is the entire reason feature flags matter more on mobile than on the server. On the server, a bad deploy is fixed by rolling back. On mobile, a bad binary is fixed by a flag or it is not fixed for days. Which is why the rule is: every new feature ships behind a remote-config flag, defaulted off, and turned on server-side after the binary is in the field.

That inverts the usual sequencing and is worth stating explicitly:

Server:  deploy the code, then release the feature. Rollback available.
Mobile:  ship the binary DARK (flag off), let it propagate for days,
         then turn the feature on server-side at 1%, 5%, 25%, 100%.
         Now the feature rollout is reversible even though the binary
         is not.

If there is no kill switch

1. Rollout stays halted. Affected population is frozen at whatever
   already received it.

2. Fix, and ship an expedited build.
   Apple: request expedited review. Typically hours rather than the
          usual timeline, but it is a request and not a guarantee,
          and using it frivolously costs you credibility for when
          you genuinely need it.
   Google: review is usually faster; a halted rollout can be replaced
          by a new release at a low percentage.

3. Communicate. Affected users are seeing a broken app and the store
   reviews arrive within hours. In-app messaging (if the app still
   launches), a status page, and support scripts.

4. For the users already on the bad build: they get the fix only when
   they update. Auto-update covers most Android users within days;
   iOS is faster on average. A forced-upgrade prompt shortens the tail
   for the rest, which is another thing you need to have built in
   advance.

The metric definitions that matter

Crash-free sessions = 1 - (sessions with a crash / total sessions)
Crash-free users    = 1 - (users with >=1 crash / total users)

Industry-typical healthy values: 99.5% to 99.9% sessions.
A drop of 1.9pp in sessions is large: it means roughly 1 in 50
sessions is crashing.

ANRs (Android) are counted separately and are often the bigger
problem, because an app that hangs is worse for the user than one
that crashes and restarts, and it does not appear in the crash metric
at all.

Watch ANR rate alongside crash rate. Google Play's vitals treat excessive ANRs as a policy problem with search-ranking consequences, and a change that moves work onto the main thread produces ANRs rather than crashes.

A worked example

A commerce app, 8 million monthly active users. Version 7.4 rolls out on Play.

Day 1  5%  crash-free sessions 99.81%   normal
Day 2 10%  99.78%                       normal
Day 3 20%  99.4%                        watch
Day 4 40%  97.9%                        ALERT

Minute 0: halt. Play Console, halt rollout. Exposure frozen at 40 percent, roughly 3.2 million users.

Minute 3: get both metrics.

crash-free sessions  97.9%   (-1.9pp)
crash-free users     94.2%   (-5.4pp)

Users dropped more than sessions, which means the crash recurs for affected users rather than being a one-off. That points at a startup or a persistent-state path, not a rare interaction.

Minute 8: segment.

by OS version:      Android 14: 91.2% crash-free users
                    Android 13: 99.7%
                    Android 12: 99.8%
by device:          spread across OEMs, so not a vendor issue
by upgrade path:    upgraded from 7.3: 91.0%
                    fresh install:     99.8%
by locale:          uniform

Two segments are decisive. Android 14 only, and upgrades only. Fresh installs are fine. That is a data-migration bug that only manifests under Android 14's behaviour change.

Minute 12: the stack trace.

java.lang.SecurityException: One of RECEIVER_EXPORTED or
RECEIVER_NOT_EXPORTED should be specified when a receiver isn't
being registered exclusively for system broadcasts
  at app.sync.SyncScheduler.restorePendingSync(SyncScheduler.kt:88)

Android 14 (API 34) requires apps targeting it to declare export behaviour when registering a runtime broadcast receiver. Version 7.4 bumped targetSdk to 34. The receiver is only registered on the restore path, which runs when there is pending sync state from a previous version, which is why fresh installs never hit it and why every internal test passed: the test devices all had clean installs.

Minute 15: the decision.

Is there a kill switch for the sync-restore path?  YES.
   remote_config: sync.restore_pending.enabled = false
   Flip it. Crash stops immediately for all 3.2 million users,
   including those already on 7.4.

Cost of the flag being off: pending syncs from 7.3 are deferred
   until the fix ships. Some users see stale cart data for a day.
   That is a far smaller problem than a crash loop.

Minute 20: crash-free users recovering as devices fetch the new config. Full recovery within about 90 minutes as the config TTL expires across the fleet.

Then, with the pressure off:

Day 4  Fix: register the receiver with RECEIVER_NOT_EXPORTED.
       Ship 7.4.1 at 1%. Verify Android 14 upgrade path specifically,
       on a device with 7.3 state, which is now a permanent test case.
Day 5  7.4.1 to 10%, 25%.
Day 6  Resume to 50%, 100%. Re-enable the flag at 5%, then fully.

The postmortem findings, which are the part worth rehearsing because they are the staff-level content:

The test matrix had no upgrade-path devices on the newest OS. Every internal device was a clean install. Adding "upgrade from N-1 on the newest OS" as a required pre-release check is the specific fix.
The alert threshold was too slow. Crash-free sessions was alerting on an aggregate at 99.5 percent; the Android 14 segment was already at 91 percent on day 3 at 20 percent rollout. Alert per OS version, not on the aggregate, because staged rollout dilutes any segmented failure into an aggregate that looks acceptable.
The targetSdk bump was bundled with feature work. A targetSdk change alters platform behaviour across the whole app and deserves its own release with its own rollout, because when something breaks you want one variable.
The kill switch is what made this a 20-minute incident instead of a 3-day one, and it existed because of a policy rather than because someone anticipated this bug.

Production evidence

Google Play's staged rollout documentation states that the rollout percentage cannot be decreased and that halting stops further distribution without removing the app from users who have it. That constraint is the foundation of the whole discipline.

Apple's phased release runs over seven days with a fixed daily schedule and a pause control, and expedited review exists as an explicit escape hatch, which Apple's own guidance asks developers to reserve for critical issues.

Google Play's Android vitals define the crash-rate and ANR-rate thresholds that affect store visibility (the "bad behaviour" thresholds), which is why ANR rate is a release-blocking metric and not just a quality metric.

Firebase Remote Config and Crashlytics are the standard pairing for exactly this workflow: Crashlytics for segmented crash-free metrics and stack traces, Remote Config for the kill switch, with the documented pattern of shipping features dark and enabling them server-side.

Android 14's runtime-receiver export requirement (API 34) is a real documented behaviour change and a common source of upgrade-path crashes, which is why it makes a good example: the failure is invisible to fresh installs by construction.

The debate

The case for slow, heavily-gated rollouts: you cannot roll back, so the only protection is limiting exposure and having a server-side off switch. Every hour of staging is an hour of information you get before the blast radius grows.

The case against over-gating: every feature behind a flag means combinatorial state, flags that never get cleaned up, and code paths that are never exercised together. A mobile codebase with 200 live flags has a testing problem that is worse than the rollback problem it solved. And a rollout so slow that it takes three weeks means you are shipping three-week-old fixes.

My position: every new feature ships dark behind a remote-config flag, and the flag is deleted within two releases. The first half is non-negotiable on mobile because it is the only reversibility available. The second half is what stops it becoming its own problem, and it needs enforcement in CI rather than good intentions, because flag debt on mobile is worse than on the server: the old binary with the old flag semantics stays in the field for months.

Two specific commitments beyond that. Alert per segment, not on the aggregate, and specifically per OS version and per upgrade path, because staged rollout mathematically dilutes a segmented failure: a crash affecting 100 percent of one OS version at 20 percent rollout is a small aggregate number that looks like noise. That dilution is the thing that makes mobile incidents get caught late. And ship platform-level changes, particularly targetSdk bumps, in their own release, because they alter behaviour across the entire app and you want one variable when something breaks.

The thing I would push back on hardest is treating the rollout percentage as the safety mechanism. It bounds the damage; it does not stop it. The kill switch is what stops it, and the difference between having one and not is the difference between a 20-minute incident and a 3-day one.

Follow-up Q&A

"Crash-free sessions dropped at 40 percent rollout. What do you do?" Halt the rollout first, before diagnosing, because it costs nothing and stops the exposure growing. Then get crash-free users alongside sessions, because the ratio tells you whether the crash recurs. Then segment by OS version, device, locale, and crucially by upgrade versus fresh install, which is decisive far more often than people expect. Then the key question: is there a server-side kill switch for the affected path? If yes, flip it and the crash stops for everyone including users already on the bad build. If no, the only fix is a new build through review, which is days.

"Why can't you just roll back?" Google Play will not let you decrease a staged rollout percentage, and users who already have the update keep it. App Store phased release can be paused but not reversed. So halting freezes the affected population; it does not shrink it. The users who are crashing keep crashing until they install a new build. That single asymmetry is why mobile release engineering looks so different from server deployment.

"What makes this a 20-minute incident instead of a 3-day one?" A remote-config kill switch on the affected feature. That is the whole difference. On the server a bad deploy is fixed by rolling back; on mobile a bad binary is fixed by a flag or it is not fixed for days. Which is why the policy is that every new feature ships dark, the binary propagates for days with the feature off, and then it is enabled server-side at 1, 5, 25 and 100 percent. The feature rollout is reversible even though the binary is not.

"Why did the alert fire so late?" Because it was on the aggregate. The Android 14 segment was already at 91 percent crash-free users on day 3 at 20 percent rollout, and the aggregate looked like 99.4 percent, which reads as noise. Staged rollout mathematically dilutes a segmented failure: a crash affecting 100 percent of one OS version at 20 percent rollout is 20 percent of that OS version's share of the aggregate. So alerts have to be per OS version and per upgrade path, or the staging that was meant to protect you is what hides the problem.

"Fresh installs were fine and upgrades crashed. What class of bug is that?" A migration or restored-state bug: code that only runs when there is data written by a previous version. It is invisible to every internal test because test devices are clean installs, and it is invisible to fresh users in the field. The specific fix is a required pre-release check on a device that has been upgraded from N-1, on the newest OS, and making that a permanent test case rather than a one-off.

"How do you get the fix to affected users faster?" Expedited review on iOS, which is a request rather than a guarantee and should be reserved for genuine emergencies because using it frivolously costs credibility. Then auto-update covers most users within days. For the tail, a forced-upgrade prompt driven by a server-side minimum-version check, which is another thing you need to have built before you need it. And in-app messaging if the app still launches, plus support scripts, because store reviews start arriving within hours.

"What would you change structurally after this?" Four things. Upgrade-path testing on the newest OS as a required gate. Per-segment alerting instead of aggregate. Ship targetSdk bumps in their own release, because they change behaviour across the whole app and you want one variable. And verify the kill switch exists for every new feature as a release checklist item, since in this case the flag is what turned a three-day incident into twenty minutes and it existed by policy rather than by foresight.

Common misconceptions

"Halting the rollout fixes it." It freezes the affected population. Everyone who already has the build still has it.

"You can decrease the rollout percentage." Google Play does not allow it. Halt is the only control.

"Crash-free sessions and crash-free users are interchangeable." The ratio between them tells you whether the crash recurs, which is diagnostic information you get for free.

"Staging is the safety mechanism." Staging bounds the damage. A kill switch stops it. They are not substitutes.

"ANRs are a subset of crashes." They are counted separately, they are often worse for the user, and a main-thread regression produces ANRs with no change in crash rate at all.

Interview delivery note

Lead with the constraint that defines the whole problem, because it is what the question is testing: "The first thing I'd say is that I can't roll back. Google Play won't let me decrease a staged rollout percentage and users who have the build keep it. So halting freezes the affected population, it doesn't shrink it, and the users who are crashing keep crashing until a new build reaches them."

Then the sequence: "Halt first, before diagnosing, because it costs nothing. Then get crash-free users alongside sessions, because the ratio tells me whether it recurs. Then segment by OS version, device, locale, and upgrade versus fresh install. That last one is decisive more often than people expect: if fresh installs are clean and upgrades crash, it's a migration bug that no internal test could have caught, because test devices are clean installs."

Then the question that determines everything: "And then the decision: is there a server-side kill switch for that path? If yes, I flip it and the crash stops for everyone including users already on the bad build, and I've turned a three-day incident into twenty minutes. If no, the only fix is a new build through review."

The depth signal is the alerting observation: "and I'd expect the postmortem to find the alert fired late, because staged rollout mathematically dilutes a segmented failure. A crash hitting a hundred percent of one OS version at twenty percent rollout looks like noise in the aggregate. So alerts have to be per OS version and per upgrade path, or the staging that was meant to protect you is exactly what hides the problem."

The Staff & Team Lead Interview Reference