Cell-based architecture and blast radius

What it is

Partitioning a service into independent, fully-stacked cells, each serving a fixed subset of customers, so that a failure is contained to one cell rather than affecting everyone.

        ┌──────────── CELL ROUTER ────────────┐
        │  customer -> cell, stable mapping    │
        └──┬──────────┬──────────┬──────────┬─┘
           │          │          │          │
      ┌────▼───┐ ┌────▼───┐ ┌────▼───┐ ┌────▼───┐
      │ CELL 1 │ │ CELL 2 │ │ CELL 3 │ │ CELL 4 │
      │ api    │ │ api    │ │ api    │ │ api    │
      │ workers│ │ workers│ │ workers│ │ workers│
      │ cache  │ │ cache  │ │ cache  │ │ cache  │
      │ db     │ │ db     │ │ db     │ │ db     │
      └────────┘ └────────┘ └────────┘ └────────┘
       25% of      25%        25%        25%
       customers

The defining property: a cell is a complete, independent instance of the service, including its data store. Not a shard of one database with shared application servers, and not an availability zone. If cell 2's database is corrupted, cells 1, 3 and 4 do not know and do not care.

Commonly confused with sharding. Sharding partitions data behind a shared application tier; cells partition the entire stack. A shared application tier means a bad deploy, a poisoned cache entry or a memory leak affects every shard, which is exactly the failure cells exist to contain.

Also commonly confused with multi-region. Cells are usually within a region, and the two compose: several cells per region, several regions.

The problem it solves

Most large outages are not infrastructure failures. They are the blast radius of a change or a poison input reaching everything at once.

The failure modes cells contain:

BAD DEPLOY           deployed cell by cell, so a broken release
                     affects 25% (or 5%, or 1%) of customers and
                     is caught before it reaches the rest.

POISON INPUT         one customer's malformed request triggers a
                     crash loop. Contained to their cell.

NOISY NEIGHBOUR      one customer's traffic spike exhausts a
                     connection pool. Their cell degrades;
                     others are untouched.

DATA CORRUPTION      a bug corrupts records. One cell's data
                     store, one restore.

RESOURCE EXHAUSTION  a leak fills memory or disk. One cell.

CONFIG ERROR         a bad configuration push. Rolled out cell
                     by cell like a deploy.

The one it does not contain is the router itself, and that is the central design tension: you have concentrated all the risk into the one component every request passes through.

Mechanics

Cell sizing, and the two competing pressures

SMALLER CELLS                       LARGER CELLS
+ Smaller blast radius              + Fewer cells to operate
+ Faster to rebuild or restore      + Better resource utilisation
+ Better isolation granularity      + Less per-cell overhead
- More cells to operate             - Larger blast radius
- Worse utilisation (each cell      - Slower to rebuild
  needs headroom)                   - Higher chance any given
- More cross-cell coordination        customer is in the broken one

The sizing rule: a cell should be large enough to be efficient and small enough that losing one is survivable, and "survivable" is a business decision rather than an engineering one.

WORKED
  10,000 customers, and the business says losing 5% of
  customers for an hour is tolerable and 25% is not.
  -> 20 cells of 500 customers each.

  Each cell must handle the largest customer plus headroom.
  If the largest customer is 3% of total traffic, a cell
  sized at 5% of total capacity cannot hold them.
  *** Cell size is bounded below by the largest tenant. ***

  So: 20 cells, and the top 5 customers get dedicated cells,
  which is a common and sensible asymmetry.

The largest-tenant constraint is the one people miss. You cannot have cells smaller than your largest customer, which means a service with one customer at 30 percent of traffic cannot have more than about three effective cells unless that customer is isolated.

The router: the thing you cannot cell

Every request must be mapped to a cell, and the mapper is
shared. This is the single point of failure the architecture
creates.

DESIGN PRINCIPLES FOR THE ROUTER

1. MAKE IT AS SIMPLE AS POSSIBLE.
   A lookup, not logic. Customer id -> cell id. No business
   rules, no computation, no dependencies beyond the mapping
   itself.

2. MAKE THE MAPPING STATIC AND CACHEABLE.
   Cell assignment changes rarely (onboarding, rebalancing),
   so the mapping can be cached aggressively at every layer
   and can survive the mapping service being down.

3. FAIL STATIC, NOT CLOSED.
   If the router cannot reach the mapping store, it serves
   from its last-known-good cache. A stale mapping is almost
   always correct; an unavailable router is a total outage.

4. DO NOT PUT IT IN THE DATA PATH IF YOU CAN AVOID IT.
   The strongest version: resolve the cell at DNS or at the
   client, so the request goes DIRECTLY to the cell and the
   router is not on the hot path at all.

Point 4 is the design that removes the SPOF rather than mitigating it. If the client resolves cell-07.api.example.com from a cached mapping and connects directly, the router is a control-plane component that can be down for an hour without affecting a single request.

Cell independence: the rules that make it real

Cells stop being independent the moment they share something, and sharing creeps in.

A CELL MUST HAVE ITS OWN
  application instances
  database (its own cluster, not a shared one)
  cache
  queues
  object storage prefix or bucket

A CELL MAY SHARE
  the container image registry (read-only, and a failure
    delays deploys rather than breaking serving)
  the identity provider (though this is a shared dependency
    worth scrutinising)
  observability collection (with per-cell labels)
  the deployment pipeline (with per-cell targeting)

A CELL MUST NOT SHARE
  a database, even a "shared read replica for reporting"
  a cache cluster
  a message queue
  any component whose failure affects request serving

The "shared read replica for reporting" is how this erodes, because it is genuinely convenient and it creates a component whose failure or overload affects every cell. The discipline is that a shared component must have its own availability argument, and "it is only for reporting" is not one, because reporting queries can and do take down databases.

Deploying cell by cell

This is where most of the operational value is realised.

DEPLOYMENT WAVES

  wave 0:  internal / canary cell (staff traffic only)
  wave 1:  1 cell   (5% of customers)     bake 1 hour
  wave 2:  3 cells  (20%)                 bake 1 hour
  wave 3:  8 cells  (60%)                 bake 30 min
  wave 4:  remaining cells

  Each wave gates on the cell's own health metrics, not the
  aggregate, because the aggregate dilutes a single cell's
  problem exactly the way a fleet percentile hides one bad
  instance.

And the property that makes this better than a percentage-based rollout: a cell rollout is naturally reversible and naturally scoped. Rolling back one cell affects only its customers, and the customers who were affected are a known, enumerable set you can contact.

Config changes go through the same waves. A configuration push that skips the wave structure is the most common way a cell-based architecture experiences a global outage anyway, because config is often not treated as a deploy.

Cell migration and rebalancing

Customers occasionally need to move: a cell is over capacity,
a customer has grown, or a cell is being decommissioned.

The mechanics are a data migration plus a mapping change,
and the mapping change is the easy part.

  1. Replicate the customer's data to the target cell.
  2. Enter a read-only or dual-write window.
  3. Verify.
  4. Flip the mapping.
  5. Drain the source cell of in-flight work.
  6. Delete the source data after a retention window.

The hard requirement this imposes: THE APPLICATION MUST BE
ABLE TO EXPORT AND IMPORT A SINGLE CUSTOMER'S COMPLETE STATE.
That capability is worth building for its own sake, because
it is also what enables customer-requested data export,
compliance deletion, and per-customer restore.

Naming that migration is a first-class capability rather than an exceptional operation is the design insight, because a cell architecture where migration is a two-week manual project will not rebalance and will drift into imbalance.

A worked example: cells versus availability zones

A team proposes "cells" that are actually availability zones:
three cells, one per AZ, each with app servers and a replica
of the same database cluster.

WHY THAT IS NOT CELLULAR
  The database is one cluster spanning AZs. A schema
  migration that corrupts data corrupts it in all three.
  A poison record is in all three. A bad query pattern
  saturates all three.
  -> AZ redundancy protects against INFRASTRUCTURE failure.
     Cells protect against SOFTWARE and DATA failure.
     They are orthogonal, and conflating them means you have
     one and believe you have both.

THE ACTUAL DESIGN
  8 cells, each spanning 3 AZs internally.
    -> AZ failure: each cell loses a third of its capacity
       and stays up. Infrastructure redundancy.
    -> Bad deploy, poison input, data corruption: contained
       to one cell, 12.5% of customers. Software isolation.

COST
  8 database clusters instead of 1. At small scale that is a
  real multiple; at large scale each cluster is smaller so
  the total is similar, plus per-cluster overhead.
  The honest framing: cells cost utilisation and operational
  surface, and buy blast radius. Below a certain size the
  trade is bad.

The orthogonality is the point to make: AZ redundancy and cellular isolation protect against different failure classes, and a team that has one and believes it has both is exposed to the entire class it has not addressed, which in practice is the more common one.

Production evidence

AWS's cell-based architecture guidance (in the Well-Architected Framework and their published "reducing the scope of impact with cell-based architecture" material) is the primary reference, and AWS uses the pattern internally across many services. Their guidance explicitly names the cell router as the component requiring the most care.

Amazon's Route 53 and S3 are documented as using cellular partitioning, and AWS's descriptions of their control-plane and data-plane separation reflect the "keep the router out of the data path" principle.

Slack's published work on cell-based architecture (2023) describes their migration to per-AZ cells specifically to contain a class of failure where a single AZ's network degradation affected the whole service, and their write-up is honest about the migration cost.

Salesforce's "pods" and Shopify's "pods" are the same pattern under a different name: fully independent stacks each serving a subset of merchants, with a routing layer mapping merchant to pod, and both are documented as the mechanism for both blast radius and scaling.

Facebook's 2021 BGP outage is the canonical counter-example: a configuration change propagated globally with no cellular staging, and the recovery was slowed because the tooling needed to fix it depended on the network that was down. It is the argument for cell-by-cell config rollout in a single incident.

The debate

The case for cells: most large outages are blast radius rather than infrastructure, and cells are the only structural answer. A bad deploy, a poison input or a data corruption bug affects a known fraction of customers instead of all of them, and the affected set is enumerable so you can communicate with them.

The case against: the cost is real and it is paid continuously. N database clusters instead of one, N sets of operational surface, worse resource utilisation because every cell needs headroom, and cross-cell operations (analytics, global search, anything aggregating across customers) become genuinely hard. For a service with a hundred customers it is over-engineering.

The case for just doing progressive rollout: most of the deploy-related benefit comes from gradual rollout with automated analysis, which is far cheaper. Cells add isolation of data and runtime, which matters less if your failures are predominantly deploy-related.

My position: cells above roughly a thousand customers or wherever a single-tenant failure would be a headline, and route at the client or DNS layer so the router is not in the data path.

The threshold matters because cells are expensive continuously and their benefit is occasional. Eight database clusters instead of one is eight sets of upgrades, backups, monitoring and capacity decisions, forever, against a blast-radius benefit that materialises during incidents. Below a certain scale that trade is bad and progressive rollout captures most of the deploy-related value for a fraction of the cost.

The design decision I would defend hardest is keeping the router out of the data path. The architecture's whole purpose is eliminating a single point of failure, and the naive implementation creates one that every request traverses. Resolving the cell at DNS or in the client, from a cached mapping that changes rarely, makes the mapping service a control-plane component that can be down for an hour without affecting a request. That is the difference between mitigating the SPOF and removing it.

The discipline that determines whether it stays real is what cells are allowed to share. A "shared read replica for reporting" is genuinely convenient, and it creates a component whose overload affects every cell, which reintroduces the thing you paid for. The rule I would hold: any shared component needs its own availability argument, and "it is only for reporting" is not one, because reporting queries take down databases regularly.

And the distinction I would make unprompted: AZ redundancy and cellular isolation are orthogonal. AZ redundancy protects against infrastructure failure; cells protect against software and data failure. A team with three AZs and one database cluster has infrastructure redundancy and no software isolation, and the failures that actually cause headline outages are overwhelmingly in the second class.

Where I would push back on an enthusiastic proposal: cell migration must be a routine operation, not a project. If moving one customer between cells is a two-week manual effort, the architecture will drift into imbalance and nobody will rebalance it. The capability that makes it routine, exporting and importing one customer's complete state, is worth building regardless, because it is also compliance deletion, per-customer restore and customer data export.

Follow-up Q&A

"What is a cell and how is it different from a shard?" A cell is a complete, independent instance of the service including its own data store, serving a fixed subset of customers. Sharding partitions data behind a shared application tier, so a bad deploy, a poisoned cache entry or a memory leak still affects every shard. Cells partition the entire stack, which is what contains software and data failures rather than only capacity.

"What does it actually protect against?" Blast radius, which is what most large outages actually are. A bad deploy affects one cell because you deploy cell by cell. A poison input that crash-loops a service is contained to the customer's cell. A data corruption bug means one restore. A noisy neighbour exhausts one cell's pool. Notably it does not protect against a failure in the router, which is the single point of failure the architecture creates.

"So how do you handle the router?" Four principles, and the fourth is the one that matters. Keep it simple: a lookup, not logic. Keep the mapping static and cacheable, because cell assignment changes rarely. Fail static rather than closed, so a router that cannot reach the mapping store serves its last-known-good cache. And ideally keep it out of the data path entirely by resolving the cell at DNS or in the client, so the mapping service becomes a control-plane component that can be down for an hour without affecting a request. That removes the SPOF rather than mitigating it.

"How do you size cells?" From the business answer to "how many customers can we lose for an hour". If 5 percent is tolerable and 25 percent is not, that is twenty cells at ten thousand customers. But there is a constraint people miss: cell size is bounded below by your largest tenant, because a cell has to hold them. If the largest customer is 3 percent of traffic, you cannot have cells sized at 5 percent unless you isolate them, which is why dedicated cells for the top few customers is a common and sensible asymmetry.

"Isn't three availability zones already three cells?" No, and conflating them is the common error. If the three AZs share one database cluster, then a schema migration that corrupts data corrupts it in all three, a poison record is in all three, and a bad query pattern saturates all three. AZ redundancy protects against infrastructure failure; cells protect against software and data failure. They are orthogonal and they compose: eight cells, each spanning three AZs internally.

"What can cells share?" As little as possible on the serving path. Their own application instances, database, cache, queues and storage, definitely. They may reasonably share a read-only image registry, since its failure delays deploys rather than breaking serving, and observability collection with per-cell labels. What erodes the architecture is the shared component that seems harmless: a "shared read replica for reporting" is convenient and creates something whose overload affects every cell. Any shared component needs its own availability argument.

"How do you deploy?" In waves, gated on each cell's own health rather than the aggregate, because the aggregate dilutes one cell's problem exactly the way a fleet percentile hides one bad instance. Internal cell, then one cell, then three, then eight, then the rest, with a bake between waves. And config changes go through the same waves, because config that skips the wave structure is the most common way a cellular architecture still has a global outage.

"What about operations that span cells?" They become genuinely hard, and that is a real cost to state. Global search, analytics across all customers, anything aggregating tenant data: each needs either a fan-out across cells or a separate aggregation pipeline reading from all of them. And that pipeline is a shared component, so it needs its own availability argument and it must not be on any serving path.

"When is it not worth it?" Below roughly a thousand customers, or when the cost of N database clusters is a large multiple rather than a similar total. Cells are paid for continuously in operational surface and utilisation, and the benefit materialises during incidents. Progressive rollout with automated canary analysis captures most of the deploy-related value for a fraction of the cost, and if your failures are predominantly deploy-related that may be the whole answer.

Common misconceptions

"Cells are shards." Sharding partitions data behind a shared application tier. Cells partition the whole stack, which is what contains software and data failures.

"Our AZs are our cells." AZs give infrastructure redundancy. If they share a database cluster they give no software isolation at all.

"The router is just a lookup so it's fine." It is the single point of failure the architecture creates. Keep it out of the data path if you possibly can.

"Cells only matter for deploys." They contain poison inputs, data corruption, noisy neighbours and resource exhaustion, none of which progressive rollout addresses.

"One shared reporting replica is harmless." Reporting queries take down databases regularly, and that replica is now a component whose failure crosses every cell.

Interview delivery note

Define it against sharding immediately, because that is the distinction being tested: "A cell is a complete independent instance of the service, including its own database, serving a fixed subset of customers. That's different from sharding, which partitions data behind a shared application tier, so a bad deploy or a poisoned cache still hits every shard. Cells partition the whole stack."

Name what it protects against, because "blast radius" alone is vague: "It contains the failures that actually cause headline outages: a bad deploy, because you roll cell by cell; a poison input that crash-loops a service; a data corruption bug, which becomes one restore; a noisy neighbour exhausting a pool. None of those are infrastructure failures, which is why AZ redundancy doesn't help with any of them."

Volunteer the router problem, because it is the obvious objection and having the answer is the signal: "And the architecture creates one single point of failure, which is the router. So: keep it a lookup rather than logic, make the mapping static and cacheable, fail static rather than closed, and ideally resolve the cell at DNS or in the client so the router isn't in the data path at all. That removes the SPOF rather than mitigating it."

Give the sizing constraint people miss: "Sizing comes from the business answer to how many customers you can lose for an hour. But there's a floor: cell size is bounded below by your largest tenant, because a cell has to hold them. If one customer is three percent of traffic you can't have five percent cells unless you isolate them, which is why dedicated cells for the top few is common."

Close on the orthogonality, which is the most useful correction to make: "and I'd separate AZ redundancy from cellular isolation explicitly, because teams conflate them. Three AZs sharing one database cluster is infrastructure redundancy with no software isolation, and the failures that make headlines are overwhelmingly the second class."

The Staff & Team Lead Interview Reference