Expand and contract: changing a schema across three deploys

"Rename a column in a table that three service versions read from, with no downtime. And: a rollback fails because the new version wrote a cache format the old one can't parse. What went wrong?"

What it is

Expand and contract (also called parallel change) is the discipline that makes a breaking change to shared state non-breaking, by splitting it into deploys that each preserve compatibility with the version before and after.

Phase 1  EXPAND    Add the new thing. Old and new both work.
Phase 2  MIGRATE   Move readers and writers to the new thing. Both still work.
Phase 3  CONTRACT  Remove the old thing, once nothing uses it.

The rule that generates every step: at no point may a deployed version of the code be unable to run against the deployed state. Not just the new version against the new schema. The old version against the new schema too, because rollback is a deploy backwards and it must work.

Commonly confused with "run the migration in the deploy pipeline", which is the thing that breaks. A single deploy that renames a column and ships the code that uses the new name is atomic in your head and is not atomic in production: there is a window, however brief, where old pods are running against the new schema. During a rolling update that window is minutes; during a canary it is hours.

Also commonly confused with backward compatibility alone. You need both directions: new code reading old data (backward) and old code reading new data (forward). Rollback safety is the forward direction, and it is the one people forget.

The problem it solves

Three failures, all common, all avoidable.

The rolling-update window. A deploy replaces pods over 2 to 10 minutes. During that window both versions serve traffic. ALTER TABLE ... RENAME COLUMN applied at the start means every old pod throws column "email_address" does not exist until it is replaced. That is a partial outage with an error rate proportional to how far the rollout has progressed.

The rollback trap. The deploy succeeds, and 40 minutes later you find a bug and roll back. The old code now runs against the migrated schema and fails, or worse, writes data the new code cannot interpret. Your rollback, the thing you rely on, is the thing that is broken, and you discover it under incident conditions.

Cross-format state. Databases are only one kind of shared state. Caches, serialised sessions, published events, message payloads and files on object storage all have formats, and all of them can be written by one version and read by another.

Mechanics

Renaming a column across three deploys

Concretely: users.email becomes users.email_address, with three service versions in the field (an old one on a slow-rolling canary, the current one, and the new one).

Deploy 1: expand the schema only. No application change.

-- Additive, non-blocking. Old code never sees it.
ALTER TABLE users ADD COLUMN email_address TEXT;

-- Backfill in batches, never one statement over 40M rows.
-- Batching keeps lock duration and WAL volume bounded.
DO $$
DECLARE last_id BIGINT := 0;
BEGIN
  LOOP
    WITH batch AS (
      SELECT id FROM users
      WHERE id > last_id AND email_address IS NULL
      ORDER BY id LIMIT 5000
    )
    UPDATE users u SET email_address = u.email
    FROM batch b WHERE u.id = b.id;
    EXIT WHEN NOT FOUND;
    SELECT max(id) INTO last_id FROM (
      SELECT id FROM users WHERE id > last_id ORDER BY id LIMIT 5000) s;
    COMMIT;
    PERFORM pg_sleep(0.05);   -- let replicas catch up
  END LOOP;
END $$;

-- Keep the two in sync for writes that arrive during the transition.
CREATE FUNCTION sync_email() RETURNS trigger AS $$
BEGIN
  IF NEW.email IS DISTINCT FROM OLD.email THEN
    NEW.email_address := NEW.email;
  ELSIF NEW.email_address IS DISTINCT FROM OLD.email_address THEN
    NEW.email := NEW.email_address;
  END IF;
  RETURN NEW;
END $$ LANGUAGE plpgsql;

CREATE TRIGGER users_sync_email BEFORE INSERT OR UPDATE ON users
  FOR EACH ROW EXECUTE FUNCTION sync_email();

The trigger is the piece that makes the middle phase safe: whichever column a given version writes, both stay correct. Without it, writes from the old version are invisible to the new one.

Deploy 2: application writes both, reads the old.

class User(Base):
    email = Column(Text)              # still the source of truth
    email_address = Column(Text)      # kept in sync

    def set_email(self, value: str) -> None:
        self.email = value
        self.email_address = value    # belt and braces alongside the trigger

    def get_email(self) -> str:
        return self.email             # reads still come from the old column

Rollback from here is free: the previous version only knows about email, and email is still correct.

Deploy 3: application reads the new, still writes both.

    def get_email(self) -> str:
        return self.email_address     # the switch, and the only risky line

This is the deploy that can be wrong, because it is the first time the new column is load-bearing. If the backfill missed rows, you find out here. Which is why it is its own deploy: a rollback restores the old read path immediately, and there is no schema change to undo.

Deploy 4: stop writing the old column. Now the old column is dead but present. Wait long enough that no version writing to it can still be deployed. In practice: one release cycle, or however long your longest-lived canary or mobile client survives.

Deploy 5: contract.

DROP TRIGGER users_sync_email ON users;
DROP FUNCTION sync_email();
ALTER TABLE users DROP COLUMN email;

Five deploys to rename a column, and every one of them is individually reversible. The instinct is that this is excessive. The arithmetic that justifies it: a rolling update is a multi-minute window at 100 percent of traffic, and the cost of getting it wrong is a partial outage plus a rollback that does not work.

The shortcut worth knowing: if you can tolerate a brief lock, a VIEW gets you there faster. Rename the table, create a view with both column names. It works, it is much less code, and it fails on anything that writes through the view in a way the view cannot express. I would use the full sequence for a hot table and the view trick for a small, low-traffic one, and say which and why.

The generalisation: any shared state

The same three phases apply wherever two versions share a format.

Events. Add a field: consumers must ignore unknown fields (Protobuf and Avro do this by construction; JSON needs discipline). Remove a field: consumers deploy first, producers second. Rename: emit both, migrate consumers, stop emitting the old. The ordering rule is the reverse of intuition, and stating it correctly is a signal: for additions, producers first; for removals, consumers first.

API responses. Adding a field is safe if clients ignore unknown fields. Removing one is a breaking change no matter how unused it looks, and "no one calls it" is a claim to verify with access logs over a full seasonal cycle, not an assumption.

Files and object storage. A version field in the payload, and readers that handle every version still present in the bucket.

The cache format trap, which is drill 64

This one deserves its own treatment because it is the specific way rollbacks fail in practice.

v2 deploys. It serialises the session object with a new field layout
   and writes it to Redis under the same key: session:{id}

Bug found. Roll back to v1.

v1 reads session:{id}, gets bytes it cannot deserialise.
   Best case: exception, user logged out, error rate spike.
   Worst case: partial parse, wrong values, silent corruption.

The rollback did not fail because the code was wrong. It failed because the new version left state the old version cannot read. And the cache is full of it: a 24-hour TTL means 24 hours of poisoned entries, so the rollback does not recover on its own.

Three fixes, and I would use the first two together:

1. Version the cache key.

CACHE_SCHEMA_VERSION = 3   # bump on ANY change to the serialised shape

def session_key(session_id: str) -> str:
    return f"session:v{CACHE_SCHEMA_VERSION}:{session_id}"

v2 writes session:v3:*, v1 reads session:v2:*, and they cannot collide. Rollback is instant and clean: v1's entries are still there, still valid. The cost is a cold cache for the new version, which is a stampede consideration and connects directly to cache stampede.

2. Version the payload and tolerate both.

def deserialize(raw: bytes) -> Session:
    obj = json.loads(raw)
    v = obj.get("_v", 1)
    if v == 1:
        return Session(user_id=obj["uid"], roles=obj["roles"], tenant=None)
    if v == 2:
        return Session(user_id=obj["uid"], roles=obj["roles"],
                       tenant=obj["tenant"])
    # Unknown future version, written by a newer deploy: treat as a miss
    # rather than as an error, so the rollback degrades to a cache miss.
    raise CacheMiss()

The last branch is the important one. Unknown version means cache miss, not exception. That single decision converts a rollback failure into a performance dip.

3. Treat the cache as strictly disposable. If nothing in the cache is authoritative and a miss is always safe, flush it on rollback and take the origin load. This only works if you have modelled the origin load and know it survives a cold cache, which is exactly the calculation most teams have not done.

The rollback checklist

Before any deploy, ask what state the new version writes that the old cannot read:

[ ] Database schema     -> expand/contract, never a bare ALTER
[ ] Cache entries       -> version the key, or version the payload
[ ] Serialised sessions -> version the payload; unknown version = miss
[ ] Published events    -> additive only; removals go consumers-first
[ ] Object storage      -> version field in the payload
[ ] Feature flag state  -> old code must tolerate a flag it does not know
[ ] Queue message shape -> both versions must parse both shapes

Running this list takes five minutes and it is the single highest-value habit in this whole area. A deploy is not safe because it works; it is safe because the deploy before it still works after it.

A worked example

A payments service adds multi-currency support: amount_cents (integer) becomes amount (decimal) plus currency. Four consumers read the table, and the ledger is audited, so silent divergence is unacceptable.

Deploy 1  Schema expand
  ALTER TABLE payments ADD COLUMN amount NUMERIC(19,4);
  ALTER TABLE payments ADD COLUMN currency CHAR(3);
  Backfill in 5,000-row batches: amount = amount_cents / 100.0,
                                 currency = 'CAD'
  Trigger keeps all three columns consistent in both directions.
  Duration: 40M rows, ~90 minutes with replica-lag pauses.
  Verification: a checksum query comparing the columns, run to zero
                mismatches BEFORE deploy 2. This gate is not optional.

Deploy 2  Service writes all three, reads amount_cents.
  Rollback: free.

Deploy 3  Service reads amount + currency, writes all three.
  Rollback: free, one deploy back to reading amount_cents.
  This is the load-bearing deploy. Canary it at 5% for an hour and
  watch a reconciliation metric, not just the error rate.

Deploy 4  Consumers 1-4 migrated, one per week, each independently
  reversible. The database is unchanged throughout, which is what
  makes four independent migrations tractable.

Deploy 5  Stop writing amount_cents. Wait one full release cycle.

Deploy 6  DROP TRIGGER; ALTER TABLE payments DROP COLUMN amount_cents;

The cache dimension. The service caches payment summaries in Redis for 6 hours. The summary object gains a currency field.

Without care: v2 writes {amount: 12.34, currency: "CAD"}.
              Roll back to v1, which reads obj["amount_cents"] -> KeyError
              on every cached summary, for 6 hours.

With key versioning: v2 writes payment_summary:v4:{id},
                     v1 reads payment_summary:v3:{id}.
                     Rollback is instant. v2's entries expire on their own.

The event dimension. The service publishes PaymentCompleted. Adding currency is additive, so producers can go first, and consumers using Protobuf ignore the unknown field until they are updated. Removing amount_cents from the event is a removal, so every consumer deploys first, and only then does the producer stop emitting it. The ordering is the reverse of the addition case and getting it backwards takes down every consumer at once.

Total: six deploys plus four consumer migrations, over about six weeks, to change a column type. That is the honest cost, and stating it plainly is better than pretending it is quick. What you get for it: every step is individually reversible, there is no maintenance window, and at no point is a rollback unavailable.

Production evidence

Martin Fowler's ParallelChange (2011) is the canonical write-up of the expand/migrate/contract pattern and the name most teams use.

GitHub's gh-ost and Percona's pt-online-schema-change exist because MySQL ALTER TABLE historically locked the table; both build a shadow table, backfill it, apply ongoing changes via triggers or the binlog, and cut over atomically. They are mechanised expand-and-contract, and the fact that two independent tools converged on the same shape is strong evidence for the pattern.

Stripe's API versioning pins each account to the version it integrated against and translates responses at the edge, which is the same idea applied to the public API surface: never break a deployed client, translate instead.

Protobuf and Avro's compatibility rules are the pattern encoded in a type system. Avro's schema resolution defines reader-writer compatibility explicitly, and Confluent's Schema Registry enforces BACKWARD, FORWARD or FULL compatibility at registration time, so an incompatible schema is rejected before it can be published.

Postgres's own behaviour is why the batched backfill exists: a single UPDATE over tens of millions of rows holds a long transaction, generates enormous WAL, blocks autovacuum from cleaning up, and causes replica lag. The batching is not superstition.

The debate

The case for full expand-and-contract every time: it is the only approach where every intermediate state is safe and every step is reversible. Downtime is not acceptable, rollback must always work, and the discipline is what makes continuous deployment possible at all.

The case against, honestly: it is six deploys and several weeks for a column rename. On a small table in a low-traffic internal service, a 200-millisecond lock during a quiet hour is genuinely fine, and the ceremony costs more engineering time than the risk it removes. Teams that apply the full sequence uniformly spend a meaningful share of their capacity on migrations that did not need it.

My position: the sequence is the default for anything on a user-facing path, and I size it by the rolling-update window rather than by the table. The question I ask is: during the deploy, how long will both versions be live, and how much traffic hits the changed path in that window? If the answer is minutes at production volume, do the full sequence. If it is an internal tool with three users, take the lock and move on, and say out loud that you are taking a shortcut so it is a decision rather than an oversight.

The part I would not compromise on is the rollback checklist, because it costs five minutes and it catches the failure mode that hurts most. A schema migration is visible and gets reviewed; a cache format change is a serialisation detail in a pull request that nobody flags, and it breaks the rollback silently. The habit of asking "what state does this version write that the previous version cannot read" is worth more than any individual technique here.

Follow-up Q&A

"Rename a column across three service versions with no downtime." Five deploys. Add the new column and backfill in batches with a trigger keeping both in sync. Deploy code that writes both and reads the old. Deploy code that reads the new and still writes both, which is the load-bearing step and gets a canary. Deploy code that stops writing the old, then wait a full release cycle. Then drop the trigger and the column. Every step is individually reversible, and the reason it is five rather than one is that a rolling update means both versions serve traffic simultaneously for minutes, so a bare RENAME produces errors on every pod that has not been replaced yet.

"Your rollback failed because the new version wrote a cache format the old one can't parse. What went wrong and how do you prevent it?" The deploy left state that the previous version cannot read, so rolling back the code did not roll back the world. And because the cache has a TTL, it does not self-heal: every poisoned entry stays until it expires. Two fixes together. Version the cache key, so v2 writes session:v3:* and v1 reads session:v2:* and they cannot collide, at the cost of a cold cache for the new version. And version the payload with a rule that an unknown version is treated as a cache miss rather than an exception, which downgrades a rollback failure into a performance dip.

"What's the ordering rule for events?" For additions, producers deploy first and consumers ignore the unknown field, which Protobuf and Avro do by construction. For removals, consumers deploy first, then the producer stops emitting. It is the reverse of intuition and getting it backwards takes down every consumer at once. A schema registry with FULL compatibility enforcement makes the mistake impossible to publish rather than merely discouraged.

"How do you verify the backfill actually worked?" A checksum query comparing the old and new columns, run to zero mismatches before the deploy that starts reading the new column. That gate is the whole reason the read switch is its own deploy: if the backfill missed rows, you want to find out from a query rather than from customers. For a payments table I would also run a reconciliation metric during the canary rather than watching the error rate alone, because a wrong-but-parseable value does not raise an exception.

"Isn't six deploys for one column change excessive?" Often, yes, and I would say so. I size it by the rolling-update window: how long are both versions live, and how much traffic hits the changed path in that time. Minutes at production volume justifies the full sequence. An internal tool with three users does not, and a brief lock in a quiet hour is the right answer there. What I would not skip regardless is the rollback checklist, because that costs five minutes and catches the case that hurts most.

"What about a mobile client, where you cannot deploy the old version away?" That is the hardest version of the problem, because the old client persists for months regardless of what you do, and a fraction of users never update. So the contract phase is measured in quarters rather than weeks, gated on install-base telemetry rather than on a release cycle, and you need a server-side kill switch and a forced-upgrade path for the case where you genuinely must drop support. The practical consequence is that mobile API surfaces should be additive-only by policy, because a removal you cannot take back is a different category of decision.

Common misconceptions

"The deploy is atomic." A rolling update runs both versions for minutes. That is the entire reason this pattern exists.

"Rollback is always safe." Rollback is safe only if the new version left no state the old version cannot read. Schema, caches, sessions, events and files all count.

"Adding a nullable column is always safe." Adding the column is. The backfill is where the danger is: a single UPDATE over 40 million rows holds a long transaction, generates enormous WAL, blocks autovacuum and lags replicas.

"Nobody uses that field." Verify with access logs over a full seasonal cycle. The quarterly reporting job that reads it does not appear in a week of traffic.

"The cache is just a cache." It is state written by one version and read by another, which makes it exactly as dangerous as the database for rollback purposes, and much less likely to be reviewed.

Interview delivery note

Lead with why one deploy is not one deploy, because that reframes the whole question: "The reason this takes several deploys is that a rolling update isn't atomic. Both versions serve traffic for minutes, so a bare RENAME COLUMN means every pod that hasn't been replaced yet throws 'column does not exist'. And rollback is a deploy backwards, so the old version has to work against the new schema too."

Then the sequence, quickly: "Add the column and backfill in batches with a trigger keeping both in sync. Then write both, read old. Then read new, still write both, and that's the load-bearing deploy so it gets a canary. Then stop writing the old one, wait a release cycle, then drop it. Five deploys, each individually reversible."

For the cache-rollback version, name the root cause precisely: "the rollback didn't fail because the code was wrong, it failed because the new version left state the old one couldn't read, and with a TTL the cache doesn't self-heal. I'd version the cache key so the two versions can't collide, and version the payload so an unknown version is treated as a cache miss rather than an exception. That turns a broken rollback into a performance dip."

The habit worth volunteering, and the thing that separates staff from senior here: "before any deploy I'd run a short checklist of what state this version writes that the previous one can't read: schema, cache, sessions, events, object storage, queue messages. Schema migrations get reviewed because they're visible. A serialisation change in a cached object is a detail in a pull request that nobody flags, and it's the one that breaks the rollback."

And show judgement about when not to: "I'd size it by the rolling-update window, not by the table. Minutes at production volume justifies the full sequence. An internal tool with three users doesn't, and I'd say out loud that I'm taking the lock deliberately."

The Staff & Team Lead Interview Reference