Data residency, and the dependency audit

What it is

Two constraints that decide whether a multi-region design is actually deployable.

Data residency is a legal or contractual requirement that specific data stays in, or is processed in, a specific jurisdiction. It turns region placement from a latency decision into a correctness one.

The technical shape it forces:

  a user has a HOME REGION, recorded at signup
  their personal data lives only there
  requests are routed there
  and the region becomes part of the identity: it is in the
  key, the token, and the URL, not looked up on every request

The dependency audit is the question nobody asks until the failover fails: what does your failover itself depend on, and is any of it in the region you just lost?

Failing over needs, at minimum:
  DNS or the global load balancer's control plane
  the identity provider (to log in and to authorise the
    action)
  the secrets manager (to start anything)
  the container registry (to pull images)
  CI/CD (if the runbook deploys)
  the certificate authority or cert store
  the config and feature-flag service
  observability (to verify the failover worked)

If any of those is single-region and that region is the one
that failed, the runbook stops at the step that needs it.

What this is confused with: residency and encryption. Encrypting data does not make it resident. Most residency regimes are about where data is stored and processed, and an encrypted copy in another jurisdiction is still a copy there, though key location and access control matter for some regimes and for contractual commitments.

Also confused: a dependency audit and an architecture diagram. The diagram shows the request path. The audit asks what the recovery path needs, which includes things the request path never touches: the registry, CI, the console you log into.

The problem it solves

Residency retrofitted is a rewrite; residency designed in is a routing rule.

A system with a global user table, a single primary in
us-east-1, and a European customer requiring EU residency.

Retrofit cost:
  - split the user table by region, which means every foreign
    key crossing that boundary is now a cross-region reference
  - every query that joins users to anything else
  - every background job that iterates all users
  - every report
  - the id scheme, if ids do not encode the region
  - and a backfill that must run while the system is live

Observed shape: two to four quarters, and it competes with
everything else.

Designed in from the start: a home_region column, a routing
rule, and region-scoped ids. Weeks.

And the dependency failure is the one that turns a rehearsed failover into a stuck one:

Region A fails.
Step 1 of the runbook: log into the cloud console.
The SSO identity provider runs in region A.

Nobody can log in to perform the failover that would restore
the identity provider.

This is a circular dependency and it is extremely common,
because the identity provider was deployed like any other
service, by a team that was not thinking about the recovery
path.

Mechanics

Residency: what the regimes actually require

GDPR (EU/EEA)
  Does NOT require storage in the EU. It restricts TRANSFER to
  third countries without an adequacy decision or appropriate
  safeguards (standard contractual clauses, and since the
  Schrems II judgment, a transfer impact assessment).
  So "GDPR requires EU data residency" is wrong as stated, and
  EU residency is nonetheless what many EU customers demand
  contractually, which produces the same engineering
  requirement by a different route.

CANADA
  PIPEDA is federal and does not mandate residency. Some
  PROVINCIAL public-sector legislation does: British
  Columbia's FIPPA and Nova Scotia's PIIDPA have historically
  restricted storage and access of personal information held
  by public bodies to within Canada, with amendments over
  time. For a Toronto company selling to provincial public
  sector, this is the requirement that appears in the
  procurement questionnaire.

OTHERS worth knowing by name
  China's PIPL and the Cybersecurity Law: localisation for
    certain operators and data categories, with a security
    assessment for cross-border transfer.
  India's DPDP Act: transfer permitted except to
    government-notified restricted countries.
  Russia: localisation of citizens' personal data.
  Sector-specific: financial regulators and health regimes
    frequently impose their own, independent of general
    privacy law.

THE PRACTICAL POINT: the requirement usually arrives as a
CONTRACT CLAUSE from a customer, not as a statute you read.
Design for the capability, and let sales tell you which
regions.

Implementing residency

1. HOME REGION AS IDENTITY, not as a lookup.

   Put the region in the identifier:
     user id:  eu1_01HQ8Z... (region prefix + UUIDv7)
     or a token claim: {"sub": "...", "hr": "eu1"}

   Why: every service that receives an id can route without a
   lookup, and a lookup is itself a cross-region dependency
   that will be in the request path forever.

2. ROUTE AT THE EDGE, once.

   The edge resolves the home region from the token or the id
   prefix and forwards to that region. Downstream services
   never make the decision, so they cannot disagree.

   Unauthenticated requests (signup, login) route by a
   different rule: signup uses the user's stated country;
   login has to find the account, which means a small GLOBAL
   directory of (identifier -> home region) and nothing else.

3. THE GLOBAL DIRECTORY IS THE ONLY GLOBAL PERSONAL DATA, and
   it must be minimal: an email hash or a username hash mapped
   to a region. Not the email, not the name. Replicated
   everywhere, tiny, and defensible because a hash of a
   login identifier is the minimum needed to route.

4. SPLIT THE DATA MODEL EXPLICITLY.

   PERSONAL (region-pinned): users, orders, addresses,
     messages, uploads, audit logs referencing a person
   SHARED (replicated everywhere): the product catalogue,
     pricing rules, feature flags, configuration, ML model
     artifacts
   DERIVED (needs a decision): aggregates and analytics.
     Anonymised or aggregated above a threshold can usually
     move; per-user rows cannot.

5. CROSS-REGION FEATURES need a designed answer, not an
   accident:
     global search   -> search within the user's region only,
                        or a global index of shared data only
     admin tooling   -> an operator in one region viewing
                        another region's personal data IS a
                        transfer. Route the operator instead of
                        the data, and log it.
     reporting       -> aggregate per region, combine the
                        aggregates centrally
     a user moving   -> a documented migration, not an UPDATE.
                        Export, import, verify, delete, with an
                        audit record.

The admin-tooling case is the one that gets missed, because an internal tool feels like it is outside the system, and an operator in Toronto opening an EU customer's record is a cross-border transfer regardless of how the tool is built.

The alternative when full pinning is too expensive:

TOKENISATION / PSEUDONYMISATION

  Keep the identifying fields in the resident region behind a
  token vault; ship only tokens to the global system.

    EU region:    vault { tok_9f3 -> "ada@example.com", name,
                          address }
    Global system: order { customer: tok_9f3, sku, amount }

  The global system can compute, report and aggregate without
  holding personal data, and re-identification requires a call
  into the resident region.

  Cost: every display path needs a resolve call, and the
  vault is a single point of failure for anything
  user-facing.
  Right when: the personal data is a small fraction of the
  model and the processing is mostly non-personal.

The dependency audit

Enumerate what the recovery path needs, then check each item's own regional posture.

THE CHECKLIST, and each line has produced a real stuck
failover somewhere:

  IDENTITY
    [ ] Can you authenticate to the cloud console if the
        primary region is down? (SSO provider location)
    [ ] Is there a break-glass account with MFA that does NOT
        depend on the SSO provider, stored where you can reach
        it?
    [ ] Do your services' machine identities (tokens, certs)
        issue from a multi-region authority?

  DNS AND ROUTING
    [ ] Is your DNS provider multi-region? (Nearly all managed
        ones are anycast; a self-hosted one may not be.)
    [ ] Is the global load balancer's control plane
        independent of the failed region?

  SECRETS
    [ ] Is the secrets manager replicated? Can services in
        region B start without region A?
    [ ] Are the DR credentials themselves in the secrets
        manager you might not be able to reach? (circular)

  ARTIFACTS
    [ ] Is the container registry replicated, or does region B
        pull from a registry in region A?
    [ ] Are images CACHED in region B, so a registry outage
        does not block a scale-up?

  BUILD AND DEPLOY
    [ ] Does the runbook require CI/CD? Is CI single-region?
    [ ] Can you deploy without it, from a laptop, with
        credentials you can reach?

  CONFIG AND FLAGS
    [ ] Does the flag service have a regional cache and a
        safe default if it is unreachable, or does an
        unreachable flag service mean an unstartable service?

  CERTIFICATES
    [ ] Where does TLS issuance happen? Can region B renew
        without region A?
    [ ] What is the expiry horizon? A cert expiring during a
        multi-day incident is a documented compound failure.

  OBSERVABILITY
    [ ] If your metrics and logs pipeline is in the failed
        region, you are failing over blind.

  STATE YOU FORGOT
    [ ] Message queues, and whether in-flight messages are
        lost or replicated
    [ ] Scheduled jobs and cron: do they run in both regions,
        neither, or twice?
    [ ] Object storage: cross-region replication configured,
        and is it one-way?
    [ ] Third-party SaaS in the critical path (payment
        gateway, email, SMS): what is THEIR regional posture,
        and do you have a documented answer if they are down?

Run it as a tabletop, not as a form. Walk the runbook step by step and ask "what does this step need, and where does that thing live", because the checklist finds known dependencies and the walkthrough finds the ones nobody listed.

And the specific pattern to look for:

CIRCULAR DEPENDENCY: the recovery of X requires Y, and Y
requires X.

  the SSO provider you need to log in to fix the SSO provider
  the secrets manager holding the credentials for the secrets
    manager's own failover
  the deploy pipeline that deploys the deploy pipeline
  the observability stack you need to verify the fix to the
    observability stack

The fix is always the same shape: a BREAK-GLASS PATH that
depends on nothing in the loop. A local credential, a
pre-pulled image, a static configuration file, a documented
manual procedure. Stored somewhere reachable without the thing
it is breaking glass on, tested quarterly, and with its use
alerting loudly.

A worked example: an audit that found three stuck steps

A B2B SaaS company, two regions, a rehearsed failover runbook, and a residency requirement arriving with a European enterprise deal.

The dependency tabletop, run as a two-hour walkthrough of the existing runbook:

STEP: "Log into the AWS console and promote the replica."
  Needs: Okta SSO -> deployed in us-east-1 only.
  STUCK. If us-east-1 is the failed region, nobody can log in.
  Nobody had noticed because every game day had failed over
  us-west-2, the standby, which nobody logs in through.

STEP: "Scale up the us-west-2 deployment."
  Needs: pulls images from ECR in us-east-1.
  Cross-region replication was configured for the production
  repository and NOT for the three internal base images.
  STUCK on a scale-up requiring a fresh pull.
  Cached images on running nodes meant this had never
  surfaced: it only fails when a new node joins, which is
  exactly what a scale-up does.

STEP: "Verify with the dashboards."
  Needs: the observability stack, single-region, us-east-1.
  NOT STUCK, but blind: the failover would proceed with no
  way to confirm it had worked.

STEP: "Rotate the compromised credential."
  Needs: the secrets manager, replicated, fine.
  But the break-glass admin credential for the secrets manager
  was stored IN the secrets manager.
  Circular, and it had been that way for two years.

Three stuck or blind steps in a runbook that had been rehearsed four times, because every rehearsal failed over in the direction that did not exercise the dependencies.

"Every game day failed over in the same direction" is the finding worth generalising: a rehearsal that always moves from A to B never tests the dependencies that live in A.

The fixes:

IDENTITY
  - a break-glass IAM user per account, hardware-MFA, with
    credentials in a physical safe and in a separate password
    manager tenant hosted outside both regions
  - its use pages the security team automatically
  - tested quarterly, which found in the first test that two
    of the four listed holders had left the company

IMAGES
  - ECR replication extended to all repositories including
    base images
  - plus a pull-through cache in each region, so a registry
    outage does not block a scale-up at all

OBSERVABILITY
  - the metrics and logs pipeline made multi-region: the
    Collector gateway runs in both, and the backend is a
    managed multi-region service
  - plus a deliberately minimal "is it up" dashboard hosted
    entirely outside both regions, on the status-page
    provider, so a total loss of observability still leaves
    one signal

SECRETS
  - the break-glass credential moved out of the secrets
    manager into the physical/offline path
  - a documented rule: no credential required to recover
    system X may be stored in system X

GAME DAYS
  - alternate direction. Every other exercise now fails over
    FROM the primary, which is what surfaces the dependencies
    that live there.

"No credential required to recover system X may be stored in system X" is a one-line rule that generalises the whole circular-dependency category, and it is checkable.

The residency work, driven by the enterprise deal:

REQUIREMENT (contractual, not statutory): customer personal
data stored and processed in the EU, with a documented list of
any sub-processors outside it.

The data model, split in a two-day workshop:

  PERSONAL, region-pinned:
    users, organisations, documents, comments, activity logs,
    uploaded files, search indexes over the above
  SHARED, replicated:
    plan definitions, feature flags, templates, the ML model
    artifacts (trained on aggregated, anonymised data), the
    public help content
  DERIVED, decided case by case:
    usage analytics -> aggregated per region above a
      threshold of 50 organisations, then combined centrally
    billing -> the INVOICE lives in the EU; the payment
      gateway is a named sub-processor with its own regional
      posture, documented in the contract

IMPLEMENTATION:
  - user and organisation ids gained a region prefix
    (eu1_, us1_) rather than a lookup, because a lookup is a
    permanent cross-region dependency in the request path
  - the edge resolves the region from the id prefix or the
    token claim, once, and forwards
  - a global directory holding ONLY sha256(lowercased email)
    -> region, replicated everywhere, ~40 bytes per user
  - admin tooling: operators are routed to the region rather
    than data being routed to operators, and every access to
    personal data is logged with the operator, the subject and
    the reason

TIMELINE: 11 weeks, against an estimate of 2 quarters for the
retrofit-everything approach that had been assumed.

The saving came almost entirely from the id prefix: because
the region was in the identifier, no service needed a lookup
and no foreign key had to change meaning.

Putting the region into the identifier rather than into a lookup table is the decision that made it eleven weeks instead of two quarters, and it is the one thing worth doing before you have a residency requirement, because it costs nothing at the start and is the expensive part later.

Two things that were harder than expected:

1. BACKGROUND JOBS. A nightly job iterated all users to send
   digest emails. Split by region, it now runs twice, and the
   two runs disagreed about a shared rate limit for the email
   provider, briefly exceeding it.
   Fix: per-region quotas allocated from a shared budget,
   which is the general shape for any shared external
   resource under a regional split.

2. SEARCH. The search index over documents is personal data,
   so it had to be regional. A "search everything you have
   access to" feature for organisations with users in both
   regions became a scatter-gather across regions with
   per-region authorisation, which is slower and more complex
   than the single index it replaced.
   Accepted, with the latency cost measured and communicated,
   because the alternative was replicating personal data.

Production evidence

GDPR's actual mechanism is a restriction on transfers to third countries (Chapter V), not a storage mandate, and the Schrems II judgment of the Court of Justice of the EU invalidated Privacy Shield and required transfer impact assessments alongside standard contractual clauses. The commercial reality that EU customers demand EU residency contractually is separate from the statute and produces the same engineering requirement.

British Columbia's Freedom of Information and Protection of Privacy Act and Nova Scotia's PIIDPA have historically restricted the storage and access of personal information held by public bodies to within Canada, with subsequent amendments; they are the provisions that appear in Canadian public-sector procurement.

Cloud providers' residency controls (AWS Control Tower data residency guardrails, Azure's data residency documentation, Google Cloud's Assured Workloads) exist because the requirement is common enough to be a product, and their existence is the clearest evidence that customers ask for it contractually.

Circular dependencies in recovery paths are documented in AWS's own guidance on static stability and in post-incident writeups across the industry; the general principle, that a recovery path should not depend on the control plane it is recovering, is the reason "static stability" is a named design property.

Break-glass access with hardware MFA, offline storage and automatic alerting on use is standard practice in cloud security baselines (CIS benchmarks, cloud providers' well-architected security guidance), specifically because SSO is a single point of failure for administrative access.

Pull-through caches for container registries are provided by AWS ECR, Google Artifact Registry and Harbor precisely so that a registry outage or a cross-region dependency does not block a scale-up, which is the failure mode that only appears when a new node joins.

The debate

Is residency a legal requirement or a sales requirement? Usually the latter, and it does not matter for the engineering. GDPR restricts transfers rather than mandating storage, and most residency work is driven by a contract clause a customer's procurement team wrote, so the right posture is to build the capability and let sales tell you which regions, rather than to argue about the statute.

Region in the identifier, or a lookup table? In the identifier. A lookup is a cross-region dependency in the request path forever, and it becomes a single point of failure for routing. The counter-argument is that ids become opaque and a user cannot move regions without a new id, which is true and is the correct trade: moving regions should be an explicit migration with an audit record, not an UPDATE.

Should you tokenise instead of pinning? When the personal data is a small fraction of the model and the processing is mostly non-personal, tokenisation gives you a global system with a small resident vault. The cost is a resolve call on every display path and a vault that is a single point of failure for anything user-facing, so it is a genuine alternative rather than a shortcut.

How thorough should the dependency audit be? Thorough enough to walk the runbook line by line and ask what each step needs. A checklist finds the dependencies you already know about; the walkthrough finds the ones nobody listed, and in the worked example the walkthrough found three stuck steps in a runbook that had been rehearsed four times.

Is break-glass access a security risk? It is a deliberate, monitored one, and the alternative is worse. A break-glass credential with hardware MFA, offline storage, quarterly testing and automatic paging on use is a controlled risk; having no path when SSO is down is an uncontrolled outage. The quarterly test matters: in the worked example the first test found that half the listed credential holders had left the company.

Do game days need to alternate direction? Yes, and this is under-appreciated. A rehearsal that always fails over from the standby to the primary, or that always goes the same way, never exercises the dependencies living in the region you would actually lose. Alternating is free and it is what surfaced all three stuck steps.

Follow-up Q&A

"Does GDPR require EU data residency?"

No. It restricts transfers to third countries without an adequacy decision or appropriate safeguards, and after Schrems II that means standard contractual clauses plus a transfer impact assessment. What produces the engineering requirement is usually a contract clause from an enterprise customer's procurement team, which is a commercial fact rather than a statutory one and creates the same work. The correct posture is to build the capability, keep a documented sub-processor list, and let sales tell you which regions.

"How do you implement residency without a rewrite?"

Put the home region in the identifier rather than in a lookup table, so every service can route from an id or a token claim without a cross-region call. Resolve the region once at the edge and forward. Keep a minimal global directory of hashed login identifier to region, which is the only globally replicated personal data and is defensible because it is the minimum needed to route. Then split the model explicitly into personal, shared and derived, and give the cross-region features a designed answer: route operators to the data rather than data to operators, aggregate per region and combine the aggregates, and treat a user changing region as a migration with an audit record rather than an UPDATE. In one case the id-prefix decision took an eleven-week project that had been estimated at two quarters.

"What is the dependency audit and how do you run it?"

Walk the failover runbook line by line and ask, for each step, what it needs and where that thing lives. Identity, DNS, secrets, container registry, CI/CD, certificates, config and flags, observability, message queues, cron, object storage replication, and any third-party SaaS in the path. Run it as a tabletop rather than a form, because a checklist finds the dependencies you already know about and a walkthrough finds the ones nobody listed. In one case that found three stuck or blind steps in a runbook rehearsed four times.

"Give an example of a circular dependency in a recovery path."

The SSO provider deployed only in the primary region: step one of the runbook is to log into the cloud console, and nobody can, because the identity provider is in the region that failed. Or the break-glass admin credential for the secrets manager stored inside that secrets manager. Or a deploy pipeline that is required to deploy the deploy pipeline. The fix always has the same shape: a break-glass path depending on nothing in the loop, stored somewhere reachable without the thing it breaks glass on, tested quarterly, and paging loudly when used. The generalisable rule is that no credential required to recover system X may be stored in system X.

"Why did a rehearsed runbook still have stuck steps?"

Because every game day failed over in the same direction. Rehearsing a failover from the standby to the primary, or always exercising one path, never touches the dependencies that live in the region you would actually lose. In one case that meant nobody discovered that the SSO provider, the base container images and the entire observability stack were single-region in the primary, because no rehearsal had ever required them to be unavailable. Alternating direction is free and it surfaced all three.

"What breaks when you split a system by region?"

Two things people miss. Background jobs now run once per region, so anything with a shared external quota needs per-region allocation from a shared budget, which one team discovered by briefly exceeding an email provider's rate limit. And any global index over personal data becomes regional, so cross-region search for an organisation with users in both regions turns into a scatter-gather with per-region authorisation, which is slower and more complex than the single index it replaced. That cost was accepted and measured, because the alternative was replicating personal data.

Common misconceptions

"GDPR requires data to stay in the EU." It restricts transfers without safeguards. The residency requirement usually comes from a customer contract, which produces the same work by a different route.

"Encrypting it makes it non-resident." A copy in another jurisdiction is still a copy there for most regimes, though key location and access control matter contractually and for some rules.

"Look up the user's region from a table." That is a cross-region dependency in the request path forever, and a single point of failure for routing. Put it in the identifier.

"Internal admin tools are outside the residency boundary." An operator viewing personal data across a border is a transfer. Route the operator, and log the access.

"Our failover is rehearsed." In one case rehearsed four times, with three stuck or blind steps, because every rehearsal went the same direction and never required the primary region's dependencies to be unavailable.

"Break-glass credentials are a security risk we should avoid." The controlled version, hardware MFA, offline, quarterly-tested, paging on use, is a managed risk. Having no path when SSO is down is an uncontrolled outage.

Interview delivery note

Say this verbatim: "Put the home region in the identifier rather than in a lookup table, because a lookup is a cross-region dependency in the request path forever. That one decision took an eleven-week residency project that had been estimated at two quarters, since no service needed a lookup and no foreign key had to change meaning." It is a specific, cheap, early decision with a measurable consequence.

The senior-versus-staff separator is auditing the recovery path rather than the request path. A senior engineer diagrams the system. A staff engineer walks the failover runbook line by line asking what each step needs and where that lives, finds that step one requires logging into a console through an SSO provider deployed only in the region that just failed, and generalises it into a rule that no credential required to recover system X may be stored in system X. The circular dependency is invisible on any architecture diagram because it is not in the request path.

The second signal is noticing that every game day went the same direction. Saying "the runbook had been rehearsed four times and still had three stuck steps, because we always failed over toward the standby and never required the primary's dependencies to be unavailable" shows you evaluate the test as well as the system, which is the same instinct as auditing a load-test rig before its results.

The Staff & Team Lead Interview Reference