Backup hygiene: 3-2-1, immutability, restore testing

What it is

Backup hygiene is the set of properties that separate backups that will work from backups that exist.

3-2-1        3 copies of the data
             on 2 different media or storage types
             with 1 offsite

3-2-1-1-0    the modern extension:
             + 1 immutable or air-gapped copy
             + 0 errors after automated verification

The threat model that added the extra two: the credential that
can write your backups is usually the credential that can
delete them, and ransomware operators know this. An attacker
with your admin credentials deletes the backups first.

The single number that matters and is almost never known: how long a restore takes. An RTO claimed from backups is fiction until someone has restored the full dataset and timed it.

What this is confused with: replication and backup. A replica applies your mistakes instantly. A DROP TABLE replicates in milliseconds; a ransomware encryption replicates too. Replication is availability; backup is a point in the past you can return to, and you need both.

Also confused: a snapshot and a backup. A snapshot in the same account, same region, same credential boundary as the data protects against instance failure and nothing else. A backup crosses a trust boundary, or it is a copy that dies with the original.

The problem it solves

Three failures, and they are in increasing order of how common they are.

Hardware or region loss. The case backups were invented for, and the one modern infrastructure handles best.

Human error and application bugs.

A migration with a bad WHERE clause deletes 400,000 rows.
Replicas: also missing 400,000 rows, within 200ms.
Point-in-time recovery to 30 seconds before the migration:
  the only mechanism that helps.

This is far more common than a region loss, and it is why
retention granularity (PITR to the second) matters more than
retention duration for most systems.

Malicious deletion, which is the case that added immutability.

An attacker with cloud admin credentials:
  1. deletes the snapshots
  2. deletes the cross-region copies (same credential)
  3. encrypts or deletes the primary
  4. then negotiates

Every copy reachable by one credential is one copy.

The control is not "more copies", it is a copy that the
compromised credential CANNOT delete: object lock in
compliance mode, a separate account with a different trust
root, or offline media.

And the failure that turns any of the three into an outage rather than an inconvenience:

"We have backups."
"How long to restore?"
"We've never done a full restore."

Restore time for 4 TB at a sustained 200 MB/s is about 5.5
hours of transfer alone, before provisioning, index rebuild,
cache warm-up and verification. If your documented RTO is
four hours, it is wrong, and the incident is when you find
out.

Mechanics

3-2-1-1-0, made concrete

3 COPIES
  the primary, plus two backups. Not the primary plus a
  replica plus a backup, because a replica is not a copy in
  the sense that matters: it shares your mistakes.

2 MEDIA / STORAGE TYPES
  the original intent was tape and disk. The cloud version is
  two independent failure domains: a managed snapshot service
  AND object storage, or two providers. The point is that a
  bug or an outage in one storage system does not take both.

1 OFFSITE
  a different region at minimum, and a different ACCOUNT is
  more important than a different region, because the account
  is the blast radius for a credential compromise.

1 IMMUTABLE
  object lock in compliance mode (which even the root
  credential cannot shorten), or a vault-lock policy, or
  genuinely offline media. The test: name the credential that
  could delete it. If you can, it is not immutable.

0 ERRORS
  automated verification after every backup, and a scheduled
  full restore test with a business-level assertion.

Immutability, and what "immutable" has to mean

S3 Object Lock, COMPLIANCE mode
  no user, including the account root, can delete or shorten
  the retention period until it expires.
  GOVERNANCE mode is different: a user with the
  BypassGovernanceRetention permission CAN delete, which means
  it protects against accident and not against a compromised
  admin. Know which one you have.

SEPARATE ACCOUNT / SUBSCRIPTION
  backups written into an account whose credentials are not
  present in the production environment, ideally with the
  write being a push from production and the delete
  permission held only by a role production cannot assume.
  This is the strongest widely-practical control.

AIR-GAPPED
  genuinely offline. Rare outside regulated industries and
  still the only thing immune to a control-plane compromise.

THE TEST, and it is one sentence:
  "Which single credential, if compromised, could destroy
   every copy?"
  If the answer is not "none", you have one copy.

Restore testing, which is the whole point

A backup that has never been restored is a hypothesis.

Three levels, and you need all three:

1. INTEGRITY VERIFICATION, every backup, automated.
   Checksums, and for a database, a restore-and-open that
   confirms the file is readable and consistent. Cheap,
   catches corrupt or truncated backups.

2. SCHEDULED FULL RESTORE, monthly or quarterly, to a scratch
   environment, TIMED.
   The output is a number: minutes to restore, which becomes
   the RTO you are allowed to claim.

3. BUSINESS-LEVEL ASSERTION on the restored data, because a
   restore that produces a readable but wrong database is the
   worst outcome.
     - row counts within tolerance of the source at that time
     - a known query returning a known answer
     - referential integrity checks on the largest relations
     - the newest record's timestamp, which verifies the RPO

def verify_restore(restored, source_at_backup_time) -> list[str]:
    """Assert the restore is USABLE, not merely readable.

    A restore that opens cleanly and is missing a table, or has
    an empty partition, passes every technical check and fails
    the business. These assertions are what catch that.
    """
    problems = []

    # RPO: how old is the newest record we recovered?
    newest = restored.query("SELECT max(created_at) FROM orders")
    age = source_at_backup_time - newest
    if age > RPO_TARGET:
        problems.append(f"RPO breach: newest record is {age} old")

    # Silent partial restores show up as a count mismatch.
    for table, expected in source_at_backup_time.counts.items():
        actual = restored.count(table)
        if abs(actual - expected) / max(expected, 1) > 0.001:
            problems.append(f"{table}: {actual} rows, expected ~{expected}")

    # A known answer catches a restore that is complete and wrong.
    total = restored.query("SELECT sum(amount_cents) FROM ledger_entries")
    if total != source_at_backup_time.ledger_total:
        problems.append("ledger total mismatch")

    # Referential integrity, on the relations that matter.
    orphans = restored.query(
        "SELECT count(*) FROM order_lines ol "
        "LEFT JOIN orders o ON o.id = ol.order_id WHERE o.id IS NULL")
    if orphans:
        problems.append(f"{orphans} orphaned order_lines")

    return problems

The restore test's real output is a measured RTO, and it is the only honest number to put in a DR document.

What to back up beyond the database

Everything the recovery needs, which is more than the data:

  DATABASE              obviously, with PITR where available
  OBJECT STORAGE        versioning plus cross-region
                        replication; note that replication is
                        NOT a backup (a delete replicates)
  CONFIGURATION         the actual applied config, not the
                        repo's version of it
  IaC STATE             Terraform state is a small file whose
                        loss makes your infrastructure
                        unmanageable. Versioned, locked,
                        backed up, in a separate account.
  SECRETS               the values, not just the structure.
                        Encrypted, with the key escrowed
                        somewhere the secrets manager is not.
  CONTAINER IMAGES      the exact digests currently deployed,
                        so a registry loss does not prevent a
                        rebuild
  DNS ZONES             exported, versioned. A zone
                        reconstructed from memory during an
                        incident is its own outage.
  CERTIFICATES and the ability to reissue
  MESSAGE QUEUES        or an explicit decision that in-flight
                        messages are acceptable losses
  CI/CD PIPELINE
    DEFINITIONS         if they live only in a SaaS product

Terraform state is the item most often missing, and its loss means every subsequent apply proposes recreating your entire infrastructure.

Retention, PITR, and the erasure tension

GRANULARITY beats DURATION for the common cases.
  PITR to the second for 7 to 35 days handles human error,
  which is the most frequent cause.
  Daily snapshots for 90 days handle "we noticed last month".
  Monthly for a year or seven handles compliance.

A typical defensible policy:
  PITR              35 days
  daily             90 days
  monthly           13 months
  yearly, immutable 7 years (if a regulation requires it)

THE ERASURE TENSION, which you should be able to discuss:
  a GDPR erasure request says delete the person's data. An
  immutable backup cannot be edited, by design.

  The accepted positions:
    - backups are excluded from immediate erasure, with a
      documented retention horizon after which they age out,
      and a commitment that erased data is re-erased on any
      restore
    - CRYPTO-SHREDDING: encrypt per-subject with a per-subject
      key, and delete the key. The backup still contains the
      ciphertext and it is unrecoverable, which most
      regulators accept as erasure.

  Crypto-shredding is the stronger answer and it has to be
  designed in from the beginning, because retrofitting
  per-subject encryption is a data migration.

The measurements

Four numbers, on a dashboard, not in a document:

  BACKUP SUCCESS RATE, per job, alerting on any failure. A
    silently failing backup job is the classic finding, and
    the classic reason is a credential expiry that nobody
    monitored.
  BACKUP AGE, per dataset, alerting when it exceeds the RPO.
    This catches the job that succeeds and produces nothing.
  RESTORE TEST RECENCY, alerting when it exceeds the interval.
  MEASURED RESTORE TIME, trended, because it grows with the
    dataset and a documented RTO silently becomes wrong.

Backup age rather than backup success is the alert that catches the most, because a job can exit zero having backed up an empty database.

A worked example: three findings from a first restore test

A SaaS company, 6 TB across a primary Postgres and object storage, documented RTO of 4 hours and RPO of 15 minutes, backups running nightly for three years, never restored.

The first full restore test, run in a scratch account:

FINDING 1: the restore took 9 hours 40 minutes.

  transfer of 6 TB from cross-region object storage   5h 10m
  Postgres restore and WAL replay                     2h 20m
  index rebuild (4 large indexes not in the base
    backup path)                                      1h 35m
  application start, cache warm, verification           35m

  Documented RTO: 4 hours. Actual: 9h 40m.

  Nobody had been dishonest. The 4 hours was written in 2022
  when the dataset was 1.4 TB, and had never been revisited.

FINDING 2: the object storage backup was replication, not
  backup.

  Cross-region replication was configured and versioning was
  NOT enabled on the destination. A delete in the source
  replicated as a delete marker.
  So the "backup" of 40 million user-uploaded files provided
  zero protection against deletion, which is the most likely
  way to lose them.

FINDING 3: the ledger totals did not match.

  Row counts matched. The database opened cleanly. Every
  technical check passed.
  Sum of ledger_entries.amount_cents differed from the source
  by a small amount, traced to a table excluded from the
  backup by a filter added 14 months earlier for a table that
  had since been repurposed. The exclusion list had never been
  reviewed.

  A restore would have produced a complete-looking, readable,
  WRONG financial database.

Finding 3 is the one that justifies business-level assertions, because it is invisible to every technical verification: the backup was valid, the restore was successful, and the data was wrong.

The remediation:

RESTORE TIME (9h40m -> 1h05m)
  - a warm standby replica kept continuously, so the common
    case is a promotion (90 seconds) rather than a restore
  - restore-from-backup retained for the case the replica is
    also bad (logical corruption, ransomware), and optimised:
      * backups moved to same-region storage with a
        cross-region immutable copy, so the restore reads
        locally: 5h10m -> 40m
      * indexes included in the physical backup rather than
        rebuilt: 1h35m -> 0
      * parallel restore jobs: 2h20m -> 25m
  - documented RTO corrected to a MEASURED 1h05m, with a note
    that it is re-measured quarterly and grows with the
    dataset

OBJECT STORAGE
  - versioning enabled on the destination bucket
  - a separate backup account with object lock in compliance
    mode, 90-day retention, written by a push from production
    with no delete permission granted to any production role
  - lifecycle to cheaper storage after 30 days

THE EXCLUSION LIST
  - reviewed, and one further stale exclusion found
  - moved from a hand-maintained list to an explicit ALLOW
    list generated from the schema, so a new table is
    included by default and an exclusion requires a comment
    with a reason and a date

VERIFICATION
  - integrity check after every backup
  - a full restore test monthly, automated, into a scratch
    account, with the four business assertions
  - the four dashboard numbers, with backup AGE alerting
    rather than job success

Changing the exclusion list from a denylist to an allowlist is the structural fix, because a denylist silently omits every new table nobody remembers to add, and the failure is invisible until a restore.

Two things caught in the first six months of monthly testing:

1. Month 2: the restore test failed. The backup job had been
   succeeding and producing a 12 KB file for nine days,
   because a credential rotation had removed the permission to
   read one tablespace and pg_dump was exiting zero with a
   warning.
   Backup SUCCESS RATE was 100% for those nine days. Backup
   AGE was fine (the file was fresh). Only the restore test
   caught it, and it is the reason the third dashboard number
   (restore test recency) exists.

2. Month 5: the restore succeeded and the ledger assertion
   failed by a large margin. Cause: a schema migration had
   added a column with a default that the restore applied
   differently because of a Postgres minor-version difference
   between the backup source and the scratch environment.
   Not a data-loss bug, and it revealed that the scratch
   environment's version was not pinned to production's,
   which would have mattered during a real restore.

"Backup success was 100 percent while the backup was 12 kilobytes" is the finding that should end any argument about whether restore testing is worth it, and it went undetected for nine days by every metric except an actual restore.

The cost, stated:

Monthly automated restore test:
  scratch environment, ~6 hours of compute per run   ~$180/mo
  engineering time to build it                       ~2 weeks
  ongoing maintenance                                ~2 h/month

Against: a documented RTO that was wrong by 5h40m, an object
store with no deletion protection for 40 million files, a
financial dataset that would have restored incorrectly, and a
nine-day window where the backup was empty.

Production evidence

The 3-2-1 rule originates in Peter Krogh's photography-archive work and became general backup practice; the 3-2-1-1-0 extension adding an immutable or air-gapped copy and zero verification errors is promoted by backup vendors and by ransomware-response guidance specifically because reachable-by-one- credential copies proved insufficient.

S3 Object Lock's compliance versus governance modes are documented by AWS with the explicit distinction that compliance mode cannot be overridden by any user including the root account, while governance mode can be bypassed by a principal holding BypassGovernanceRetention. Knowing which one you have is the difference between protection against accident and protection against compromise.

Ransomware guidance from national cyber agencies (CISA, the UK NCSC, the Canadian Centre for Cyber Security) consistently identifies backup deletion as an early step in the attack chain and recommends offline or immutable copies with separate credentials as the primary control.

Point-in-time recovery in managed databases (RDS, Cloud SQL, Azure SQL) with continuous WAL or transaction-log archiving is what makes recovery from human error possible at second granularity, which is why granularity often matters more than duration.

Crypto-shredding as a GDPR erasure mechanism for immutable backups is a documented pattern in event-sourcing and data-protection engineering literature: encrypt per subject, delete the key, and the ciphertext remaining in backups is treated as erased.

Terraform state as a critical artifact is emphasised in HashiCorp's own guidance on remote state with versioning and locking, and its loss is a well-documented operational failure because subsequent applies propose recreating existing infrastructure.

The debate

Is a replica a backup? No, and this is the most common substantive error. A replica applies your mistakes at replication speed: a bad DELETE, a bad migration or a ransomware encryption all arrive in milliseconds. A replica is availability. You need both, and the warm standby is the fast path while the backup is the correct path when the data itself is bad.

Is immutability necessary? Once the threat model includes a compromised credential, yes, and it does. The test is naming the single credential that could destroy every copy, and if you can name one, more copies do not help. Compliance-mode object lock or a separate account with no delete permission reachable from production are the practical answers; air-gapping is stronger and rarely practical.

How often should you test restores? Monthly if automated, quarterly at minimum. The argument for monthly is the nine-day window in the worked example, where every metric said the backups were healthy and the file was 12 kilobytes, which only a restore could detect. The argument against is cost, and at roughly $180 a month of scratch compute it is not a serious objection.

Should the restore test include business assertions? Yes, and this is the part usually skipped. A restore can be technically perfect and produce a wrong financial dataset, which is what a stale exclusion filter did in the worked example: row counts matched, the database opened, and the ledger total was wrong. Row counts, a known query with a known answer, referential integrity and the newest record's timestamp are four cheap assertions that catch it.

Denylist or allowlist for what gets backed up? Allowlist, generated from the schema. A denylist silently omits every new table nobody remembers to add, and the omission is invisible until a restore. The counter-argument is noise from genuinely excludable tables, which is handled by requiring a comment and a date on each exclusion.

How do you reconcile erasure requests with immutable backups? Either a documented retention horizon after which backups age out, with a commitment to re-apply erasure on restore, or crypto-shredding with per-subject keys. Crypto-shredding is the stronger answer and must be designed in, because retrofitting per-subject encryption is a full data migration.

Follow-up Q&A

"Why is a replica not a backup?"

Because it applies your mistakes at replication speed. A DROP TABLE, a migration with a bad WHERE clause, or a ransomware encryption all reach the replica in milliseconds, so the replica is missing the same 400,000 rows you are. A replica gives you availability; a backup gives you a point in the past you can return to. You need both, and for most systems the more common disaster is human error rather than region loss, which is why point-in-time recovery granularity often matters more than retention duration.

"What does 3-2-1-1-0 add, and why?"

The extra one is an immutable or air-gapped copy and the zero is verification with no errors. Both were added because the threat model changed: an attacker with cloud admin credentials deletes the snapshots and the cross-region copies first, since the credential that can write your backups is usually the credential that can delete them. Every copy reachable by one credential is effectively one copy. The test is a single sentence: which credential, if compromised, could destroy every copy? If you can name one, you do not have the protection you think you do.

"What is the difference between compliance and governance mode object lock?"

Compliance mode cannot be overridden by any user, including the account root, until the retention period expires. Governance mode can be bypassed by a principal holding the bypass permission. So governance mode protects against accident and compliance mode protects against a compromised administrator, which is the threat that motivated immutable backups in the first place. Knowing which one you have configured is the whole question.

"What should a restore test assert?"

Three levels. Integrity on every backup, cheap and automated, catching corrupt or truncated files. A full timed restore monthly or quarterly, whose output is the measured RTO you are allowed to claim. And business-level assertions on the restored data, because a restore can open cleanly, match row counts and still be wrong: in one case a stale table exclusion added fourteen months earlier meant the ledger totals differed from the source, and every technical check passed. Four assertions cover most of it: row counts within tolerance, a known query returning a known answer, referential integrity on the largest relations, and the newest record's timestamp, which verifies the RPO.

"What is the most common backup failure you have seen?"

A job that succeeds while producing nothing. In one case a credential rotation removed permission to read a tablespace, pg_dump exited zero with a warning, and the backup was 12 kilobytes for nine days. Backup success rate was 100 percent. Backup age was fine, because the file was fresh. Only the monthly restore test caught it. That is why the alert should be on backup age and, more importantly, why restore-test recency is itself a monitored number.

"How do you handle a GDPR erasure request against immutable backups?"

Two accepted positions. Exclude backups from immediate erasure with a documented retention horizon after which they age out, plus a commitment that erased data is re-erased on any restore. Or crypto-shred: encrypt each subject's data under a per-subject key and delete the key, so the ciphertext remaining in the backup is unrecoverable, which most regulators accept as erasure. Crypto-shredding is the stronger answer and it has to be designed in from the start, because retrofitting per-subject encryption is a full data migration.

Common misconceptions

"We have a replica, so we have a backup." The replica has your mistake too, within milliseconds.

"Cross-region replication protects the object store." A delete replicates as a delete. Without versioning on the destination and a separate immutable copy, it protects against region loss and nothing else.

"Backups succeeded, so we are covered." A job can exit zero having produced an empty file. Alert on backup age, and test restores, because success rate is the metric that missed a nine-day empty-backup window.

"A snapshot is a backup." In the same account, same region and same credential boundary, it protects against instance failure only. A backup crosses a trust boundary.

"Our RTO is four hours." Only if someone has restored the full dataset and timed it. In one case the measured number was 9 hours 40 minutes against a documented 4, written when the dataset was a quarter of its current size.

"The restore worked, so the data is fine." A technically successful restore of an incomplete backup produces a readable, complete-looking, wrong database. Assert on the business data.

Interview delivery note

Say this verbatim: "A replica is not a backup, because it applies your mistakes in milliseconds, and a backup nobody has restored is a hypothesis. The first full restore test on a three-year-old backup regime found the real RTO was nine hours forty against a documented four, that cross-region replication of the object store gave no protection against deletion, and that the ledger totals did not match because of a stale table exclusion nobody had reviewed." Three findings from one test is a compact argument for the practice.

The senior-versus-staff separator is asserting on the business data rather than on the restore. A senior engineer verifies that the restore completed and the row counts match. A staff engineer adds a known query with a known answer, referential integrity on the largest relations and the newest record's timestamp, because a stale exclusion filter produces a restore that opens cleanly, matches row counts and is financially wrong, which is the worst possible outcome and is invisible to every technical check.

The second signal is naming the credential test for immutability. Asking "which single credential, if compromised, could destroy every copy?" reframes backup strategy from counting copies to naming a trust boundary, and it is the question that distinguishes compliance-mode object lock, which the root account cannot override, from governance mode, which a bypass permission defeats.

The Staff & Team Lead Interview Reference