Infrastructure as code: state, drift, and the module boundary
What it is
Infrastructure as code declares the desired state of infrastructure in version-controlled files and reconciles reality to match. The three things that decide whether it works at scale are how you split state, how you detect drift, and where you draw module boundaries.
STATE the mapping from your configuration to real
resources. Terraform keeps it explicitly in a state
file; Pulumi and CloudFormation keep an equivalent.
Lose it and your infrastructure becomes
unmanageable, because the tool no longer knows
which real resource corresponds to which
declaration.
DRIFT reality diverging from the declaration. Someone
changed something in the console, another tool
modified a resource, or a provider changed a
default.
MODULE the unit of reuse. The question is not "how do I
BOUNDARY make a module" but "what should a module
encapsulate", and the common answer (one module per
resource) is the wrong one.
What this is confused with: declarative configuration and idempotence. A Terraform apply is not idempotent in the sense people assume: it computes a diff against recorded state, so two applies from different state files against the same infrastructure produce conflicting plans. The state file is the source of truth about what you own, not the cloud.
Also confused: IaC and GitOps. IaC is the declaration. GitOps is the operating model where the repository is the trigger and a controller continuously reconciles. You can have IaC with a human running apply from a laptop, and that is the configuration most drift comes from.
The problem it solves
Three failures, in the order teams meet them.
One state file for everything.
A monolithic state with ~3,000 resources.
terraform plan 8 to 14 minutes
every change refreshes every resource
a change to a DNS record locks the state file that also
contains the production database
two engineers cannot work simultaneously
a bad apply can affect anything in the file
and the blast radius of a mistake is the whole estate
Observed consequence: people stop using it. Changes get made
in the console because a 12-minute plan for a one-line change
is intolerable, which produces drift, which makes the next
plan dangerous, which makes people more reluctant.
Drift, and the spiral it causes.
Someone increases an RDS instance size in the console during
an incident. Correct decision, correct urgency.
Nobody updates the Terraform.
Three weeks later, an unrelated apply proposes to shrink the
instance back, because the declaration still says the old
size. Either it happens (an outage) or the engineer aborts and
learns not to trust plan output.
The second outcome is worse and more common: a plan nobody
trusts is a plan nobody reads, and the next real problem in a
plan gets approved along with everything else.
Modules that abstract nothing.
module "s3_bucket" {
source = "./modules/s3"
bucket_name = var.name
versioning = var.versioning
encryption = var.encryption
lifecycle_rules = var.lifecycle_rules
... 40 more passthrough variables
}
This is `aws_s3_bucket` with extra steps. It adds a layer of
indirection, a version to manage, and no decision. Every
change to the underlying resource requires a change to the
module, and every consumer must upgrade.
A module should encapsulate a DECISION, not a resource.
Mechanics
Splitting state
Split by blast radius and by rate of change, which usually coincide.
A workable decomposition for one workload in one environment:
00-account IAM baseline, org-level settings.
Changes yearly.
10-network VPC, subnets, transit gateway attachments,
DNS zones. Changes quarterly.
20-data RDS, ElastiCache, S3 buckets holding state.
Changes rarely, and destroying it is
catastrophic.
30-platform EKS cluster, node groups, shared controllers.
Changes monthly.
40-app-<name> per-application resources: queues, topics,
IAM roles, secrets. Changes weekly.
Each is a separate state file with its own lifecycle.
Three properties fall out:
1. PLAN TIME collapses. A change to an application's queue
refreshes 40 resources, not 3,000.
2. BLAST RADIUS is bounded. An apply in 40-app cannot destroy
the database, because the database is not in that state.
3. PERMISSIONS can differ per layer. The application team can
apply 40-app; only the platform team can apply 20-data.
This is impossible with one state file.
How layers reference each other, and the trade:
REMOTE STATE DATA SOURCE
data "terraform_remote_state" "network" { ... }
Reads the other layer's outputs directly.
Simple, and it couples you to the other layer's state
format and requires read access to its state, which
contains secrets.
DATA SOURCES / TAGS
data "aws_vpc" "main" { tags = { Name = "prod" } }
Looks up the real resource. Decoupled from state, and it
depends on a naming or tagging convention holding.
AN EXPLICIT CONTRACT
the lower layer writes its outputs to SSM Parameter Store
or a similar registry; upper layers read named parameters.
More machinery, and it is the version that scales, because
the contract is explicit and versionable and does not
require state access.
PREFER the third at scale, the second early, and use the
first knowing that state files contain secrets in plaintext.
State file hygiene, which is not optional:
REMOTE, always. A state file on a laptop is a single point of
failure for the estate.
VERSIONED. S3 with versioning, so a corrupted state can be
rolled back. This has saved more estates than any other
single control.
LOCKED. DynamoDB or the backend's native locking, so two
concurrent applies cannot interleave.
ENCRYPTED. State contains secrets in plaintext: database
passwords, generated keys, anything a provider returns.
Treat the state bucket as a secrets store, with the same
access controls.
BACKED UP to a separate account. It is a small file whose
loss makes your infrastructure unmanageable, and it is the
item most commonly missing from a backup inventory.
Drift: detect, alert, decide
CAUSES, in rough order of frequency:
a console change during an incident
another tool: an autoscaler changing desired capacity, a
Kubernetes controller creating a load balancer, a
certificate manager rotating something
a provider changing a default in a new version
a resource modified by a managed service on your behalf
a partial apply that failed halfway
DETECTION: run `terraform plan -detailed-exitcode` on a
schedule, per state file, in CI.
exit 0 no changes
exit 2 drift detected
Alert on exit 2 with the plan attached.
The metric: NUMBER OF STATE FILES WITH NON-EMPTY PLANS, and
its trend. A rising number means people are working around
the tooling, and that is a leading indicator rather than a
lagging one.
Should drift be auto-corrected? No.
Auto-apply on drift will, sooner or later, revert a change
someone made deliberately in an emergency, at a moment when
that change is what is keeping the service up.
The correct response to detected drift is a decision:
- the console change was right -> update the code to
match, and ask why
the code path was
not usable
- the console change was wrong -> apply, deliberately
- it is expected external mutation -> add a lifecycle
ignore_changes with
a COMMENT explaining
why, so it is a
decision rather than
an accumulation
The third case is the one that rots. An ignore_changes block
without a comment is indistinguishable from someone silencing
an alert.
Reducing drift at the source is better than detecting it:
- REMOVE console write access in production. If the only way
to change production is a pull request, drift from console
changes is structurally impossible.
- Provide a BREAK-GLASS path (documented, paging, time-boxed)
so the emergency case has an answer that is not "use the
console quietly".
- Make the code path FAST. Most console changes happen
because the plan takes 12 minutes and the console takes 12
seconds. Splitting state fixes the cause.
The module boundary
A module should encapsulate a decision, not a resource.
BAD: module "s3" wrapping aws_s3_bucket with 40 passthrough
variables. No decision, pure indirection.
GOOD: module "document_store" that encodes the organisation's
opinions:
- versioning on
- encryption with the customer-managed key
- a bucket policy denying unencrypted transport
- access logging to the log-archive account
- a lifecycle rule to infrequent access at 30 days
- the standard tag set
Interface: name, retention_days, and whether it is
public-read (which it will not be).
The second one is worth versioning, because upgrading it
propagates a decision. The first one is worth deleting.
# A module interface with three inputs is a decision.
# A module interface with forty is a passthrough.
module "document_store" {
source = "git::ssh://git@github.com/acme/tf-modules.git//document-store?ref=v3.2.0"
name = "customer-uploads"
retention_days = 2555 # 7 years, compliance
data_class = "personal" # drives the KMS key and the
# replication destination
}
Signals that a module boundary is wrong:
- more than about 15 input variables -> passthrough
- a boolean input that switches large -> two modules
amounts of behaviour ("create_dns")
- consumers routinely need a resource -> the boundary is
the module does not expose in the wrong place
- every consumer passes the same value -> it should be
for a variable inside the module
- the module contains an entire -> untestable, and
environment nothing reuses it
Versioning and pinning:
Pin module versions and provider versions. An unpinned module
means an apply's behaviour changes because someone else
merged.
source = "...//document-store?ref=v3.2.0" yes
source = "...//document-store" no
Provider versions in a lock file, committed. A provider minor
version can change a default and produce a diff on every
resource of that type, which is the "why does my plan want to
change 400 things" experience.
Plan and apply discipline
PLAN IN THE PULL REQUEST, posted as a comment, so review is
review of the diff rather than of the intent.
APPLY ON MERGE, from CI, with a credential no human holds.
NEVER APPLY TO PRODUCTION FROM A LAPTOP. The reason is not
purity: a laptop apply uses whatever provider version and
whatever local state that machine has, and it is the
single largest source of state corruption.
REQUIRE APPROVAL for plans that DESTROY, and make destruction
visible in the PR comment. `terraform plan` output buries a
destroy among a hundred lines of change.
A LOCK on the state during apply, and a documented procedure
for a stale lock, because a CI job killed mid-apply leaves
one and the next engineer will force-unlock without
understanding what half-applied.
Refactoring without destroying:
Renaming a resource in code, or moving it into a module,
makes Terraform plan a destroy and a create, because the
address changed.
moved {
from = aws_s3_bucket.uploads
to = module.document_store.aws_s3_bucket.this
}
`moved` blocks (and `import` blocks) express this in code
rather than as an out-of-band `terraform state mv`, so the
refactor is reviewable, reproducible and applies in CI. Before
they existed, refactoring meant a human running state
commands, which is exactly the operation that corrupts state.
A worked example: a 14-minute plan and the drift spiral
A platform team, one Terraform state file, 3,100 resources, four years of accumulation, and a stated problem that "nobody trusts the plans".
Measured before:
plan time (full) 14 min
resources in state 3,100
state file size 38 MB
concurrent applies possible 1
non-empty plans on a scheduled run every run, 60 to 90
resources drifted
console changes in production
(from CloudTrail, per month) ~40
applies per week 3
median time from PR to applied 2.5 days
The diagnosis, which inverted the team's assumption:
The team's belief: engineers were undisciplined and made
console changes.
CloudTrail analysis of the ~40 monthly console changes:
31 were changes an engineer needed to make in under 5
minutes, where the Terraform path was a 14-minute plan
plus a review plus a queue behind two other applies
6 were genuine emergencies
3 were people who did not know the resource was managed
So 31 of 40 were a RESPONSE to the tooling being unusable,
not a cause of it. The drift was a symptom.
"The drift was a symptom of the plan time" is the finding that changed the plan of work, because the proposed remediation had been a policy (no console access) which would have removed the workaround without fixing the reason for it.
The state split, over six weeks:
1 state -> 19 states:
1 account baseline
1 network
3 data (per environment)
3 platform (per environment)
11 per-application, per-environment
Migration mechanism: `terraform state mv` between state files
for the bulk move, then `moved` blocks in code for everything
subsequent. Done one layer at a time, lowest first, with the
old state left intact until the new one had produced a clean
plan twice.
Results:
plan time, application layer 14 min -> 25 s
plan time, network layer 14 min -> 90 s
plan time, data layer 14 min -> 2 min
concurrent applies 1 -> 19
applies per week 3 -> 41
median PR to applied 2.5 days -> 40 min
Applies per week going from 3 to 41 is the number that mattered, because the console changes had been a queue-avoidance behaviour and the queue was gone.
Then the drift work:
DETECTION: a scheduled plan per state file, every 6 hours,
alerting on exit code 2 with the plan attached to the alert.
FIRST RUN found 3 categories in the 60 to 90 drifted
resources:
41 autoscaling group desired_capacity, changed constantly
by the autoscaler. EXPECTED external mutation.
-> lifecycle { ignore_changes = [desired_capacity] }
with a comment naming the autoscaler.
19 genuinely drifted from console changes over years.
-> reconciled: 14 by updating code to match reality
(the console change had been right), 5 by applying.
7 provider default changes from a version bump nobody had
pinned.
-> provider versions pinned, lock file committed.
After reconciliation: scheduled plans clean.
Six months later: 2 to 4 drifted resources on a typical run,
each investigated within a day.
CONSOLE ACCESS in production removed AFTER the plan time was
fixed, with a break-glass role that pages on assumption.
Break-glass used 3 times in six months, all genuine.
Removing console access only after the code path was fast is the sequencing that made the policy stick, and the team's earlier attempt at the policy alone had been abandoned in three weeks.
The module work, which was the smallest change and the most contested:
Before: 34 modules, of which 22 were single-resource wrappers.
Audit by input-variable count:
22 modules with >20 variables (passthroughs)
8 modules with 4 to 12 variables (real abstractions)
4 modules containing an entire environment (untestable,
each used exactly once)
Actions:
22 passthrough modules DELETED; consumers use the resource
directly. -4,100 lines.
4 environment modules deleted; their contents became the
per-environment state files.
8 kept, versioned and pinned.
3 new ones written, each encoding a decision:
document_store (encryption, logging, lifecycle,
replication by data class)
service_account (IAM role, boundary policy, the
standard trust relationship)
postgres (parameter group, backup retention,
monitoring, subnet group, the
encryption key by data class)
The postgres module has 5 inputs. It replaced 400 lines
repeated across 11 places, each subtly different, three of
which had backup retention set to the default 1 day.
Three of eleven databases had a one-day backup retention because the setting was copied and pasted rather than encapsulated, which is the argument for the module boundary in its most concrete form: a module encoding a decision makes the decision uniform, and a passthrough makes it repeated.
Two things that went wrong:
1. THE FIRST STATE SPLIT was done by resource type (all the
IAM in one state, all the networking in another, all the
compute in another). It looked tidy and it was wrong:
creating a new application touched all three states, so
every change needed three coordinated applies.
Redone by lifecycle and blast radius, which is the split
above. The lesson: split by what changes together, not by
what looks similar, which is the same argument as vertical
slices in application code.
2. AN EARLY ATTEMPT AT AUTO-REMEDIATION applied the plan
automatically when drift was detected. In week two it
reverted an instance-size increase that an engineer had
made 40 minutes earlier during a load spike, causing a
second incident.
Removed. Drift now alerts and a human decides.
The auto-remediation incident is the concrete form of why drift correction must be a decision, and it happened exactly as the theory predicts: the drift that matters most is the one someone made deliberately under pressure.
Production evidence
Terraform's documented guidance on state covers remote backends with locking, the fact that state contains sensitive values in plaintext, and the recommendation to separate state by lifecycle and blast radius rather than keeping a monolithic configuration.
moved and import blocks (Terraform 1.1 and 1.5 respectively) exist specifically so that
refactoring and adoption of existing resources are expressed in code and applied through CI, rather than
through out-of-band terraform state commands, which were the documented source of state corruption.
terraform plan -detailed-exitcode returning 2 for a non-empty plan is the documented mechanism for
scheduled drift detection, and it is what every drift-detection tool wraps.
Provider and module version pinning with a committed .terraform.lock.hcl is Terraform's own
recommendation, and the failure it prevents, a provider minor version changing a default and producing a
diff on every resource of that type, is a widely reported experience.
GitOps controllers (Argo CD, Flux) implement continuous reconciliation for Kubernetes, and their documented drift behaviour is instructive by contrast: reconciliation is opt-in per application and self-heal is a deliberate setting, precisely because automatic reversion of a deliberate emergency change is a known hazard.
Open Policy Agent and Sentinel integrations for policy-as-code on Terraform plans are the standard mechanism for preventive guardrails at the plan stage, evaluated before apply, which is the IaC-layer equivalent of a service control policy.
The debate
One state file or many? Many, split by lifecycle and blast radius. The monolithic state's failure is not aesthetic: a 14-minute plan produces console changes, which produce drift, which makes plans untrustworthy. The counter-argument is cross-state references and coordination cost, which is real and is addressed with an explicit output contract rather than by keeping everything together.
Split by resource type or by lifecycle? By lifecycle. Splitting by type (all IAM here, all networking there) looks tidy and means every new application touches every state, which is the same mistake as organising application code by layer instead of by feature.
Should drift be auto-remediated? No for infrastructure with a human-operable console. Auto-apply
will eventually revert a deliberate emergency change at the worst moment, which is exactly what happened
in week two of one implementation. Detect, alert, and require a decision, with ignore_changes plus a
comment for genuinely expected external mutation.
Is removing console access the answer to drift? Only after the code path is fast. In one analysis 31 of 40 monthly console changes were a response to a 14-minute plan and a two-day queue, so the policy alone removes the workaround without fixing its cause and gets abandoned. Fix the plan time, then remove the access, then provide break-glass.
How big should a module be? Big enough to encode a decision, small enough to test. The input-variable count is a reliable smell: above about fifteen it is a passthrough, and a passthrough adds indirection and a version to manage while abstracting nothing. Delete those; keep the ones whose upgrade propagates an opinion.
Should Terraform manage everything? No. Resources mutated continuously by another controller
(autoscaler desired capacity, Kubernetes-created load balancers, certificate rotations) should be
explicitly excluded with a comment, and resources whose lifecycle genuinely belongs to an application
deployment pipeline usually belong there instead. The failure is an ignore_changes list that
accumulates without comments, which is indistinguishable from silencing an alarm.
Follow-up Q&A
"How do you split Terraform state, and why?"
By lifecycle and blast radius, which usually coincide: account baseline, network, data, platform, then per application per environment. Three properties fall out. Plan time collapses, because an application change refreshes forty resources rather than three thousand. Blast radius is bounded, because an apply in the application layer cannot destroy the database that is not in its state. And permissions can differ per layer, which is impossible with one state file. Do not split by resource type: putting all IAM in one state and all networking in another means every new application touches every state, which is the same mistake as organising code by layer rather than by feature.
"Where does drift actually come from?"
Mostly from the tooling being unusable. In one CloudTrail analysis of about forty monthly production console changes, thirty-one were changes an engineer needed to make in under five minutes where the Terraform path was a fourteen-minute plan plus review plus a queue behind two other applies. Six were genuine emergencies and three were people who did not know the resource was managed. So the drift was a symptom, and the proposed remedy of removing console access would have removed the workaround without fixing its cause, which is why that policy had already been tried and abandoned in three weeks.
"Should detected drift be corrected automatically?"
No. Auto-apply on drift will eventually revert a change someone made deliberately in an emergency, at the
moment that change is keeping the service up. One implementation did exactly this in week two, reverting
an instance-size increase made forty minutes earlier during a load spike and causing a second incident.
Detect on a schedule with plan -detailed-exitcode, alert with the plan attached, and require a human
decision: update the code, apply deliberately, or add ignore_changes with a comment naming the external
controller. The comment matters, because an ignore list without one is indistinguishable from silencing an
alarm.
"What makes a good Terraform module?"
It encapsulates a decision rather than a resource. A module wrapping a bucket with forty passthrough variables is the resource with extra steps: indirection, a version to manage, and no abstraction. A document-store module with three inputs that encodes encryption with the right key, access logging to the audit account, a lifecycle rule and the standard tags is worth versioning, because upgrading it propagates an opinion. The reliable smell is the input count: above about fifteen it is a passthrough. In one audit twenty-two of thirty-four modules were passthroughs and were deleted, removing 4,100 lines.
"What is the concrete cost of not having a real module?"
Copied configuration that drifts. In one estate a Postgres module with five inputs replaced four hundred lines repeated across eleven places, each subtly different, and three of the eleven had backup retention left at the one-day default because the setting had been copied rather than encapsulated. A module encoding a decision makes the decision uniform; a passthrough makes it repeated, and repetition is where defaults survive.
"How do you refactor without destroying resources?"
With moved blocks, which express a change of address in code so Terraform updates state rather than
planning a destroy and create. Before they existed, refactoring meant a human running terraform state mv
out of band, which is exactly the operation that corrupts state, and it was neither reviewable nor
reproducible. The same applies to import blocks for adopting existing resources. The general rule is that
any state manipulation you would do by hand should be expressible in code and applied through CI.
Common misconceptions
"Terraform is idempotent, so applies are safe." It computes a diff against recorded state. Two applies from different state files against the same infrastructure produce conflicting plans, and the state file, not the cloud, is the source of truth about what you own.
"One repository, one state, one source of truth." A monolithic state produces a plan time that makes people work around the tool, which produces the drift that makes the state untrustworthy.
"Drift means people are undisciplined." Usually it means the code path is slower than the console. Measure where the changes come from before writing a policy.
"Auto-remediate drift." It will revert the deliberate emergency change, at the worst possible moment.
"A module per resource is good hygiene." It is indirection with a version number. A module should encode a decision, and the input-variable count tells you which kind you have.
"State files are just metadata." They contain provider-returned secrets in plaintext. Treat the state bucket as a secrets store, and back it up to a separate account, because its loss makes the estate unmanageable.
Interview delivery note
Say this verbatim: "Drift is usually a symptom of plan time. We found thirty-one of forty monthly production console changes were people avoiding a fourteen-minute plan and a two-day queue, so the fix was splitting state by lifecycle and blast radius, which took the application-layer plan to twenty-five seconds and applies per week from three to forty-one. Then we removed console access, and that time the policy stuck." It names the causal direction most people get backwards and gives the sequencing.
The senior-versus-staff separator is measuring where the console changes come from before writing the policy. A senior engineer proposes removing production console access to stop drift. A staff engineer pulls CloudTrail, classifies forty changes into thirty-one queue avoidance, six emergencies and three ignorance, concludes the drift is a symptom, fixes the plan time first, and only then removes the access with a break-glass path. The earlier attempt at the policy alone had been abandoned in three weeks, which is the evidence that ordering matters.
The second signal is knowing why drift auto-remediation is wrong. Saying "it will eventually revert the change someone made deliberately under pressure, and in our case it did so in week two during a load spike" shows you understand that the drift most worth investigating is the drift most dangerous to correct automatically.
Further reading
- Terraform's documentation on remote state, state locking, and the guidance to separate configurations by lifecycle.
- Terraform
movedandimportblock documentation, for refactoring and adoption expressed in code. terraform plan -detailed-exitcode, for the scheduled drift-detection mechanism every tool wraps.- Argo CD and Flux documentation on self-heal and drift, for the contrasting continuous-reconciliation model and why self-heal is opt-in.
- The well-architected pillars and landing zones page, whose account boundaries are the natural state-file boundaries too.