Infrastructure as code: state, drift and safe applies

Infrastructure as code promises that your servers, databases and networks can be reviewed, versioned and recreated like any other code. It mostly delivers, but it adds a new way to cause an outage: a small, innocent looking change to a file that deletes a production database. Almost every such incident comes from misunderstanding one thing, the state file, and from not reading one thing, the plan.

This article explains how a tool like Terraform decides what to change, what the state file is and why it matters so much, how to read a plan, how to rename and reorganise code without destroying anything, what drift is and how to catch it, how to limit the damage any one mistake can do, and a workflow that makes applying changes uneventful. Examples use Terraform; OpenTofu, its open-source fork, works the same way, and Pulumi and CloudFormation share the same core ideas.

Desired state and the plan

Infrastructure code is declarative: you describe what should exist, not the steps to create it. “There is a Postgres database called orders-db of this size” rather than “call the API to create a database”. The tool’s job is to work out the steps that turn what exists into what you described.

It does that in two phases. plan compares your code with what currently exists and lists the changes it would make: create this, update that, delete the other. apply carries them out. The plan is the most important safety feature in the whole system, because it is the last moment a human can see exactly what is about to happen before it happens.

State: the map from code to the cloud

To compare code with reality, Terraform needs to know which real object each block of code refers to. Your code says resource "aws_db_instance" "main"; the cloud has a database with the ID orders-db. Nothing in either one says they are connected. The state file is that connection: a record that the address aws_db_instance.main means the real database orders-db, along with the attributes it had last time Terraform looked.

Everything about state follows from that role:

• It must be shared. If two engineers each keep their own copy, they each have a different idea of what exists. State lives in a remote backend, such as an S3 bucket or Terraform Cloud.
• It must be locked. Two applies running at once against the same state can both read it, both make changes, and each overwrite the other’s record. Backends take a lock for the duration of a plan or apply.
• It must be protected. State stores every attribute of every resource, including database passwords and private keys, in plain text. Encrypt it, restrict who can read it, and keep its history (bucket versioning), because a corrupted or deleted state file is a very bad day.
• It is the source of identity. Remove an entry from state and Terraform forgets the resource exists; it still runs and still costs money, but Terraform will try to create a duplicate. Change an address in code without changing it in state, and Terraform sees one resource to delete and one to create.

Reading a plan

A plan marks each resource with a symbol. The symbol tells you more about risk than anything else in the output, so it is the first thing to read:

The dangerous one is replace, because it looks like an edit. Some attributes can’t be changed on a live resource: the cloud API only allows them at creation. Change one, such as a database’s storage encryption setting or the subnet group it lives in, and Terraform plans to delete the resource and create a new one, marked -/+ with a note saying which attribute “forces replacement”. For a stateless server that may be fine. For a database, it means an empty database.

Large plans hide small disasters. A plan of 300 lines with one destroy in the middle is easy to approve without seeing it. Read the summary line (Plan: 3 to add, 12 to change, 1 to destroy) first, and if the destroy count is not what you expected, find out why before anything else.

Refactoring without destroying

Renaming things is routine in application code. In infrastructure code it can be destructive, because a resource’s name in code is its address in state. Rename main to primary and Terraform sees an address in state with no code (delete it) and an address in code with no state (create it):

Terraform now has blocks for each kind of refactor, so they can be reviewed in a pull request like any change:

• moved re-addresses a resource: a rename, moving it into a module, or changing from count to for_each.
• import adopts something that already exists, such as a bucket someone created by clicking in the console, so Terraform manages it from now on.
• removed stops managing a resource without deleting it, for example when handing it over to another team’s code.

A related trap is count. A list of three queues created with count is addressed by position: [0], [1], [2]. Remove the first item from the list and every other item shifts down one position, so Terraform plans to change or replace them all. With for_each they are addressed by name, and removing one touches only that one. It is the same problem as using array indexes as keys in a UI list, and has the same fix.

Drift

Drift is any difference between what the code says and what really exists, caused by changes made outside Terraform: someone resizing a database in the console during an incident, an autoscaler changing a setting, another tool editing the same resource. Drift is dangerous in a quiet way. The next person to run apply, for a completely unrelated change, also undoes the console fix, because Terraform faithfully restores what the code says.

Handling it is mostly a matter of habit:

• Detect it on a schedule. A nightly terraform plan -refresh-only with -detailed-exitcode exits with code 2 if anything changed outside Terraform. Alert on it.
• Make console changes the exception. Read-only access by default in production, with a break-glass role for emergencies, and a rule that every emergency change is written back into code the same day.
• Declare what others own. If an autoscaler manages a group’s desired size, lifecycle { ignore_changes = [desired_capacity] } tells Terraform not to fight it.

Blast radius and how to split state

If every resource in production lives in one state, every apply can, in principle, touch every resource. Plans become slow because Terraform has to check thousands of objects, and a mistake in one corner can reach everything else. The blast radius of a change is how much it could break, and splitting state is how you shrink it.

Useful lines to split along:

• Environment. Production and staging never share a state, so a staging change can’t touch production at all.
• Rate of change. Networks and DNS change rarely and break everything when they go wrong; application resources change weekly. Keeping them apart means the risky foundation is touched only when someone means to.
• Ownership. One state per team or service, so teams apply independently and can’t block each other on the lock.

The cost is wiring: one state needs values from another, such as the network’s subnet IDs. Read them through outputs or data sources rather than hard-coding them, and keep the dependencies pointing one way, from applications down to foundations.

A workflow that makes applies boring

1. Plan on every pull request, in CI, and post the plan as a comment. Reviewers review the plan, not just the code: the code says what you meant, the plan says what will happen.
2. Check the plan automatically. Export it as JSON and run policy checks with a tool such as Open Policy Agent (via Conftest) or Sentinel: no deletes of stateful resources without an extra approval, no public buckets, no security groups open to the internet.
3. Apply exactly the reviewed plan. Save it with plan -out and apply that file. A fresh plan at apply time might include changes nobody reviewed, from drift or from another merge.
4. Apply from CI, not laptops. One place with the credentials, one audit log, one queue, and no “it worked on my machine” provider versions.
5. Guard what can’t be recreated. prevent_destroy in Terraform, and deletion protection in the cloud itself, which still holds if someone bypasses Terraform.

Secrets, modules and versions

• Keep secrets out of state where you can. Let the cloud generate and store database passwords (for example in a secrets manager) rather than passing them through Terraform. Recent Terraform versions add ephemeral values and write-only attributes, which are used during an apply but never written to state.
• Keep modules small and versioned. A module is a reusable package of resources. A small module with a clear purpose (“a Postgres database with our backup and alarm defaults”) is easy to review and upgrade. A giant module with fifty options becomes a second programming language. Version modules, and upgrade one environment at a time.
• Pin providers. Providers translate your code into cloud API calls, and a new version can change defaults or behaviour. Commit .terraform.lock.hcl and upgrade deliberately, reading the plan for unexpected changes.

The guardrails in code

Locked, encrypted remote state; a database protected twice, by Terraform and by the cloud; a rename done with moved; adopting a hand-made bucket with import; letting go of one with removed; named collections with for_each; and the CI commands for reviewed applies and nightly drift checks.

# Remote state: shared, encrypted, and locked so two applies can't run at once.
terraform {
  required_version = ">= 1.10"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 6.0" }   # exact versions pinned in .terraform.lock.hcl
  }
  backend "s3" {
    bucket       = "acme-terraform-state"
    key          = "prod/orders/terraform.tfstate"   # one state per environment and component
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true                              # S3-native locking, no DynamoDB table needed
  }
}

resource "aws_db_instance" "primary" {
  identifier          = "orders-db"
  engine              = "postgres"
  instance_class      = "db.r7g.xlarge"
  deletion_protection = true             # the cloud API refuses to delete it, whatever Terraform says
  lifecycle {
    prevent_destroy = true               # and Terraform refuses to plan its destruction
  }
}

# The resource used to be called "main". Re-address the state instead of destroying it.
moved {
  from = aws_db_instance.main
  to   = aws_db_instance.primary
}

# Adopt a bucket someone created by hand, so Terraform manages it from now on.
import {
  to = aws_s3_bucket.exports
  id = "acme-orders-exports"
}
resource "aws_s3_bucket" "exports" {
  bucket = "acme-orders-exports"
}

# Stop managing a resource without deleting it.
removed {
  from = aws_s3_bucket.legacy_reports
  lifecycle { destroy = false }
}

# Key collections by name, not position: removing one entry can't shift the others.
resource "aws_sqs_queue" "jobs" {
  for_each = toset(["emails", "invoices", "exports"])
  name     = "orders-${each.key}"
}

# In CI:
#   terraform plan -out=tfplan -detailed-exitcode     # exit code 2 means there are changes
#   terraform show -json tfplan > plan.json           # policy check: fail on any delete without approval
#   terraform apply tfplan                            # apply exactly what was reviewed
# Nightly drift check:
#   terraform plan -refresh-only -detailed-exitcode   # exit 2: something changed outside Terraform

References

1. HashiCorp. Terraform Language Documentation: State, Refactoring (moved), Import and Removing Resources. developer.hashicorp.com
2. HashiCorp. Backend Type: s3 (state locking with use_lockfile). developer.hashicorp.com
3. Brikman, Y. (2022). Terraform: Up & Running, 3rd ed. O’Reilly. terraformupandrunning.com
4. Morris, K. (2020). Infrastructure as Code, 2nd ed. O’Reilly. oreilly.com
5. OpenTofu. OpenTofu Documentation. opentofu.org/docs
6. Open Policy Agent. Terraform policy with Conftest. openpolicyagent.org