Production Checklist

The idea in one sentence

Shipping Terraform and Terragrunt to production is not one big decision but a checklist of smaller ones — each covered earlier in this course — that all have to hold true at the same time.

The checklist

None of these are new ideas at this point. What is new is seeing them side by side as one list, because production readiness is really just all of them being true simultaneously, not any single one of them done especially well.

Remote S3 backend with locking configured from day one. use_lockfile = true on the S3 backend, set up before the first apply, not retrofitted after a corrupted state file forces the issue. Covered in State Management.
Provider and module versions pinned, not floating. The committed .terraform.lock.hcl pins exact provider versions and checksums; module source references use a specific tag or version constraint, never a branch name that can move underneath you. Covered in Foundations and Modules.
Every plan reviewed in CI before every apply, with reviewers specifically watching for unexpected -/+ resource replacements rather than skimming the plan like a code diff. Covered in this module’s CI/CD lesson.
Sensitive values kept out of variables and, where possible, out of state, pulled from a secrets manager via a data source instead of typed into a tfvars file, with the state backend itself encrypted at rest. Covered in State Management and this module’s secrets lesson.
tflint, a security scanner, and ideally policy-as-code wired into CI, so known-bad patterns and organizational rules are enforced mechanically instead of depending on a human catching every case by eye. Covered in this module’s policy-as-code lesson.
Terragrunt dependency blocks driving rollout order, not a person remembering that the VPC has to apply before the database. Covered in Terragrunt Fundamentals.
terraform test / .tftest.hcl coverage for any module doing something non-trivial, so a refactor of a module’s internals gets caught by an assertion instead of by a surprised downstream consumer. Covered in Modules.
lifecycle.prevent_destroy on genuinely irreplaceable resources, so a config change that would destroy production data fails the plan loudly instead of quietly succeeding. Proven below.
precondition blocks validating assumptions before they become resources, catching a bad input at plan time with a message that names the actual problem, instead of a cryptic provider error mid-apply. Proven below.
A rehearsed force-unlock runbook and a real state-backup habit, so a stuck lock or a bad apply has a practiced recovery path instead of becoming an incident where someone improvises against production state for the first time. Proven below.

Two of these — prevent_destroy and precondition — are both lifecycle block, not two separate mechanisms, but they guard different moments: prevent_destroy blocks a specific action (destroy) unconditionally; precondition blocks any action when an assumption about the configuration turns out false.

Guardrails that are actually enforced: `prevent_destroy` and `precondition`

prevent_destroy = true makes Terraform refuse a plan that would destroy the resource, full stop — no flag bypasses it short of removing the lifecycle block itself. Real proof, local_file standing in for any resource a mistake could otherwise destroy:

resource "local_file" "critical" {
  filename = "${path.module}/generated/critical.conf"
  content  = "important=true\n"

  lifecycle {
    prevent_destroy = true
  }
}

$ terraform destroy -input=false -auto-approve
...
Plan: 0 to add, 0 to change, 1 to destroy.

Error: Instance cannot be destroyed

  on main.tf line 3:
   3: resource "local_file" "critical" {

Resource local_file.critical has lifecycle.prevent_destroy set, but the
plan calls for this resource to be destroyed. To avoid this error and
continue with the plan, either disable lifecycle.prevent_destroy or reduce
the scope of the plan using the -target option.

Real refusal, real exit code 1 — a terraform destroy (or a plan that would replace this resource) simply cannot proceed. precondition is the complementary guardrail: it validates an assumption about input, before Terraform commits to the resource at all:

variable "environment" {
  type    = string
  default = "prod"
}

resource "local_file" "config" {
  filename = "${path.module}/generated/app.conf"
  content  = "env=${var.environment}\n"

  lifecycle {
    precondition {
      condition     = contains(["dev", "staging", "prod"], var.environment)
      error_message = "environment must be one of dev, staging, prod — got \"${var.environment}\"."
    }
  }
}

$ terraform plan -var="environment=bogus"
...
Error: Resource precondition failed

  on main.tf line 21, in resource "local_file" "config":
  21:       condition     = contains(["dev", "staging", "prod"], var.environment)
    ├────────────────
    │ var.environment is "bogus"

environment must be one of dev, staging, prod — got "bogus".

Real failure, and a real success with a valid environment value (exit 0, plan proceeds normally) — proving both branches of the same condition, not just the failure case. The error message names the actual bad value ("bogus") and the actual rule, which is the whole point: a plan-time failure a reviewer can act on immediately, instead of a provider-level error deep into apply.

The `force-unlock` runbook — proven, including a real surprise

A stuck lock (a CI job killed mid-apply, a laptop that lost network mid-run) blocks every subsequent plan/apply until someone runs terraform force-unlock. The real runbook, tested against this course’s local backend with a genuinely held lock (time_sleep holding the state locked for 20 seconds while a background apply runs):

# real lock file present while the background apply holds it
$ cat .terraform.tfstate.lock.info
{"ID":"564d6b33-65bd-5b30-f952-c2d2a2297461","Operation":"OperationTypeApply", ...}

$ terraform force-unlock -force 00000000-0000-0000-0000-000000000000
Failed to unlock state: LocalState not locked

$ terraform force-unlock -force 564d6b33-65bd-5b30-f952-c2d2a2297461
Failed to unlock state: LocalState not locked

The real surprise: even the correct lock ID against a genuinely active lock produces the same LocalState not locked message. This is not a harness bug — terraform force-unlock --help explains exactly why: "Local state files cannot be unlocked by another process." The local backend’s lock is an OS-level file lock (flock) scoped to the process holding it, not a named, backend-mediated lock the way S3 native locking or a DynamoDB lock table is — so a second terraform process running force-unlock has nothing it can meaningfully release. This course cannot verify what a real ID-mismatch message from S3 native locking looks like (that needs a real S3 bucket), so that specific message is not claimed here — but the runbook lesson is real and provider-agnostic: force-unlock is a remote-backend recovery tool. Against local state, the actual fix for a stuck process is killing the process or removing the stale .terraform.tfstate.lock.info file directly, never force-unlock.

A real backup, before the risky operation — not after

Two backup mechanisms are real and worth knowing apart. First, automatic: the local backend writes terraform.tfstate.backup on every state-changing operation, holding the previous state — proven by applying a config change twice and diffing both files:

$ cat terraform.tfstate.backup | grep content
      "content": "listen=8080\n",

$ cat terraform.tfstate | grep content
      "content": "listen=9090\n",

Second, deliberate: terraform state pull > backup-$(date +%Y%m%d-%H%M%S).tfstate before any operation an operator is nervous about (a manual state rm, an import, a provider major-version upgrade) — a real, timestamped snapshot an operator chose to take, independent of whatever Terraform’s own automatic .backup file happens to hold. the-state-file.mdx already proved state push enforces lineage/serial before accepting a restore, which is exactly the safety net that makes this backup usable: pushing back an old snapshot cannot silently clobber newer state by accident.

One coherent system: a real repo layout

Put all of it together and this course adds up to a single, concrete shape — not a list of disconnected facts, but one repository that actually looks like this:

infra/
├── root.hcl
├── vpc/
│   └── terragrunt.hcl
└── rds/
    └── terragrunt.hcl

Production root.hcl holds a real S3 backend with locking — the shape remote-backends-and-locking.mdx already proved real against a live S3 bucket:

# Production shape only — needs a real S3 bucket, so it is not collected/run by this lesson's harness proof below.
remote_state {
  backend = "s3"
  config = {
    bucket       = "acme-terraform-state"
    key          = "${path_relative_to_include()}/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true
  }
}

This lesson’s own proof below uses a local backend instead, specifically so the whole tree — root.hcl, both units, and both modules — validates and plans fully offline, with no S3 bucket required:

remote_state {
  backend = "local"
  config = {
    path = "${get_parent_terragrunt_dir()}/${path_relative_to_include()}/terraform.tfstate"
  }
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = <<EOF
provider "aws" {
  region                      = "us-east-1"
  access_key                  = "mock"
  secret_key                  = "mock"
  skip_credentials_validation = true
  skip_requesting_account_id  = true
  skip_metadata_api_check     = true
  skip_region_validation      = true
}
EOF
}

generate "backend" {
  path      = "backend.tf"
  if_exists = "overwrite"
  contents  = <<EOF
terraform {
  backend "local" {}
}
EOF
}

Terragrunt’s own docs are explicit about why a generate "backend" block belongs at the root rather than hand-written per module: a remote_state block has no effect unless the underlying Terraform code declares a matching backend block, and generate is what writes that block into every unit automatically, so an individual module’s own .tf files never need to know which backend a given environment uses at all.

include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "../modules/vpc"
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

resource "aws_vpc" "this" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_subnet" "private" {
  vpc_id     = aws_vpc.this.id
  cidr_block = "10.0.1.0/24"
}

output "vpc_id" {
  value = aws_vpc.this.id
}

output "private_subnet_ids" {
  value = [aws_subnet.private.id]
}

include "root" {
  path = find_in_parent_folders("root.hcl")
}

terraform {
  source = "../modules/rds"
}

dependency "vpc" {
  config_path = "../vpc"

  mock_outputs = {
    vpc_id             = "mock-vpc-id"
    private_subnet_ids = ["mock-subnet-id"]
  }
  mock_outputs_allowed_terraform_commands = ["plan"]
}

inputs = {
  vpc_id    = dependency.vpc.outputs.vpc_id
  subnet_id = dependency.vpc.outputs.private_subnet_ids[0]
}

# modules/rds/terraform.tfvars — harness fixture only, so this lesson's raw
# `terraform plan` proof for modules/rds can run standalone offline; a real
# deployment gets these values from Terragrunt's dependency block instead
# (proven below via terragrunt run --all plan, using vpc's real outputs).
vpc_id    = "mock-vpc-id"
subnet_id = "mock-subnet-id"

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

variable "vpc_id" {
  type = string
}

variable "subnet_id" {
  type = string
}

resource "aws_db_instance" "this" {
  identifier          = "app-db"
  engine              = "postgres"
  instance_class      = "db.t3.micro"
  allocated_storage   = 20
  username            = "app"
  password_wo         = "mock"
  password_wo_version = 1
  skip_final_snapshot = true
}

Real terragrunt run --all plan against this exact tree:

$ terragrunt run --all plan --non-interactive
INFO  The following units will be run, starting with dependencies and then their dependents:
.
╰── vpc
    ╰── rds
...
[rds] terraform: Plan: 1 to add, 0 to change, 0 to destroy.

❯❯ Run Summary  2 units  20s
   ────────────────────────────
   Succeeded    2

vpc planned first, computing a real vpc_id; rds planned second, consuming vpc’s mock_outputs for the plan command (real vpc outputs would flow through automatically once vpc has actually been applied) — the DAG order in the log (vpc before rds) is Terragrunt’s dependency graph, not anyone’s memory of which unit goes first. Every piece here traces back to an earlier lesson: use_lockfile = true in the production root.hcl above is the State Management module’s locking story; root.hcl plus include is Terragrunt Fundamentals’ DRY backend pattern; the dependency block is what lets rds reference vpc’s real output without a person remembering which unit has to apply first; and the ../../modules/rds source, pinned in a real repo to a tag rather than left floating, is Modules’ versioning discipline.

The whole tree ships through exactly the CI/CD shape from earlier in this module: terragrunt run --all plan runs on every pull request touching infra/, posts its output for review, and terragrunt run --all apply runs only after merge to main, using the same OIDC-federated role rather than a stored key. Nothing here is a new mechanism — it is the same pieces from every earlier module, wired together into one system that deploys vpc, then rds behind it, in the order the dependency blocks describe, gated by a plan a human actually read.

flowchart TB
  root["root.hcl: shared backend, locking enabled"]
  root --> vpc["vpc unit"]
  root --> rds["rds unit"]
  vpc -->|dependency block: vpc_id, subnet_ids| rds
  pr["Pull request"] -->|terragrunt run --all plan| review["Reviewed for -/+ replacements"]
  review -->|merge to main| apply["terragrunt run --all apply"]
  apply --> vpc
  apply --> rds

root.hcl plus dependent units, deployed through a CI pipeline that gates plan and apply

Mental Model

Every item on this checklist is a guardrail against a specific, named failure mode — not abstract good practice. prevent_destroy exists because a plan destroying the wrong resource is a real, provable failure mode (proven above). precondition exists because a plan-time typo reaching apply as a cryptic provider error is real (proven above). force-unlock exists because a lock outliving the process that took it is real — and proving it against local backend surfaced a sharper version of the lesson than expected: the tool’s own --help text draws the exact line between what force-unlock can and cannot do. State backups exist because “restore the previous state” needs to already be a rehearsed action, not something invented live during an incident. None of these are optional polish; each closes one specific, demonstrated gap.

Trade-offs

Choice	Benefit	Cost
`lifecycle.prevent_destroy` on critical resources	Proven above: a destroy or a forced replacement is refused outright, no flag bypasses it	Must be deliberately removed (a real, reviewable diff) before a genuinely intended destroy can proceed — friction that is the entire point
`precondition` blocks	Proven above: a bad input fails at plan time with a message naming the actual problem	Every precondition is one more thing to keep in sync as the module’s real constraints evolve
Rehearsing `force-unlock` against `local` state instead of assuming it	Proven above: this course discovered `force-unlock` genuinely cannot act on `local` state before ever hitting the failure in production	The real lesson only applies fully to remote backends (S3, etc.) — this course cannot verify the exact ID-mismatch message S3 native locking produces without a real bucket
A deliberate `state pull` backup before a risky operation	An operator-controlled snapshot independent of whatever the automatic `.tfstate.backup` happens to hold	One more manual step to remember before `state rm`/`import`/a provider upgrade — easy to skip under time pressure, which is exactly when it matters most

Common Misconceptions

“prevent_destroy stops any change to the resource.” Proven above: it specifically blocks destroy (including the destroy half of a forced replacement) — in-place updates that do not require destroying the resource proceed normally.
“A precondition failure is the same kind of error as a provider rejecting a request.” Proven above: precondition fails at plan time, before Terraform ever talks to a provider, with a message the author wrote naming the actual rule — a fundamentally earlier and clearer failure than an apply-time provider error.
“force-unlock is the correct fix for any stuck Terraform run.” Proven above, with a real surprise: against local state, force-unlock cannot release a lock a different process holds — terraform force-unlock --help says so directly. The actual fix for local state is stopping the stuck process or removing its lock file; force-unlock is a remote-backend tool.
“Terraform’s automatic .tfstate.backup is a sufficient backup strategy.” Proven above: it only ever holds the single previous state, overwritten on every subsequent operation — a deliberate state pull snapshot before a risky operation is the only backup that survives more than one apply.

Reading AI-generated code

When reviewing an AI-generated “production-ready” Terraform/Terragrunt setup, check for:

Stateful resources (databases, KMS keys, anything holding data) with no lifecycle.prevent_destroy at all — proven above, this is the guardrail that turns an accidental destroy into a loud, refused plan instead of a silent, catastrophic one.
Input validation happening only in application code or a wiki page, never as a precondition — proven above, a plan-time precondition catches the mistake before any provider call, not after one fails confusingly.
A runbook or comment claiming terraform force-unlock <id> “just works” against whatever backend is in use, with no distinction between local and remote state — proven above, that claim is false for local state specifically, and this course could not verify a claim about S3’s exact mismatch behavior either, so it says so.
A dependency block referencing an upstream unit’s outputs with no mock_outputs/mock_outputs_allowed_terraform_commands — proven above, terragrunt run --all plan needs those to plan a downstream unit before the upstream one has ever actually been applied.
A module source pinned to a branch name (ref=main) instead of a tag or commit SHA — the same unpinned-dependency risk this checklist already flags for provider versions, just one directory level up.

🐛 Spot the bug

resource "aws_db_instance" "main" {
identifier         = "app-prod"
engine             = "postgres"
instance_class     = "db.t3.micro"
allocated_storage  = 20
skip_final_snapshot = true
}

Show the answer

This production database resource has no lifecycle { prevent_destroy = true } block at all. Proven above, that block is what turns a plan that would destroy this resource into a loud, refused error — without it, a config change that forces replacement (a changed identifier, an argument only settable at creation) or a stray terraform destroy against the wrong workspace proceeds exactly as planned, silently, with no extra confirmation beyond the normal apply prompt. skip_final_snapshot = true makes this worse, not better — it explicitly disables the one safety net (a final RDS snapshot) that might otherwise soften the blow. The fix is the lifecycle block proven above, deliberately absent from anything genuinely disposable and deliberately present on anything that is not.

🏗️ The one-line takeaway

Production readiness is a checklist, not a single decision: a locked remote backend from day one, pinned provider and module versions, every plan reviewed in CI for unexpected replacements, secrets kept out of variables and state where possible with the backend encrypted at rest, tflint/a scanner/policy-as-code wired into CI, Terragrunt dependency blocks driving rollout order instead of human memory, terraform test coverage for non-trivial modules, prevent_destroy/precondition guardrails (both proven above with real refusals), a rehearsed force-unlock runbook (proven above with a real, provider-agnostic lesson: it cannot act on local state, by design), and a deliberate state-backup habit beyond Terraform’s own single-previous-version .tfstate.backup. Put together, it is one coherent system — a root.hcl shared backend, units wired together with dependency blocks, deployed through a pipeline that runs terragrunt run --all plan on every pull request and terragrunt run --all apply only after merge (proven above end to end: Run Summary 2 units, Succeeded 2) — not a pile of separate facts.

Why does pinning module versions to a specific tag matter, rather than pointing source at a floating branch It has no real effect since Terraform always re-downloads the latest code anyway An unpinned module can change underneath a configuration between runs with no diff to review, so a plan can shift for reasons that have nothing to do with your own changes Pinning versions is only relevant for the Terraform binary itself, not modules Floating branches are actually the recommended production pattern

Why should a remote backend with locking be configured from day one rather than added later Locking only matters once a team exceeds fifty engineers Local state does not survive a second teammate or a CI run touching the same infrastructure, and retrofitting a backend after a corrupted or conflicting state file is far harder than starting with one Locking is only needed for destroy operations, not apply S3 backends cannot be configured until after the first apply

Why do Terragrunt dependency blocks matter for a multi-unit rollout instead of relying on an engineer applying units in the right order manually dependency blocks are purely cosmetic and do not affect apply order They let one unit reference another real, computed output directly, and drive apply order automatically, instead of depending on a person remembering that vpc must apply before rds dependency blocks replace the need for a remote backend entirely dependency blocks only work within a single terragrunt.hcl file, never across units

Why does reviewing a plan in CI specifically for -/+ replacements matter as part of this checklist Because -/+ always indicates a syntax error in the configuration Because a forced replacement destroys and recreates a resource, which can mean real data loss or downtime if an unreviewed plan is approved and applied Because -/+ only appears when using Terragrunt, never plain Terraform Because CI cannot run terraform plan without first seeing a -/+ line

What does this real, proven error demonstrate about lifecycle.prevent_destroy?

$ terraform destroy -input=false -auto-approve
...
Error: Instance cannot be destroyed

Resource local_file.critical has lifecycle.prevent_destroy set, but the
plan calls for this resource to be destroyed.

It only warns; the destroy still proceeds after the warning It refuses the plan outright — no flag bypasses it short of removing or changing the lifecycle block itself, a change that is its own reviewable diff It only applies to resources created in the current apply, not existing ones It silently converts the destroy into an in-place update instead

This is the CORRECT lock ID against a genuinely active lock, and it still fails this way. What does this real, proven result mean for a force-unlock runbook?

$ cat .terraform.tfstate.lock.info
{"ID":"564d6b33-...","Operation":"OperationTypeApply", ...}
$ terraform force-unlock -force 564d6b33-...
Failed to unlock state: LocalState not locked

force-unlock is broken and should never be used on any backend Against local state specifically, force-unlock cannot release a lock a different process holds — terraform force-unlock --help confirms this directly ("Local state files cannot be unlocked by another process"); force-unlock is a remote-backend recovery tool, and the local fix is stopping the stuck process or removing the lock file The lock ID must always be entered in uppercase for force-unlock to work This only happens when time_sleep is used to create the lock