Skip to content
Terraform & Terragrunt for AWS — From Zero to Hero

CI/CD for Infrastructure

The idea in one sentence

Section titled “The idea in one sentence”

A standard infrastructure pipeline runs plan automatically on every pull request for a human to review, and reserves apply for after merge to the main branch — using short-lived, OIDC-federated AWS credentials instead of a long-lived access key stored in CI.

Plan on every pull request, apply only after merge

Section titled “Plan on every pull request, apply only after merge”

The pattern that most teams converge on mirrors application CI/CD, with one important difference in what each stage actually does:

  • Pull request opened or updated → run terraform plan (or terragrunt run --all plan for a multi-unit Terragrunt setup) and post the resulting diff as a comment on the pull request. Nothing in AWS changes yet — this is a dry run whose only job is to make the proposed change visible to a reviewer before anyone approves it.
  • Merge to main → run terraform apply (or terragrunt run --all apply) from the pipeline itself, not from anyone’s laptop. For production specifically, this step is frequently gated behind a manual approval — a named reviewer or team has to click “approve” in the CI system before the apply job is allowed to run, even though the plan was already reviewed at the pull-request stage.
name: terraform


on:
  pull_request:
    paths:
      - 'infra/**'
  push:
    branches:
      - main
    paths:
      - 'infra/**'


permissions:
  id-token: write
  contents: read
  pull-requests: write


jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::111122223333:role/github-actions-terraform
          aws-region: us-east-1
      - uses: hashicorp/setup-terraform@v4
      - run: terragrunt run --all plan -no-color | tee plan.txt
      - run: gh pr comment ${{ github.event.pull_request.number }} --body-file plan.txt
        env:
          GH_TOKEN: ${{ github.token }}


  apply:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v7
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::111122223333:role/github-actions-terraform
          aws-region: us-east-1
      - uses: hashicorp/setup-terraform@v4
      - run: terragrunt run --all apply --non-interactive

The environment: production line is what supplies the manual approval gate — GitHub Actions holds the apply job until someone with permission on that environment approves the run. Notice that neither job ever checks out a stored AWS access key; both assume a role through OIDC, covered next.

Why a plan deserves sharper review than a typical code diff

Section titled “Why a plan deserves sharper review than a typical code diff”

A pull request against application code is reviewed for logic, tests, and style — worst case, a bug ships and gets rolled back. An infrastructure apply can do something a code deploy cannot: destroy real, stateful AWS resources and rebuild them from scratch, sometimes because of a single one-line change.

Terraform calls this a forced replacement. Certain arguments on certain resources cannot be updated in place — changing them means the provider has no API call for “modify this attribute,” only “destroy and create a new one.” An RDS instance’s identifier, an EBS volume’s availability_zone, or a resource’s own name in some providers all fall into this category. When that happens, terraform plan marks the resource with -/+ instead of the in-place ~:

# terraform plan output — this is the line to stop and read carefully
  # aws_db_instance.main must be replaced
-/+ resource "aws_db_instance" "main" {
      ~ identifier     = "app-db" -> "app-db-renamed" # forces replacement
        engine         = "postgres"
        instance_class = "db.t3.micro"
    }

A -/+ on a stateful resource like a database or a persistent volume can mean real data loss or real downtime the moment someone clicks approve — the old resource is gone before the new one exists. This is exactly why reviewing an infrastructure plan cannot be a skim. A reviewer has to specifically scan for -/+ lines and ask whether that replacement is expected and safe, not just read the plan the way they would read a code diff for style and logic.

Short-lived OIDC credentials instead of a stored access key

Section titled “Short-lived OIDC credentials instead of a stored access key”

Older pipelines stored a long-lived AWS access key and secret as CI secrets and exported them as environment variables before every run. That key works for plan and apply equally well, which is exactly the problem: if it ever leaks — from a misconfigured log, a compromised dependency, or a workflow file that echoes environment variables — it keeps working until someone manually rotates or revokes it.

Current best practice replaces that key with OIDC federation. GitHub Actions can present a short-lived, cryptographically signed identity token to AWS for each workflow run. An IAM OIDC identity provider in the AWS account trusts GitHub’s token issuer, and an IAM role’s trust policy allows only specific repositories and branches to assume it. aws-actions/configure-aws-credentials exchanges that token for temporary AWS credentials scoped to the assumed role, valid for the lifetime of the job and nothing beyond it.

permissions:
  id-token: write
  contents: read


jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7


      - name: Assume AWS role via OIDC
        uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::111122223333:role/github-actions-terraform
          aws-region: us-east-1


      - name: Terraform plan
        run: terraform plan

No AWS access key ever needs to exist as a stored CI secret at all — there is nothing long-lived to leak, rotate, or forget about. permissions: id-token: write is what allows the workflow to request that identity token in the first place; without it, the OIDC exchange has nothing to present to AWS.

Per GitHub’s own docs (docs-sourced, not run — no AWS account exists to create this in): “For the provider URL: Use https://token.actions.githubusercontent.com” and “For the ‘Audience’: Use sts.amazonaws.com”, then “Edit the trust policy, adding the sub field to the validation conditions” on the IAM role — docs.github.com/en/actions/how-tos/secure-your-work/security-harden-deployments/oidc-in-aws. The sub claim is what a trust policy scopes to a specific repository/branch (e.g. repo:my-org/my-infra:ref:refs/heads/main) — without it, any repository in the GitHub organization that can mint an OIDC token for sts.amazonaws.com could assume the role.

Linting the workflow itself with actionlint

Section titled “Linting the workflow itself with actionlint”

The workflow YAML above is proven by running actionlint (rhysd/actionlint) against it in a one-shot container — the same tool a pre-commit hook or a separate CI job would run on every workflow change:

$ docker run --rm -v "$PWD:/repo" -w /repo rhysd/actionlint:latest -color

Real output against both workflows in this lesson (the plan/apply pipeline and the drift-detection job below): no output at all, and a 0 exit code — actionlint prints nothing when it finds no problems, which is itself the proof there are no undefined ${{ }} expressions, invalid runs-on values, unpinned uses: refs missing a version, or shellcheck-level issues in the embedded run: scripts.

Applying the exact plan that was reviewed, not a fresh one

Section titled “Applying the exact plan that was reviewed, not a fresh one”

A subtle gap in the two-job pipeline above: the apply job runs terragrunt run --all apply --non-interactive, which computes its own plan at apply time. If anything changed between the PR being reviewed and the merge landing — a teammate’s parallel merge, a provider registry update, drift in the account — the plan a human approved and the plan that actually gets applied are not provably the same plan. terragrunt run --all plan --out-dir <dir> (confirmed via terragrunt run --help: --out-dir value Directory to store plan files.) closes that gap by writing one saved plan file per unit, which terragrunt run --all apply --out-dir <dir> then applies directly instead of re-planning:

$ terragrunt run --all plan --out-dir plan-output --non-interactive
...
Saved the plan to:
.../plan-output/tfplan.tfplan
To perform exactly these actions, run the following command to apply:
    terraform apply ".../plan-output/tfplan.tfplan"


❯❯ Run Summary  1 units
   Succeeded    1


$ terragrunt run --all apply --out-dir plan-output --non-interactive
local_file.config: Creating...
local_file.config: Creation complete after 0s [id=f572d396fae9206628714fb2ce00f72e94f2258f]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.


❯❯ Run Summary  1 units
   Succeeded    1

Real Terragrunt 1.1.0 output, proven locally against a local backend: the second command applied the file the first command saved — no re-plan, no prompt. In CI, the plan job uploads plan-output/ as a build artifact keyed to the commit SHA, and the apply job downloads that same artifact before running apply --out-dir, so the thing that gets applied is provably the thing a reviewer read.

Gating apply with -detailed-exitcode, and a scheduled drift check

Section titled “Gating apply with -detailed-exitcode, and a scheduled drift check”

terraform plan -detailed-exitcode (terraform plan -help: “Return detailed exit codes when the command exits… 0 - Succeeded, diff is empty (no changes)”) changes exit code 0 to mean no changes, 1 to mean an error, and 2 to mean changes present. A pipeline step can use that to skip an unnecessary apply entirely, or — run on a schedule instead of on push — to detect drift with no apply step at all:

All three real exit codes, proven locally with a saved plan file (-out=tfplan) so the artifact-reuse pattern above and the exit-code check below are the same command:

# 2 — changes present (aws_security_group + aws_security_group_rule, mock credentials)
$ terraform plan -input=false -out=tfplan -detailed-exitcode
...
Plan: 2 to add, 0 to change, 0 to destroy.
Saved the plan to: tfplan
$ echo $?
2


# 1 — a real error (a data source AWS actually rejects, even with mock credentials)
$ terraform plan -input=false -detailed-exitcode
╷
│ Error: reading EC2 VPC: operation error EC2: DescribeVpcs, https response
│ error StatusCode: 401, ... api error AuthFailure: AWS was not able to
│ validate the provided access credentials
│
│   with data.aws_vpc.missing,
│   on main.tf line 26, in data "aws_vpc" "missing":
╵
$ echo $?
1


# 0 — no changes (same config applied once already, local_file resource)
$ terraform plan -input=false -detailed-exitcode
...
Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.
$ echo $?
0

Three separate real runs, three different real exit codes — this is what a script branching on -detailed-exitcode actually has to distinguish (the SpotTheBug below shows the common way that distinction gets lost). Note the 1 case: mock credentials (skip_credentials_validation) satisfy plan’s own auth check, but a data source that has to actually read from the AWS API (aws_vpc, DescribeVpcs) still makes a real HTTPS call and still fails — mock credentials make plan possible offline for resources that don’t need a read-back, not universally credential-free.

terraform show -json of a saved plan file

Section titled “terraform show -json of a saved plan file”

The saved tfplan from the -out run above is a binary file — terraform show -json tfplan renders it as the same documented, versioned JSON schema the-state-file.mdx uses for state (format_version, separate from state’s own version field), except resource_changes[].change.actions describes a proposed change instead of resources[].instances[].attributes describing an applied one. Real output (trimmed) from the tfplan above:

{
  "format_version": "1.2",
  "terraform_version": "1.16.4",
  "resource_changes": [
    {
      "address": "aws_security_group.web",
      "change": { "actions": ["create"] }
    },
    {
      "address": "aws_security_group_rule.https",
      "change": { "actions": ["create"] }
    }
  ]
}

resource_changes[].change.actions is exactly the field policy-as-code-and-static-analysis.mdx runs OPA/conftest rego policies against — a policy that wants to block a delete-then-create replacement, or forbid a specific resource type from ever being destroyed, reads this array rather than trying to diff two state snapshots itself. It is the same -json flag as terraform show -json on state (no plan file argument), just pointed at a plan file instead of the current state.

name: drift-detection


on:
  schedule:
    - cron: '0 6 * * *' # every day at 06:00 UTC
  workflow_dispatch:


permissions:
  id-token: write
  contents: read


jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::111122223333:role/github-actions-terraform
          aws-region: us-east-1
      - uses: hashicorp/setup-terraform@v4
      - name: Check for drift
        id: plan
        run: |
          set +e
          terragrunt run --all plan -detailed-exitcode -no-color
          echo "exitcode=$?" >> "$GITHUB_OUTPUT"
      - name: Fail if drift detected
        if: steps.plan.outputs.exitcode == '2'
        run: |
          echo "::error::Drift detected — real infrastructure no longer matches configuration"
          exit 1

This job never runs apply — its only job is to fail loudly (and, in a real setup, page someone or post to a chat channel) the moment plan’s exit code says the account has drifted from what’s committed, catching drift on a schedule instead of only at the next deliberate apply.

flowchart LR
  pr["Pull request"] -->|triggers| plan["terragrunt run --all plan"]
  plan -->|posted as| comment["PR comment for review"]
  comment -->|approved and merged| main["Merge to main"]
  main -->|triggers, gated by approval| apply["terragrunt run --all apply"]
  oidc["GitHub OIDC provider"] -->|short-lived credentials| plan
  oidc -->|short-lived credentials| apply
A pull request triggers plan for review; merge to main triggers apply behind an approval gate; OIDC supplies short-lived credentials to both

Mental Model

Section titled “Mental Model”

An infrastructure pipeline has exactly two jobs, and confusing them is the root cause of most incidents: compute a plan for a human (fast, safe, read-only against AWS — nothing changes) and execute an already-reviewed plan (the only step allowed to touch real resources, and only after both automated gates — -detailed-exitcode, policy checks — and a human gate have passed). Treat every other mechanism in this lesson as serving one of those two jobs: OIDC exists so the execute job never needs a standing credential; --out-dir exists so execute runs the exact bytes compute produced instead of re-deriving them; the scheduled drift job is the compute job running with no execute job behind it at all, purely to surface when reality has quietly diverged from the plan.

Trade-offs

Section titled “Trade-offs”
ChoiceBenefitCost
OIDC federation instead of a stored access keyNo long-lived AWS credential exists anywhere to leak or forget to rotateRequires one-time IAM OIDC provider + trust-policy setup per AWS account, and only works from a CI system that can present an OIDC token
--out-dir plan/apply artifact reuseApply is provably the plan a human reviewed, not a fresh re-planAn extra artifact-upload/download round trip between the plan job and the apply job, and the artifact must be treated as sensitive (a plan can contain values a data source resolved)
-detailed-exitcode gatingSkips a no-op apply, and turns a scheduled plan into a drift alarm with no apply step at allExit code 1 (a real error) and exit code 2 (real changes) both need explicit handling — treating any non-zero exit as “changes” silently swallows real plan errors
Manual approval environment: production gateA named human must explicitly approve before any production resource is touched, even after every automated check passesAdds latency to every production change, and is only as good as the reviewers actually reading the plan rather than reflexively clicking approve

Common Misconceptions

Section titled “Common Misconceptions”
  • “Running apply again in the apply job re-produces the same plan a reviewer saw on the pull request.” Proven above: without --out-dir, the apply job computes an independent plan at apply time — if anything changed in the account or the module registry between review and merge, that plan can differ from the one a human actually read.
  • “-detailed-exitcode returning non-zero always means something is wrong.” It returns 2 for a perfectly healthy “there are changes to apply” and 1 only for a genuine error — treating both the same way in a script either blocks a completely normal deploy or, worse, silently ignores a real failure.
  • “OIDC removes the need for an IAM role.” It removes the need for a long-lived access key — a real IAM role, with its own trust policy scoped to the right repository and branch, is still required and is the actual authorization boundary.
  • “A scheduled drift-detection job needs an apply step to be useful.” The job proven above never applies anything — its entire value is plan -detailed-exitcode failing loudly on a schedule, which catches drift long before the next deliberate change would have surfaced it.

Reading AI-generated code

Section titled “Reading AI-generated code”

When reviewing an AI-generated CI/CD workflow for Terraform/Terragrunt, check for:

  • A stored AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY secret instead of aws-actions/configure-aws-credentials with role-to-assume — the giveaway of a long-lived key that should be OIDC federation instead.
  • An apply job with no environment: (or equivalent required-reviewers gate) on the job — nothing stops a merged PR from applying to production unattended.
  • terraform apply (no -input=false, no saved plan file) run directly in CI instead of applying a plan artifact produced by an earlier plan step — this is exactly the gap the --out-dir proof above closes.
  • A drift-detection or CI plan step that checks if: failure() instead of a captured -detailed-exitcode value — failure() cannot distinguish “there are changes” (exit 2) from “plan actually errored” (exit 1).
  • Missing permissions: id-token: write on a workflow that otherwise looks like it uses OIDC — without it, the token exchange has nothing to present and the job fails at the credentials step, not silently.

🐛 Spot the bug

- name: Check for drift
id: plan
run: terragrunt run --all plan -detailed-exitcode -no-color
continue-on-error: true
- name: Fail if drift detected
if: steps.plan.outcome == 'failure'
run: exit 1
Show the answer

-detailed-exitcode makes the step’s shell command exit non-zero for two different reasons: exit code 2 means real changes exist (the case this job wants to catch), but exit code 1 means the plan itself failed — a bad credential, a syntax error, a provider timeout. Both make continue-on-error: true mark steps.plan.outcome as 'failure', so this workflow cannot tell “infrastructure drifted” apart from “the drift check itself is broken,” and a broken check gets reported as drift every single day. The fix, proven above, is to capture the literal exit code with echo "exitcode=$?" >> "$GITHUB_OUTPUT" under set +e (no continue-on-error needed at all) and branch on steps.plan.outputs.exitcode == '2' specifically.

In a standard IaC CI/CD pipeline, what should trigger terraform plan versus terraform apply
Why does an infrastructure plan need more careful review than a typical application code diff
What does this real exit code of 2 mean, and how does it differ from 1?
$ terraform plan -detailed-exitcode -input=false
...
Plan: 2 to add, 0 to change, 0 to destroy.
$ echo $?
2
Why is OIDC federation preferred over a long-lived AWS access key stored as a CI secret
What is the real bug in this drift-detection step, proven in this lesson's SpotTheBug?
- name: Check for drift
  id: plan
  run: terragrunt run --all plan -detailed-exitcode -no-color
  continue-on-error: true
- if: steps.plan.outcome == 'failure'
  run: exit 1
What real problem does terragrunt run --all plan --out-dir <dir> followed by run --all apply --out-dir <dir> (same directory) solve, proven in this lesson with a real Terragrunt 1.1.0 run?