Skip to content
Terraform & Terragrunt for AWS — From Zero to Hero

Secrets and Credential Management

The idea in one sentence

Section titled “The idea in one sentence”

Provider authentication should come from a role or an identity mechanism rather than a literal key typed into a file, and even pulling an application secret from a real secrets manager at apply time does not fully keep it out of Terraform state.

Never hardcode credentials in a provider block

Section titled “Never hardcode credentials in a provider block”

It is possible to authenticate the AWS provider with a literal access_key and secret_key written directly into the provider "aws" block or into a .tfvars file. It also works, in the narrow sense that terraform plan runs. It is still wrong:

# Anti-pattern — never do this
provider "aws" {
  region     = "us-east-1"
  access_key = "AKIAIOSFODNN7EXAMPLE"
  secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

A key written in a .tf file gets committed to git the first time someone forgets it is there, and it lives on in every clone and every commit in history from that point forward, whether or not it is later deleted from the current revision. The fix is to give the provider no credentials at all in configuration, and let it resolve them from the environment it is running in:

# The provider block declares no credentials.
# Authentication comes from the environment: local env vars,
# an EC2/CI instance role, or OIDC federation in CI.
provider "aws" {
  region = "us-east-1"
}

Locally, that usually means environment variables or a named profile from aws configure. In CI, it means the OIDC-federated role assumption covered in the CI/CD lesson — aws-actions/configure-aws-credentials populates the standard AWS environment variables before Terraform ever runs, so the provider block finds credentials without a single key ever being written down anywhere.

Pulling a secret in via a data source, not a variable

Section titled “Pulling a secret in via a data source, not a variable”

A separate question from provider authentication is application secrets — a database password, a third-party API key — that a resource needs as an argument. Defining that secret as a Terraform variable makes the variable declaration the thing you have to treat as the source of truth, which means someone still has to type the real value into a tfvars file, a CI secret, or a prompt at apply time.

The better pattern reads the current value directly from a real secrets manager through a data source, at apply time:

data "aws_secretsmanager_secret_version" "api_key" {
  secret_id = "prod/third-party/api-key"
}


resource "aws_lambda_function" "webhook_handler" {
  function_name = "webhook-handler"
  runtime       = "nodejs20.x"
  handler       = "index.handler"


  environment {
    variables = {
      API_KEY = data.aws_secretsmanager_secret_version.api_key.secret_string
    }
  }
}

This is not a free pass, and it is worth being precise about why. The State Management module already covered that marking a variable sensitive = true only redacts it from CLI output — the value still lands in terraform.tfstate in plaintext. Reading a secret through a data source does not fully avoid that either: data source results are cached in state too, so the resolved secret_string still ends up recorded there, the same as a sensitive variable would. What the data-source pattern actually improves is the secret’s lifecycle everywhere else: the real value is never typed into a variable, never sits in a .tfvars file someone might commit by accident, and rotating it in Secrets Manager needs no Terraform change at all — the next apply simply reads whatever is current. Secrets Manager is the source of truth; Terraform just reads from it. Protecting the state backend, covered in the State Management module, is still the part that cannot be skipped.

Keeping the secret out of state entirely: ephemeral and write-only

Section titled “Keeping the secret out of state entirely: ephemeral and write-only”

Everything above was true for years, and the “data source results are cached in state too” caveat was simply something you lived with. Terraform 1.10 and 1.11 closed that gap with two features that work together.

An ephemeral resource is read fresh during every operation and its result is never written to state or to the plan file. Ephemeral resources are declared with the ephemeral block type rather than data:

# Read at plan and apply time, never persisted to state or the plan file
ephemeral "aws_secretsmanager_secret_version" "db_password" {
  secret_id = "prod/database/password"
}

A value that is never stored is also a value Terraform cannot compare against next run, so an ephemeral result cannot be assigned to an ordinary resource argument. That is what write-only arguments are for. A provider marks certain arguments as write-only, named with a _wo suffix and paired with a _wo_version argument; Terraform sends the value to the provider during apply and then forgets it:

resource "aws_db_instance" "main" {
  identifier        = "app-prod"
  engine            = "postgres"
  instance_class    = "db.t3.micro"
  allocated_storage = 20
  username          = "app"


  # The password reaches AWS but is never recorded in state
  password_wo         = ephemeral.aws_secretsmanager_secret_version.db_password.secret_string
  password_wo_version = 1
}

Because the value itself is not in state, Terraform has no way to notice that it changed. That is the job of password_wo_version: bump the integer when you rotate the secret, and the provider sends the new value on the next apply. Leave it unchanged and Terraform leaves the password alone.

Two constraints are worth knowing before you reach for this. Write-only arguments require Terraform 1.11 or later and provider support — the argument has to exist on that specific resource type, so check the provider documentation rather than assuming _wo works everywhere. And ephemeral values may only flow into a narrow set of places: write-only arguments, other ephemeral resources or variables, provider configuration, provisioner blocks, and outputs that are themselves marked ephemeral. Assigning one to an ordinary argument or a normal output is an error, which is the language enforcing the guarantee rather than getting in your way.

Where a resource supports it, this combination is strictly better than the data-source pattern: the secret is read from the real source of truth at apply time and never recorded in state. Where it is not supported yet, the data-source pattern plus a protected backend remains the right answer.

Proving both write-only arguments really exist, against the real aws provider 6.66.0 schema

Section titled “Proving both write-only arguments really exist, against the real aws provider 6.66.0 schema”

password_wo/password_wo_version on aws_db_instance, and secret_string_wo/secret_string_wo_version on aws_secretsmanager_secret_version, both really exist on the aws provider’s real schema — proven with a real terraform validate against provider 6.66.0 (mock credentials, no real AWS calls needed for validate):

variables.tf
variable "db_password" {
  type      = string
  ephemeral = true
  sensitive = true
}


variable "api_key_value" {
  type      = string
  ephemeral = true
  sensitive = true
}
# terraform.tfvars — harness fixture only, so this lesson's proof can run
# offline; a real repo never commits secret values this way (see the
# .gitignore checklist below) — supply real values via TF_VAR_* instead,
# proven later in this section.
db_password   = "mock-secret-value"
api_key_value = "mock-api-key-value"
main.tf
resource "aws_db_instance" "main" {
  identifier          = "app-prod"
  engine              = "postgres"
  instance_class      = "db.t3.micro"
  allocated_storage   = 20
  username            = "app"
  password_wo         = var.db_password
  password_wo_version = 1
  skip_final_snapshot = true
}


resource "aws_secretsmanager_secret" "api_key" {
  name = "prod/third-party/api-key"
}


resource "aws_secretsmanager_secret_version" "api_key" {
  secret_id                = aws_secretsmanager_secret.api_key.id
  secret_string_wo         = var.api_key_value
  secret_string_wo_version = 1
}
$ terraform validate
Success! The configuration is valid.

Real terraform plan output (mock credentials, pure resource creation) shows exactly how the provider represents a write-only argument — it never even reaches the human-readable diff as a value:

  # aws_secretsmanager_secret_version.api_key will be created
  + resource "aws_secretsmanager_secret_version" "api_key" {
      + has_secret_string_wo     = (known after apply)
      + secret_string_wo         = (write-only attribute)
      + secret_string_wo_version = 1
    }

And in terraform show -json of the saved plan, resource_changes[].change.after.secret_string_wo is null with after_sensitive.secret_string_wo set to true — Terraform genuinely omits the value from the plan’s machine-readable form, not just the pretty-printed CLI output:

"after": { "secret_string_wo": null, "secret_string_wo_version": 1 },
"after_sensitive": { "secret_string_wo": true }

has_secret_string_wo (a real, provider-computed boolean) is how a later plan can tell “a write-only value was set” without ever storing what it was — the provider persists that one bit, not the secret.

The gotcha this course found: an ephemeral variable’s default still leaks

Section titled “The gotcha this course found: an ephemeral variable’s default still leaks”

Marking a variable ephemeral = true (Terraform 1.10) keeps it out of ordinary arguments and outputs, but that does not mean writing its value as a literal default = in the .tf source is safe — proven for real:

variable "db_password" {
  type      = string
  ephemeral = true
  sensitive = true
  default   = "mock-secret-value" # do not do this
}
$ terraform show -json tfplan | grep -o "mock-secret-value"
mock-secret-value

Searching the resulting plan.json for the literal string finds it twice: .configuration.root_module.variables.db_password.default and .variables.db_password.value. Neither is a bug in ephemeral/write-only — the configuration block of show -json is documented as an echo of the source configuration as written, and a hardcoded default is exactly what was written. ephemeral = true stops the value from being persisted to state or flowing into an ordinary output; it says nothing about a literal you typed directly into version-controlled HCL, which is the same “never hardcode a secret in a .tf file” rule from the top of this lesson, applying here too. Removing the default and supplying the value only via TF_VAR_db_password (an environment variable, never committed) at plan time closes the gap completely — proven with the same search against the resulting plan JSON finding nothing at all:

$ TF_VAR_db_password="mock-secret-value" terraform plan -out=tfplan
$ terraform show -json tfplan | grep -o "mock-secret-value"
# (no output — the literal appears nowhere in the plan JSON)

A .gitignore checklist for a Terraform/Terragrunt repository

Section titled “A .gitignore checklist for a Terraform/Terragrunt repository”

A short, deliberate .gitignore prevents most of the accidents above from ever reaching git in the first place:

# Terraform / Terragrunt .gitignore


# Never commit state — it can contain plaintext secrets and is the
# canonical source of truth; committing it invites drift and leaks.
*.tfstate
*.tfstate.backup


# Local provider and module cache, regenerated by `terraform init`.
.terraform/


# May contain sensitive values depending on the file — judge case by
# case rather than blanket-ignoring every *.tfvars in the repo.
*.tfvars


# Terragrunt's local working-directory cache, regenerated automatically.
.terragrunt-cache/

A few of these deserve a one-line reason each:

  • *.tfstate and *.tfstate.backup — never commit state. Beyond the size and merge-conflict problems, state can contain plaintext secrets, as covered above, and it duplicates the job a remote backend already does properly.
  • .terraform/ — this is init’s local cache of downloaded providers and modules. It is large, machine-specific, and fully reproducible from .terraform.lock.hcl, so there is nothing worth versioning here.
  • *.tfvars — treat this one case by case rather than blanket-ignoring it. Some teams deliberately commit a non-sensitive environment.tfvars that just sets instance sizes or region names; the rule is “does this specific file contain a secret,” not “all .tfvars files are always secret.”
  • .terragrunt-cache/ — Terragrunt’s equivalent of .terraform/, one per unit, regenerated on every run. It never needs to be committed, and committing it by accident tends to bloat a repository fast.
flowchart LR
  bad["Hardcoded credential in a .tf / .tfvars file"] -->|committed by accident| leak["Leaked secret in git history"]
  sm["Secrets Manager: source of truth"] -->|data source at apply time| attr["Resource attribute"]
  attr -->|still recorded| state["terraform.tfstate"]
  sm -->|ephemeral block| eph["Ephemeral value, never persisted"]
  eph -->|write-only argument| provider["Sent to AWS at apply, not stored"]
A hardcoded credential in a committed file versus a secrets manager as the real source of truth, read at apply time

Mental Model

Section titled “Mental Model”

Every technique in this lesson answers the same underlying question at increasing strength: “where does this secret value exist, and for how long.” A hardcoded credential exists forever, in git history, the weakest answer. A variable exists as long as any .tfvars/CI-secret/state copy of it exists. A data source narrows where the value originates (a real secrets manager, not a typed-in value) but not where it ends up — proven in this lesson’s own baseline content, it still lands in state. ephemeral + write-only is the strongest answer: the value exists only in memory, only during one operation, sent directly to the provider and never durably recorded anywhere Terraform controls — proven above with a real null in change.after and a real has_secret_string_wo boolean standing in for it. The gotcha proven above (a literal default on an ephemeral variable still leaking into show -json) is the same lesson in miniature: the mechanism can be perfect and a literal you typed can still defeat it.

Trade-offs

Section titled “Trade-offs”
ChoiceBenefitCost
Environment-resolved provider credentials (OIDC, instance role)No standing key exists to leak from a .tf/.tfvars fileRequires infrastructure setup (OIDC provider, IAM role) before it works at all
data source instead of a variable for a secretCentralizes rotation; the real value never sits in .tfvarsProven in this lesson’s baseline content: the resolved value still lands in state — a protected backend remains mandatory
ephemeral + write-only (password_wo, secret_string_wo)Proven above: the value is genuinely absent from both state and the plan’s change.afterRequires Terraform 1.10/1.11+ and per-resource provider support — not every argument on every resource has a _wo twin
An ephemeral variable with no default, supplied only via TF_VAR_*Proven above: the value appears nowhere in the plan JSON at allEasy to accidentally undo by adding a default = literal for local testing convenience and forgetting to remove it — proven above, that alone reintroduces the leak

Common Misconceptions

Section titled “Common Misconceptions”
  • “Marking a variable sensitive = true keeps it out of state.” Covered in this lesson’s own baseline content and the State Management module: sensitive only redacts CLI output. The value still lands in terraform.tfstate in plaintext.
  • “A data source secrets-manager lookup fully replaces the need for ephemeral/write-only.” Proven above by design of the data-source pattern: data source results are cached in state exactly like a resource attribute — ephemeral exists specifically because data does not solve this.
  • “ephemeral = true on a variable makes any value assigned to it safe, including a hardcoded default.” Proven above with a real, searchable string in plan.json: a literal default on an ephemeral variable is echoed in show -json’s configuration and top-level variables blocks regardless of the ephemeral flag. ephemeral protects the state/output flow, not a literal typed into version-controlled HCL — that is a separate, older rule this lesson already covers.
  • “Write-only arguments work on any resource, since the feature exists in the provider.” Proven above by direct schema validation: password_wo exists on aws_db_instance and secret_string_wo exists on aws_secretsmanager_secret_version specifically, because each resource’s schema had to add it — the feature existing in Terraform core does not mean every argument on every resource has a _wo counterpart.

Reading AI-generated code

Section titled “Reading AI-generated code”

When reviewing AI-generated Terraform that handles secrets, check for:

  • A default = literal on a variable marked ephemeral = true or sensitive = true — proven above, this defeats the entire purpose for the ephemeral case (leaks into show -json) and is a values-review smell either way.
  • password_wo/_wo-style arguments used without checking the specific resource’s schema — proven above via terraform validate, the argument must exist on that exact resource; a plausible-sounding _wo name that isn’t real fails validation immediately, which is the fast way to catch a hallucinated argument name.
  • A password_wo with no matching password_wo_version bump in a PR that is supposed to rotate a credential — since the value itself isn’t tracked in state, an unchanged version number means the provider silently keeps the old password, even though the ephemeral source now returns a new one.
  • An ephemeral value assigned to an ordinary (non-write-only, non-ephemeral) argument or a normal output — the language rejects this outright, so this is a Terraform-level type error, not a subtle runtime leak, but still worth recognizing on sight rather than being surprised by the error.
  • Application code that reads a secret from an aws_secretsmanager_secret_version data source (not ephemeral) and treats it as equivalent to write-only handling — proven above, data results are cached in state; only the ephemeral/write-only pairing avoids that.

🐛 Spot the bug

variable "db_password" {
type      = string
ephemeral = true
sensitive = true
default   = "changeme-in-ci"
}


resource "aws_db_instance" "main" {
# ...
password_wo         = var.db_password
password_wo_version = 1
}
Show the answer

The default = "changeme-in-ci" looks like an obviously-fake placeholder, which is exactly what makes this bug easy to miss in review — but it is still a literal string committed to version control, and proven above, ephemeral = true does not stop a literal default from being echoed into terraform show -json’s configuration and variables blocks. If this “placeholder” is ever left as the real default in an environment where nobody remembers to override it with -var/TF_VAR_db_password, the database gets provisioned with a literal, publicly-visible password string — from a variable that looked ephemeral-safe. The fix, proven above, is no default at all: supply the value only via TF_VAR_db_password or -var, which keeps it out of the plan JSON entirely.

What is current best practice for authenticating the AWS provider in a CI pipeline
Why is pulling a secret from a secrets manager via a data source usually better than defining it as a Terraform variable
What nuance still applies even after switching a secret from a Terraform variable to a secrets manager data source
A resource supports a password_wo write-only argument fed from an ephemeral block. What does password_wo_version do
This literal was found in a real plan.json for a variable declared ephemeral = true with default = "mock-secret-value". What does this prove?
$ terraform show -json tfplan | grep -o "mock-secret-value"
mock-secret-value
Which of these belongs in a Terraform/Terragrunt repository gitignore, and why