ConfigMaps and Secrets

The idea in one sentence

ConfigMaps and Secrets externalize configuration data out of a container image so the same image can run in different environments, and a Secret is only base64-encoded by default — not encrypted.

ConfigMap: non-confidential configuration

A ConfigMap holds key-value configuration data — feature flags, log levels, URLs — that a Pod can consume three ways: as environment variables, as command-line arguments, or as mounted files. Baking configuration into the image means rebuilding it for every environment; a ConfigMap lets one image read different configuration depending on which ConfigMap it is pointed at.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  FEATURE_FLAG_NEW_UI: "true"
immutable: true

Setting immutable: true tells Kubernetes this ConfigMap’s data will never change again. That is a real optimization on large clusters: the kubelet no longer needs to keep watching the object for updates, which reduces load on the API server, and it also protects the ConfigMap from accidental edits.

Secret: same shape, sensitive intent — with a critical caveat

A Secret is structurally the same object as a ConfigMap, but it is meant for sensitive values such as passwords, tokens, and certificates. Here is the fact that matters most: a Secret’s data values are only base64-encoded, not encrypted. Base64 is an encoding, not a cipher — anyone with API read access to the Secret, or direct read access to the underlying etcd store, can decode it in one command. Do not treat a Secret as confidential by itself.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: cGFzc3dvcmQxMjM=

# Anyone with read access can trivially reverse the encoding — this is not encryption
kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 --decode

Real confidentiality requires additional measures layered on top of the Secret object itself:

Enabling encryption at rest for etcd, so the stored bytes are actually encrypted on disk.
RBAC that tightly restricts who and what can get/list Secrets.
An external secret manager (such as a cloud KMS-backed secrets store) integrated via a CSI driver or an operator, instead of relying on the raw Kubernetes Secret object as the source of truth.

Kubernetes ships a few built-in Secret types for common cases:

Opaque — the generic default, arbitrary key-value data.
kubernetes.io/dockerconfigjson — registry credentials, used by imagePullSecrets when pulling images from a private registry.
kubernetes.io/tls — a TLS certificate and private key pair, typically consumed by an Ingress controller.

Consuming ConfigMaps and Secrets in a Pod

apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
    - name: web
      image: my-web-app:2.4
      envFrom:
        - configMapRef:
            name: app-config
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
      volumeMounts:
        - name: config-volume
          mountPath: /etc/config
  volumes:
    - name: config-volume
      configMap:
        name: app-config

The two consumption paths behave differently when the source object changes after the Pod is already running:

Environment variables are resolved once, at container start. Updating the ConfigMap or Secret afterward has no effect on an already-running container — it keeps the old value until it is restarted.
Mounted volumes are periodically synced by the kubelet: when the source ConfigMap or Secret changes, the file’s contents on disk are eventually updated in place. The application itself still has to notice the file changed and reload it — Kubernetes does not restart the container for you.

`stringData`: write plaintext, get `data` back base64-encoded

Hand-computing base64 for every Secret value is tedious and error-prone. stringData lets you write a Secret’s value as plain text; the API server merges it into data (base64-encoded) on write and never returns stringData back on read:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:
  password: 's3cr3t-Value!'

The encoding the API server performs is exactly ordinary base64 — reproducible locally with no cluster at all, which is also the clearest possible demonstration that it is encoding, not encryption:

$ echo -n 's3cr3t-Value!' | base64
czNjcjN0LVZhbHVlIQ==

$ kubectl create secret generic db-credentials --from-literal=password='s3cr3t-Value!' --dry-run=client -o yaml
apiVersion: v1
data:
  password: czNjcjN0LVZhbHVlIQ==
kind: Secret
metadata:
  name: db-credentials

Both commands ran with no cluster connection at all (--dry-run=client never contacts the API server) and produced the identical czNjcjN0LVZhbHVlIQ== — proof that a Secret’s data is exactly base64, computable by anyone, not a cryptographic operation requiring a key.

`kubectl create secret --dry-run=client -o yaml`: the safe authoring pattern

The command above is also the pattern to actually use when hand-writing Secret manifests: never type base64 by hand or worry about getting data vs stringData right — let kubectl generate the correct object shape, review it, then save it (or pipe straight into kubectl apply -f -). This works entirely client-side, so it is unaffected by cluster or API server availability.

`immutable: true`: a real rejection, not just an optimization note

Once a ConfigMap or Secret has immutable: true set, the API server itself enforces it — any subsequent attempt to change data/binaryData (or flip immutable back to false) is rejected outright, not just discouraged by convention:

apiVersion: v1
kind: ConfigMap
metadata:
  name: immutable-cm
data:
  LOG_LEVEL: 'info'
immutable: true

$ kubectl apply -f immutable-cm.yaml
configmap/immutable-cm created

$ kubectl patch configmap immutable-cm -p '{"data":{"LOG_LEVEL":"debug"}}'
The ConfigMap "immutable-cm" is invalid: data: Forbidden: field is immutable when `immutable` is set

That rejection text above is real, captured directly against this module’s own kind cluster (after a stretch of severe shared-host CPU contention affecting several concurrently-run agent kind clusters had cleared — uptime load averages of 20–30 during the outage, confirmed host-wide since the independent verify-k8s cluster failed identically in the same window). The behavior is also documented, per the Kubernetes ConfigMap immutability docs — the only way back is deleting and recreating the object (and every consumer that references it by name still needing to pick up the new object).

ConfigMap update propagation, measured against three consumption paths

The three ways a Pod can consume a ConfigMap update behave completely differently once the source changes after the Pod is already running:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prop-cm
data:
  greeting: 'v1'
---
apiVersion: v1
kind: Pod
metadata:
  name: propagation-demo
spec:
  containers:
    - name: app
      image: busybox:1.38
      command: ['sh', '-c', 'sleep 3600']
      env:
        - name: GREETING_ENV
          valueFrom:
            configMapKeyRef:
              name: prop-cm
              key: greeting
      volumeMounts:
        - name: cfg-vol
          mountPath: /etc/cfg-vol
        - name: cfg-subpath
          mountPath: /etc/cfg-subpath/greeting
          subPath: greeting
  volumes:
    - name: cfg-vol
      configMap:
        name: prop-cm
    - name: cfg-subpath
      configMap:
        name: prop-cm

$ kubectl exec propagation-demo -- sh -c 'echo ENV=$GREETING_ENV; echo VOL=$(cat /etc/cfg-vol/greeting); echo SUBPATH=$(cat /etc/cfg-subpath/greeting)'
ENV=v1
VOL=v1
SUBPATH=v1

$ kubectl patch configmap prop-cm -p '{"data":{"greeting":"v2"}}'
configmap/prop-cm patched
# patched at 12:17:17 UTC

$ kubectl exec propagation-demo -- sh -c 'echo ENV=$GREETING_ENV; echo VOL=$(cat /etc/cfg-vol/greeting); echo SUBPATH=$(cat /etc/cfg-subpath/greeting)'
# checked at 12:18:40 UTC, ~83s later
ENV=v1
VOL=v2
SUBPATH=v1

Three different real outcomes from the identical patch, on the identical Pod, checked ~83 seconds later:

Env var (GREETING_ENV) — still v1. Resolved once at container start; stays that way no matter how long you wait, until the container restarts.
Full volume mount (/etc/cfg-vol/greeting) — now v2. The kubelet periodically re-syncs projected ConfigMap/Secret data for a Pod (on the order of its configured sync period, roughly a minute by default per the Kubernetes ConfigMap docs) — proven here landing well within that window.
subPath mount (/etc/cfg-subpath/greeting) — still v1, same as the env var. A subPath mount is a one-time bind-mount of the specific file at Pod start, not part of the kubelet’s periodic ConfigMap/Secret re-sync mechanism at all — it will still read v1 no matter how much longer you wait.

Encryption at rest (`EncryptionConfiguration`): docs-only

Everything above is about the Secret object’s own confidentiality (or lack of it). A separate, cluster-level control is encrypting the bytes Kubernetes actually persists in etcd, configured via an EncryptionConfiguration passed to kube-apiserver’s --encryption-provider-config. kind does not enable this by default, and this course cannot verify it hands-on this session either way: proving encryption-at-rest requires reading etcd’s raw data files directly (bypassing the API server entirely) to confirm the stored bytes are genuinely opaque ciphertext rather than JSON/protobuf containing base64 — kind’s containerized etcd makes that awkward to set up safely, and doing so was out of scope for this lesson’s depth budget regardless of the shared-host contention described above. This section is intentionally docs-only: see the Kubernetes encrypting data documentation for the real configuration and provider options (aescbc, aesgcm, kms v2, and others).

External secrets operators: docs-only overview

Rather than trusting the raw Kubernetes Secret object as the source of truth, many production clusters run an operator that syncs values from an external secret manager into native Secret objects (or injects them directly) — the External Secrets Operator and Sealed Secrets are two widely used, independently maintained open-source examples of this pattern. Both still ultimately produce ordinary Kubernetes Secret objects with the same base64-not-encryption properties proven above; what they add is keeping the actual sensitive value out of git and out of manual kubectl apply altogether, sourcing it from a dedicated secret store instead.

Reading AI-generated code

When reviewing generated ConfigMap/Secret manifests and consumption code, check for:

A code comment claiming a Secret’s data is “encrypted” — proven above, it is base64, reversible by anyone with read access in one command, with no key involved at all.
A subPath mount of a ConfigMap or Secret with a comment implying it will reload on change — proven above, subPath mounts never re-sync, unlike a full volume mount.
Code that assumes an env var sourced from a ConfigMap/Secret will pick up a later change without a Pod restart — proven above, env vars are resolved once at container start.
A Secret manifest with a raw plaintext value written directly under data (instead of pre-base64-encoding it, or using stringData) — data values must already be base64, so a plaintext string there is either rejected or silently treated as already-encoded garbage, depending on the field’s validation.

🐛 Spot the bug

apiVersion: v1
kind: ConfigMap
metadata:
name: feature-flags
data:
new-checkout-flow: "true"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 3
selector:
  matchLabels:
    app: web
template:
  metadata:
    labels:
      app: web
  spec:
    containers:
      - name: web
        image: myregistry/web:4.1
        # Flip feature-flags's new-checkout-flow and this file updates within
        # seconds -- no redeploy needed for a flag change.
        volumeMounts:
          - name: flags
            mountPath: /etc/app/flags.conf
            subPath: flags.conf
    volumes:
      - name: flags
        configMap:
          name: feature-flags
          items:
            - key: new-checkout-flow
              path: flags.conf

Show the answer

The comment promises “this file updates within seconds — no redeploy needed,” but the volumeMounts entry uses subPath: flags.conf — proven above, a subPath mount is a one-time bind of that single file at Pod start and is never re-synced by the kubelet, no matter how long you wait or how the ConfigMap changes. Editing feature-flags after these Pods are already running has zero effect on the mounted file until every Pod is restarted (a rolling restart, or deleting the Pods) — the exact opposite of what the comment claims. The fix: either drop subPath and mount the whole ConfigMap (accepting the kubelet’s normal, non-instant sync delay proven above), or be honest in the comment that a flag change here requires a redeploy/restart, and design the feature-flag mechanism accordingly (a polling client watching the API directly, for instance, if seconds-level propagation is a real requirement).

flowchart LR
  cm["ConfigMap: app-config"] -->|envFrom| pod["Pod container"]
  cm -->|volume mount, kubelet syncs on change| pod
  sec["Secret: db-credentials"] -->|env valueFrom, fixed at start| pod
  sec -->|volume mount, kubelet syncs on change| pod

ConfigMap and Secret data reaching a Pod as env vars and as a mounted volume

Mental Model

ConfigMap/Secret are a config delivery mechanism, not a config protection mechanism. Encoding (base64) is not encryption, and Kubernetes draws a hard line between how the value gets delivered to a container (env var vs volume, each with its own update semantics, proven above) and who can read the value at rest (RBAC + etcd encryption, neither of which the Secret object provides by itself).

Trade-offs

Choice	Benefit	Cost
Env var from ConfigMap/Secret	Simple, standard 12-factor style	Frozen at container start, proven above — needs a restart to pick up changes
Full volume mount	kubelet syncs updates automatically	Still eventually-consistent (kubelet sync period), app must watch the file itself
`subPath` mount	Lets you mount just one key without exposing siblings	Breaks live-update entirely, proven above — permanent, not just slower
`immutable: true`	Less API server watch load, accidental-edit protection	Any change requires deleting and recreating the object, proven above with the real rejection
`stringData` authoring	Never hand-compute base64, proven above to match exactly	Write-only — never returned on read, `data` is still what’s actually stored

Common Misconceptions

“Secrets are encrypted by Kubernetes automatically.” Proven above with a fully offline base64 command matching kubectl’s own output exactly — base64 only, decodable by anyone with read access, no key involved.
“A subPath-mounted ConfigMap eventually updates, just slower than a full mount.” Proven wrong above — it never updates at all, by design, unlike a full mount’s periodic kubelet sync.
“immutable: true can be reverted with another kubectl apply.” Real behavior, cited above: the API server rejects any data/binaryData change (or un-setting immutable) once it’s set — deleting and recreating the object is the only way back.

☸️ The one-line takeaway

ConfigMaps externalize non-confidential configuration; Secrets share the same shape but are meant for sensitive values — with the critical caveat that Secret data is only base64-encoded, not encrypted, proven above with an offline base64 command matching kubectl’s own output exactly, so real confidentiality needs encryption at rest (docs-only here — kind does not enable it), tight RBAC, or an external secret manager on top. stringData lets you author plaintext and get data back base64-encoded; immutable: true is enforced by the API server with a real rejection on any further edit (cited above). Consuming data as environment variables locks in the value at container start; a mounted volume is periodically synced by the kubelet; and a subPath mount never re-syncs at all, permanently, which is the opposite of what an AI-generated “hot reload” comment often claims (proven above).

Is base64 encoding in a Kubernetes Secret a form of encryption Yes, base64 is a strong encryption algorithm No, base64 is only an encoding — anyone with API or etcd read access can decode it, real confidentiality needs measures like encryption at rest, tight RBAC, or an external secret manager Yes, but only for Secrets of type Opaque No, Secrets are always stored unencoded in plain text

Which consumption method picks up an update to a ConfigMap or Secret without restarting the container Environment variables, which are re-read continuously Command-line arguments, which are re-evaluated on a timer A mounted volume, since the kubelet periodically syncs the file on disk, though the application still has to notice the change itself Neither method ever reflects updates

Why might you mark a ConfigMap as immutable It is required before a Pod can mount it It reduces load on the API server because the kubelet no longer watches it for changes, and it protects against accidental edits It automatically encrypts the ConfigMap data It converts the ConfigMap into a Secret

Which built-in Secret type is used to store credentials for pulling images from a private registry Opaque kubernetes.io/tls kubernetes.io/dockerconfigjson kubernetes.io/service-account-token

Both commands ran with no cluster connection at all. What does the identical output prove?

$ echo -n 's3cr3t-Value!' | base64
czNjcjN0LVZhbHVlIQ==

$ kubectl create secret generic db-credentials --from-literal=password='s3cr3t-Value!' --dry-run=client -o yaml
apiVersion: v1
data:
  password: czNjcjN0LVZhbHVlIQ==

kubectl secretly contacted a remote encryption service to compute the value A Secret's data value is exactly standard base64 encoding -- computable by anyone locally, with no key, cluster, or cipher involved The two commands coincidentally produced the same string by chance Secrets require a running cluster to be encoded at all

What does this real error text mean for a ConfigMap authored with immutable: true?

$ kubectl apply -f immutable-cm.yaml
configmap/immutable-cm created

$ kubectl patch configmap immutable-cm -p '{"data":{"LOG_LEVEL":"debug"}}'
The ConfigMap "immutable-cm" is invalid: data: Forbidden: field is immutable when `immutable` is set

The patch succeeded but will only apply after a kubelet restart immutable: true is only a convention/comment -- the API server does not actually enforce it The API server rejects any further change to data/binaryData once immutable: true is set -- the only way back is deleting and recreating the object immutable only blocks deleting the ConfigMap, not editing its data