Helm and Package Management
The idea in one sentence
Section titled “The idea in one sentence”Helm is the package manager for Kubernetes: it turns a directory of templated manifests into a versioned, parameterized Chart that gets installed as a tracked Release.
The problem: raw YAML does not parameterize or version well
Section titled “The problem: raw YAML does not parameterize or version well”A handful of Deployment, Service, and ConfigMap YAML files works fine for one environment. It falls apart the moment there is more than one: dev, staging, and prod need different replica counts, different resource limits, different image tags, and different hostnames, but the underlying shape of the manifests is identical. Copy-pasting the YAML three times and hand-editing each copy means every future change has to be applied three times, correctly, in sync. There is also no built-in notion of “this set of 12 manifests is one versioned unit” — kubectl apply treats each file independently, with no shared history and no easy way to package the whole set for someone else to install.
Helm solves both problems at once. It packages a set of related manifests into a single unit called a Chart, replaces hardcoded values with template placeholders, and tracks every install of that Chart as a Release with its own version history.
Anatomy of a Chart
Section titled “Anatomy of a Chart”A Chart is a directory with three key pieces:
mychart/├── Chart.yaml # name, version, and other chart metadata├── values.yaml # default configuration values└── templates/ ├── deployment.yaml ├── service.yaml └── configmap.yamlChart.yaml holds metadata (name, version, description). values.yaml holds the default configuration. Everything under templates/ is a Kubernetes manifest written with Go template syntax, where {{ .Values.xxx }} pulls a value in from values.yaml at render time:
# templates/deployment.yaml (excerpt)apiVersion: apps/v1kind: Deploymentmetadata: name: {{ .Release.Name }}-webspec: replicas: {{ .Values.replicaCount }} template: spec: containers: - name: web image: '{{ .Values.image.repository }}:{{ .Values.image.tag }}'replicaCount: 2image: repository: myregistry/web tag: '1.4.0'helm template (renders locally without installing) or helm install fills in those placeholders and produces plain Kubernetes YAML — Helm never invents a new deployment mechanism, it just generates the manifests kubectl would apply anyway.
Supplying environment-specific configuration
Section titled “Supplying environment-specific configuration”The same Chart deploys to dev, staging, and prod by overriding values.yaml at install time, either with a dedicated override file or individual --set flags:
# Render the manifests without installing, to sanity-check the outputhelm template myapp ./mychart -f values-prod.yaml
# Install into prod using a prod-specific values filehelm install myapp ./mychart -f values-prod.yaml
# Override a single value on the command line insteadhelm install myapp ./mychart --set replicaCount=5replicaCount: 5image: tag: '1.4.0'The Chart’s templates never change between environments — only the values fed into them do.
Releases and lifecycle commands
Section titled “Releases and lifecycle commands”Every helm install creates a Release: one named, tracked instance of a Chart running in the cluster. Helm keeps a revision history for each Release, so upgrades and rollbacks operate on the whole set of resources the Chart manages, not on one Deployment at a time — conceptually the same idea as kubectl rollout history for a single Deployment, just scoped to everything a Chart installed together.
# Upgrade an existing Release to a new chart version or new valueshelm upgrade myapp ./mychart -f values-prod.yaml
# Roll an entire Release back to a previous revisionhelm rollback myapp 2
# See the revision history for a Releasehelm history myapp
# Remove a Release and the resources it createdhelm uninstall myappHelm 4 versus Helm 3 — what actually changed
Section titled “Helm 4 versus Helm 3 — what actually changed”This course is written against Helm 4.3.0. Helm 4’s own GitHub release for v4.0.0 is explicit about scope: “Helm v4 is a major version with backward incompatible changes including to the flags and output of the Helm CLI and to the SDK,” but adds that “the changes are not as extensive as those from Helm v2 to v3, with the goal that the majority of workflows remain compatible between Helm v3 and v4.” Existing Chart apiVersion v2 charts — the vast majority in the wild — “will continue to be supported in Helm v4” with no rewrite required. The release notes list, among other things, a redesigned plugin system (with WebAssembly-based plugin support), post-renderers implemented as plugins, first-class server-side apply support, kstatus-based resource-readiness watching, and local content-based chart caching.
Two concrete CLI differences a course or an AI assistant trained mostly on Helm 3 will get wrong, proven directly against the installed v4.3.0 binary:
$ helm install --help | grep -A1 "server-side" --server-side object updates run in the server instead of the client (default true)
$ helm upgrade --help | grep -A1 "take-ownership" --take-ownership if set, upgrade will ignore the check for helm annotations and take ownership of the existing resourcesServer-side apply is the default in Helm 4 (--server-side defaults to true on helm install; helm upgrade takes a tri-state --server-side auto|true|false) — the exact same Server-Side Apply mechanism this course already covers for kubectl apply --server-side, with the same managedFields/field-manager machinery and the same --force-conflicts escape hatch (Helm’s flag is spelled --force-conflicts too). --take-ownership is new in Helm 4: it lets helm install/helm upgrade adopt resources that already exist in the cluster without the meta.helm.sh/release-name annotation Helm normally requires before it will touch an object — useful for importing hand-applied manifests into a Release, dangerous if pointed at the wrong object.
A third change is a real, working-directory-breaking one, proven by running the command directly:
$ helm upgrade --help | grep -A1 -- "--atomic"Error: unknown flag: --atomicFlag --atomic has been deprecated, use --rollback-on-failure instead--atomic still parses (it is accepted with a deprecation warning) but its replacement, --rollback-on-failure, is the flag Helm 4 documents going forward — a Helm-3-trained AI assistant reaching for --atomic in new pipeline code is not wrong, just behind.
helm template versus helm install —dry-run=server: a real, provable diff
Section titled “helm template versus helm install —dry-run=server: a real, provable diff”helm template never talks to a cluster — it is a pure client-side Go-template render. helm install --dry-run=server renders the same manifests but then actually sends them to the API server with dry-run=server, so schema validation, admission plugins, and defaulting all run for real, exactly like kubectl apply --dry-run=server elsewhere in this course. Run back-to-back against the identical Chart and values:
$ helm template orders ./orders-api---apiVersion: v1kind: Service...---# Source: orders-api/templates/deployment.yamlapiVersion: apps/v1kind: Deployment...---# Source: orders-api/templates/tests/test-connection.yamlapiVersion: v1kind: Podmetadata: name: "orders-orders-api-test-connection" annotations: "helm.sh/hook": test...
$ helm install orders ./orders-api --dry-run=serverNAME: ordersLAST DEPLOYED: Sat Sep 26 18:15:10 2026NAMESPACE: defaultSTATUS: pending-installREVISION: 1DESCRIPTION: Dry run completeHOOKS:---# Source: orders-api/templates/tests/test-connection.yamlapiVersion: v1kind: Podmetadata: name: "orders-orders-api-test-connection" annotations: "helm.sh/hook": test...
MANIFEST:---# Source: orders-api/templates/service.yaml...---# Source: orders-api/templates/deployment.yaml...helm template dumps every rendered document — including the helm.sh/hook: test Pod — as one flat stream with no notion of Release state. helm install --dry-run=server separates HOOKS from MANIFEST (a hook is not part of the tracked Release manifest, it only runs on demand via helm test) and reports the exact NAME/STATUS/REVISION metadata a real install would produce, because it genuinely round-trips through the API server rather than just filling in a template.
Values precedence, proven with a real chart
Section titled “Values precedence, proven with a real chart”Helm merges values from lowest to highest precedence: the chart’s own values.yaml, then any -f/--values files in the order given (later files win on overlapping keys), then --set/--set-string/--set-file last. Proven with a chart whose values.yaml sets replicaCount: 2 and image.tag: 1.27-alpine, an override file setting replicaCount: 3 and image.tag: "1.27", and a --set flag overriding the tag once more:
$ cat values-prod.yamlreplicaCount: 3image: tag: "1.27"
$ helm upgrade --install orders ./orders-api -f values-prod.yaml --set image.tag=1.27-alpine --rollback-on-failure --waitRelease "orders" has been upgraded. Happy Helming!NAME: orders...REVISION: 6
$ helm get values ordersUSER-SUPPLIED VALUES:image: tag: 1.27-alpinereplicaCount: 3
$ helm get values orders -aCOMPUTED VALUES:image: repository: nginx tag: 1.27-alpinereplicaCount: 3resources: requests: cpu: 100m memory: 64MireplicaCount ends up 3 (the -f file beat the chart default of 2, and no --set touched it), while image.tag ends up 1.27-alpine (the --set flag beat the -f file’s "1.27", which had already beaten the chart default) — exactly the documented values.yaml → -f (in order) → --set precedence, not a guess about it.
helm rollback and a real, messy revision history
Section titled “helm rollback and a real, messy revision history”helm upgrade --install --rollback-on-failure --wait (the Helm 4 name for --atomic) automatically rolls a failed upgrade back — proven here across two real, back-to-back upgrade failures on the exact same Release (the first failure was worker nodes not yet Ready, the second was a genuinely bad, non-existent image tag causing ImagePullBackOff):
$ helm history ordersREVISION UPDATED STATUS CHART APP VERSION DESCRIPTION1 Sat Sep 26 18:15:18 2026 superseded orders-api-0.1.0 1.0.0 Install complete2 Sat Sep 26 18:15:25 2026 failed orders-api-0.1.0 1.0.0 Upgrade "orders" failed: resource Deployment/default/orders-orders-api not ready...3 Sat Sep 26 18:16:55 2026 superseded orders-api-0.1.0 1.0.0 Rollback to 14 Sat Sep 26 18:17:08 2026 failed orders-api-0.1.0 1.0.0 Upgrade "orders" failed: resource Deployment/default/orders-orders-api not ready...5 Sat Sep 26 18:20:08 2026 superseded orders-api-0.1.0 1.0.0 Rollback to 36 Sat Sep 26 18:20:28 2026 deployed orders-api-0.1.0 1.0.0 Upgrade completeEvery failed upgrade (revisions 2 and 4) produced its own automatic rollback revision (3 and 5) with no operator intervention — --rollback-on-failure genuinely protects a Release from being left half-upgraded, but it is not a substitute for fixing the underlying cause: an upgrade with a typo’d, non-existent image tag will loop through exactly this fail-then-rollback cycle forever if a CI pipeline blindly retries it, since the rollback restores the previous working state, not the intended new one. A manual helm rollback orders <revision> does the same thing on demand, without needing a failure to trigger it.
helm lint and helm test
Section titled “helm lint and helm test”helm lint statically checks a chart’s structure and templates before anything is rendered against a cluster:
$ helm lint ./orders-api==> Linting orders-api[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failedA chart can also ship its own helm.sh/hook: test Pod under templates/tests/ — helm test runs it against the already-installed Release and reports pass/fail, which is a real smoke test of the live Service, not a lint of the template source:
$ helm test ordersNAME: orders...TEST SUITE: orders-orders-api-test-connectionLast Started: Sat Sep 26 18:23:09 2026Last Completed: Sat Sep 26 18:23:16 2026Phase: Succeededhelm diff: seeing the change before you make it
Section titled “helm diff: seeing the change before you make it”The built-in helm upgrade already shows what would change via --dry-run, but the community helm-diff plugin renders a proper unified diff against the live Release. It is not bundled — installing it in Helm 4 surfaces the plugin system’s new signature-verification step directly:
$ helm plugin install https://github.com/databus23/helm-diffError: plugin source does not support verification. Use --verify=false to skip verification
$ helm plugin install https://github.com/databus23/helm-diff --verify=falseWARNING: Skipping plugin signature verificationDownloading https://github.com/databus23/helm-diff/releases/download/v3.15.15/helm-diff-macos-arm64.tgzInstalled plugin: diff
$ helm diff upgrade orders ./orders-api -f values-prod.yaml --set image.tag=1.27-alpine --set replicaCount=4default, orders-orders-api, Deployment (apps) has changed: spec:- replicas: 3+ replicas: 4 ...Helm 4 verifies plugin signatures by default and refuses an unsigned source outright rather than silently trusting it — --verify=false is an explicit, auditable opt-out, not a default you fall into.
OCI registries: charts as OCI artifacts
Section titled “OCI registries: charts as OCI artifacts”Modern Helm distributes charts the same way container images are distributed — as OCI artifacts, addressed with an oci:// reference instead of a classic chart-repository index. Proven end-to-end against a throwaway local registry:
$ helm package ./orders-api -d .Successfully packaged chart and saved it to: orders-api-0.1.0.tgz
$ helm push orders-api-0.1.0.tgz oci://localhost:5050/charts --plain-httpPushed: localhost:5050/charts/orders-api:0.1.0Digest: sha256:23c5fd204fbcc25d18e0111a2b1846d6335a2657e98c7644635fb3213b1a95cb
$ helm pull oci://localhost:5050/charts/orders-api --version 0.1.0 -d ./pulled --plain-httpPulled: localhost:5050/charts/orders-api:0.1.0Digest: sha256:23c5fd204fbcc25d18e0111a2b1846d6335a2657e98c7644635fb3213b1a95cb
$ helm show chart oci://localhost:5050/charts/orders-api --version 0.1.0 --plain-httpapiVersion: v2appVersion: 1.0.0name: orders-apiversion: 0.1.0--plain-http is only needed here because the demo registry has no TLS certificate — a real registry (ghcr.io, ECR, Artifact Registry, a private Harbor) is addressed the same way over HTTPS with no extra flag, and helm install/helm upgrade accept an oci:// reference exactly like a classic repo/chart reference.
A tiny chart, a library chart, and why the templates themselves are excerpts
Section titled “A tiny chart, a library chart, and why the templates themselves are excerpts”The chart proven throughout this lesson is genuinely two charts: orders-api (the application chart) declaring a dependencies: entry on a second chart, common, whose Chart.yaml sets type: library. A library chart ships no installable templates of its own — only reusable, defined helpers other charts include:
# charts/common/templates/_helpers.tpl — excerpt, not a standalone manifest{{- define "common.fullname" -}}{{ .Release.Name }}-{{ .Chart.Name }}{{- end -}}
{{- define "common.labels" -}}app.kubernetes.io/name: {{ .Chart.Name }}app.kubernetes.io/instance: {{ .Release.Name }}{{- end -}}orders-api’s own templates/deployment.yaml then calls {{ include "common.fullname" . }} and {{- include "common.labels" . | nindent 4 }} — every .tpl helper file and every chart template containing {{ }} Go-template syntax is an intentional excerpt in this lesson, not a complete standalone manifest: the harness that validates this course’s # k8s/... fences against a real API server skips them on purpose, because {{ .Release.Name }} is not valid YAML until Helm has rendered it. The proof that the templates are correct is the rendered helm template/helm install --dry-run=server output shown above, not a raw apply of the template source.
flowchart LR values["values.yaml (defaults)"] --> render["helm install / helm template"] overrides["values-prod.yaml or --set"] --> render templates["templates/ (Go-templated manifests)"] --> render render --> manifests["Rendered Kubernetes manifests"] manifests --> cluster["Applied to cluster as Release myapp, revision N"]
Mental Model
Section titled “Mental Model”Helm is a template renderer plus a revision tracker, nothing more mystical than that: values.yaml (and its overrides) is the only input that ever changes, templates/ is the fixed Go-template source, and every helm install/upgrade/rollback is really “render, then let the API server reconcile the result,” proven above by helm install --dry-run=server genuinely round-tripping through the API. Helm 4 changed how carefully that reconciliation happens (server-side apply by default, a signature-verified plugin system, --rollback-on-failure in place of --atomic) but not the basic model — a Release’s revision history is exactly kubectl rollout history scoped to everything the Chart manages, not a separate concept.
Trade-offs
Section titled “Trade-offs”| Choice | Benefit | Cost |
|---|---|---|
helm template | Fast, no cluster needed, safe to run anywhere | Never catches a real API-server rejection — proven above, only --dry-run=server does |
helm install/upgrade --dry-run=server | Proven above: real schema validation and a real NAME/STATUS/REVISION preview | Requires live cluster access, slower than a pure template render |
--rollback-on-failure (Helm 4 name for --atomic) | Proven above: automatically undoes a failed upgrade, no half-deployed Release | Proven above: a bad image tag loops fail→rollback forever if blindly retried — it protects state, it does not diagnose the cause |
helm-diff plugin | Proven above: real unified diff against the live Release before upgrading | Third-party, unsigned by default — proven above, Helm 4 requires an explicit --verify=false to install it |
Library chart (type: library) for shared helpers | One _helpers.tpl, reused by every consuming chart, no copy-paste | Cannot be installed on its own — it only exists as a dependency, and its own templates are unrenderable excerpts by design |
OCI (oci://) chart registry | Proven above: same registry infrastructure as container images, no separate chart-repo index to host | Needs --plain-http or a real TLS cert — an OCI registry with neither simply refuses the connection |
Common Misconceptions
Section titled “Common Misconceptions”- “helm template and helm install —dry-run=server prove the same thing.” Proven above:
helm templatenever contacts the cluster at all; only--dry-run=serversends the render to the real API server and gets real schema validation and admission behavior back. - “—atomic is still the flag to reach for in Helm 4.” Proven above with the real CLI:
--atomicstill parses with a deprecation warning, but Helm 4’s documented flag is--rollback-on-failure. - “Values from a -f file always win over —set, since the file is more explicit.” Proven above:
--setis the highest-precedence source, applied after every-ffile, not before. - “A library chart is just a normal chart with no default values.” Proven above:
type: libraryinChart.yamlmeans the chart has no installable templates of its own at all — onlydefined helpers, never applied directly to a cluster. - “helm lint would have caught the ImagePullBackOff from a bad image tag.”
helm lintonly checks chart structure and template syntax statically — proven above, it never contacts a registry or a cluster, so a non-existent image tag lints clean and only fails at realhelm upgrade --waittime.
Reading AI-generated code
Section titled “Reading AI-generated code”When reviewing generated Helm charts, CI pipeline steps, or upgrade scripts, check for:
helm install/helm upgradewith no--rollback-on-failure(or the older--atomic) in a CI pipeline — proven above, a failed upgrade with neither flag leaves the Release in afailedstate and the old Pods potentially already torn down, with nothing rolling it back automatically.--atomicused in new Helm 4 pipeline code — proven above, it still works but is deprecated;--rollback-on-failureis the current flag.- A chart template (
.tplfile, or any file undertemplates/containing{{ }}) applied directly withkubectl apply -finstead of rendered first — Go-template syntax is not valid YAML, proven above by this course’s own harness skipping such fences on purpose. helm plugin installfrom a URL with--verify=falseused as a default habit rather than a deliberate, reviewed exception — proven above, Helm 4’s default is to require a verifiable signature.- A chart with no
helm.sh/hook: testPod and nohelm teststep in the deploy pipeline — the difference between “the chart rendered” and “the running Service actually answers a request” is exactly whathelm testproves and a template-only review cannot. helm upgradepiped straight from CI with no precedinghelm diff(or at minimum--dry-run) step — proven above,helm diffshows the exact live-Release change before it happens, catching an unintendedreplicas/image drop before it reaches production.
🐛 Spot the bug
#!/usr/bin/env bashset -euo pipefail
helm upgrade --install orders ./orders-api \-f values-prod.yaml \--set image.tag="${IMAGE_TAG}" \--wait --timeout 5mShow the answer
This upgrade has no --rollback-on-failure (Helm 4’s current name; --atomic still works but is deprecated). Proven above with this course’s own Release history: two real upgrade failures each needed an automatic rollback to avoid leaving the Deployment half-updated — without either flag, a failed --wait here (a bad IMAGE_TAG, a probe that never turns healthy) leaves the Release in STATUS: failed with whatever partial rollout already happened, and nothing reverts it. The fix is --rollback-on-failure added to the same command.