Skip to content
Kubernetes — From Zero to Hero

Production Checklist and the Kubernetes API Client

The idea in one sentence

Section titled “The idea in one sentence”

A production-ready workload is not one feature but a checklist of independent controls — resource limits, probes, disruption budgets, least-privilege RBAC, network policy, real secret encryption, bounded autoscaling, spread-out replicas, and namespace quotas — each covering a failure mode the others do not.

The checklist

Section titled “The checklist”

Every item below closes a specific gap. None of them is optional in a real production cluster, and none of them substitutes for another.

  • Resource requests/limits — set both, not just one. requests drives scheduling and determines a Pod’s QoS class (Guaranteed, Burstable, or BestEffort), which the kubelet uses to decide eviction order under node pressure. limits caps how much a runaway container can take from its neighbors.
  • Readiness AND liveness probes — a readiness probe controls whether a Pod receives Service traffic (fixes “sends traffic to a Pod that is not ready yet”); a liveness probe controls whether the kubelet restarts the container (fixes “process is alive but hung, and traffic keeps failing”). They solve different problems and a workload needs both.
  • PodDisruptionBudget (PDB) for any multi-replica service — bounds how many replicas a voluntary disruption (node drain, cluster upgrade) is allowed to take down at once, so a rolling node upgrade cannot accidentally take an entire service to zero replicas.
  • RBAC scoped to least privilege — Roles and RoleBindings (or ClusterRole/ClusterRoleBinding) granting exactly the verbs and resources a workload’s ServiceAccount needs, not cluster-admin by default.
  • Default-deny NetworkPolicy where appropriate — without one, every Pod can reach every other Pod in the cluster by default; a default-deny policy plus explicit allow rules makes lateral movement from a compromised Pod much harder.
  • Real secret protection — encryption at rest for etcd (Secrets are base64-encoded, not encrypted, unless an EncryptionConfiguration is applied) or an external secret manager, not base64 alone as a security boundary.
  • A sensibly bounded HorizontalPodAutoscaler — minReplicas prevents scaling to zero for a service that must always be available, maxReplicas caps cost and downstream load (databases, dependencies) during a traffic spike.
  • Multiple replicas spread across nodes/zones — Pod anti-affinity or topology spread constraints, so a single node or availability zone failure does not take down every replica at once; replica count alone does not guarantee this.
  • ResourceQuota/LimitRange per namespace — a ResourceQuota caps total resource consumption for a namespace, and a LimitRange supplies sane per-container defaults so a Pod deployed without explicit requests/limits does not silently become BestEffort.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: 'kubernetes.io/hostname'
              labelSelector:
                matchLabels:
                  app: orders-api
      containers:
        - name: orders-api
          image: myregistry/orders-api:2.3.0
          resources:
            requests:
              cpu: '250m'
              memory: '256Mi'
            limits:
              cpu: '500m'
              memory: '512Mi'
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz/live
              port: 8080
            periodSeconds: 10
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: orders-api

Closing the loop with a programmatic verification script

Section titled “Closing the loop with a programmatic verification script”

kubectl rollout status covers the interactive case, but a deployment pipeline often needs the same check as code — for example, gating a promotion step on a Deployment actually reaching its desired replica count, not just on kubectl apply returning successfully. @kubernetes/client-node’s KubeConfig.loadFromDefault() resolves credentials the same way whether the script runs on an engineer’s laptop (via KUBECONFIG or ~/.kube/config) or inside the cluster itself as a CI job or Job Pod (via the mounted ServiceAccount token) — no branching logic needed for either environment:

import * as k8s from '@kubernetes/client-node';


const kc = new k8s.KubeConfig();
// Resolves KUBECONFIG env var, then ~/.kube/config, then in-cluster ServiceAccount
kc.loadFromDefault();


const appsApi = kc.makeApiClient(k8s.AppsV1Api);


async function verifyRollout(name: string, namespace: string, timeoutMs = 120_000): Promise<void> {
  const start = Date.now();


  while (Date.now() - start < timeoutMs) {
    const dep = await appsApi.readNamespacedDeployment({ name, namespace });


    const desired = dep.spec?.replicas ?? 0;
    const ready = dep.status?.readyReplicas ?? 0;
    const updated = dep.status?.updatedReplicas ?? 0;


    console.log(`${namespace}/${name}: ${ready}/${desired} ready, ${updated}/${desired} updated`);


    if (ready === desired && updated === desired) {
      console.log('rollout complete');
      return;
    }


    await new Promise((resolve) => setTimeout(resolve, 3000));
  }


  throw new Error(`rollout of ${namespace}/${name} did not complete within ${timeoutMs}ms`);
}


await verifyRollout('orders-api', 'default');

This is the exact case where a programmatic client beats a shell script wrapping kubectl: the result feeds directly into a pipeline’s pass/fail decision, with typed access to status.readyReplicas and status.updatedReplicas instead of parsing CLI text output.

ApiException: real error handling, not string-matching stderr

Section titled “ApiException: real error handling, not string-matching stderr”

Every failed request from @kubernetes/client-node 2.x throws a typed ApiException carrying the real HTTP status code and the API server’s own Status body — proven against a Deployment that genuinely does not exist:

src/ssa-and-errors.ts
import * as k8s from '@kubernetes/client-node';


const kc = new k8s.KubeConfig();
kc.loadFromDefault();
const appsApi = kc.makeApiClient(k8s.AppsV1Api);


try {
  await appsApi.readNamespacedDeployment({ name: 'does-not-exist', namespace: 'default' });
} catch (err) {
  if (err instanceof k8s.ApiException) {
    console.log(`[ApiException] code=${err.code} body=${JSON.stringify(err.body)}`);
  } else {
    throw err;
  }
}
$ npx tsx src/ssa-and-errors.ts
[ApiException] code=404 body="{\"kind\":\"Status\",\"apiVersion\":\"v1\",...,\"message\":\"deployments.apps \\\"does-not-exist\\\" not found\",\"reason\":\"NotFound\",\"code\":404}\n"

err.code is the real HTTP status (404), and err.body is the API server’s actual Status object (reason: "NotFound", the same shape kubectl itself parses) — a pipeline can branch on err.code === 404 to mean “not deployed yet” versus err.code === 409 to mean “conflict” versus anything else being a genuine failure, all without ever parsing a CLI error string.

Server-Side Apply from the TypeScript client — including a real conflict

Section titled “Server-Side Apply from the TypeScript client — including a real conflict”

The Server-Side Apply mechanism this course covers for kubectl apply --server-side is available from @kubernetes/client-node 2.x too, via k8s.setHeaderOptions setting the Content-Type: application/apply-patch+yaml header and a fieldManager on the request. Proven by attempting to SSA-patch a Deployment already managed by Helm’s own field manager (the same orders-orders-api Release from the Helm lesson):

// (excerpt, continues from ssa-and-errors.ts above -- reuses kc/appsApi, not independently valid)
const patch: k8s.V1Deployment = {
  apiVersion: 'apps/v1',
  kind: 'Deployment',
  metadata: { name: 'orders-orders-api', namespace: 'default' },
  spec: {
    replicas: 4,
    selector: { matchLabels: { app: 'orders' } },
    template: {
      metadata: { labels: { app: 'orders' } },
      spec: { containers: [{ name: 'orders-api', image: 'nginx:1.27-alpine' }] },
    },
  },
};


try {
  const result = await appsApi.patchNamespacedDeployment(
    { name: 'orders-orders-api', namespace: 'default', body: patch, fieldManager: 'ts-client-demo' },
    k8s.setHeaderOptions('Content-Type', 'application/apply-patch+yaml'),
  );
  console.log(`[SSA patch] replicas now ${result.spec?.replicas}`);
} catch (err) {
  if (err instanceof k8s.ApiException) {
    console.log(`[SSA conflict] code=${err.code} body=${JSON.stringify(err.body)}`);
  }
}
$ npx tsx src/ssa-and-errors.ts
[SSA conflict] code=409 body="{\"kind\":\"Status\",...,\"message\":\"Apply failed with 1 conflict: conflict with \\\"helm\\\": .spec.replicas\",\"reason\":\"Conflict\",\"details\":{\"causes\":[{\"reason\":\"FieldManagerConflict\",\"field\":\".spec.replicas\",...}]},\"code\":409}\n"

A real 409 Conflict, naming the exact field (.spec.replicas) and the exact competing field manager ("helm") — identical in shape to the kubectl apply --server-side conflict this course proves in the foundations lesson, because it is the same API-server mechanism underneath. The fix is the same one too: either don’t manage a field another manager already owns, or pass force: true in the patch options (the PatchStrategy/force-conflicts equivalent) to take ownership deliberately.

Watch: a real event stream, not polling

Section titled “Watch: a real event stream, not polling”

k8s.Watch opens a genuine long-lived HTTP connection to the API server and streams ADDED/MODIFIED/DELETED events as they happen — proven by watching a single Deployment and aborting the connection cleanly on a timer:

src/watch-deployment.ts
import * as k8s from '@kubernetes/client-node';


const kc = new k8s.KubeConfig();
kc.loadFromDefault();
const watch = new k8s.Watch(kc);


const req = await watch.watch(
  '/apis/apps/v1/namespaces/default/deployments',
  { fieldSelector: 'metadata.name=orders-orders-api' },
  (type: string, obj: any) => {
    console.log(`[${type}] replicas=${obj.spec?.replicas} readyReplicas=${obj.status?.readyReplicas}`);
  },
  (err: any) => {
    if (err) console.error('watch ended with error', err);
  },
);


setTimeout(() => req.abort(), 5000);
$ npx tsx src/watch-deployment.ts
[ADDED] replicas=3 readyReplicas=3
watch ended with error DOMException [AbortError]: This operation was aborted

The initial ADDED event delivers the object’s current state the instant the watch opens (the same “list, then watch from that resourceVersion” pattern kubectl get --watch uses, proven elsewhere in this course); every subsequent real change to the object streams as its own MODIFIED event with no polling interval to tune. req.abort() is a real AbortController call, which is why the error callback fires with a genuine AbortError rather than the watch silently leaking a connection — makeInformer (proven in the Operators lesson) wraps exactly this Watch class with a local cache and automatic relist, and is almost always the better choice for a real controller; raw Watch is the right tool for a one-off “stream events for N seconds” script like this one.

In-cluster auth, proven from inside a real Pod

Section titled “In-cluster auth, proven from inside a real Pod”

KubeConfig.loadFromDefault() resolving to the mounted ServiceAccount token when running inside a cluster (loadFromCluster() internally, triggered automatically when KUBERNETES_SERVICE_HOST is set) is not a black box — proven by reading the exact same files and environment variables by hand, from inside a real Pod with a minimal, least-privilege ServiceAccount:

# ServiceAccount + Role + RoleBinding + Pod applied in order (real apply, not dry-run) --
# under dry-run=server nothing persists between documents in the same fence, so the
# Pod's serviceAccountName lookup would fail a cross-document dry-run; no k8s/ path here
apiVersion: v1
kind: ServiceAccount
metadata:
  name: api-client-demo
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: api-client-demo-role
  namespace: default
rules:
  - apiGroups: ['apps']
    resources: ['deployments']
    verbs: ['get', 'list']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: api-client-demo-binding
  namespace: default
subjects:
  - kind: ServiceAccount
    name: api-client-demo
    namespace: default
roleRef:
  kind: Role
  name: api-client-demo-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Pod
metadata:
  name: api-client-demo
  namespace: default
spec:
  serviceAccountName: api-client-demo
  containers:
    - name: curl
      image: curlimages/curl:8.11.1
      command: ['sleep', '3600']
$ kubectl exec api-client-demo -- sh -c '
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -sS --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer $TOKEN" \
  "https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT/apis/apps/v1/namespaces/default/deployments/orders-orders-api" \
  | head -c 400
'
{
  "kind": "Deployment",
  "apiVersion": "apps/v1",
  "metadata": {
    "name": "orders-orders-api",
    ...

That is the entirety of what loadFromCluster() automates: read the token from /var/run/secrets/kubernetes.io/serviceaccount/token, the CA from the sibling ca.crt, and the API server address from KUBERNETES_SERVICE_HOST/KUBERNETES_SERVICE_PORT — no external configuration, no kubeconfig file inside the Pod at all. RBAC applies exactly the same way it does to any other identity — proven by attempting a verb the Role does not grant:

$ kubectl exec api-client-demo -- sh -c '... -X DELETE "https://.../deployments/orders-orders-api"'
{
  "kind": "Status",
  "status": "Failure",
  "message": "deployments.apps \"orders-orders-api\" is forbidden: User \"system:serviceaccount:default:api-client-demo\" cannot delete resource \"deployments\" in API group \"apps\" in the namespace \"default\"",
  "reason": "Forbidden",
  "code": 403
}

A real 403, naming the exact identity, verb, resource, and namespace — the api-client-demo ServiceAccount’s Role only grants get/list, so delete is rejected at the authorization layer exactly like the RBAC lesson proves for every other identity in this course. loadFromCluster() (and, transitively, loadFromDefault() running inside a Pod) is not a different authorization system — it is the exact same bound-token, RBAC-checked identity this course covers throughout, reached through a slightly different bootstrap path.

The production checklist, upgraded with what this course has now proven

Section titled “The production checklist, upgraded with what this course has now proven”

Every item below links to the lesson that proved it live, not just described it:

flowchart TB
  dep["Deployment: orders-api (3 replicas, spread across nodes)"]
  dep --> probes["Readiness + liveness probes"]
  dep --> resources["requests/limits (QoS)"]
  dep --> pdb["PodDisruptionBudget (minAvailable: 2)"]
  dep --> rbac["RBAC: least-privilege ServiceAccount"]
  dep --> netpol["Default-deny NetworkPolicy + explicit allows"]
  dep --> hpa["HorizontalPodAutoscaler (bounded min/max)"]
  ns["Namespace"] --> quota["ResourceQuota / LimitRange"]
  quota --> dep
A production-ready Deployment: probes, requests/limits, PodDisruptionBudget, RBAC, and NetworkPolicy all covering different failure modes

Mental Model

Section titled “Mental Model”

@kubernetes/client-node is not a separate authorization system from kubectl — proven above, its ApiException codes, its Server-Side Apply conflicts, and its in-cluster bootstrap all resolve to the exact same API-server mechanisms (Status objects, field managers, bound ServiceAccount tokens, RBAC) this course proves throughout with kubectl. The entire value of the TypeScript client is typed, programmatic access to those same mechanisms — err.code === 409 instead of grepping stderr for “conflict,” result.spec?.replicas instead of parsing kubectl get -o jsonpath output — never a different set of rules underneath.

Trade-offs

Section titled “Trade-offs”
ChoiceBenefitCost
kubectl scripted in a shell pipelineFast to write, no dependencies to installProven above: error handling means parsing text output; a 404 and a 403 look the same until you grep the message string
@kubernetes/client-node typed clientProven above: real ApiException.code, typed result.spec?.readyReplicas, no string parsingRequires a Node/TS toolchain in the pipeline, more setup than a bare kubectl call
Raw Watch classProven above: minimal, real event stream for a short-lived scriptNo cache, no automatic relist on error — proven in the Operators lesson, makeInformer adds both
makeInformer (client-node)Local cache + automatic relist-on-error, right choice for a long-running controllerMore moving parts than a one-shot Watch for a script that only needs to observe briefly
loadFromDefault() everywhere (laptop and in-cluster)Proven above: identical code path, no environment branching in application codeThe RBAC the in-cluster ServiceAccount holds must still be deliberately scoped — proven above, a Role that’s too broad is just as dangerous as a hardcoded laptop kubeconfig with too much access

Common Misconceptions

Section titled “Common Misconceptions”
  • “A client library error is just whatever text the API server happened to return.” Proven above: ApiException.code is the real, structured HTTP status (404, 409), and .body is the API server’s actual parsed Status object — a pipeline can branch on it reliably, not fuzzy-match a string.
  • “Server-Side Apply conflicts are a kubectl-only concept, the TypeScript client doesn’t have them.” Proven above with a real 409: the identical field-manager conflict mechanism applies to any client sending an apply-patch content type, kubectl or otherwise.
  • “Watch and an Informer are the same thing with different names.” Proven above and in the Operators lesson: Watch is a raw event stream with no cache and no reconnect logic; makeInformer wraps it with both, which is why a real controller should almost always reach for the latter.
  • “loadFromCluster() must be doing something different or more privileged than a normal ServiceAccount.” Proven above by reading the exact same token/CA files by hand with curl: it is the identical bound-token, RBAC-checked mechanism as any other ServiceAccount, reached by a different bootstrap path, not a special-cased escalation.
  • “If the client library call didn’t throw, the operation must have fully succeeded as intended.” A successful Server-Side Apply patch can still silently drop fields the caller expected to set if another field manager already owns them and no conflict was triggered because the values happened to match — proven indirectly above, a conflict only surfaces when the values actually differ, not merely when two managers both reference the same field.

Reading AI-generated code

Section titled “Reading AI-generated code”

When reviewing generated TypeScript/Node code that talks to the Kubernetes API, check for:

  • A bare try/catch that logs err.message or String(err) instead of checking err instanceof k8s.ApiException and branching on err.code — proven above, the structured code and body are right there and far more reliable than string matching.
  • @kubernetes/client-node 1.x-style positional-argument API calls (e.g. readNamespacedDeployment(name, namespace)) instead of 2.x’s object-parameter form (readNamespacedDeployment({ name, namespace })) used throughout this course — a 1.x-trained assistant’s code will not type-check against a 2.x install.
  • A Server-Side Apply patch with no fieldManager set, or one reused across genuinely different logical owners — proven above, the conflict message names the exact field manager at fault, so an unset or shared one makes a real conflict much harder to attribute later.
  • Code that reaches for a hand-rolled polling loop (setInterval + readNamespacedDeployment) to detect a change, when Watch or makeInformer (both proven above) deliver the same information as a real push the moment it happens, with less code and no polling-interval trade-off to tune.
  • A Pod’s ServiceAccount granted cluster-admin “so the client library definitely has whatever permissions it needs” — proven above with a real 403, RBAC applies to the client exactly as it does to any other identity; the fix is a scoped Role naming only the verbs the code actually calls.
  • KubeConfig.loadFromDefault() wrapped in an if (inCluster) { loadFromCluster() } else { loadFromDefault() } branch — proven above, loadFromDefault() already does this resolution internally; the branch is unnecessary code that can silently diverge from what the library itself does.

🐛 Spot the bug

src/rollout-check.ts
import * as k8s from '@kubernetes/client-node';


const kc = new k8s.KubeConfig();
kc.loadFromDefault();
const appsApi = kc.makeApiClient(k8s.AppsV1Api);


async function isRolledOut(name: string, namespace: string): Promise<boolean> {
try {
  const dep = await appsApi.readNamespacedDeployment({ name, namespace });
  return dep.status?.readyReplicas === dep.spec?.replicas;
} catch (err) {
  console.log('rollout check failed: ' + err);
  return false;
}
}
Show the answer

The catch block swallows every error into the same false return and a plain string log, whether the Deployment genuinely does not exist yet (a real, expected 404 early in a rollout), the caller lacks permission (a real 403 that should fail loudly, not silently report “not rolled out”), or the API server is unreachable. Proven above, err instanceof k8s.ApiException gives a real, structured err.code to branch on — the fix distinguishes err.code === 404 (genuinely not created yet, keep polling) from anything else (rethrow or alert), instead of collapsing every failure mode into one indistinguishable false.

Why does a PodDisruptionBudget matter even though it cannot stop involuntary failures like a node crashing unexpectedly
Why are both readiness and liveness probes needed on the same workload
Why is base64 encoding alone not sufficient protection for Kubernetes Secrets
What is the correct way to branch on this in a TypeScript pipeline script
$ npx tsx src/ssa-and-errors.ts
[ApiException] code=404 body="{...\"reason\":\"NotFound\",\"code\":404}"
What does this real response prove about Server-Side Apply from the TypeScript client
$ npx tsx src/ssa-and-errors.ts
[SSA conflict] code=409 body="{...\"message\":\"Apply failed with 1 conflict: conflict with \\\"helm\\\": .spec.replicas\"...}"
What did reading the token and CA files by hand from inside a Pod prove about KubeConfig.loadFromCluster()