Architecture: Control Plane and Nodes
The idea in one sentence
Section titled “The idea in one sentence”A cluster’s control plane makes decisions and holds state, while its nodes run Pods, and every one of those decisions and every byte of that state passes through a single front door: the API server.
The control plane: the brain
Section titled “The control plane: the brain”The control plane is a small set of components, usually run together, that together decide what should happen in the cluster and remember what has already been decided.
- kube-apiserver is the front door of the cluster. It exposes a REST API that every other component — and every
kubectlcommand you run — talks to. It validates and authenticates requests, and it is the only component that talks directly to etcd. - etcd is a distributed, consistent key-value store that holds all cluster state: every object you have created, its current status, everything. If etcd is lost without a backup, the cluster’s memory is lost with it.
- kube-scheduler watches for newly created Pods that have no node assigned yet, and decides which node each one should run on, based on resource requirements, constraints, and current cluster load.
- kube-controller-manager runs the built-in controllers as a single process — the node controller (notices when nodes go unreachable), the replication controller (keeps the right number of Pod replicas running), the endpoint controller (keeps Service endpoints up to date), and others. This is where the reconciliation loops from the previous lesson actually live.
- cloud-controller-manager is optional, and only present on managed cloud clusters — it integrates cluster-level concerns like load balancers and node lifecycle with the underlying cloud provider’s API.
You can see the control plane’s own Pods on a kubeadm-style cluster like any other workload:
kubectl get pods -n kube-systemNode components: where Pods actually run
Section titled “Node components: where Pods actually run”Every node in the cluster runs three things:
- kubelet is the node agent. It is the component that actually ensures the containers described in the Pod specs assigned to this node are running and healthy — it starts them, restarts them if they die, runs their probes, and reports the node’s and its Pods’ status back to the API server. Nothing runs on a node without kubelet making it happen.
- kube-proxy maintains the network rules on the node that make Services work — routing traffic addressed to a Service’s virtual IP to one of the matching Pods, wherever they happen to be running.
- A container runtime, such as containerd or CRI-O, is what actually pulls images and runs containers, speaking to kubelet through the Container Runtime Interface (CRI).
Inspect a node’s components and status directly:
kubectl describe node <node-name>Everything talks to etcd through the API server
Section titled “Everything talks to etcd through the API server”This is the architectural fact worth memorizing: kube-scheduler, kube-controller-manager, kubelet, and everything else never touch etcd directly. They all read and write cluster state exclusively by calling the API server, which is the single source of truth and authority for the entire cluster. This gives Kubernetes one consistent point for authentication, authorization, validation, and watch notifications — every controller and every kubelet is really just a client of the same REST API you use with kubectl.
How controllers actually notice a change: watches, not polling
Section titled “How controllers actually notice a change: watches, not polling”Every controller and every kubectl get --watch opens a long-lived HTTP connection to the API server and receives a stream of ADDED/MODIFIED/DELETED events the instant something changes — it is never polling in a loop. --output-watch-events shows the event type alongside the usual columns. Proof: a watch started on a namespace, then a Pod created and deleted while it runs:
kubectl get pods -n foundations-demo --watch --output-watch-events -o wideReal stream (existing Pods arrive as ADDED first, then a brand-new Pod’s full lifecycle plays out as a sequence of MODIFIED events, ending in DELETED):
EVENT NAME READY STATUS RESTARTS AGE IP NODEADDED web-69d5569566-fmcq2 1/1 Running 0 19s 10.244.2.10 foundations-workerADDED watchtest 0/1 Pending 0 0s <none> <none>MODIFIED watchtest 0/1 Pending 0 0s <none> foundations-worker2MODIFIED watchtest 0/1 ContainerCreating 0 0s <none> foundations-worker2MODIFIED watchtest 1/1 Running 0 1s 10.244.1.9 foundations-worker2MODIFIED watchtest 1/1 Terminating 0 3s 10.244.1.9 foundations-worker2MODIFIED watchtest 0/1 Completed 0 3s 10.244.1.9 foundations-worker2DELETED watchtest 0/1 Completed 0 4s 10.244.1.9 foundations-worker2Every one of those lines is the API server pushing a change down an already-open connection — this is exactly the mechanism kube-controller-manager, kube-scheduler, and kubelet all use to react to cluster state instead of asking “did anything change yet?” in a loop. The deeper mechanics — resourceVersion, watch bookmarks, and how an informer builds a local cache on top of this same stream — belong to the internals module; this lesson only needs you to recognize that “controllers react” means “controllers watch.”
kubectl get --raw: talking to the API server with nothing in between
Section titled “kubectl get --raw: talking to the API server with nothing in between”kubectl get --raw <path> sends a request straight at an API server path and prints the raw response — no object formatting, no table view — useful for endpoints that aren’t a normal resource at all:
$ kubectl get --raw /healthzok
$ kubectl get --raw /version{ "major": "1", "minor": "37", "gitVersion": "v1.37.0", "gitCommit": "f54c212e3a2f75d674b717a9b29052b20b60aefc", "buildDate": "2026-08-26T10:44:25Z", "goVersion": "go1.26.6", "platform": "linux/arm64"}-v=8: watching kubectl’s own HTTP requests
Section titled “-v=8: watching kubectl’s own HTTP requests”Every kubectl command is, underneath, an HTTP call to the API server — -v=8 prints the real request line, headers, and response body kubectl sends and receives, which is the fastest way to stop guessing what a command “really does”:
$ kubectl get pods -n foundations-demo -v=8...I0926 15:34:31.288997 round_trippers.go:527] "Request" verb="GET" url="https://127.0.0.1:52255/api/v1/namespaces/foundations-demo/pods?limit=500"I0926 15:34:31.296968 round_trippers.go:632] "Response" status="200 OK" headers=<...>I0926 15:34:31.297057 helper.go:113] "Response Body" body="{\"kind\":\"Table\",\"apiVersion\":\"meta.k8s.io/v1\", ...}"kubectl get without -o actually requests meta.k8s.io/v1, Kind=Table from the server — the column formatting you see in a normal kubectl get pods is computed server-side and sent back as structured data, not guessed by the client from a plain object list.
Where etcd actually lives in a kind cluster
Section titled “Where etcd actually lives in a kind cluster”kind runs the control plane the same way kubeadm does anywhere else: as static Pods, defined by plain manifest files that kubelet watches directly on disk, not through the API server. On the control-plane node:
$ docker exec foundations-control-plane ls /etc/kubernetes/manifests/etcd.yamlkube-apiserver.yamlkube-controller-manager.yamlkube-scheduler.yamlReal (trimmed) etcd.yaml — this is the exact command line etcd starts with, no abstraction in between:
$ docker exec foundations-control-plane cat /etc/kubernetes/manifests/etcd.yamlapiVersion: v1kind: Podmetadata: name: etcd namespace: kube-systemspec: containers: - command: - etcd - --advertise-client-urls=https://172.19.0.7:2379 - --data-dir=/var/lib/etcd - --initial-cluster=foundations-control-plane=https://172.19.0.7:2380 - --listen-client-urls=https://127.0.0.1:2379,https://172.19.0.7:2379The kind node image ships no etcdctl binary, so member list isn’t available inside the node the way it might be on a kubeadm cluster you built yourself with the full etcd tooling installed — the honest fallback for “is etcd healthy” on kind is watching its static Pod through the API server like anything else:
$ kubectl -n kube-system get podsNAME READY STATUS RESTARTS AGEetcd-foundations-control-plane 1/1 Running 0 3m24skube-apiserver-foundations-control-plane 1/1 Running 0 3m24skube-controller-manager-foundations-control-plane 1/1 Running 0 3m23skube-scheduler-foundations-control-plane 1/1 Running 0 3m24skubelet and the container runtime, on a real node
Section titled “kubelet and the container runtime, on a real node”On a worker node, kubelet and containerd are separate host processes — kubelet never runs a container itself, it tells containerd (via CRI) to do it, and containerd hands each container off to a containerd-shim-runc-v2 process:
$ docker exec foundations-worker sh -c 'ps aux | grep -E "kubelet|containerd" | grep -v grep'root 130 /usr/local/bin/containerdroot 242 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yamlroot 294 /usr/local/bin/containerd-shim-runc-v2 -namespace k8s.io -id bf542e6a6f4bd107dc83be43e581da8a6b297fb39d17fbfb1291a33f75834384One shim process per running container, each a direct child of containerd, not of kubelet — kubelet’s job stops at telling containerd what should be running; it is containerd (and the shim underneath it) that actually owns the running process tree.
flowchart TB
subgraph cp["Control Plane"]
api["kube-apiserver"]
etcd["etcd"]
sched["kube-scheduler"]
cm["kube-controller-manager"]
end
subgraph n1["Node"]
kubelet["kubelet"]
proxy["kube-proxy"]
cri["container runtime (containerd / CRI-O)"]
end
sched --> api
cm --> api
api --> etcd
kubelet --> api
proxy --> api Reading AI-generated code
Section titled “Reading AI-generated code”When reviewing generated architecture diagrams, runbooks, or debugging instructions, check for:
- Any instruction to
curlor otherwise talk to etcd directly from an application or a script — proven above, only the API server is meant to touch etcd; a healthy cluster’s etcd port is not something your own workloads should ever be pointed at. - “Poll the API server every N seconds to detect changes” as a design for a controller or automation script — proven above, the real mechanism every built-in controller uses is a watch (a long-lived connection receiving
ADDED/MODIFIED/DELETEDevents), not polling; a generated controller sketch that polls is describing something much slower and heavier than what Kubernetes itself does. - Debugging instructions for a managed cloud cluster (EKS/GKE/AKS) that say to
docker execinto a control-plane node or read/etc/kubernetes/manifests/etcd.yaml— proven above, that path is real for a kubeadm-style cluster (including kind), but a managed cluster’s control plane is not run as Pods on nodes you can shell into at all; the equivalent commands don’t exist there. etcdctlassumed to be present on every Kubernetes node — proven above, it is not part of the kind node image; the fallback (kubectl -n kube-system get pods) is a real, working substitute worth knowing, not a compromise.- Confusing kubelet with the container runtime — proven above, kubelet only calls containerd (via CRI); it is containerd and its shim process that actually own the running container process tree.
🐛 Spot the bug
# From a generated "custom health checker" scriptETCD_ENDPOINT="https://172.19.0.7:2379"curl --cacert /etc/kubernetes/pki/etcd/ca.crt "$ETCD_ENDPOINT/health"# ...then feed the result straight into a scaling decisionShow the answer
This script talks to etcd directly from outside the control plane, bypassing the API server entirely. Proven above, in a real kind cluster: kube-scheduler, kube-controller-manager, kubelet, and every kubectl command all reach etcd through exactly one door, the API server — nothing else is meant to open a client connection to it. Even setting aside that etcd’s client certs (/etc/kubernetes/pki/etcd/*) are not something an application should ever hold, basing a scaling decision on a raw etcd health check throws away every authentication, authorization, validation, and watch-notification guarantee the API server exists to provide. The fix is kubectl get --raw /healthz (or the equivalent client-go call against the API server), proven above to be the real, supported way to ask “is the API reachable” without touching etcd at all.
Mental Model
Section titled “Mental Model”Treat the control plane as a database (etcd) with exactly one front door (the API server) and a small crowd of clients standing outside that door, all doing the same two things forever: reading the current state through a watch, and writing back the small delta needed to move it toward what’s desired. kube-scheduler, kube-controller-manager, kubelet, kube-proxy, and kubectl itself are architecturally identical from etcd’s point of view — every one of them is just another watch-and-patch client of the same REST API, with no back door and no special access. Once that clicks, “why can’t I read etcd directly on my managed cluster” and “why does kubectl get --watch never seem to stop” both answer themselves: there is no other way in, and there is no polling loop to stop.
Trade-offs
Section titled “Trade-offs”| Choice | Benefit | Cost |
|---|---|---|
| Single API server as the only etcd client | One consistent point for auth, validation, and watch notifications (proven above) | Every component’s availability is bounded by API server availability — no component has a private fallback path to state |
| Watches instead of polling | Near-instant reaction to change, proven above with a real event stream; no wasted “did anything change?” requests | A controller must handle a dropped/expired watch connection and re-list, which is more state to manage correctly than a naive poll loop |
| Static Pods for the control plane (kubeadm/kind) | Simple, inspectable — literally a YAML file kubelet watches on disk, provable with docker exec + cat above | Not how a managed cloud cluster works at all; debugging instructions written against kind/kubeadm do not transfer |
| kubelet delegates to containerd via CRI instead of running containers itself | Runtime is swappable (containerd, CRI-O) without changing kubelet | One more process boundary and one more thing that can be out of sync (kubelet up, containerd degraded, or vice versa) |
Common Misconceptions
Section titled “Common Misconceptions”- “An application or script can talk to etcd directly for a quick read, as long as it has the certs.” Proven above: nothing but the API server is meant to touch etcd, and the SpotTheBug above shows exactly why that matters — every other guarantee (auth, validation, watch notifications) is bypassed the moment you do.
- “Controllers detect change by polling the API server on a timer.” Proven above with a real
--output-watch-eventsstream: the actual mechanism is a long-lived watch connection receiving push events, not a request-response poll loop. - “Every Kubernetes cluster has an
/etc/kubernetes/manifests/directory you can inspect for the control plane.” Proven above: true for kind and any kubeadm-built cluster, but not for a managed cloud cluster (EKS/GKE/AKS), where the control plane isn’t Pods on nodes you can reach at all. - “kubelet runs containers.” Proven above: kubelet calls containerd through the Container Runtime Interface; it is containerd (and its per-container shim process) that actually owns the running process.