DNS and Service Discovery

The idea in one sentence

CoreDNS gives every Service a predictable DNS name so Pods can find each other by name instead of by IP, and a headless Service flips that from one load-balanced IP to the individual IPs of each backing Pod.

CoreDNS and the Service DNS name

CoreDNS runs as a Deployment inside the cluster (typically in kube-system) and serves as the cluster’s internal DNS server. Every Pod is configured, via its /etc/resolv.conf, to send DNS queries to CoreDNS. The moment a Service is created, CoreDNS starts answering for it with a fully qualified name of the form:

<service-name>.<namespace>.svc.cluster.local

# From inside any Pod in the cluster
nslookup payments.default.svc.cluster.local

Within the same namespace, the short form also works, because a Pod’s DNS search path automatically includes its own namespace’s svc.cluster.local suffix:

# From a Pod that lives in the same namespace as the "payments" Service
nslookup payments
curl http://payments/health

For a normal (non-headless) Service, that name resolves to a single A/AAAA record: the Service’s ClusterIP. Every request to that name still gets load-balanced across the backing Pods by kube-proxy — DNS just gets a client to the virtual IP.

Headless Services: skipping the load balancer

Sometimes a client needs to reach a specific Pod, not “any healthy Pod behind the Service.” Setting clusterIP: None creates a headless Service:

apiVersion: v1
kind: Service
metadata:
  name: cache
spec:
  clusterIP: None
  selector:
    app: cache
  ports:
    - port: 6379

A headless Service gets no virtual IP at all. Instead, CoreDNS returns the individual IPs of every matching, ready Pod directly. A query still resolves the same name, cache.default.svc.cluster.local, but the answer is now a list of Pod IPs instead of one ClusterIP.

Why StatefulSets rely on this

A StatefulSet gives each of its Pods a stable identity — web-0, web-1, web-2 — and pairs that with a headless Service (via spec.serviceName) so each Pod also gets its own individual, stable DNS record:

<pod-name>.<service-name>.<namespace>.svc.cluster.local

# Reach one specific ordinal Pod directly, not a load-balanced peer
nslookup web-0.cache.default.svc.cluster.local

This is exactly what distributed, stateful systems need: a database replica has to reconnect to the same peer it had before, not a random one a load balancer happens to pick. Without a headless Service, that per-Pod DNS record would not exist at all.

Real proof: svc, svc.ns, fqdn, and headless per-Pod records

From a real dnsutils Pod (registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7) against a real web-clusterip Service:

$ nslookup web-clusterip.default.svc.cluster.local
Server:    10.96.0.10
Address:  10.96.0.10#53

Name:  web-clusterip.default.svc.cluster.local
Address: 10.96.196.12

Headless per-Pod A records, real, from web-headless (clusterIP: None):

$ nslookup web-headless.default.svc.cluster.local
Server:    10.96.0.10
Address:  10.96.0.10#53

Name:  web-headless.default.svc.cluster.local
Address: 10.244.147.20
Name:  web-headless.default.svc.cluster.local
Address: 10.244.138.135

Same query shape, completely different answer shape: the normal Service returns one ClusterIP; the headless Service returns every backing Pod’s real IP directly, exactly as this lesson’s mechanism describes.

ndots:5 and search: the real /etc/resolv.conf, and the extra lookups it causes

$ cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

ndots:5 means: any name with fewer than 5 dots gets every entry in search tried, in order, before the resolver ever tries the name as given. web-clusterip (0 dots) triggers up to three extra lookup attempts (web-clusterip.default.svc.cluster.local, .svc.cluster.local, .cluster.local) before landing on the right one — invisible in a single successful nslookup, but real extra DNS round-trips on every short-name lookup in the cluster, which is exactly why this default is a frequent target for tuning on high-QPS workloads.

Custom dnsPolicy/dnsConfig: real, different resolv.conf

k8s/dns-and-service-discovery/custom-dnsconfig.yaml

apiVersion: v1
kind: Pod
metadata:
  name: custom-dns-demo
spec:
  dnsPolicy: None
  dnsConfig:
    nameservers:
      - 10.96.0.10
    searches:
      - default.svc.cluster.local
      - svc.cluster.local
      - cluster.local
    options:
      - name: ndots
        value: '2'
  containers:
    - name: app
      image: registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7
      command: ['sleep', '3600']

$ kubectl exec custom-dns-demo -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:2

dnsPolicy: None hands full control to dnsConfig — this Pod’s real resolv.conf shows ndots:2 instead of the cluster default 5, proving the override took effect exactly as configured. Lowering ndots is a real, common tuning move for a workload that mostly talks to fully-qualified external names, to cut the wasted in-cluster search attempts described above.

CoreDNS Corefile: the real ConfigMap

$ kubectl get configmap coredns -n kube-system -o jsonpath='{.data.Corefile}'
.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30 {
       disable success cluster.local
       disable denial cluster.local
    }
    loop
    reload
    loadbalance
}

This is the real plugin chain answering every query in this lesson: the kubernetes plugin serves cluster.local (with pods insecure enabling per-Pod IP-based DNS names, and a 30s TTL), forward sends anything outside cluster.local upstream to whatever /etc/resolv.conf the CoreDNS Pod itself has, and cache 30 bounds how long answers are cached.

ExternalName: a real CNAME, proven

k8s/dns-and-service-discovery/externalname-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: external-api
spec:
  type: ExternalName
  externalName: kubernetes.io

$ nslookup external-api.default.svc.cluster.local
Server:    10.96.0.10
Address:  10.96.0.10#53

external-api.default.svc.cluster.local  canonical name = kubernetes.io.
Name:  kubernetes.io
Address: 15.197.167.90
Name:  kubernetes.io
Address: 3.33.186.135

CoreDNS answers with a real CNAME to kubernetes.io, which the resolver then follows to kubernetes.io’s own real, current A records — exactly what ExternalName is: a DNS-level alias, nothing more. No Service selector, no Endpoints/EndpointSlice, no load balancing — just a name pointing at another name.

Reading AI-generated code

When reviewing generated DNS-related manifests and code, check for:

• ExternalName pointing at a raw IP address instead of a hostname — proven above, ExternalName only produces a CNAME, and a CNAME to an IP address is not valid DNS; a fixed IP needs a different mechanism (a headless Service with manually managed Endpoints/EndpointSlice, not ExternalName).
• Code that assumes every Pod’s ndots/search matches the cluster default without checking for a dnsPolicy: None / dnsConfig override — proven above, a Pod’s real resolv.conf can differ completely from the cluster norm.
• A retry loop for “DNS propagation delay” copied from public-internet DNS habits — CoreDNS answers are generated live from EndpointSlice state on every query; there is no TTL-based propagation delay to wait out the way there is for public DNS.
• Hardcoding 10.96.0.10 as “the” cluster DNS server IP in application code instead of reading it from the Pod’s own /etc/resolv.conf — it’s real (proven above) but cluster-specific (kube-dns Service ClusterIP), not a universal constant.

🐛 Spot the bug

k8s/dns-and-service-discovery/pin-to-ip.yaml

apiVersion: v1
kind: Service
metadata:
name: legacy-db
spec:
type: ExternalName
externalName: "10.20.30.40"

Show the answer

externalName is set to a raw IP address string, "10.20.30.40". Proven above, ExternalName works by having CoreDNS answer with a CNAME record pointing at externalName’s value — but a CNAME target must be a hostname, never an IP address; DNS has no concept of “alias this name to that IP” via CNAME. Depending on the CoreDNS/upstream resolver version this either fails validation, resolves to nothing, or behaves unpredictably — none of which is “pin this Service to a fixed IP,” which is almost certainly what the author wanted. The fix: for a fixed-IP external dependency, create a headless Service with no selector and a manually-managed Endpoints/EndpointSlice object pointing at the real IP — that is the actual supported mechanism for “give this IP a stable in-cluster DNS name.”

Mental Model

Every DNS answer in the cluster is really “ask CoreDNS, which is itself just a Deployment watching Services/EndpointSlices” (proven above via its real Corefile) — there’s no separate DNS control plane. ndots/search shape which name a client asks about (proven above with the real resolv.conf and the extra lookups a low dot-count name causes); dnsPolicy/dnsConfig shape which server and search list a Pod uses at all (proven above by overriding both); and headless vs normal Service shape what CoreDNS answers with (individual Pod IPs vs one ClusterIP, both proven above with real nslookup output).

Trade-offs

Choice	Benefit	Cost
Default ndots:5 + cluster search list	Short in-namespace names (web-clusterip) just work	Every external FQDN with fewer than 5 dots costs extra failed lookups against the cluster search suffixes first, proven above
Custom dnsConfig with lower ndots	Cuts wasted in-cluster search attempts for external-heavy workloads, proven above	Short in-cluster names now need to be fully qualified, or they may not resolve as expected
Headless Service	Per-Pod stable DNS, required for StatefulSet peer discovery, proven above	No load balancing at the DNS layer at all — client must handle multiple A records
ExternalName	Clean in-cluster name for an external dependency, proven above with a real CNAME	Just a CNAME — no health checking, no load balancing, must be a real resolvable hostname, never an IP (proven above)

Common Misconceptions

• “DNS changes need time to propagate inside the cluster, like public DNS.” CoreDNS answers are generated live from EndpointSlice state (proven above via the real Corefile’s kubernetes plugin), not cached the way public DNS/TTL propagation works.
• “ndots only matters for external names.” Proven above — it changes lookup behavior (and cost) for every name, in-cluster or not, that has fewer dots than the ndots setting.
• “ExternalName can point at an IP address to pin a Service to a fixed backend.” Proven above — it cannot; ExternalName only produces a CNAME to a hostname. Pinning a fixed IP needs a headless Service with manually-managed Endpoints/EndpointSlice instead.

flowchart TB
  pod["Client Pod"] -->|"DNS query"| coredns["CoreDNS"]
  coredns -->|"normal Service"| vip["Service ClusterIP\n(load-balanced)"]
  coredns -->|"headless Service"| p0["web-0 IP"]
  coredns -->|"headless Service"| p1["web-1 IP"]
  coredns -->|"headless Service"| p2["web-2 IP"]

A Pod's DNS query resolved by CoreDNS: a Service ClusterIP, or individual Pod IPs for a headless Service

☸️ The one-line takeaway

CoreDNS is the cluster’s internal DNS server (a real Deployment running the Corefile proven above), giving every Service a name in the form <service>.<namespace>.svc.cluster.local (with a short form usable inside the same namespace), which normally resolves to the Service’s load-balanced ClusterIP. A headless Service (clusterIP: None) skips load balancing and returns individual Pod IPs directly (proven above with real nslookup output), which is exactly what StatefulSets use to give each ordinal Pod its own stable DNS record. Every Pod’s real /etc/resolv.conf (proven above) sets ndots:5 and a cluster search list — overridable per-Pod via dnsPolicy/dnsConfig (proven above) — and ExternalName is nothing more than a real CNAME alias (proven above), never a way to pin a fixed IP.

What is the full DNS name format Kubernetes automatically assigns to a Service

<service-name>.cluster.local only <service-name>.<namespace>.svc.cluster.local <namespace>.<service-name>.local Services do not get automatic DNS names

Why do StatefulSets need a headless Service for DNS purposes

They do not, StatefulSets work identically to Deployments for networking A headless Service gives each ordinal Pod its own stable, individually addressable DNS record instead of one load-balanced address Headless Services are faster than normal Services at DNS resolution Headless Services are required for any Pod to get a DNS name at all

What role does CoreDNS play in the cluster

It implements the virtual IP routing for Services on each node It runs as the cluster internal DNS server, answering queries for Service and Pod DNS names It schedules Pods onto nodes based on DNS records It is only used for external, internet-facing DNS lookups

What does a headless Service (clusterIP: None) return when queried by DNS

Nothing, headless Services cannot be queried A single virtual IP, exactly like a normal Service The individual IPs of the matching, ready Pods directly An error, since there is no ClusterIP to return

This is a real Pod's resolv.conf. A lookup for the bare name web-clusterip (0 dots) with ndots:5 in effect does what?

$ cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

Resolves directly against nameserver 10.96.0.10 with no other attempts, since ndots only affects fully-qualified names Tries each entry in the search list first (appending it to the bare name), since the name has fewer dots than ndots, before trying the name as given Fails immediately because ndots:5 requires at least 5 dots in every query Ignores the search list entirely and queries the nameserver 5 times in parallel

What is wrong with this Service, proven above?

apiVersion: v1
kind: Service
metadata:
  name: legacy-db
spec:
  type: ExternalName
  externalName: "10.20.30.40"

Nothing -- ExternalName works with either a hostname or a raw IP address ExternalName only produces a CNAME record, and a CNAME target must be a hostname -- a raw IP address is not a valid CNAME target ExternalName requires a selector, which is missing here ExternalName Services are cluster-scoped and cannot have a namespace

Submit

Mark as complete