The Datadog Agent and Integrations

The idea in one sentence

The “Agent” is not one process but a small family of them — core Agent, Trace Agent, Process Agent, Security Agent, and, on Kubernetes, a fifth cluster-scoped Cluster Agent — each collecting a different kind of telemetry, with checks and Autodiscovery deciding what gets collected and from where. This lesson runs a real datadog/agent:7 container (tag 7, resolved to Agent v7.83.3 — verified below) to prove every claim instead of just describing it.

The Agent’s five component processes

When people say “install the Datadog Agent,” they mean a package that brings up several cooperating processes, each with a narrow job:

• Core Agent — collects host/system metrics (CPU, memory, disk, network) and runs checks against configured integrations. This is the process that reads your conf.d/ configuration.
• Trace Agent — listens locally for spans emitted by your application’s tracing library, batches them, and forwards them to Datadog’s APM intake. Your app never talks to Datadog directly for traces — it talks to the Trace Agent.
• Process Agent — the logical component that collects live process-level data (what’s running, its resource usage) for the Live Processes view. In current Agent 7 (7.83.3, proven below) its checks run inside the core Agent process by default, so there is no separate process-agent PID; see Agent Architecture for the full proof.
• Security Agent (optional) — runs Cloud Security workloads: misconfiguration/posture scanning and runtime threat detection, marketed today as Cloud Security Management (CSM). Earlier course drafts called this “Cloud Security Posture Management” — that term names only the posture/misconfiguration half of what the Security Agent now does; CSM is the current umbrella product name.
• Cluster Agent (Kubernetes only, the fifth process) — a single deployment per cluster (not per node) that runs cluster-scoped work the per-node Agent shouldn’t duplicate: the Kubernetes State Metrics core check, the External Metrics Provider that backs Horizontal Pod Autoscaler queries, the Admission Controller, and coordination for cluster checks. Every node Agent talks to the Cluster Agent instead of hitting the Kubernetes API server directly, which is what keeps API server load flat as a cluster scales to hundreds of nodes.

These run as one installed unit. Most are separate OS processes underneath (core, Trace, Security), which is why the status command (below) reports on each of them individually; the Process Agent is the exception in 7.83.3, where it is a component of the core Agent process.

flowchart TB
    subgraph Node["Host / Node"]
      Core["Core Agent
(metrics + checks)"]
      Trace["Trace Agent
(receives spans)"]
      Proc["Process Agent
(live processes, runs inside core by default)"]
      Sec["Security Agent
(optional)"]
    end
    subgraph Cluster["Kubernetes cluster (one per cluster)"]
      CA["Cluster Agent
(cluster checks, HPA metrics, Admission Controller)"]
    end
    App["Instrumented app"] -->|spans| Trace
    Core -->|runs process checks in-process| Proc
    Core -->|checks read| ConfD["conf.d/*.d/conf.yaml"]
    Core -->|node-local queries| CA
    CA -->|cluster-scoped queries| K8sAPI["Kubernetes API server"]
    Core --> Backend["Datadog Backend"]
    Trace --> Backend
    Proc --> Backend
    Sec --> Backend
    CA --> Backend

Agent component processes

Proof: a real agent status walk-through

A long-running datadog/agent:7 container (DD_API_KEY a 32-zero fake, DD_DD_URL=http://127.0.0.1:1 so nothing can leave this machine even by accident) gives real agent status output, trimmed to the parts that matter:

$ docker run -d --name dd-fnd \
    -e DD_API_KEY=00000000000000000000000000000000 \
    -e DD_DD_URL=http://127.0.0.1:1 \
    -e DD_HOSTNAME=lesson-fnd-host \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    datadog/agent:7
$ docker exec dd-fnd agent status
===============
Agent (v7.83.3)
===============
  ...
  Build arch: arm64

$ docker exec dd-fnd agent version
Agent 7.83.3 - Commit: 8c639c92581 - Serialization version: v5.0.207 - Go version: go1.26.7

Real docker exec dd-fnd ps aux shows the process family directly — s6-supervise (the container’s process supervisor) has one entry per component, proving the “several cooperating processes” claim at the OS level, not just in prose:

root       424  s6-supervise agent
root       425  s6-supervise trace
root       426  s6-supervise process
root       427  s6-supervise sysprobe
root       429  s6-supervise security
root       430  agent run
root       431  trace-loader /etc/datadog-agent/datadog.yaml /opt/datadog-agent/embedded/bin//trace-agent --config=...
root       433  system-probe-lite run --socket ...
root       578  agent-data-plane --config ... run

agent status itself confirms two of those component processes are alive and independently reporting in over the Agent’s internal gRPC bus:

2 remote agent(s) registered:
  Name: Agent Data Plane
  Flavor: agent_data_plane
  Last seen: 25.619486636s ago

  Name: Security Agent
  Flavor: security_agent
  Last seen: 24.788567469s ago

🐶 No Cluster Agent in this container — and that's expected

This single datadog/agent:7 container run as a plain Docker container never shows a Cluster Agent, because there is nothing to run it against: the Cluster Agent is a separate Helm/Operator-deployed workload that only makes sense with a real Kubernetes cluster behind it (it talks to the Kubernetes API server for cluster-scoped state). This lesson does not stand up a kind cluster to run one live — the Helm chart and DatadogAgent CRD sections below are validated with real chart/CRD schema files instead, which is the honest way to prove config shape without a live cluster.

The APM (Trace) Agent and Process Agent sections report Not running or unreachable in this minimal container — real output, not a bug (the two have different causes, explained after the block):

=========
APM Agent
=========
  Status: Not running or unreachable on localhost:5012.
  Error: Get "https://localhost:5012/debug/vars": dial tcp [::1]:5012: connect: connection refused

Their s6-supervise entries exist (proving the process family is provisioned), but their own debug/status endpoints aren’t reachable yet at the moment agent status polled them in this idle, trace-free, process-collection-light container — a reminder that “the process exists” and “the process has fully reported in” are two different things agent status distinguishes for you.

The Process Agent line has a different cause. With DD_PROCESS_AGENT_ENABLED=true on a --network none container (Agent 7.83.3), ps shows no process-agent binary at all, agent status still prints Process Agent → Status: Not running or unreachable, and directly below it a Process Component section lists Enabled Checks: [process rtprocess] with a process check in the core Agent’s collector. The checks run inside the core Agent; the separate process s6 service exits 0. Both lines are true at once, so “Not running” here does not mean Live Processes collection is off. Full proof in Agent Architecture.

Checks and integrations

A check is a small piece of code (Python, bundled with the Agent) that knows how to talk to a specific service — Postgres, Redis, NGINX, Kafka, and hundreds of others — and turn its internal state into Datadog metrics. Each check is configured through a YAML file under the Agent’s conf.d/ directory, in a folder named after the integration:

conf.d/postgres.d/conf.yaml

init_config:

instances:
  - host: localhost
    port: 5432
    username: datadog
    password: "<PASSWORD>"
    tags:
      - "env:prod"
      - "service:orders-db"

conf.d/redis.d/conf.yaml

init_config:

instances:
  - host: localhost
    port: 6379
    tags:
      - "env:prod"
      - "service:orders-cache"

Once that file exists and the Agent restarts (or picks up the change), it starts running the postgres or redis check on a schedule and submitting metrics like postgresql.connections or redis.mem.used — tagged with whatever you put under tags, plus the Agent’s own host tags.

A real configcheck error: instances: written as a dict, not a list

AI-generated conf.d/ YAML routinely collapses instances: from a list of one item into a bare mapping — it looks equivalent, and for a single instance it even reads more naturally. It is not equivalent, and the real Agent rejects it at config-load time with a real Go type error:

# @expect-error conf.d/broken.d/conf.yaml -- instances: is a mapping, not a []map
init_config:

instances:
  host: localhost
  port: 6379

$ docker exec dd-fnd-broken agent configcheck
=== Configuration errors ===

broken: yaml: unmarshal errors:
  line 4: cannot unmarshal !!map into []integration.RawMap

instances is always a list of instance maps — even with exactly one instance, the leading - is load-bearing. This is a real, reproducible error from this lesson’s own broken container, not a paraphrase.

Autodiscovery: checks and container logs

Static conf.d/ files work fine for a host that runs the same services forever. They break down the moment services are containers that start, stop, and reschedule constantly — you can’t hand-write a config file for a Postgres pod that might not exist five minutes from now. Autodiscovery solves this: the Agent watches for containers/pods and matches them against config templates attached as annotations (Kubernetes) or labels (plain Docker), generating (and tearing down) check configs automatically as the matching containers come and go.

The current (v2) Kubernetes annotation format uses one annotation per container per purpose. This is exactly what an annotation block like this is doing on the Pod template in front of NGINX — for checks:

k8s/nginx-checks-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    ad.datadoghq.com/nginx.checks: |
      {
        "nginx": {
          "init_config": {},
          "instances": [
            {
              "nginx_status_url": "http://%%host%%:81/nginx_status/"
            }
          ]
        }
      }
spec:
  selector:
    app: nginx
  ports:
    - port: 81

and the sibling annotation for container logs, ad.datadoghq.com/<CONTAINER_NAME>.logs, on a Pod template:

k8s/orders-deployment-logs.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  # WRONG PLACE: an annotation here, on the Deployment's own metadata,
  # is invisible to Autodiscovery -- see the Callout below.
spec:
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
      annotations:
        ad.datadoghq.com/orders-api.logs: |
          [{"source": "nodejs", "service": "orders-api"}]
    spec:
      containers:
        - name: orders-api
          image: orders-api:latest

The Agent reads ad.datadoghq.com/<container-name>.checks and ad.datadoghq.com/<container-name>.logs, substitutes template variables like %%host%% with the actual pod IP at runtime, and starts the matching check or log source against whichever pod currently matches — no manual conf.d/ editing, no restart-per-pod.

🐶 Deployment metadata vs Pod template metadata — docs, quoted

Per docs.datadoghq.com/containers/kubernetes/log/: “If you define your Kubernetes Pods indirectly (with replication controllers, ReplicaSets, or Deployments), add Pod annotations to the Pod template under .spec.template.metadata.” An annotation on the Deployment’s own metadata (top of the manifest) is metadata about the Deployment object — Autodiscovery watches running Pods/containers, and a Deployment is not a Pod. The annotation has to be on .spec.template.metadata (what the manifest above does correctly) or it is silently never seen. This is proven below without a cluster, using the equivalent Docker-labels mechanism, where the same “wrong object” mistake is a typo’d label key that the running Agent simply never matches.

Proving discovery with Docker labels — no cluster required

Kubernetes annotations and Docker labels are the same underlying Autodiscovery mechanism with a different attachment point. The Docker form is com.datadoghq.ad.checks, one label holding the full check config as JSON (confirmed against docs.datadoghq.com/containers/docker/integrations/?tab=dockerlabels). Three real containers, one Agent (dd-fnd, restarted with /var/run/docker.sock mounted so it can see container labels), prove three distinct outcomes:

# 1. Correct label key, correct check name "redisdb"
$ docker run -d --name ad-redis-correct \
    --label 'com.datadoghq.ad.checks={"redisdb":{"instances":[{"host":"%%host%%","port":6379}]}}' \
    redis:7-alpine

# 2. Correct label key, WRONG check name "redis" (a very AI-plausible guess --
#    the Python package and Docker image are both called "redis"; the
#    Datadog integration is named "redisdb")
$ docker run -d --name ad-redis-good \
    --label 'com.datadoghq.ad.checks={"redis":{"instances":[{"host":"%%host%%","port":6379}]}}' \
    redis:7-alpine

# 3. WRONG label key ("check", singular -- the "wrong object" mistake,
#    Docker's equivalent of annotating the Deployment instead of the Pod template)
$ docker run -d --name ad-redis-bad \
    --label 'com.datadoghq.ad.check={"redisdb":{"instances":[{"host":"%%host%%","port":6379}]}}' \
    redis:7-alpine

agent configcheck after all three are running:

$ docker exec dd-fnd agent configcheck
=== redisdb check ===
Configuration provider: kubernetes-container-allinone
Configuration source: container:docker://bac373c7ed70e24c64dd646113a99696947388f1e310acababa4ce54a779f33c
Config for instance ID: redisdb:533fb300e611a572
host: 172.17.0.5
port: 6379
Auto-discovery IDs:
* docker://bac373c7ed70e24c64dd646113a99696947388f1e310acababa4ce54a779f33c
===

=== redis check ===
Configuration provider: kubernetes-container-allinone
Configuration source: container:docker://5bf7d03f5930c062d23327d960d51d39082cc1968c8d41a5e948ae4a3708bc96
Config for instance ID: redis:2461dfba582d49ec
host: 172.17.0.3
port: 6379
Auto-discovery IDs:
* docker://5bf7d03f5930c062d23327d960d51d39082cc1968c8d41a5e948ae4a3708bc96
===

Container 1 (ad-redis-correct, id bac373c7...) is discovered as a working redisdb check. Container 2 (ad-redis-good, id 5bf7d03f...) is discovered too — Autodiscovery matched the label fine — but as a check literally named redis, and agent status shows it never actually runs:

  Loading Errors
  ==============
    redis
    -----
      Core Check Loader:
        Check redis not found in Catalog

      Python Check Loader:
        unable to load check redis: unable to load python module datadog_checks.redis:
        unable to import module 'datadog_checks.redis': No module named 'datadog_checks.redis'

Container 3 (ad-redis-bad, id f4db23d2...) never appears anywhere in configcheck output — not as a working check, not as a loading error, nothing. The Agent’s Docker listener never matches com.datadoghq.ad.check (singular) against anything, exactly the same class of “silently invisible” failure the Kubernetes Deployment-metadata mistake produces, just without needing a cluster to prove it.

🐶 Built-in default templates are a separate mechanism

The same configcheck output also showed two more redisdb check entries with Configuration provider: file and Configuration source: file:/etc/datadog-agent/conf.d/redisdb.d/auto_conf.yaml — those came from the Agent’s own built-in default Autodiscovery templates (it ships a template for common images like redis), matched purely by image name, completely independent of any label this lesson wrote. Don’t confuse a check appearing in configcheck with “my Autodiscovery config worked” — check the Configuration source line; container:docker://<id> means your label/annotation matched, file:.../auto_conf.yaml means the Agent’s own built-in template matched instead.

Kubernetes: Helm chart vs the Datadog Operator + DatadogAgent CRD

Two supported ways to run the Agent family on Kubernetes exist today, and AI-generated manifests mix their shapes constantly because both are legitimate and well-documented:

Helm (datadog/datadog chart, real pinned version 3.250.2, appVersion: "7") installs the DaemonSet/Deployment resources directly from chart templates. Real keys from this exact chart’s values.yaml — not guessed:

datadog-values.yaml

datadog:
  apiKey: "<DATADOG_API_KEY>"
  site: datadoghq.com
  logs:
    enabled: true
    containerCollectAll: false

clusterAgent:
  enabled: true
  image:
    name: cluster-agent
    tag: 7.82.3   # the chart pins an exact patch by default -- see "image tags" below

agents:
  image:
    name: agent

clusterAgent.enabled: true is the literal key that turns on the fifth component described above — Helm treats it as one deployment inside the same chart install, not a separate product.

The Datadog Operator (DataDog/datadog-operator, real latest release v1.30.0) is the alternative Datadog recommends for larger/more complex clusters: install the Operator once, then manage the Agent declaratively through a DatadogAgent custom resource instead of a helm upgrade. This lesson fetched the Operator’s real datadoghq.com_datadogagents_v2alpha1.json OpenAPI schema straight from github.com/DataDog/datadog-operator at tag v1.30.0 and validated a real manifest offline with kubeconform (already pulled, ghcr.io/yannh/kubeconform, -schema-location pointed at the extracted schema — no cluster, no API server, nothing sent anywhere):

k8s/datadog-agent.yaml

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
  namespace: datadog-agent
spec:
  global:
    site: datadoghq.com
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: api-key
  features:
    apm:
      enabled: true
    logCollection:
      enabled: true
      containerCollectAll: true
  override:
    nodeAgent:
      image:
        name: agent
        tag: "7"

$ docker run --rm -v "$PWD":/work ghcr.io/yannh/kubeconform \
    -schema-location '/work/k8s-schema/{{.Group}}/{{.ResourceAPIVersion}}/{{.ResourceKind}}.json' \
    -summary -verbose /work/datadog-agent-cr.yaml
/work/datadog-agent-cr.yaml - DatadogAgent datadog is valid
Summary: 1 resource found in 1 file - Valid: 1, Invalid: 0, Errors: 0, Skipped: 0

The v2alpha1 CRD is strict ("additionalProperties": false) — a manifest shaped like the CRD’s older v1alpha1 API (a top-level spec.agent: block, a pattern AI training data still contains plenty of) fails real, offline schema validation immediately:

$ docker run --rm -v "$PWD":/work ghcr.io/yannh/kubeconform \
    -schema-location '/work/k8s-schema/{{.Group}}/{{.ResourceAPIVersion}}/{{.ResourceKind}}.json' \
    -summary -verbose /work/datadog-agent-cr-broken.yaml
/work/datadog-agent-cr-broken.yaml - DatadogAgent datadog is invalid: problem validating schema.
Check JSON formatting: jsonschema validation failed with '...#' - at '/spec': additional properties 'agent' not allowed
Summary: 1 resource found in 1 file - Valid: 0, Invalid: 1, Errors: 0, Skipped: 0

Current shape uses spec.global (site, credentials), spec.features (per-product toggles like apm, logCollection), and spec.override (per-component overrides like nodeAgent.image.tag) — not a flat spec.agent/spec.clusterAgent split from the older API version.

Agent 5, 6, and 7: legacy patterns you might see in AI-generated code

The course only teaches Agent 7 — that’s correct for anything shipping today — but AI training data spans years of Datadog docs and blog posts, so hallucinated or outdated snippets referencing older generations show up in generated code. Two real, docs-confirmed transitions, and where each one actually happened (this corrects a looser “Agent 6 vs 7” framing from an earlier pass of this course — the config-file change below happened at 5→6, not 6→7):

• Agent 5 → 6: config format changed from INI datadog.conf to YAML datadog.yaml (per docs.datadoghq.com/agent/faq/agent_v6_changes/: “Agent v5 datadog.conf –> Agent v6 datadog.yaml”); config directory moved from /etc/dd-agent to /etc/datadog-agent; the binary/command renamed from dd-agent to datadog-agent; three separate processes (Forwarder, Collector, DogStatsD) consolidated into one Agent process; per-process log files (collector.log, forwarder.log, dogstatsd.log) consolidated into one agent.log.
• Agent 6 → 7: Python 2 support for checks was removed. Per the official upgrade guide (docs.datadoghq.com/agent/versions/upgrade_to_agent_v7/): “Agent 7 only supports Python 3 custom checks” — and critically, “The upgrade process does not automatically move custom Agent checks”, so any hand-written Python 2 check has to be ported, not just re-deployed. If AI-generated code implies Python 2 syntax in a custom check (print statements without parens, unicode() calls), that check predates Agent 7 and will not load.

# Illustrative only -- NOT a checked fence (INI syntax, not YAML): Agent 5's
# datadog.conf. If you see this shape in AI output, it's generating for a
# Datadog Agent generation that has been end-of-life for years.
[Main]
api_key: <API_KEY>
hostname: my-host

Install methods, image tags, and troubleshooting

The Agent ships through a few common paths, chosen by where it needs to run:

# Host package install (Debian/Ubuntu, one-line installer)
DD_API_KEY=<API_KEY> DD_SITE="datadoghq.com" bash -c \
  "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

# Docker Agent container
docker run -d --name datadog-agent \
  -e DD_API_KEY=<API_KEY> -e DD_SITE="datadoghq.com" \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  gcr.io/datadoghq/agent:7

# Kubernetes, via the official Helm chart
helm repo add datadog https://helm.datadoghq.com
helm install datadog-agent datadog/datadog \
  --set datadog.apiKey=<API_KEY> --set datadog.site="datadoghq.com"

Every example above pins the image tag to 7, not latest. Real proof of why that distinction matters even though they currently agree:

$ docker inspect datadog/agent:7 --format '{{.Id}}'
sha256:da43d3ecfad79906df4d32c21c3dcee8c681ba8f0afa9a59f6663d8f5489d70c
$ docker inspect datadog/agent:latest --format '{{.Id}}'
sha256:da43d3ecfad79906df4d32c21c3dcee8c681ba8f0afa9a59f6663d8f5489d70c
$ docker inspect datadog/agent:7.83.3 --format '{{.Id}}'
sha256:da43d3ecfad79906df4d32c21c3dcee8c681ba8f0afa9a59f6663d8f5489d70c

Right now, 7, latest, and 7.83.3 all resolve to the identical image. That will not stay true: latest moves to whatever the newest tag is across every major version (including an eventual Agent 8 with its own breaking changes, the same kind this lesson just documented for the 5→6 and 6→7 transitions), while 7 only ever moves within the 7.x line — a real, intentional floor against an unannounced major-version jump, for the cost of a docker pull picking up a new minor/patch on every restart instead of a byte-identical image. Pin the full 7.83.3 instead if you need a byte-identical image across restarts.

Once it’s running, two commands answer almost every “is the Agent working?” question:

# Full status: which checks ran, when, any errors, each component process's health
sudo datadog-agent status

# Run one check once, in the foreground, with verbose output — the fastest
# way to see exactly what a specific integration is doing (or failing to do)
sudo datadog-agent check postgres

status is the first thing to run when metrics seem missing — proven above, it shows every check’s last run time, loading errors (the redis-vs-redisdb case above), and each component process’s reachability. check <name> is the tool for iterating on a single integration’s config without waiting for the next scheduled run.

🐶 The one-line takeaway

The Agent is really five cooperating components (four per node, one per cluster) — core Agent for host metrics and checks, Trace Agent for spans, Process Agent (logical; its checks run inside the core Agent in 7.83.3) for live processes, Security Agent for Cloud Security Management, and (Kubernetes only) the cluster-scoped Cluster Agent — proven above via real agent status/ps aux output. Checks/integrations (static in conf.d/, or dynamic via Autodiscovery labels/annotations) decide what actually gets collected, and a wrong check name or a misplaced annotation/label is silently invisible, not an error — proven above three different ways. On Kubernetes, the Helm chart and the Datadog Operator’s DatadogAgent CRD are both current, supported paths with different (and non-interchangeable) manifest shapes — validate CRD manifests against the real CRD schema, not intuition.

Reading AI-generated code

🐛 Spot the bug

# AI comment: "Autodiscovery config for the orders Redis cache"
apiVersion: v1
kind: Pod
metadata:
name: orders-cache
annotations:
  ad.datadoghq.com/orders-cache.checks: |
    {
      "redis": {
        "init_config": {},
        "instances": [
          { "host": "%%host%%", "port": 6379 }
        ]
      }
    }
spec:
containers:
  - name: orders-cache
    image: redis:7-alpine

Show the answer

This looks completely reasonable — a real, current-format (.checks) annotation, correctly placed on the Pod, with a plausible-looking check name. The bug, proven above with a real container: the Datadog integration for Redis is named redisdb, not redis — redis is the name of the Python package and the Docker image, which is exactly why an AI model conflates them. Real agent configcheck still shows this container as a discovered “redis check” (Autodiscovery matched the annotation fine), but real agent status shows it failing to load: Check redis not found in Catalog / unable to load python module datadog_checks.redis. The check is silently dead — no metrics, no error surfaced anywhere except the Loading Errors section of agent status, which nothing alerts on by default. Fix: "redis" → "redisdb" in both the annotation JSON key and, if migrating a static config, the conf.d/ folder name (redisdb.d/).

Mental Model

Think of the Agent install as a household of specialists sharing one address, not one worker doing everything. Core Agent is the one who reads the mail (conf.d/) and runs errands (checks) on a schedule. Trace Agent only answers the door for spans. Process Agent only watches what’s running (in 7.83.3 it works from inside the core Agent’s own house). Security Agent only handles security scanning. The Cluster Agent doesn’t even live in the same house — it’s a single office serving the whole neighborhood (cluster), so every household doesn’t independently hammer the city record office (Kubernetes API server) for the same cluster-wide facts. Autodiscovery is the household’s mail-sorting rule: a label/annotation is an envelope, and if it’s addressed to the wrong door (wrong key) or dropped at the wrong building (Deployment metadata instead of the Pod template), it isn’t returned to sender — it just never arrives, and nothing tells you that.

Trade-offs

• Helm vs the Datadog Operator — Helm is one command, well-understood by anyone who already runs Helm, and upgrades are helm upgrade; the Operator adds a CRD and a controller to manage, but gives you kubectl get datadogagent status reporting, drift correction, and a cleaner declarative story for GitOps — at the cost of one more moving part (the Operator itself) that also needs upgrading.
• Image tag 7 vs a pinned patch (7.83.3) vs latest — proven above, all three are identical today; 7 self-updates within the major version (convenient, small supply-chain risk of an unreviewed minor bump landing on restart) while a pinned patch is fully reproducible but requires you to manually bump it to get fixes; latest gives you zero warning before an eventual Agent 8 breaking change ships into a plain restart.
• Static conf.d/ vs Autodiscovery — static files are simpler to read and diff in version control, and fail loudly (file doesn’t exist, easy to notice); Autodiscovery is required for anything containerized and reschedulable, but its failure mode — proven above three times — is silence, not an error, which makes a typo far more expensive to find.

Common Misconceptions

• “The Datadog Agent is one process.” No — proven above: ps aux inside a real container shows separate s6-supervise entries for agent, trace, process, and security, plus two of them (Agent Data Plane, Security Agent) independently registering as “remote agents” with the core Agent over its internal gRPC bus.
• “An Autodiscovery annotation on a Deployment configures every Pod it creates.” No — docs, quoted above: the annotation must be on .spec.template.metadata, not the Deployment’s own metadata; proven above with the Docker-label equivalent, a mistyped label key produces a container that never appears anywhere in configcheck output at all.
• “If agent configcheck shows my check, it’s working.” Partially wrong — proven above: a check with a wrong integration name (redis instead of redisdb) is discovered and listed by configcheck, but real agent status shows it failing to load entirely, with zero metrics ever submitted.
• “The Datadog Operator replaces the Helm chart.” No — both are current, Datadog-supported, actively maintained paths (Helm chart 3.250.2, Operator v1.30.0, both released within the last few weeks of this writing); they use different manifest shapes for the same underlying Agent, and a manifest written for one is not valid input for the other, proven above by a real, strict kubeconform rejection.
• “docker pull datadog/agent:7 always gives you the same image.” No — proven by definition above: tag 7 floats to the newest 7.x release on every pull; it matched 7.83.3 at the moment this lesson was written, but will not match it forever, unlike a pinned patch tag.

How many separate cooperating processes make up "the Datadog Agent" on a Kubernetes cluster, and which one is cluster-scoped rather than per-node?

Four: core, Trace, Process, Security — all per-node, each always its own PID Five components: core Agent, Trace Agent, Process Agent, and Security Agent (all per-node; in 7.83.3 the Process Agent's checks run inside the core Agent process), plus the Cluster Agent, which runs once per cluster One process that handles everything internally Five, but the Cluster Agent runs once per node like the others

A real broken conf.d/*.d/conf.yaml fence writes instances: as a bare mapping instead of a list. What does real agent configcheck report?

It silently treats it as a one-item list yaml: unmarshal errors: line 4: cannot unmarshal !!map into []integration.RawMap A warning only, the check still runs configcheck crashes with no output

Proven above with three real Docker-labeled containers, what happened to the container whose label key was com.datadoghq.ad.check (singular) instead of com.datadoghq.ad.checks?

It ran the check anyway, Datadog normalizes the key It appeared in configcheck as a Loading Error It never appeared anywhere in configcheck output — Autodiscovery never matched the label at all The Agent container crashed

A container is labeled with the JSON check name "redis" (not "redisdb") using the correct com.datadoghq.ad.checks key. What does this lesson's real container prove happens?

Datadog aliases "redis" to "redisdb" automatically Autodiscovery matches the label and configcheck lists a "redis check", but agent status shows it failing to load: "Check redis not found in Catalog" The Agent refuses to start It is silently ignored like the wrong-label-key case

Proven above with a real kubeconform run against the Datadog Operator's actual v2alpha1 CRD schema, what happens to a DatadogAgent manifest using the older spec.agent: shape?

kubeconform accepts it since Kubernetes CRDs are always backward compatible It is rejected: "additional properties 'agent' not allowed", because the current v2alpha1 CRD only allows spec.global / spec.features / spec.override It silently falls back to v1alpha1 behavior kubeconform cannot validate custom resources at all

What real, docs-confirmed fact corrects a common oversimplification of "Agent 6 vs 7" legacy patterns?

The datadog.conf -> datadog.yaml config format change happened between Agent 5 and 6, not between 6 and 7 -- the real Agent 6->7 breaking change is dropping Python 2 support for custom checks Agent 6 and 7 are identical except for a version number datadog.conf is still valid in Agent 7 Python 2 checks still work in Agent 7 as long as ddtrace-run is used

Submit

Mark as complete