Skip to content
Datadog — From Zero to Hero

Worked review 2: the Agent on a Kubernetes cluster

The feature

Section titled “The feature”

The prompt was: “Deploy the Datadog Agent to our EKS cluster with Helm — APM, logs from every container, a Cluster Agent, and a Redis integration for the cache Deployment. Give me the Operator version as well, we might switch.” What follows is the first draft. It is three artifacts: a Helm values.yaml, a DatadogAgent manifest for the Operator, and the Redis Deployment with Autodiscovery annotations. helm lint passes on the first; the other two look like YAML that any reviewer would nod at.

What was used to review it, and the limits of each:

  • helm template against the real chart, datadog/datadog 3.250.3 (Helm v4.3.0) — the rendered DaemonSet is the ground truth of what the values do. Nothing was installed anywhere.
  • The Operator’s DatadogAgent CRD, taken from the datadog-operator chart 2.27.0 (Operator app version 1.30.0, CRD sub-chart 2.24.0). The CRD serves exactly one version, v2alpha1. It was converted to a strict JSON Schema (additionalProperties: false wherever the CRD does not set x-kubernetes-preserve-unknown-fields), which emulates what kubectl apply --validate=strict asks the API server to do. It was not run against a live API server — there is no cluster, and the course harness’s --k8s mode needs a kind cluster that does not exist here.
  • agent configcheck from a real datadog/agent 7.83.3 container (fake API key, started with --network none so nothing can leave the container, removed afterwards) with the fixed Redis template mounted from disk.
  • The Kubernetes 1.37 apps/v1 Deployment JSON schema from the Kubernetes course, for the Deployment.

The generated code

Section titled “The generated code”
# @expect-error as generated -- see the review below (helm lint passes; helm template renders)
datadog:
  apiKey: "<your-api-key>"
  site: https://api.datadoghq.com
  tags: env:prod
  apm:
    enabled: true
  logs:
    enabled: true
    collectAll: true
  dogstatsd:
    port: 8125
agents:
  image:
    tag: latest
clusterAgent:
  enabled: true
# @expect-error the Operator manifest as generated -- v1alpha1-shaped fields under v2alpha1
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
  namespace: datadog
spec:
  credentials:
    apiKey: "<your-api-key>"
  global:
    clusterName: prod-eks-1
    site: https://api.datadoghq.com
  features:
    apm:
      enabled: true
    logCollection:
      enabled: true
      containerCollectAll: true
  agent:
    image:
      name: datadog/agent:latest
  clusterAgent:
    enabled: true
# @expect-error Autodiscovery annotations on the Deployment, keyed to the wrong container name
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache
  namespace: shop
  annotations:
    ad.datadoghq.com/redis.checks: |
      {
        "redisdb": {
          "init_config": {},
          "instances": [{ "host": "%%host%%", "port": "6379" }]
        }
      }
    ad.datadoghq.com/redis.logs: '[{"source": "redis", "service": "cache"}]'
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cache
  template:
    metadata:
      labels:
        app: cache
    spec:
      containers:
        - name: cache
          image: redis:latest

The review walk

Section titled “The review walk”

Part 1 — the Helm values. Every finding here is a line of the rendered DaemonSet.

site: https://api.datadoghq.com — a URL where a domain goes

Section titled “site: https://api.datadoghq.com — a URL where a domain goes”

helm lint passed (1 chart(s) linted, 0 chart(s) failed) and helm template exited 0. The rendered agent container:

DD_SITE = https://api.datadoghq.com

The chart passes it straight through, and so does the Agent. Started with that value, agent status shows how the Agent builds its intake hostnames (the same behaviour as bug catalog entry 6, re-run for this review):

$ docker exec dd-reading-b agent status      # DD_SITE=https://api.datadoghq.com
    https://process.https://api.datadoghq.com. - API Key ending with:
$ docker exec dd-reading-b agent status      # DD_SITE=datadoghq.com
    https://process.datadoghq.com. - API Key ending with:

🐛 Spot the bug

datadog:
site: https://api.datadoghq.com
Show the answer

site is a Datadog site domain (datadoghq.com, datadoghq.eu, …), not an API URL; the intake hostnames are built by prefixing it (list of sites: docs.datadoghq.com/getting_started/site/). Neither Helm nor the Agent validates it — the failure is an Agent that runs, and never delivers. Fix: site: datadoghq.com.

The API key committed in values.yaml

Section titled “The API key committed in values.yaml”

apiKey: "<your-api-key>" — a placeholder here, but the shape is a literal key in a file that lives in git. Rendered, the chart turns it into a Secret whose api-key is just base64 of the string:

dd-datadog  api-key  <your-api-key>          (base64-decoded from the rendered Secret)

The chart’s own values.yaml documents the alternative: datadog.apiKeyExistingSecret — “Use existing Secret which stores API key … The value should be set with the api-key key inside the secret”, and “If set, this parameter takes precedence over apiKey”. With it set, the rendering contains no Secret object with the key at all (0 kind: Secret documents in the fixed render).

🐛 Spot the bug

datadog:
apiKey: "<your-api-key>"
Show the answer

A working key in a values file means the key is in git history, in CI logs that print helm template, and in every review screenshot. The Operator manifest below repeats the mistake in a place where the schema allows it. Fix: create the Secret out of band (kubectl create secret generic datadog-secret --from-literal api-key=...) and reference it with apiKeyExistingSecret.

tags: env:prod — a string where a list belongs

Section titled “tags: env:prod — a string where a list belongs”

The chart documents datadog.tags as a “List of static tags”. Helm did not complain about a string, and the chart does toJson on whatever it is given:

values  tags: env:prod        ->  DD_TAGS = "env:prod"        (string, JSON-quoted)
values  tags: [env:prod]      ->  DD_TAGS = ["env:prod"]

What the Agent does with each, from agent config and agent status in a real 7.83.3 container started with exactly those two DD_TAGS values:

DD_TAGS='"env:prod"'      tags:
                            - '"env:prod"'
DD_TAGS='["env:prod"]'    tags:
                            - env:prod            host tags: env:prod

The scalar form produces a tag whose text contains the double quotes — it will never match env:prod in a monitor scope or a dashboard filter.

🐛 Spot the bug

datadog:
tags: env:prod
Show the answer

The type is wrong and nothing enforces it: the chart’s values.schema.json only describes a handful of keys (top-level properties are registryMigrationMode and datadog, and under datadog just instrumentationCrd, networkPath, apm, appsec), so tags is unchecked. Fix: a YAML list — tags: then - env:prod. Better still, do not put env here at all; unified service tagging sets it per workload.

logs.collectAll — an option the chart does not have

Section titled “logs.collectAll — an option the chart does not have”

The request was “logs from every container”. The rendered container:

DD_LOGS_ENABLED = true
DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL = false

The real key is datadog.logs.containerCollectAll (default false; the chart comment: “Enable this to allow log collection for all containers”). collectAll is ignored. The proof is a diff: rendering logs: {enabled: true, collectAll: true} against logs: {enabled: true} produced identical output apart from the random Cluster Agent token, checksum and install_id lines. And because unknown values keys are simply carried along by Helm, helm lint cannot see it either.

With enabled: true the log plumbing is all there — the rendered DaemonSet mounts /var/log/pods, /var/log/containers and the pointer directory — so the failure is subtle. The docs (docs.datadoghq.com/containers/kubernetes/log/) state the rule: “When set to false (default), you need to specify Autodiscovery log configurations to enable log collection.” Every container without its own logs annotation is silently skipped.

🐛 Spot the bug

  logs:
  enabled: true
  collectAll: true
Show the answer

collectAll looks right and is silently ignored. The key is containerCollectAll: true. Turning it on has a cost the assistant did not mention: every container’s stdout, including noisy sidecars and the Agent itself, is ingested; use datadog.containerExcludeLogs to trim it (the trade-off is in the section below).

agents.image.tag: latest

Section titled “agents.image.tag: latest”

Rendered: registry.datadoghq.com/agent:latest on the agent, trace-agent, system-probe, init-* and seccomp-setup containers. The chart’s own default, with no tag set, is a pinned registry.datadoghq.com/agent:7.82.3 for chart 3.250.3 — the assistant made the deployment less reproducible than doing nothing. The tags do exist on Docker Hub (docker manifest inspect datadog/cluster-agent:7.83.3 resolves; the Agent tag was pulled and run for this review). latest moves under a running DaemonSet: the next node that pulls gets a different Agent from its neighbours.

🐛 Spot the bug

agents:
image:
  tag: latest
Show the answer

Not broken today - the risk is that nobody can say which Agent produced a given metric, and a node restart mid-week can silently upgrade one node. Pin agents.image.tag and clusterAgent.image.tag (the Cluster Agent has its own image setting, which the AI left unset), and bump both on purpose. Fix: tag: 7.83.3.

The Cluster Agent token is different on every render

Section titled “The Cluster Agent token is different on every render”

clusterAgent.enabled: true with no token and no tokenExistingSecret. The chart comment: “autogenerated if empty, needs to be at least 32 characters”. Two consecutive helm template runs of the same values:

render 1  dd-datadog-cluster-agent  token  PIjV...      checksum/clusteragent_token: b2b26b5d...
render 2  dd-datadog-cluster-agent  token  qlPr...      checksum/clusteragent_token: 82e66f08...

(only the first characters of each token are shown). The token changes on every render, and the value is hashed into a pod-template annotation, so every helm upgrade rolls the whole DaemonSet and the Cluster Agent even when nothing else changed. With tokenExistingSecret set, the fixed render is byte-identical across runs except the unrelated install_id.

🐛 Spot the bug

clusterAgent:
enabled: true
Show the answer

The pre-shared token between node Agents and the Cluster Agent has to be stable. Fix: create a Secret with a token key (32+ characters) once, and set clusterAgent.tokenExistingSecret. This is the same class of bug as a hard-coded API key, with the opposite polarity: here the generated value is too fresh.

No clusterName

Section titled “No clusterName”

The rendered DaemonSet has no DD_CLUSTER_NAME at all (its absence was checked, not assumed: the template only emits it inside if .Values.datadog.clusterName). The chart’s own comment on the key: “Set a unique cluster name to allow scoping hosts and Cluster Checks easily”. The tag reference (docs.datadoghq.com/containers/kubernetes/tag/) lists kube_cluster_name with source “DD_CLUSTER_NAME envvar or cloud provider integration” — so on a cluster with no cloud integration matching, there is no cluster tag to scope a monitor to “prod-eks-1” rather than “any cluster”.

🐛 Spot the bug

datadog:
site: https://api.datadoghq.com
tags: env:prod
Show the answer

Nothing in this block names the cluster. The Operator draft did set clusterName, which is why the two artifacts disagree. Fix: datadog.clusterName: prod-eks-1 (lowercase letters, digits, hyphens; the chart itself fails on an invalid name).

dogstatsd.port: 8125 — sets the port, exposes nothing

Section titled “dogstatsd.port: 8125 — sets the port, exposes nothing”

The rendered agent container ports:

agent        [{'containerPort': 8125, 'name': 'dogstatsdport', 'protocol': 'UDP'}]
trace-agent  [{'containerPort': 8126, 'hostPort': 8126, 'name': 'traceport', 'protocol': 'TCP'}]

APM has a hostPort (the deprecated apm.enabled: true still enables it — see Misconceptions); DogStatsD has none, because port is the container port and the chart default for datadog.dogstatsd.useHostPort is false. Applications that send metrics to status.hostIP:8125 — the same pattern as bug 11 of the Node review — reach nothing. The chart comment for useHostPort: “Sets the hostPort to the same value of the container port. Needs to be used for sending custom metrics.” With useHostPort: true the render shows 'hostPort': 8125. The other supported transport, on by default, is the Unix socket (useSocketVolume: true), which the application pod must mount.

🐛 Spot the bug

  dogstatsd:
  port: 8125
Show the answer

port: 8125 restates the default. The question that matters is how the application reaches DogStatsD — hostPort or the socket volume — and the values file never answers it. Fix: decide, and set useHostPort: true (or mount the socket in the app).

Part 2 — the Operator manifest.

v1alpha1-shaped fields under apiVersion: v2alpha1

Section titled “v1alpha1-shaped fields under apiVersion: v2alpha1”

The CRD in operator chart 2.27.0 declares exactly one version: [('v2alpha1', True, True)] (name, served, storage). So apiVersion: datadoghq.com/v1alpha1 — the shape the assistant learned from older tutorials — is not served at all (the API server’s answer to that is the generic no matches for kind "DatadogAgent" in version error; not run here). This draft got the apiVersion right and the body wrong. The strict-schema check:

$ python3 vcr.py bad-dda.yaml
apiVersion: datadoghq.com/v2alpha1
  /spec -> Additional properties are not allowed ('agent', 'clusterAgent', 'credentials' were unexpected)

spec in v2alpha1 has exactly three properties: features, global, override. credentials moved to global.credentials; agent.image became override.nodeAgent.image (with separate name and tag); the Cluster Agent’s settings live under override.clusterAgent (and its image under override.clusterAgent.image). Kubernetes’ own docs describe what the API server does with unknown fields: “Possible levels of validation are Ignore, Warn (default), and Strict” (kubernetes.io/docs/reference/using-api/api-concepts/#field-validation), and kubectl apply --help on this machine (client v1.36.1) lists --validate='strict' as the default. With those two levels the unknown fields are not an error; a CRD with a structural schema prunes them, so the stored DatadogAgent would be missing the blocks (this pruning was not run here — no API server), and an Agent with no credentials is the natural result.

🐛 Spot the bug

spec:
credentials:
  apiKey: "<your-api-key>"
agent:
  image:
    name: datadog/agent:latest
clusterAgent:
  enabled: true
Show the answer

The blocks are valid v1alpha1 fields in a v2alpha1 document. A strict-schema check flags them; a lenient apply drops them silently, which is the worst outcome — a healthy-looking DatadogAgent with none of the intended configuration. Fix: spec.global.credentials, spec.override.nodeAgent.image, spec.override.clusterAgent.image.

The schema is green and the manifest is still wrong

Section titled “The schema is green and the manifest is still wrong”

Move the three misplaced blocks to their v2alpha1 homes and change nothing else (site still a URL, the key still literal, latest still there):

$ python3 vcr.py mid-dda.yaml
apiVersion: datadoghq.com/v2alpha1
OK: no schema violations

The schema validates structure and types, not meaning. The CRD has no pattern for site, and credentials.apiKey is a legal string. The same three value problems from Part 1 — the URL site, the plaintext key, the floating tag — sail through. (A negative control shows the checker is not vacuous: containerCollectall gives Additional properties are not allowed ('containerCollectall' was unexpected) at /spec/features/logCollection, and enabled: "yes" gives 'yes' is not of type 'boolean'.)

🐛 Spot the bug

  global:
  site: https://api.datadoghq.com
  credentials:
    apiKey: "<your-api-key>"
Show the answer

“It validates” is a statement about the schema. The API key belongs in spec.global.credentials.apiSecret (secretName and keyName); the site is datadoghq.com. Both are review items no offline schema can raise.

Part 3 — Autodiscovery and the Redis Deployment.

Autodiscovery annotations on the Deployment

Section titled “Autodiscovery annotations on the Deployment”

The annotations sit on the Deployment’s own metadata. Autodiscovery reads pod annotations. From the docs (docs.datadoghq.com/containers/kubernetes/integrations/): “If you define pods indirectly (with deployments, ReplicaSets, or ReplicationControllers) add pod annotations under spec.template.metadata.” The pod template is what the Deployment creates pods from; the Deployment’s own metadata is not copied to the pods. Checking the two manifests with the Kubernetes Deployment schema and a small script:

bad-cache.yaml  schema: OK
  Deployment.metadata.annotations keys : ['ad.datadoghq.com/redis.checks', 'ad.datadoghq.com/redis.logs']
  pod template annotations keys        : []
good-cache.yaml schema: OK
  Deployment.metadata.annotations keys : []
  pod template annotations keys        : ['ad.datadoghq.com/cache.checks', 'ad.datadoghq.com/cache.logs']

Both are schema-valid. kubectl apply says deployment.apps/cache created for either; the difference is only visible on the running pods, in the node Agent’s agent configcheck. That last step needs a cluster and was not run here — the claim rests on the docs quote above.

🐛 Spot the bug

kind: Deployment
metadata:
name: cache
annotations:
  ad.datadoghq.com/redis.checks: |
    { ... }
Show the answer

A Deployment’s metadata.annotations is documentation for humans and tools; the pods get spec.template.metadata.annotations. Fix: move both annotations under spec.template.metadata. The same rule applies to StatefulSet, DaemonSet, Job and CronJob (spec.jobTemplate.spec.template.metadata for the last).

ad.datadoghq.com/redis.checks — the identifier is the container name, not the technology

Section titled “ad.datadoghq.com/redis.checks — the identifier is the container name, not the technology”

Even on the pod template, the annotation key is ad.datadoghq.com/<container-name>.checks. The container is called cache; the annotation says redis. The docs describe the identifier in the annotation form as “an identifier to match against the names (spec.containers[i].name, not spec.containers[i].image) of the containers that correspond to your integration”. The script above applied that rule to both manifests:

deployment: ad.datadoghq.com/redis.checks JSON ok; identifier 'redis' matches a container name: False
pod-template: ad.datadoghq.com/cache.checks JSON ok; identifier 'cache' matches a container name: True

The template itself, run through a real Agent as a file-based Autodiscovery config with ad_identifiers: [cache], is what the fixed annotation expands to. agent configcheck -v in the 7.83.3 container lists it, and shows exactly the unresolved form the node Agent waits to match against a running container:

Auto-discovery IDs: cache
check_name: redisdb
init_config: null
instances:
- host: '%%host%%'
  port: "6379"
logs_config: null

That listing (under “Collected configs (matched and unmatched)”) proves the template parses and is registered under the identifier cache; it does not prove a pod matched it — the container listener has no Kubernetes to look at in this run, so the companion “Services (matched and unmatched)” section is empty. The wrong identifier redis would be registered just as happily and never match.

🐛 Spot the bug

ad.datadoghq.com/redis.checks: |
...
containers:
- name: cache
Show the answer

An annotation prefix that names a technology instead of the container is the second most common Autodiscovery mistake (after wrong-object placement, above). The two errors mask each other in review because each is individually plausible. Fix: ad.datadoghq.com/cache.checks and ad.datadoghq.com/cache.logs on the pod template. The v1 three-key format (check_names, init_configs, instances) is still documented next to v2 on the same page, so it is not the bug here; v2 (.checks, Agent 7.36+) is simply the one-annotation form.

The corrected code

Section titled “The corrected code”
helm/values.yaml
datadog:
  apiKeyExistingSecret: datadog-secret
  site: datadoghq.com
  clusterName: prod-eks-1
  tags:
    - env:prod
  apm:
    portEnabled: true
  dogstatsd:
    useHostPort: true
  logs:
    enabled: true
    containerCollectAll: true
agents:
  image:
    tag: 7.83.3
clusterAgent:
  enabled: true
  image:
    tag: 7.83.3
  tokenExistingSecret: datadog-cluster-agent-token
scripts/create-datadog-secrets.sh
kubectl -n datadog create secret generic datadog-secret \
  --from-literal api-key=<your-api-key>
kubectl -n datadog create secret generic datadog-cluster-agent-token \
  --from-literal token=<random-string-of-32-or-more-characters>

helm lint reports 1 chart(s) linted, 0 chart(s) failed. Rendered, the agent container has:

DD_API_KEY = secretKeyRef datadog-secret / api-key
DD_CLUSTER_NAME = prod-eks-1
DD_TAGS = ["env:prod"]
DD_SITE = datadoghq.com
DD_CLUSTER_AGENT_AUTH_TOKEN = secretKeyRef datadog-cluster-agent-token / token
DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL = true
agent        containerPort 8125 hostPort 8125 (UDP)
trace-agent  containerPort 8126 hostPort 8126 (TCP)

and two consecutive renders differ only in the unrelated install_id UUID. The Operator equivalent, which passes the strict v2alpha1 check (OK: no schema violations):

k8s/datadog-agent.yaml
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
  namespace: datadog
spec:
  global:
    clusterName: prod-eks-1
    site: datadoghq.com
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: api-key
    clusterAgentTokenSecret:
      secretName: datadog-cluster-agent-token
      keyName: token
    tags:
      - env:prod
  features:
    apm:
      enabled: true
      hostPortConfig:
        enabled: true
        hostPort: 8126
    dogstatsd:
      hostPortConfig:
        enabled: true
        hostPort: 8125
    logCollection:
      enabled: true
      containerCollectAll: true
    admissionController:
      enabled: true
    clusterChecks:
      enabled: true
      useClusterChecksRunners: true
  override:
    nodeAgent:
      image:
        name: datadog/agent
        tag: 7.83.3
    clusterAgent:
      image:
        name: datadog/cluster-agent
        tag: 7.83.3
k8s/cache-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache
  namespace: shop
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cache
  template:
    metadata:
      labels:
        app: cache
      annotations:
        ad.datadoghq.com/cache.checks: |
          {
            "redisdb": {
              "init_config": {},
              "instances": [{ "host": "%%host%%", "port": "6379" }]
            }
          }
        ad.datadoghq.com/cache.logs: '[{"source": "redis", "service": "cache"}]'
    spec:
      containers:
        - name: cache
          image: redis:7.4
conf.d/redisdb.d/conf.yaml
ad_identifiers:
  - cache
init_config:
instances:
  - host: "%%host%%"
    port: "6379"

The last fence is the file-based twin of the annotation and is a collected fence: the course harness mounts it into a real Agent container and runs agent configcheck.

What changed and why

Section titled “What changed and why”
#BugFixProven by
1site: https://api.datadoghq.comsite: datadoghq.comrendered DD_SITE; agent status intake URL
2Literal apiKey in valuesapiKeyExistingSecretrendered Secret decodes to the key; 0 Secrets after fix
3tags: env:prod stringYAML listDD_TAGS = "env:prod"; agent config shows the quoted tag
4logs.collectAll (invented key)logs.containerCollectAllCOLLECT_ALL = false; identical render with/without the key
5agents.image.tag: latestpin Agent and Cluster Agent tagsrendered image; chart default is 7.82.3
6No Cluster Agent tokentokenExistingSecrettoken and checksum/clusteragent_token differ per render
7No clusterNamedatadog.clusterNameno DD_CLUSTER_NAME rendered
8DogStatsD port with no hostPortuseHostPort: true (or socket)rendered container ports
9v1alpha1-shaped fields in v2alpha1global.credentials, override.*strict CRD schema: 3 unexpected properties
10Schema-valid but wrong valuesapiSecret, datadoghq.comOK: no schema violations on the still-wrong manifest
11AD annotations on the Deploymentspec.template.metadata.annotationsdocs quote; both manifests schema-valid
12Annotation prefix redis, container cachead.datadoghq.com/cache.*script + configcheck -v registers the template under cache

What the reviewer writes in the PR

Section titled “What the reviewer writes in the PR”

12 findings, none of them a syntax error — helm lint is green and every manifest is valid YAML:

  1. Helm values: site is a domain (datadoghq.com), not a URL; tags is a list; there is a literal API key (use apiKeyExistingSecret); logs.collectAll does not exist — the key is containerCollectAll, and today DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL renders false.
  2. Reproducibility: pin agents.image.tag and clusterAgent.image.tag; set clusterAgent.tokenExistingSecret — today every helm upgrade regenerates the token and rolls the DaemonSet.
  3. Missing: datadog.clusterName, and a decision on how apps reach DogStatsD (useHostPort or the socket).
  4. Operator manifest: credentials, agent and clusterAgent are v1alpha1 fields; in v2alpha1 they are global.credentials, override.nodeAgent and override.clusterAgent. Run the strict schema — and note that “valid” still says nothing about the plaintext key or the URL site.
  5. Redis Autodiscovery: annotations go on spec.template.metadata, and the key prefix is the container name (cache), not redis.
  6. Not verified here, please check on a real cluster: agent configcheck on the node Agent for the cache pod, and that logs from a container without annotations actually arrive.

Mental Model

Section titled “Mental Model”
flowchart TD
  V["values.yaml"] -->|"1, 3, 5, 7: wrong value shape or missing"| R["helm template"]
  V -->|"2: literal key in git"| R
  V -->|"4: invented key ignored"| R
  V -->|"6: token differs per render"| R
  V -->|"8: port without hostPort"| R
  R --> D["DaemonSet + Cluster Agent"]
  O["DatadogAgent manifest"] -->|"9: v1alpha1 fields"| S["v2alpha1 CRD schema"]
  O -->|"10: valid but wrong values"| S
  S --> D
  A["Deployment annotations"] -->|"11: wrong object"| P["pod"]
  A -->|"12: wrong container identifier"| P
  D -->|"node Agent reads pod annotations"| P
Where each bug sits between values and a running pod

Look at what catches what. helm template makes bugs 1, 3, 4, 5, 6, 7 and 8 visible because the rendered DaemonSet is a plain text you can read — but only if you look for what is missing (a DD_CLUSTER_NAME, a hostPort). The strict CRD schema catches 9 and, by design, cannot catch 10. Bugs 11 and 12 are invisible to every offline tool in this review: both manifests validate against the Kubernetes schema. The honest position is a table with a column labelled “needs a cluster”.

Trade-offs

Section titled “Trade-offs”
  • Helm chart vs Operator. The chart renders plain manifests you can helm template and diff in CI; the Operator turns one small custom resource into those manifests at runtime, so “what will be deployed” is decided inside the cluster and you review the intent, not the result. The strict schema protects the Operator path; the render protects the chart path.
  • containerCollectAll: true. It is the fastest way to “logs from everything”, and it ships everything: noisy sidecars, health-check chatter, the Agent’s own logs. Per-workload ad.datadoghq.com/<container>.logs annotations plus exclusion filters cost more setup and control the bill.
  • hostPort vs socket for APM and DogStatsD. A hostPort is reachable with only status.hostIP and no volume mount, at the cost of opening a port on every node; the Unix socket needs a hostPath mount in the application pod but avoids the network hop. Both are supported by the chart and the Operator; the review’s job is that one was chosen.

Common Misconceptions

Section titled “Common Misconceptions”
  • “helm lint passed, so the values are right.” Proven above: lint passed for a values file with a URL as the site, an invented logs key and a string for a list.
  • “apm.enabled: true is a bug, it is deprecated.” It is deprecated (the chart comment says “DEPRECATED. Use datadog.apm.portEnabled instead”) but it works: the render shows DD_APM_ENABLED = true, DD_APM_RECEIVER_PORT = 8126 and a hostPort: 8126 on the trace-agent container. Rewrite it for the future, but do not call it broken.
  • “If the CRD schema validates, the DatadogAgent is correct.” Proven above: the still-wrong manifest returned OK: no schema violations.
  • “Autodiscovery annotations can go anywhere on the workload.” They go on the pod (spec.template.metadata for a Deployment), keyed by the container name. Both variants pass kubectl apply.
  • “v1 annotations (check_names) are removed.” The docs page still shows v1 beside v2; v2 needs Agent 7.36+, which every current Agent satisfies.
helm lint reports 0 chart(s) failed for a values file containing site: https://api.datadoghq.com. What does helm template render, and what does the Agent do with it?
A values file sets datadog.logs.collectAll: true. Comparing the render with and without that line, what is the result?
helm template run twice on the same values (no clusterAgent.token, no tokenExistingSecret) shows different checksum/clusteragent_token annotations. What is the consequence for helm upgrade?
A DatadogAgent with apiVersion: datadoghq.com/v2alpha1 has spec.credentials, spec.agent and spec.clusterAgent. What does the strict CRD schema report, and what does a lenient apply do?
After moving the misplaced blocks to spec.global and spec.override, the manifest still has site: https://api.datadoghq.com and a literal apiKey. What does the schema check print?
Redis Autodiscovery annotations are on the Deployment's metadata.annotations with the prefix ad.datadoghq.com/redis.; the container is named cache. Which statement is accurate?