How to read Datadog config you did not write

Every earlier module taught you to write Datadog: Agent configuration, tracer setup, custom metrics, logs, monitors, SLOs, Terraform. This module is the other half — reading a conf.d/*.yaml, a tracer.init({...}) call, a Terraform datadog_monitor, or a burn-rate query that someone, or something, else wrote, and deciding whether it is correct. An AI assistant hands you a config that looks exactly like the documentation, and the success is the trap: the YAML parses, the JavaScript runs, terraform validate says nothing, and the demo dashboard fills in — while the tracer silently ignored the option you spelled log_injection, the check never loaded because tags: was a string, and the burn-rate alert is missing a required window.

Three things about Datadog make this worse than most tools. First, the surface is wide and loosely validated: a JavaScript object accepts any key, a conf.d file is parsed in two stages (the Agent’s loader, then the check’s own model), a monitor query is an opaque string to every client library. Second, training data is stale and mixed: the Agent is on its 7.83 line while tutorials show older annotation formats, Python’s tracer.configure(hostname=…) API from an earlier major, and option names from one tracer pasted into another. Third, the fastest fix an assistant reaches for is whatever it last saw — and sometimes the “fix” is to a problem that does not exist. This module’s catalog includes two false alarms (a “missing init_config” and a “deprecated” DD_TRACE_SAMPLE_RATE, which is in fact honoured by Node and silently ignored by Python) that both turned out to be different from the folklore when run.

Every claim in the four lessons here that could be run was run — against Agent 7.83.3 (datadog/agent:7), dd-trace 6.17.0, ddtrace 4.15.2, @datadog/browser-rum 7.15.0, datadog-api-client 2.61.0, and the datadog Terraform provider 4.22.0 — with a fake API key and nothing ever sent to Datadog. Where only the Datadog API or the docs can decide, the lesson says so and quotes the docs URL.

flowchart TD
  A["1. Versions and pins<br/>which Agent, tracer, provider? floating tags?"] --> B["2. Agent config<br/>conf.d shape, check name, list vs scalar, DD_SITE"]
  B --> C["3. Kubernetes wiring<br/>Autodiscovery on the Pod template, container name, v2 format"]
  C --> D["4. Tracer options<br/>real option names, init order, env case"]
  D --> E["5. Custom metrics<br/>histogram vs distribution, bounded tags"]
  E --> F["6. Terraform and monitor JSON<br/>provider schema, recovery, burn-rate grammar, enums"]
  F --> G["7. Prove it<br/>configcheck, status, config dump, validate, models, harness"]

Reading order for Datadog config you did not write

1. Versions and pins

Check which versions the code assumes before judging anything else. Is the Agent image latest or a floating major instead of an exact tag or digest? Is a Terraform provider unpinned or pinned to something that lacks a resource? Does a claim like “X is deprecated” come from the docs page or from memory? The real cost here is reproducibility: on the day this module was captured the 7, 7.83.3, and latest tags were the same image, which is exactly why nobody notices. See catalog entries 7 and 13.

2. Agent configuration

Does the conf.d/<name>.d/ directory use the check’s name (redisdb, not redis)? Is instances: a list? Are tags: and every other list-typed key YAML lists? Is DD_SITE a domain and not a URL? The catch: agent configcheck prints a scalar tags, a wrong check name, and a wrong-typed value without an error and exits 0 even when it does report one — the truth is in agent status. See entries 1–6.

3. Kubernetes wiring

Are Autodiscovery annotations on the Pod template (spec.template.metadata), keyed by the container name, in the consolidated .checks format (Agent 7.36+)? Are DD_ENV/DD_SERVICE/DD_VERSION derived from one lower-case source? This step is the one most dependent on docs rather than a local run — kubectl accepts every wrong placement. See entries 8 and 12.

4. Tracer options

Does every key in tracer.init({...}) exist in TracerOptions? Is the tracer initialised before the libraries it must instrument? Does a Python file read attributes that exist and call tracer.configure with keywords that exist? JavaScript accepts any key silently, and Python accepts writes to any attribute. See entries 9–11 and 13–16.

5. Custom metrics

Is a “global percentile” requirement implemented with a distribution, not a histogram? Is every tag value bounded? These run cleanly and cost money or accuracy later; both were proven with a real Agent’s flush. See entries 17 and 18.

6. Terraform and monitor JSON

Does each resource validate with the pinned provider? Are recovery thresholds inside monitor_thresholds, block names singular, dashboard queries metric_query plus formula? Is a burn-rate query burn_rate(…).over(<SLO window>).long_window(…).short_window(…)? Are enums real? Here terraform validate and the model constructor are useful and incomplete, and the incompleteness is measured. See entries 19–27.

7. Prove it

Choose the tool that sees the layer where the bug lives, and write down what the green result does not cover. That is Verification tools, with the Review checklist as the one-page version.

What this module covers next

• The AI Bug Catalog — 27 patterns, each with the code AI typically writes, the real output captured for it, an honest note where only the API or the docs can decide, and the fix. Includes the three bugs the course’s own review found (log_injection, the flat dashboard query, the burn-rate query).
• Review Checklist — the same patterns as a seven-group checklist, each line naming the tool that proves it.
• Verification Tools — what agent configcheck, agent status, the tracer config dump, terraform validate, the API client models, and this course’s harness (npm run verify, --agent, --k8s) each prove, including the measured blind spots.
• Three worked reviews applying this reading order to full AI-generated systems: Worked Review 1 — a Node service (dd-trace, logs, unified service tagging), Worked Review 2 — a Kubernetes cluster (Agent via Helm or the Operator, Autodiscovery, the Cluster Agent), and Worked Review 3 — monitors, SLOs and dashboards as code (Terraform plus JSON). These three are written and maintained separately from this page; if one of those links 404s while that work is in progress, the reading order, catalog, checklist, and tools stand on their own.
• Every Phase 3 lesson already has a Reading AI-generated code section; this module does not repeat those — it groups a broader set around the reading order. Start from What is Datadog for the lesson-by-lesson path.

Mental Model

Reading order for Datadog config you did not write:

1. Versions and pins   -> which Agent / tracer / provider, floating tags?
2. Agent config        -> check name, list vs mapping, scalar tags, DD_SITE
3. Kubernetes wiring   -> Pod template, container name, .checks (v2)
4. Tracer options      -> real option names, init order, env case
5. Custom metrics      -> distribution vs histogram, bounded tags
6. Terraform / JSON    -> provider schema, recovery, burn-rate grammar, enums
7. Prove it            -> the tool that sees the layer where the bug lives

"It ran / it validated" answers almost none of these. Most of the 27 catalog
entries pass at least one tool and fail another.

Common Misconceptions

• “If agent configcheck is clean, the integration will run.” Proven false three ways in the catalog: scalar tags, the wrong check name, and a wrong-typed value all print a normal configuration and fail in agent status.
• “Unknown option keys are rejected.” Not in JavaScript (log_injection is silently ignored) and not on a Python Config object (a write creates an attribute); only a type checker or the harness sees them.
• “The API client models validate my monitor JSON.” They check scalar types and required fields. type: "metrics alert" and a burn-rate query with no .short_window() both construct; _unparsed catches the enum, and only the API catches the grammar.
• “A reviewer who flags more is safer.” The catalog’s entries 5 and 13 are false alarms; time spent on them is time not spent on DD_SITE as a URL.

🐶 The one-line takeaway

Read Datadog config in this order — versions, Agent config, Kubernetes wiring, tracer options, custom metrics, Terraform and JSON, then the proof — because each layer’s tool accepts what the layer above assumed, and “it ran” tells you only that the parser agreed with the file, never that Datadog will do what the author meant.

Why does the reading order put 'Versions and pins' before everything else?

Because version numbers are the most common bug Because option names, annotation formats, and defaults change between versions - a config can be correct for one Agent, tracer, or provider and meaningless for another, so you cannot judge it before knowing which one it targets Because Datadog only supports one version at a time Because pinning makes the tools run faster

tracer.init({ log_injection: false }) in a .js file on dd-trace 6.17.0 - what happens?

It throws an exception at startup It runs; the key is ignored and log injection stays at its default of true, as shown by the tracer's own config dump It disables log injection It prints a deprecation warning and works

A conf.d file has tags: env:prod (a scalar). What do agent configcheck and agent status show on Agent 7.83.3?

Both report an error configcheck prints the config with no error; agent status lists a Loading Error 'cannot unmarshal !!str env:prod into []string' Neither reports anything configcheck errors, status is clean

Which of these should a careful reviewer NOT report as a bug, having run the tool?

conf.d/redis.d/ for the redisdb check A conf.d file without init_config: - it loaded, and the only error was that the host did not resolve DD_SITE=https://api.datadoghq.com instances: written as a mapping

You want to check a burn-rate monitor's JSON with the client model. What does that prove?

That the burn_rate grammar is valid Only scalar types and required fields - the query is an opaque string to the model, so a query with the wrong .over() and no .short_window() constructs fine; the grammar is in the docs and enforced by the API That the SLO exists That the alert will page

Where do the three worked reviews of this module live, and what do they cover?

They are in this page Separate lessons: worked-review-1-node-service, worked-review-2-kubernetes-cluster, and worked-review-3-monitors-slos-as-code They are in the Phase 3 lessons They do not exist

Submit

Mark as complete