Verification tools

None of the tools below overlaps perfectly, and none is complete alone. The AI bug catalog’s Caught by line names, for each of its 27 entries, which one (if any) would have caught it — and a good share name only agent status, or nothing offline. Every output here is real: datadog/agent:7 = Agent 7.83.3 (container dd-reading-a, fake DD_API_KEY, DD_DD_URL pointed at 127.0.0.1 or a local listener, no traffic to Datadog), dd-trace 6.17.0, ddtrace 4.15.2, datadog-api-client 2.61.0, datadog Terraform provider 4.22.0 on Terraform v1.16.4 (no plan, no apply, cloud credentials unset), all run on one Mac.

flowchart TD
  F["Config file / code fence"] --> S["Shape and names<br/>(configcheck, tsc, terraform validate, model constructor)"]
  S --> V["Values and types<br/>(agent status, config dump, ApiTypeError)"]
  V --> R["Runtime behaviour<br/>(captured payloads, fake trace-agent, flush receiver)"]
  R --> M["Meaning and semantics<br/>(docs, review, the Datadog API)"]

Which tool sees which layer

agent configcheck — shape, not health

configcheck prints every configuration the Agent resolved, and a Configuration errors block for files it could not parse. It needs a running Agent (it talks to the Agent’s command socket), so docker run --rm datadog/agent:7 agent configcheck fails with unable to read artifact: open auth_token: no such file or directory; start a container, wait about 20–35 s on this Mac, then docker exec:

docker run -d --rm --name dd-reading-a \
  -e DD_API_KEY=<your-api-key> -e DD_HOSTNAME=probe -e DD_DD_URL=http://127.0.0.1:1 \
  -v "$PWD/conf.d:/etc/datadog-agent/conf.d" datadog/agent:7
sleep 30
docker exec dd-reading-a agent configcheck

=== Configuration errors ===

mysql: yaml: unmarshal errors:
  line 4: cannot unmarshal !!map into []integration.RawMap

=== http_check check ===
Configuration provider: file
Configuration source: file:/etc/datadog-agent/conf.d/http_check.d/conf.yaml
Config for instance ID: http_check:checkout:85dc6c39d0e92c1b
name: checkout
timeout: five
url: http://checkout:8080/health
~
===

Read this output carefully. The mysql file (an instances: mapping, catalog entry 1) is reported. The http_check file with timeout: "five" (entry 4) is printed as a normal configuration with no complaint, because wrong-typed values are the check’s business, not the loader’s. So configcheck proves “the loader could parse the shape” and nothing else.

Blind spot, measured: agent configcheck exits with status 0 even with a Configuration errors block (configcheck exit=0 above). A CI step that only checks the exit code will pass a broken file.

🐛 Spot the bug

# CI: validate Agent integration config
docker exec dd-reading-a agent configcheck || exit 1
echo "agent config OK"

Show the answer

The || exit 1 never fires: measured exit status is 0 for a file that configcheck itself labels Configuration errors. It also only shows the shape half of the truth (entries 2, 3, 4 pass configcheck and fail in agent status). Grep the output for the error block and read status:

scripts/reading-datadog/agent-config-gate.sh

out=$(docker exec dd-reading-a agent configcheck 2>&1)
if printf '%s\n' "$out" | grep -q "Configuration errors"; then
  printf '%s\n' "$out"
  exit 1
fi
docker exec dd-reading-a agent status 2>&1 | grep -A20 -E "Config Errors|Loading Errors" && exit 1
echo "agent config OK"

agent status — does the check load, and does it run?

agent status is where the previous tool’s blind spots become visible. It lists Running Checks with per-instance [ERROR] and the underlying exception, plus Config Errors and Loading Errors sections. Everything below is from the same container, after the checks had run at least once:

  Config Errors
  ==============
    mysql
    -----
      yaml: unmarshal errors:
  line 4: cannot unmarshal !!map into []integration.RawMap
  Loading Errors
  ==============
    postgres
    --------
      Python Check Loader:
        could not configure check instance for python check postgres: yaml: unmarshal errors:
  line 3: cannot unmarshal !!str `env:prod` into []string
    redis
    -----
      Python Check Loader:
        unable to load check redis: ... No module named 'datadog_checks.redis' ...

    http_check (13.1.0)
      Instance ID: http_check:checkout:85dc6c39d0e92c1b [ERROR]
      Error: Detected 1 error while loading configuration model `InstanceConfig`:
timeout
  Input should be a valid number, unable to parse string as a number

Two limits. First, the exit status is 0 here too (status exit=0 measured), so it is a reading tool, not a gate; parse it. Second, a check that connects to something (a database, an nginx status URL) will show a runtime [ERROR] for an unreachable host — as in catalog entry 5 (Failed to resolve 'nginx') — that is a fine result for a config review; the configuration was loaded. agent status also prints the endpoints, which is how catalog entry 6 (DD_SITE as a URL) was proven:

$ docker exec dd-reading-a agent status | grep -n "process\."
    https://process.https://api.datadoghq.com. - API Key ending with:

agent dogstatsd-stats and a local receiver — what custom metrics really become

To see what a custom metric turns into, you need something to receive the Agent’s flush. Point DD_DD_URL at a tiny local HTTP server (nothing reaches Datadog), send DogStatsD over UDP from inside the container, and read what arrives. Two settings make it readable: DD_USE_V2_API_SERIES=false (the JSON /api/v1/series body instead of protobuf) and DD_SERIALIZER_COMPRESSOR_KIND=zlib (the default serializer compression is not zlib, so a plain Python receiver could not parse the body).

datadog.yaml

dogstatsd_metrics_stats_enable: true

$ docker exec dd-reading-a agent dogstatsd-stats
Metric                                   | Tags                 | Count      | Last Seen
-----------------------------------------|----------------------|------------|---------------------
orders.bounded                           | env:prod plan:pro    | 25         | ...
orders.bounded                           | env:prod plan:free   | 25         | ...
checkout.latency_dist                    | env:prod             | 4          | ...
checkout.latency_hist                    | env:prod             | 4          | ...
orders.count                             | env:prod user_id:u10 | 1          | ...
orders.count                             | env:prod user_id:u41 | 1          | ...

The same run captured, on the receiver:

POST /api/v1/series    checkout.latency_hist.{avg,count,median,95percentile,max}   (histogram)
POST /api/beta/sketches (protobuf)                                                (distribution)
orders.count           50 series          orders.bounded   2 series

Without dogstatsd_metrics_stats_enable the command answers Dogstatsd metrics stats not enabled in the Agent configuration. This is how catalog entries 17 and 18 were proven. The limits: it needs a running Agent, and it proves what the Agent saw, not what your application intended to send.

The tracer’s startup config dump and .d.ts — option names, values, and order

dd-trace prints a DATADOG TRACER CONFIGURATION line at startup (on by default) with what it actually resolved. Compare two runs that differ by one key:

tracer.init({ service: 'probe', logInjection: false })    -> "log_injection_enabled":false
tracer.init({ service: 'probe', log_injection: false })   -> "log_injection_enabled":true
DD_ENV=Prod DD_SERVICE=Checkout DD_VERSION=1.0.0          -> "env":"Prod","service":"Checkout","dd_version":"1.0.0"
DD_TRACE_SAMPLE_RATE=0.1                                  -> "sample_rate":0.1

The types answer the names question without running anything: tsc --noEmit against dd-trace/index.d.ts rejects log_injection (TS2561 ... Did you mean to write 'logInjection'?) and agentUrl (TS2353), but reports only the first unknown key per object literal and does nothing for .js files. The course harness --options mode extracts the TracerOptions keys from the same .d.ts and checks every key in every tracer.init({...}) fence.

Neither tells you about order. Catalog entry 10 (init after require('express')) was proven by running the app against a fake trace-agent — a 10-line http.createServer on 127.0.0.1:28126 that records the span names found in each PUT /v0.4/traces:

init first  => express.request, http.request
init second => http.request, web.request

ddtrace introspection — attributes and keyword arguments, for real

ddtrace.config is a dynamic object, so a static key list is never authoritative; the only reliable check is a real getattr:

>>> config.logs_injection
AttributeError: ... object has no attribute logs_injection, logs_injection is not a valid configuration
>>> config._logs_injection            # DD_LOGS_INJECTION, default True
True
>>> sorted(inspect.signature(tracer.configure).parameters)
['apm_tracing_disabled', 'appsec_enabled', 'appsec_enabled_origin', 'compute_stats_enabled',
 'context_provider', 'iast_enabled', 'trace_processors']

Blind spot: a write to an unknown attribute succeeds (config.logs_injection = False creates a plain attribute), so introspection is a read-side check — the harness resolves the name with getattr, which is why it flags writes too. Environment variables come from a dump of SUPPORTED_CONFIGURATIONS (DD_LOGS_INJECTION present; DD_LOG_INJECTION and DD_TRACE_ANALYTICS_ENABLED absent from ddtrace 4.15.2).

terraform validate — names, blocks, and types against the pinned provider

export TF_PLUGIN_CACHE_DIR=~/.terraform.d/plugin-cache
terraform init -backend=false -input=false && terraform validate

The provider block used in every proof here — a fake key, validate = false so the provider never calls the API, and no plan or apply ever:

# provider block used for every proof (the harness generates the same one)
terraform {
  required_providers {
    datadog = {
      source  = "DataDog/datadog"
      version = "4.22.0"
    }
  }
}

provider "datadog" {
  api_key  = "<your-api-key>"
  app_key  = "<your-app-key>"
  validate = false
}

What it says, from the catalog: Unsupported argument, Unsupported block type ... Did you mean "template_variable"?, Incorrect attribute value type ... set of string required, but have string, Missing required argument. What it cannot say: whether a query string is a valid Datadog query, whether a permission.id is an ID, whether an exclusion filter is too broad. Two extra commands make it a better reading tool:

$ terraform providers schema -json | jq '.provider_schemas[].resource_schemas.datadog_monitor.block.attributes.notify_no_data.description'
"A boolean indicating whether this monitor will notify when data stops reporting. Defaults to `false`."

That single line corrected a claim in the planning review that the default was true. Treat the provider schema as documentation you can grep. Be careful with credentials: a developer shell often holds real cloud credentials, so run these proofs with fake Datadog keys, the cloud variables unset, and never a plan or apply.

datadog-api-client models — types and required fields, not enums

from datadog_api_client.v1.model.monitor import Monitor
Monitor(**data, _check_type=True)

Measured results (2.61.0), the ones that raise:

options.thresholds.critical="2"  -> ApiTypeError: Invalid type for variable 'critical'. Required value type is float and passed type was str at ['options']['thresholds']['critical']
tags="env:prod"                  -> ApiTypeError: Invalid type for variable 'tags'. Required value type is list and passed type was str at ['tags']
missing query                    -> TypeError: Monitor.__init__() missing 1 required positional argument: 'query'

and the ones that do not (each OK (constructs)): type="metrics alert", an SLO timeframe="30days", a pipeline processor type attribute_remapper, a dashboard layout_type="grid", a widget with no type, a burn_rate query with no .short_window(), a composite "12345 && not-an-id". The stricter client-side signal is the deserializer:

from datadog_api_client import ApiClient, Configuration
obj = ApiClient(Configuration()).deserialize(json_text, (Monitor,), True)
obj._unparsed   # True when an enum value or a oneOf did not match

_unparsed was True for every enum/oneOf case above. The Service Catalog check is a different mechanism: it validates against the JSON Schema published at github.com/DataDog/schema (apiVersion: v2 → 'v2' is not one of ['v3']), and it accepts unknown spec keys.

The course harness — npm run verify and its modes

tools/verify-snippets.mjs collects only fences whose first line is a path comment (# datadog.yaml, # conf.d/<check>.d/conf.yaml, // monitors/<name>.json, # terraform/<name>.tf, // src/<name>.ts, …); a first line containing @expect-error is a deliberate failure and is skipped. Any other fence is a fragment and never touched by the strict modes.

$ npm run verify                       # = --options + --tf + --json
--options: [email protected] init/configure calls: 10 checked, 0 with unknown key(s)
  [email protected] config.<x> refs: 0 checked, 0 unknown; tracer.configure(kw=) refs: 0 checked, 0 unknown
  DD_* env var refs: 8 checked, 0 unknown, 0 deprecated
--tf: provider [email protected] -- 14 fence(s) validated, 0 failed
--json: [email protected] -- 8 fence(s) classified, 0 failed

What each mode does with a bad fence — a scratch lesson with a log_injection init, a tags = "env:prod" monitor, a string critical threshold, and an instances: mapping in a conf.d file:

--options: [email protected] init/configure calls: 11 checked, 1 with unknown key(s)
  ERROR ...:fence #1: tracer.init({...}) unknown key(s) [log_injection] ([email protected] TracerOptions)
--tf: provider [email protected] -- 15 fence(s) validated, 1 failed
  ERROR ...:fence #2: line 7: Incorrect attribute value type -- Inappropriate value for attribute "tags": set of string required, but have string.
--json: [email protected] -- 9 fence(s) classified, 1 failed
  ERROR ...:fence #3 (monitor): ApiTypeError: Invalid type for variable 'critical'. Required value type is float and passed type was str at ['options']['thresholds']['critical']
--agent: datadog/agent:7 configcheck -- 3 lesson group(s) checked, 0 failed

The last line is the important one. --agent reported 0 failed for the instances: mapping, although agent configcheck itself printed Configuration errors. The harness fails a group only if the output matches \bERROR\b or is not valid, and the real message is unmarshal errors under Configuration errors with exit status 0 — neither pattern matches. This is a harness limitation found while writing this module (reported to the course coordinator, not changed here). Until it is widened, treat a green --agent as “the Agent started and parsed something” and confirm with your own grep "Configuration errors" (see the Spot-the-bug above).

Other modes:

Mode	Proves	Costs
--options	every tracer.init({...}) key exists in the pinned .d.ts; every ddtrace.config.<x> resolves; every DD_* name is in the supported list (the harness does not flag DD_TRACE_SAMPLE_RATE; being in the supported list does not mean ddtrace 4.15.2 acts on it — catalog entry 13 shows it is a silent no-op in Python)	about a second
--tf	terraform init + validate per fence with provider 4.22.0	seconds per fence, needs the provider cache
--json	Monitor / Dashboard / SLO / LogsPipeline constructor + Service Catalog JSON Schema	about a second, misses enums
--agent	a real datadog/agent:7 starts with your YAML mounted	~35 s per lesson, needs Docker, see the blind spot above
--k8s	kubectl --dry-run=server on kind-verify-k8s	optional, skipped when the cluster does not exist
--self-test	the harness’s own good/bad fixtures (jsUnknownFound, tfBadFound, jsonBadFound, agentOk, … all true)	starts a container, about a minute

What to run, in what order

1. Lint the file’s own format (YAML/JSON/HCL parse) — nothing here is Datadog-specific.
2. Names and shape: tsc --noEmit / harness --options, terraform validate, the model constructor, agent configcheck.
3. Values and loading: agent status (read it), the tracer config dump, _unparsed.
4. Behaviour: a fake trace-agent, a local flush receiver.
5. Everything else — meaning, ordering of exclusion filters, burn-rate grammar, Autodiscovery placement — is a docs URL and a reviewer. Say so in the review.

Mental Model

• Each tool answers one question. configcheck: can the loader read the shape. status: does it load and run. Dump: what value won. validate: does the schema know the name. Model: are the types right. Harness: did a human wire those into CI.
• Exit codes are not the signal. Two of the Agent’s commands returned 0 while printing errors. Read output.
• A green tool has a scope. Write the scope next to the result.

Trade-offs

• Speed vs. truth. --options/--json/--tf are seconds; --agent and agent status after a run are minutes but see what the others cannot.
• Fake keys vs. realism. A fake DD_API_KEY and a local receiver make every proof safe and offline; they cannot show a real intake response, a real monitor evaluation, or a real UI.
• Pinned vs. current. The harness pins dd-trace, ddtrace, the client, and the provider; pinned answers are reproducible but go stale on upgrade. Re-run the catalog’s probes when a pin moves.

Common Misconceptions

• “If agent configcheck returns 0, the config is fine.” It returned 0 with a Configuration errors block.
• “npm run verify covers the Agent.” It covers --options, --tf, and --json; the Agent is --agent, run separately, with the limitation above.
• “The model check is a validator.” It is a constructor that checks scalar types and required arguments.
• “terraform validate talked to Datadog.” It read the provider’s schema; the provider block sets validate = false and no request is made.

🐶 The one-line takeaway

Run the cheap tools for names and types, read agent status and the tracer’s config dump for values, capture a payload for behaviour, and for meaning write the docs URL — and never trust an exit code that configcheck and status both return as 0 while printing errors.

What exit status did agent configcheck return for a conf.d file whose instances: was a mapping, on Agent 7.83.3?

A non-zero status, so || exit 1 works 0, even though it printed a 'Configuration errors' block 2, reserved for YAML errors It hangs until timeout

Why did the harness's --agent mode report '0 failed' for a scratch lesson containing an instances: mapping?

Because the mapping is valid It only fails on output matching \bERROR\b or 'is not valid'; the real message is 'unmarshal errors' under 'Configuration errors', which matches neither, and the exit status is 0 Because --agent skips yaml fences Because Docker was not running

Which command shows that timeout: "five" is wrong, given that agent configcheck printed it without complaint?

agent version agent status - the http_check instance shows [ERROR] and 'Input should be a valid number, unable to parse string as a number' agent config docker logs shows nothing

You need to know whether an unknown key passed to tracer.init({...}) in a .js file took effect. What proves it?

tsc --noEmit, which checks every .js file The tracer's startup DATADOG TRACER CONFIGURATION line (or _tracer._config): compare log_injection_enabled with and without the key terraform validate agent status

What is the difference between Monitor(**data, _check_type=True) and ApiClient(Configuration()).deserialize(json, (Monitor,), True) on type: "metrics alert"?

None - both raise The constructor accepts it; the deserialized object has _unparsed == True, the client's signal that an enum value did not match The constructor raises, deserialize accepts Both silently coerce to 'metric alert'

What does terraform validate with validate = false on the provider block prove about permission { id = "logs_modify_indexes" }?

That the permission exists in your Datadog org Only that id is a string argument of the right type - it makes no API request, so it says nothing about whether the value is an ID or a name That the role can read logs That the role is safe

Submit

Mark as complete