The AI bug catalog

Every snippet below is written the way an AI assistant actually writes it, then run against the real tool: datadog/agent:7 (Agent 7.83.3, image digest sha256:c5bf5ec9…, container dd-reading-a, removed afterwards) with a fake API key and DD_DD_URL pointed at 127.0.0.1 or a local listener, dd-trace 6.17.0 and @datadog/browser-rum 7.15.0 with the TypeScript compiler, ddtrace 4.15.2, terraform validate with the datadog provider 4.22.0 (Terraform v1.16.4, no plan, no apply, cloud credentials unset), and the datadog-api-client 2.61.0 Python models. Nothing was ever sent to Datadog — there is no account and no real key anywhere in this module. Timings and ports are from one Mac.

Each entry ends with a Caught by line, and you will see a pattern: the tool that catches a bug is very different from entry to entry. Some are hard errors (agent configcheck, terraform validate, tsc); some are silently accepted by every offline tool and only visible in a second place (agent status, a captured payload); and a few are only catchable by the Datadog API or by reading the docs — those say so plainly and quote the doc URL rather than pretending a tool proved them.

Agent configuration & deployment

1. `instances:` written as a mapping, not a list

AI often flattens the check config, because in many READMEs the instances example is written as a single block. The Agent needs a list of instances.

# @expect-error mysql.d instances is a map, not a list -> "cannot unmarshal !!map into []integration.RawMap"
init_config:

instances:
  host: mysql
  username: datadog
  port: 3306

Symptom — real output, agent configcheck (Agent 7.83.3):

=== Configuration errors ===

mysql: yaml: unmarshal errors:
  line 4: cannot unmarshal !!map into []integration.RawMap

agent status repeats it under Config Errors, and the check simply never runs — no metrics, no [ERROR] instance line.

Fix:

init_config:

instances:
  - host: mysql
    username: datadog
    port: 3306

Caught by: agent configcheck (hard error). See The Datadog Agent and integrations.

2. `tags: env:prod` — a scalar where the check wants a list

The YAML is valid, configcheck prints the config as if it were fine — and the check still fails to load.

# @expect-error tags must be a list: "cannot unmarshal !!str `env:prod` into []string" (only visible in agent status, not configcheck)
init_config:

instances:
  - host: postgres
    port: 5432
    username: datadog
    tags: env:prod

Symptom — real output. agent configcheck lists the postgres instance with tags: env:prod and no error at all. agent status shows the truth:

  Loading Errors
  ==============
    postgres
    --------
      Core Check Loader:
        Check postgres not found in Catalog

      Python Check Loader:
        could not configure check instance for python check postgres: yaml: unmarshal errors:
  line 3: cannot unmarshal !!str `env:prod` into []string

Fix:

init_config:

instances:
  - host: postgres
    port: 5432
    username: datadog
    tags:
      - env:prod

Caught by: agent status (Loading Errors) — not agent configcheck. A clean configcheck is necessary, never sufficient.

3. `conf.d/redis.d/` — the integration is called `redisdb`

AI names the directory after the product. The Agent names it after the check.

# @expect-error the check is named redisdb, there is no redis check: "Check redis not found in Catalog"
init_config:

instances:
  - host: redis
    port: 6379

This file lives at conf.d/redis.d/conf.yaml. Symptom — real output: agent configcheck prints a normal-looking === redis check === block with the config (no error), and agent status says:

    redis
    -----
      Core Check Loader:
        Check redis not found in Catalog

      Python Check Loader:
        unable to load check redis: unable to load python module datadog_checks.redis: unable to import module 'datadog_checks.redis': No module named 'datadog_checks.redis', unable to load python module redis: unable to find a subclass of the base check in module 'redis': cannot find a subclass

Fix: move it to the real check name.

init_config:

instances:
  - host: redis
    port: 6379

Caught by: agent status (Loading Errors). Same lesson as entry 2: the directory name is data, and configcheck does not know whether a check of that name exists.

4. A right-shaped instance with a wrong-typed value

# @expect-error timeout must be a number: "Input should be a valid number, unable to parse string as a number" (agent status only)
init_config:

instances:
  - name: checkout
    url: http://checkout:8080/health
    timeout: "five"

Symptom — real output. agent configcheck prints timeout: five with no error. agent status:

    http_check (13.1.0)
    -------------------
      Instance ID: http_check:checkout:85dc6c39d0e92c1b [ERROR]
      ...
      Error: Detected 1 error while loading configuration model `InstanceConfig`:
timeout
  Input should be a valid number, unable to parse string as a number

Fix:

init_config:

instances:
  - name: checkout
    url: http://checkout:8080/health
    timeout: 5

Caught by: agent status — the check’s own pydantic model, not the Agent’s config loader.

5. The phantom bug: “missing `init_config:` will break the check”

A reviewer (human or AI) who has memorised “every conf.yaml needs init_config:” will flag this file.

# @expect-error demonstrates a NON-bug: this loads fine without init_config; the only error is that host "nginx" does not resolve
instances:
  - nginx_status_url: http://nginx:81/nginx_status/

Real output. agent configcheck prints the === nginx check === block with no configuration error; agent status shows the instance running and failing only on DNS: Failed to resolve 'nginx' ([Errno -2] Name or service not known). So the rule “missing init_config is an error” is folklore for this check on Agent 7.83.3 — the file is loaded.

The lesson is about the review, not the file. Adding init_config: is harmless and matches every reference example, so keep it, but do not file “missing init_config” as a bug without running the tool.

init_config:

instances:
  - nginx_status_url: http://nginx:81/nginx_status/

Caught by: nothing — it was never wrong. This is the entry that justifies the whole module: prove it, do not pattern-match.

6. `DD_SITE` set to a URL

AI tends to give DD_SITE the API base URL because that is what other tools ask for. DD_SITE is a site domain.

# @expect-error DD_SITE takes a site domain (datadoghq.com), not a URL
docker run -d -e DD_API_KEY=<your-api-key> -e DD_SITE=https://api.datadoghq.com datadog/agent:7

Symptom — real output (agent status, endpoints, with the local DD_DD_URL override so nothing leaves the machine):

# DD_SITE=https://api.datadoghq.com
    https://process.https://api.datadoghq.com. - API Key ending with:

# DD_SITE=datadoghq.eu
    https://process.datadoghq.eu. - API Key ending with:

The Agent does not reject the value; it builds every intake hostname by prefixing it, producing an address that cannot resolve.

Fix:

site: datadoghq.com

Caught by: agent status endpoint list (and a forwarder DNS error in the logs). No offline linter knows the site list. Site names: docs.datadoghq.com/getting_started/site/.

7. `datadog/agent:latest` in a manifest

# @expect-error not a syntax error: an unpinned, moving tag (see the digest output below)
image: datadog/agent:latest

Real output — the three tags resolved to the same image today:

$ for t in 7 7.83.3 latest; do docker image inspect datadog/agent:$t --format '{{.Id}}'; done
sha256:da43d3ecfad79906df4d32c21c3dcee8c681ba8f0afa9a59f6663d8f5489d70c
sha256:da43d3ecfad79906df4d32c21c3dcee8c681ba8f0afa9a59f6663d8f5489d70c
sha256:da43d3ecfad79906df4d32c21c3dcee8c681ba8f0afa9a59f6663d8f5489d70c

That is the honest result: nothing is broken today. The bug is reproducibility — latest and 7 will point at a different Agent next week (and 7.83.3 was the release on the day this was captured), so an incident postmortem cannot say which Agent ran. Pin an exact tag, or a digest, and bump it deliberately.

apiVersion: v1
kind: Pod
metadata:
  name: agent-image-example
spec:
  containers:
    - name: agent
      image: datadog/agent:7.83.3

Caught by: review only; docker image inspect shows the tags coincide right now, which is exactly why it goes unnoticed.

8. Autodiscovery annotation on the Deployment (and in the v1 format)

Two separate mistakes AI makes in one breath: annotating the workload object, and using the pre-7.36 three-key format.

# @expect-error Autodiscovery reads POD annotations (spec.template.metadata), not Deployment metadata; v1 keys check_names/init_configs/instances are the pre-7.36 format
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache
  annotations:
    ad.datadoghq.com/cache.check_names: '["redisdb"]'
    ad.datadoghq.com/cache.init_configs: '[{}]'
    ad.datadoghq.com/cache.instances: '[{"host":"%%host%%","port":"6379"}]'
spec:
  selector:
    matchLabels: { app: cache }
  template:
    metadata:
      labels: { app: cache }
    spec:
      containers:
        - name: cache
          image: redis:7

Not proven by a run here — docs only. This needs a cluster, and the course harness’s --k8s mode needs a kind cluster that does not exist. What the docs say (docs.datadoghq.com/containers/kubernetes/integrations/): for pods created by a Deployment “Add annotations under spec.template.metadata”; the container identifier in ad.datadoghq.com/<CONTAINER_NAME>.checks matches spec.containers[i].name; the consolidated .checks (v2) format needs Agent 7.36+, the three-key format is the earlier one. kubectl accepts both placements without complaint, so nothing offline catches it.

Fix:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cache
spec:
  selector:
    matchLabels: { app: cache }
  template:
    metadata:
      labels: { app: cache }
      annotations:
        ad.datadoghq.com/cache.checks: |
          {
            "redisdb": {
              "init_config": {},
              "instances": [{ "host": "%%host%%", "port": "6379" }]
            }
          }
    spec:
      containers:
        - name: cache
          image: redis:7

Caught by: the node Agent’s agent configcheck on a real cluster (an absent AD config for the pod) — never by kubectl apply.

dd-trace (Node.js)

9. `log_injection` — the snake_case option that does nothing

This is the real bug the course’s own Phase 3 review found. JavaScript does not reject an unknown key.

// @expect-error 'log_injection' does not exist in TracerOptions: TS2561, and at runtime it is silently ignored
import tracer from 'dd-trace';
tracer.init({ service: 'checkout', log_injection: false });

Symptom — real output. tsc (dd-trace 6.17.0 typings): error TS2561: Object literal may only specify known properties, but 'log_injection' does not exist in type 'TracerOptions'. Did you mean to write 'logInjection'? — but in plain JS it runs, and the tracer’s own startup dump says:

logInjection:false  -> "log_injection_enabled":false
log_injection:false -> "log_injection_enabled":true

The person who tries to turn injection off gets injection on. (Default is true, which is why the course’s original log_injection: true “worked” by accident.)

Fix:

import tracer from 'dd-trace';
tracer.init({ service: 'checkout', logInjection: false });

Caught by: tsc --noEmit on TypeScript; the course harness --options on any fence (it checks every tracer.init({...}) key against the .d.ts); the startup config dump on JavaScript. See Connecting logs and traces.

10. `dd-trace` initialised after the libraries it should instrument

// @expect-error init after require('express'): express is not instrumented (no express.request span)
const express = require('express');
require('dd-trace').init({ flushInterval: 200 });
const app = express();
app.get('/', (req, res) => res.send('ok'));
app.listen(3000);

Symptom — real output. A fake trace-agent on 127.0.0.1:28126 received one PUT /v0.4/traces from each variant; the span names found in the payload:

init first  (good.js) => express.request, http.request
init second (bad.js)  => http.request, web.request

The express.request span — the one that carries the route — is gone. Only the generic http layer (loaded before user code) still reports.

Fix: initialise first, or preload it.

require('dd-trace').init({ flushInterval: 200 });
const express = require('express');
const app = express();
app.get('/', (req, res) => res.send('ok'));
app.listen(3000);

For ESM, import statements are hoisted above any init() call, so preload with node --import dd-trace/initialize.mjs app.mjs (the package ships initialize.mjs and init.js; only the CommonJS ordering above was run here). Docs: “The Node.js SDK needs to be imported and initialized before any other module” — docs.datadoghq.com/tracing/trace_collection/automatic_instrumentation/dd_libraries/nodejs/.

Caught by: a captured payload only; no option checker sees import order.

11. Invented tracer options

AI invents plausible neighbours of real options. tsc, checked one key at a time against the 6.17.0 typings:

analytics       -> TS2353: ... 'analytics' does not exist in type 'TracerOptions'
traceSampleRate -> TS2561: ... Did you mean to write 'sampleRate'?
runtimeMetric   -> TS2561: ... Did you mean to write 'runtimeMetrics'?
enabled         -> TS2353
debug           -> TS2353
agentUrl        -> TS2353            (the real key is 'url')
logInjection / sampleRate / runtimeMetrics / logLevel / env / url -> OK

// @expect-error analytics, traceSampleRate, enabled, agentUrl are not TracerOptions keys
import tracer from 'dd-trace';
tracer.init({ service: 'checkout', analytics: true, traceSampleRate: 0.5, agentUrl: 'http://dd-agent:8126' });

Note the compiler reports only the first excess property per literal, so a fix-one-rerun loop is normal.

import tracer from 'dd-trace';
tracer.init({ service: 'checkout', sampleRate: 0.5, url: 'http://dd-agent:8126' });

Caught by: tsc (first key only) and the harness --options (all keys). Plain .js gets neither.

12. `DD_ENV=Prod` in one service, `prod` in another

Unified service tagging only joins if the values match. Set once with a capital by one team:

DD_ENV=Prod DD_SERVICE=Checkout DD_VERSION=1.0.0
  -> "env":"Prod","service":"Checkout"   (tracer tags, verbatim)
DD_ENV=prod DD_SERVICE=checkout
  -> "env":"prod","service":"checkout"

The tracer does not normalise — it sends what you gave it. What happens next is the platform’s rule: “Span tags and metric tags are normalized to lowercase, so avoid using camel case in tag keys” (docs.datadoghq.com/getting_started/tagging/), while log attributes are case-sensitive — so a log attribute env:Prod and a metric tag env:prod can stop joining. The fix is a convention, enforced in the deploy template:

apiVersion: v1
kind: Pod
metadata:
  name: checkout
  labels:
    tags.datadoghq.com/env: prod
    tags.datadoghq.com/service: checkout
    tags.datadoghq.com/version: 1.0.0
spec:
  containers:
    - name: checkout
      image: checkout:1.0.0
      env:
        - name: DD_ENV
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['tags.datadoghq.com/env']
        - name: DD_SERVICE
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['tags.datadoghq.com/service']

Caught by: the tracer config dump shows the raw value; the normalisation claim is docs-only (nothing was sent to Datadog). See Unified service tagging in practice.

13. “Fixing” `DD_TRACE_SAMPLE_RATE` because it ‘is deprecated’ (it is not — but it is a no-op in Python)

A reviewer who remembers an old deprecation notice rewrites DD_TRACE_SAMPLE_RATE=0.1 to something else. On Node the run says otherwise:

$ DD_TRACE_SAMPLE_RATE=0.1 node cfg.js          # dd-trace 6.17.0
{"sampleRate":0.1}   (and sampling_rules: [{ _sampler: { _rate: 0.1 } }])
$ node cfg.js '{"sampleRate":0.25}'
{"sampleRate":0.25}

DD_TRACE_SAMPLE_RATE is not deprecated: the Node and Python config pages list it without any deprecation text (“Controls the ingestion sample rate (between 0.0 and 1.0) between the Agent and the backend”, docs.datadoghq.com/tracing/trace_collection/library_config/nodejs/, .../python/), and dd-trace (Node) honours it, as above. ddtrace (Python) 4.15.2 does not — the variable appears in the package only in its supported-configurations (telemetry) list, and it has no effect on sampling:

$ DD_TRACE_SAMPLE_RATE=0.1 python  # ddtrace 4.15.2
>>> ddtrace.tracer._sampler.rules
[]
>>> # 400 root spans created in a loop: sampling_priority > 0 on 400 / 400

So the accurate review note is neither “deprecated, rewrite it” nor “supported everywhere”: it works in Node, is a silent no-op in Python 4.15.2, and DD_TRACE_SAMPLING_RULES is the setting that behaves the same in both. (The pinned harness --options mode accepts the name because it is in the supported list; it does not flag it. Being in that list is not proof that a setting does anything.) See Ingestion sampling vs. retention filters.

import tracer from 'dd-trace';
tracer.init({ service: 'checkout', sampleRate: 0.1 });

Caught by: the Node config dump and, for Python, a loop of real spans checking sampling_priority. Half of this entry is a bug in the reviewer; see entry 5.

ddtrace (Python)

14. `ddtrace.config.logs_injection = …` — writes succeed, reads raise

# @expect-error config.logs_injection is not an attribute (AttributeError on read); the write is silently accepted
from ddtrace import config
config.logs_injection = False

Symptom — real output (ddtrace 4.15.2):

>>> config.logs_injection
AttributeError: ... object has no attribute logs_injection, logs_injection is not a valid configuration
>>> config.logs_injection = False; config.logs_injection
False          # the setattr just created a new plain attribute
>>> config._logs_injection        # the real, env-driven setting
True
$ DD_LOGS_INJECTION=false python -c "from ddtrace import config; print(config._logs_injection)"
False

A write to a made-up attribute succeeds, so nothing fails at the line that is wrong — and the real setting was never touched. The supported switch is the environment variable.

Fix:

import os
# set before ddtrace is imported / ddtrace-run starts
os.environ["DD_LOGS_INJECTION"] = "false"

Caught by: harness --options (it resolves every ddtrace.config.<x> reference with a real getattr) — a read check, so it flags the write too because it looks the name up.

15. `tracer.configure(hostname=…, port=…)` — the 0.x/1.x API

# @expect-error Tracer.configure() got an unexpected keyword argument 'hostname'
from ddtrace import tracer
tracer.configure(hostname="dd-agent", port=8126)

Symptom — real output:

TypeError: Tracer.configure() got an unexpected keyword argument 'hostname'
>>> sorted(inspect.signature(tracer.configure).parameters)
['apm_tracing_disabled', 'appsec_enabled', 'appsec_enabled_origin', 'compute_stats_enabled',
 'context_provider', 'iast_enabled', 'trace_processors']

Fix: the agent address is environment configuration.

import os
os.environ["DD_TRACE_AGENT_URL"] = "http://dd-agent:8126"

Caught by: running it (a TypeError on import) and the harness --options (tracer.configure(<kw>=) check).

RUM in the browser

16. `allowedTracingOrigins`, and RUM init with no `allowedTracingUrls`

// @expect-error allowedTracingOrigins does not exist in RumInitConfiguration (TS2561: did you mean allowedTrackingOrigins?)
import { datadogRum } from '@datadog/browser-rum';
datadogRum.init({
  applicationId: '<your-application-id>',
  clientToken: '<your-client-token>',
  allowedTracingOrigins: ['https://api.example.com'],
});

Symptom — real output (@datadog/browser-rum 7.15.0, tsc):

allowedTracingUrls    -> OK (exists)
allowedTracingOrigins -> TS2561: ... does not exist in type 'RumInitConfiguration'.
trackInteractions     -> TS2561: ... Did you mean to write 'trackUserInteractions'?
sampleRate            -> TS2353
sessionSampleRate / traceSampleRate / defaultPrivacyLevel -> OK

And the silent variant: omitting allowedTracingUrls compiles (it is optional), initialises, and RUM works — but browser requests are not linked to backend traces. The docs: “Configure the allowedTracingUrls initialization parameter with the list of internal, first-party origins called by your browser application” (docs.datadoghq.com/real_user_monitoring/correlate_with_other_telemetry/apm/); the docs do not spell out the failure mode, so the consequence (no link) is inferred, not observed here.

Fix:

import { datadogRum } from '@datadog/browser-rum';
datadogRum.init({
  applicationId: '<your-application-id>',
  clientToken: '<your-client-token>',
  allowedTracingUrls: ['https://api.example.com'],
  traceSampleRate: 100,
});

Caught by: tsc for the wrong name; only a review (or a trace in the UI) for the missing one. The course harness does not check RUM options.

Custom metrics

17. `histogram` where the requirement says “global percentile”

# @expect-error semantic: HISTOGRAM aggregates per Agent, so its percentile is per host, not global
statsd.histogram("checkout.latency", latency_ms, tags=["env:prod"])

Symptom — real output. A real Agent 7.83.3 received the same four values (100, 120, 300, 900) as |h and as |d and flushed to a local receiver:

histogram  -> POST /api/v1/series : checkout.latency_hist.avg (gauge, 355) .median (120)
                                     .95percentile (900) .max (900) .count (rate 0.4)   [tags: env:prod]
distribution -> POST /api/beta/sketches   (protobuf, raw values, no per-host percentile series)

Docs: a HISTOGRAM “aggregates on the Agent during a given time interval”; a DISTRIBUTION “sends all the raw data during a time interval to Datadog. Aggregations occur on the server-side” (docs.datadoghq.com/metrics/types/). Note the .95percentile above is one Agent’s number — averaging it across hosts is not the fleet p95.

Fix:

from datadog import statsd

statsd.distribution("checkout.latency", 120.0, tags=["env:prod"])

Caught by: the captured flush (different intake path); the semantic mismatch is a review item. See Custom metrics with DogStatsD.

18. `user_id` as a metric tag

# @expect-error semantic: every distinct user_id value is a separate metric context (custom metric)
statsd.increment("orders.count", tags=["env:prod", f"user_id:{user_id}"])

Symptom — real output. Fifty sends with fifty different user_id values, next to fifty sends over two plan values, flushed by a real Agent:

orders.count   50 series  (env:prod,user_id:u0 ... u49)   each rate 0.1
orders.bounded  2 series  (env:prod,plan:pro / plan:free)  each rate 2.5
agent dogstatsd-stats: 50 rows for orders.count, one per user_id

Docs: “A custom metric is identified by a unique combination of a metric’s name and tag values (including the host tag)” (docs.datadoghq.com/metrics/custom_metrics/) — the billing consequence follows from that sentence; no price is quoted here.

Fix: bounded tags only; put the id in a log or span attribute.

from datadog import statsd

statsd.increment("orders.count", tags=["env:prod", "plan:pro"])

Caught by: agent dogstatsd-stats (needs dogstatsd_metrics_stats_enable), or the captured series count. See Metrics without limits and tag cardinality.

Terraform & monitors as code

19. `datadog_dashboard` request query written flat

The second real bug from the Phase 3 review.

# @expect-error Unsupported argument: name / data_source / query are not expected directly inside request.query
resource "datadog_dashboard" "checkout" {
  title       = "Checkout overview"
  layout_type = "ordered"

  widget {
    timeseries_definition {
      title = "CPU by host"
      request {
        query {
          name        = "query1"
          data_source = "metrics"
          query       = "avg:system.cpu.user{service:checkout} by {host}"
        }
        display_type = "line"
      }
    }
  }
}

Symptom — real terraform validate (provider 4.22.0):

Error: Unsupported argument
  on main.tf line 10, in resource "datadog_dashboard" "checkout":
  10:           name        = "query1"
An argument named "name" is not expected here.

(repeated for data_source and query.)

Fix: query { metric_query { … } } plus a formula block.

resource "datadog_dashboard" "checkout" {
  title       = "Checkout overview"
  layout_type = "ordered"

  widget {
    timeseries_definition {
      title = "CPU by host"
      request {
        query {
          metric_query {
            name        = "query1"
            data_source = "metrics"
            query       = "avg:system.cpu.user{service:checkout} by {host}"
          }
        }
        formula {
          formula_expression = "query1"
        }
        display_type = "line"
      }
    }
  }
}

Caught by: terraform validate (hard error). See Datadog as code with Terraform.

20. Hallucinated arguments and blocks on `datadog_monitor` / `datadog_dashboard`

Four different real terraform validate failures, same family:

critical_recovery = 1.5 at monitor top level
  -> Error: Unsupported argument ... An argument named "critical_recovery" is not expected here.
tags = "env:prod"
  -> Error: Incorrect attribute value type ... set of string required, but have string.
thresholds { critical = 2 }
  -> Error: Unsupported block type ... Blocks of type "thresholds" are not expected here.
template_variables { name = "env" }   (dashboard)
  -> Error: Unsupported block type ... Did you mean "template_variable"?
(monitor without query)
  -> Error: Missing required argument ... The argument "query" is required

# @expect-error critical_recovery is only valid inside monitor_thresholds; tags is a set of string
resource "datadog_monitor" "latency" {
  name              = "Checkout p95 latency"
  type              = "metric alert"
  message           = "p95 too high @slack-oncall"
  query             = "avg(last_5m):p95:trace.http.request{service:checkout} > 2"
  tags              = "env:prod"
  critical_recovery = 1.5
}

The recovery threshold is not cosmetic. Docs: “If no recovery threshold is set, the monitor recovers whenever the alert conditions are no longer met” and “Recovery thresholds reduce noise of a flapping monitor” (docs.datadoghq.com/monitors/guide/recovery-thresholds/).

resource "datadog_monitor" "latency" {
  name    = "Checkout p95 latency"
  type    = "metric alert"
  message = "p95 too high @slack-oncall"
  query   = "avg(last_5m):p95:trace.http.request{service:checkout} > 2"
  tags    = ["env:prod", "service:checkout"]

  monitor_thresholds {
    critical          = 2
    critical_recovery = 1.5
  }
}

Caught by: terraform validate. The datadog-api-client model catches the string tags too (ApiTypeError: Invalid type for variable 'tags'. Required value type is list and passed type was str).

21. `notify_no_data` on a sparse metric

A frequent AI “hardening” is notify_no_data = true on a low-traffic service, or leaving require_full_window at its default on a sparse metric. terraform validate passes. The provider’s own schema text (terraform providers schema -json, 4.22.0) says what the defaults are:

notify_no_data      "... Defaults to `false`."
no_data_timeframe   "... We recommend at least 2x the monitor timeframe for metric alerts ... Defaults to `10`."
require_full_window "Datadog strongly recommends you set this to `false` for sparse metrics,
                     otherwise some evaluations may be skipped ... Defaults to `true`."

A correction to a claim in the planning review: notify_no_data defaults to false in the provider, not true. The bug is enabling it (or ignoring require_full_window) without thinking about traffic, and only the API and time show it.

resource "datadog_monitor" "batch_errors" {
  name    = "Nightly batch errors"
  type    = "metric alert"
  message = "batch errors @slack-oncall"
  query   = "sum(last_1h):sum:batch.errors{env:prod}.as_count() > 0"

  require_full_window = false
  notify_no_data      = false
}

Caught by: nothing offline — terraform validate is green. Provider schema text is the evidence, the behaviour is the API’s.

22. Role permission by name, not by ID

# @expect-error validates cleanly but is semantic: permission.id is an ID, not the permission name; and this grants index modification
resource "datadog_role" "viewer" {
  name = "log-viewer"

  permission {
    id = "logs_modify_indexes"
  }
}

terraform validate prints Success! The configuration is valid. The schema description of permission.id is just “ID of the permission to assign”, with name as a separate attribute — so a name in the id slot is a string of the right type and only the API can reject it. Worse, if it is accepted, the permission is the wrong one for a viewer: logs_modify_indexes is “Read and modify all indexes in your account” (docs.datadoghq.com/account_management/rbac/permissions/).

data "datadog_permissions" "all" {}

resource "datadog_role" "viewer" {
  name = "log-viewer"

  permission {
    id = data.datadog_permissions.all.permissions["logs_read_index_data"]
  }
}

Caught by: review (and the API). terraform validate cannot resolve a data source value.

23. Exclusion filter written broader than the intent

# @expect-error validates; semantic: status:info at sample_rate 1.0 drops ALL info logs, including ones you meant to keep
resource "datadog_logs_index" "main" {
  name           = "main"
  retention_days = 15

  filter {
    query = "*"
  }

  exclusion_filter {
    name       = "drop-noise"
    is_enabled = true
    filter {
      query       = "status:info"
      sample_rate = 1.0
    }
  }
}

terraform validate accepts it. Docs: excluded logs “are discarded from indexes, but still flow through the Live Tail and can be used to generate metrics and archived”, and “Index filters for logs are only processed with the first active exclusion filter matched” (docs.datadoghq.com/logs/log_configuration/indexes/) — so order and breadth matter and nothing here checks them.

resource "datadog_logs_index" "main" {
  name           = "main"
  retention_days = 15

  filter {
    query = "*"
  }

  exclusion_filter {
    name       = "drop-health-checks"
    is_enabled = true
    filter {
      query       = "service:checkout @http.url_details.path:/healthz"
      sample_rate = 1.0
    }
  }
}

Caught by: review. See Indexes, exclusion filters, and retention.

Monitor, SLO, and catalog JSON

24. Burn-rate alert with the wrong `.over()` and no `.short_window()`

The third real bug from the review.

// @expect-error query string is accepted by the model; the API grammar needs .over(<SLO window>) and .short_window()
{
  "name": "Checkout SLO burn rate",
  "type": "slo alert",
  "query": "burn_rate(\"<slo_id>\").over(\"1h\").long_window(\"6h\") > 10",
  "message": "@slack-oncall budget is burning"
}

Real output: Monitor(**data, _check_type=True) on this JSON and on the correct one both print OK (constructs) — the client only checks that query is a str. The grammar comes from the docs (docs.datadoghq.com/service_management/service_level_objectives/burn_rate/): burn_rate("slo_id").over("time_window").long_window("1h").short_window("5m") > 14.4, where .over is the SLO target duration (7d/30d/90d) and .short_window “ensures quick alert recovery”; Google’s recommendation quoted there is a short window of 1/12 of the long one.

{
  "name": "Checkout SLO burn rate",
  "type": "slo alert",
  "query": "burn_rate(\"<slo_id>\").over(\"30d\").long_window(\"6h\").short_window(\"30m\") > 10",
  "message": "@slack-oncall budget is burning"
}

Caught by: only the Datadog API (or a human reading the grammar). See SLOs, error budgets, and burn rate alerts.

25. Composite monitor referencing IDs that are not IDs

// @expect-error model accepts any string; composite queries are monitor IDs joined by &&, ||, !
{
  "name": "Checkout degraded",
  "type": "composite",
  "query": "12345 && not-an-id",
  "message": "@slack-oncall both signals are red"
}

Real output: composite bad ids: OK (constructs). Docs: a composite query looks like "1234 && 5678", "!1234 || 5678"; and “Creating a downtime on an individual monitor A, which is used within a composite monitor, does not mute the composite monitor” (docs.datadoghq.com/monitors/types/composite/). The docs do not say what happens for a non-existent ID, so that outcome is untested here.

{
  "name": "Checkout degraded",
  "type": "composite",
  "query": "12345 && 67890",
  "message": "@slack-oncall both signals are red"
}

Caught by: the API at creation time (and review of the IDs against the real monitors).

26. Enum typos the constructor accepts and `_unparsed` catches

// @expect-error type "metrics alert" and timeframe "30days" are not valid enum values, yet the model constructor accepts them
{
  "name": "Checkout p95",
  "type": "metrics alert",
  "query": "avg(last_5m):p95:trace.http.request{service:checkout} > 2",
  "message": "@slack-oncall"
}

Real output (datadog-api-client 2.61.0):

Monitor(**data, _check_type=True) with type="metrics alert"        -> OK (constructs)
ServiceLevelObjective(...) with timeframe="30days"                  -> OK (constructs)
ServiceLevelObjective(...) with type="metrics"                      -> OK (constructs)
LogsPipeline(...) with processor type "attribute_remapper"          -> OK (constructs)
Dashboard(...) with layout_type="grid", or a widget with no "type"  -> OK (constructs)

ApiClient(Configuration()).deserialize(json, (Monitor,), True)
  monitor typo type / slo timeframe / pipeline underscore / layout grid / widget without type
                                                                    -> _unparsed = True

The constructor path (which is what node tools/verify-snippets.mjs --json uses) does not enforce allowed_values; the deserializer sets _unparsed = True when an enum value or a oneOf does not match, which is a much stronger — and unused by the harness — signal. The type errors it does raise:

options.thresholds.critical="2"   -> ApiTypeError: Invalid type for variable 'critical'. Required value type is float and passed type was str
options.notify_no_data="true"     -> ApiTypeError: Invalid type for variable 'notify_no_data'. Required value type is bool and passed type was str
tags="env:prod"                   -> ApiTypeError: Invalid type for variable 'tags'. Required value type is list and passed type was str
missing query / missing thresholds -> TypeError: __init__() missing 1 required positional argument

{
  "name": "Checkout p95",
  "type": "metric alert",
  "query": "avg(last_5m):p95:trace.http.request{service:checkout} > 2",
  "message": "@slack-oncall"
}

Caught by: _unparsed on a deserialized object (write that check yourself); the harness --json misses enums and says “fine”. The API rejects a bad type.

27. Service Catalog `apiVersion: v2` with v3 fields

// @expect-error apiVersion must be "v3" for the entity schema: 'v2' is not one of ['v3']
{
  "apiVersion": "v2",
  "kind": "service",
  "metadata": { "name": "checkout", "owner": "payments-team" },
  "spec": { "lifecycle": "production", "tier": "1", "dependsOn": ["service:payments-api"] }
}

Real output (the JSON Schema fetched from github.com/DataDog/schema, validated with jsonschema, the same call the harness makes):

apiVersion "v2"                    -> ['apiVersion']: 'v2' is not one of ['v3']
apiVersion "v3", dependsOn         -> ok
apiVersion "v3", depends_on (typo) -> ok      # the schema does not reject unknown spec keys
kind "microservice"                -> ok      # ...nor an unknown kind

So the schema catches the version, and not the two other mistakes AI makes with it — the schema is a floor, not a guarantee.

{
  "apiVersion": "v3",
  "kind": "service",
  "metadata": { "name": "checkout", "owner": "payments-team" },
  "spec": { "lifecycle": "production", "tier": "1", "dependsOn": ["service:payments-api"] }
}

Caught by: the harness --json (service-catalog/ fences), for the version only. See The Service Catalog and ownership.

Spot the bug

🐛 Spot the bug

resource "datadog_monitor" "checkout_latency" {
name              = "Checkout p95 latency"
type              = "metric alert"
message           = "p95 above 2s @pagerduty-checkout"
query             = "avg(last_5m):p95:trace.http.request{service:checkout} > 2"
tags              = "team:payments"
critical_recovery = 1.5
}

Show the answer

Two hard errors, both from terraform validate with provider 4.22.0: tags is a set of strings (set of string required, but have string) and critical_recovery is not a top-level argument (An argument named "critical_recovery" is not expected here) — it belongs inside monitor_thresholds. Fixed version, which validates:

resource "datadog_monitor" "checkout_latency" {
  name    = "Checkout p95 latency"
  type    = "metric alert"
  message = "p95 above 2s @pagerduty-checkout"
  query   = "avg(last_5m):p95:trace.http.request{service:checkout} > 2"
  tags    = ["team:payments"]

  monitor_thresholds {
    critical          = 2
    critical_recovery = 1.5
  }
}

🐛 Spot the bug

const express = require('express');
const tracer = require('dd-trace').init({
service: 'checkout',
env: process.env.DD_ENV,
log_injection: true,
agentUrl: 'http://dd-agent:8126',
});
const app = express();

Show the answer

Three problems, none of which throws at runtime in plain JavaScript. (1) dd-trace is initialised after require('express'), so no express.request span is produced (captured payload: http.request, web.request only). (2) log_injection is not an option (the real key is logInjection; it is a no-op here, and the default is already true). (3) agentUrl does not exist; the key is url. The fix, which type-checks and passes the harness:

import tracer from 'dd-trace';

tracer.init({
  service: 'checkout',
  env: process.env.DD_ENV,
  logInjection: true,
  url: 'http://dd-agent:8126',
});

For ESM, preload with node --import dd-trace/initialize.mjs instead of relying on statement order.

Mental Model

Accepted is not correct. Every tool here accepts something it should not: configcheck accepts a scalar tags, the model accepts "metrics alert", terraform validate accepts a role permission by name, a JS object accepts log_injection. The bug is always one layer later than the tool that stayed quiet.
Ask which layer validates what. Agent config loader (shape) → the check’s own model (agent status) → the tracer/.d.ts (option names) → the provider schema (block/attribute names) → the API (semantics). Most catalog entries live in the gap between two of those.
Silence has two flavours. A no-op (log_injection) and a wrong-but-running (DD_SITE as URL, histogram instead of distribution). The second is more expensive: it produces data that looks fine.
Do not over-correct. Entry 5 and half of entry 13 are false alarms. A review that flags folklore (“missing init_config”, “DD_TRACE_SAMPLE_RATE is deprecated”) costs as much as a missed bug.

Trade-offs

agent configcheck is fast; agent status is where the truth is. configcheck needs a running Agent (about 20–35 s on this Mac to become ready); status is only meaningful once the check has run at least once.
tsc is free for TypeScript, absent for JavaScript. The harness --options closes that gap for fenced code, but not for your repo unless you run a similar check in CI.
_unparsed is stricter than the constructor, but a moving target. It exists so a client can survive new server enum values; using it as a linter means an old client flags a valid new value. Pin the client version you check against.
Docs-only entries (7, 8, 12, 24, 25) cost nothing to write and can rot. Quote the URL and re-check on upgrade.

Common Misconceptions

“A clean agent configcheck means the check will run.” Entries 2, 3, and 4 print a normal-looking config and fail in agent status.
“The model validation in the harness proves my monitor JSON is valid.” It proves types and required fields. Not enum values (26), not query grammar (24, 25).
“Unknown option keys throw.” In JavaScript, and on a Python Config object write, they do not (9, 14).
“latest is broken.” It is not broken — it is unpinned (7). The difference decides whether you write a bug report or a policy.
“If the AI’s comment explains it, it is right.” The comment in entry 12 and the original course text for entry 9 both explained a behaviour the code did not have.

configcheck printed the config with no error. What does agent status show for this instance?

$ agent configcheck
=== postgres check ===
...
tags: env:prod

Nothing - the check runs and tags every metric env:prod A Loading Error: 'cannot unmarshal !!str env:prod into []string' - the check never loads A warning that tags should be a list, but the check still runs The check runs, but with the tag dropped

Entry 5 shows a conf.d file with no init_config: key. What actually happened on Agent 7.83.3? configcheck reported a Configuration error and the check never ran The file loaded and the instance ran; the only error was that the host 'nginx' did not resolve The Agent crashed on startup The Agent silently ignored the file

In plain JavaScript, tracer.init({ log_injection: false }) on dd-trace 6.17.0 results in which startup-dump value? "log_injection_enabled": false An exception: unknown option "log_injection_enabled": true - the key is ignored and the default applies The tracer does not start

You capture the payloads of require('express'); require('dd-trace').init() and of the reverse order. What differs? Nothing - order is irrelevant With init after require('express') there is no express.request span; only http.request and web.request are reported The spans are sent to a different endpoint Both orders produce no spans

Why can Monitor(**data, _check_type=True) say 'OK' for type: "metrics alert" and for a burn_rate query missing .short_window()? Because both are valid The constructor does not enforce enum allowed_values and treats query as an opaque string; only the API (or a check on _unparsed for the enum) catches them Because the model is out of date Because _check_type=True is ignored

Fifty sends of orders.count with user_id:u0..u49 versus fifty sends over two plan values reached a real Agent. What did the flush contain? One series each 50 series for the user_id metric and 2 for the plan metric - each unique tag combination is its own series 2 series for the user_id metric Nothing, DogStatsD drops high-cardinality tags

DD_SITE=https://api.datadoghq.com is given to the Agent. What does agent status show for the Process endpoint? https://process.datadoghq.com. https://process.https://api.datadoghq.com. - the value is used verbatim as a domain suffix The Agent refuses to start It falls back to the default site

Which entry is a false alarm that a careful reviewer should NOT report as a bug? log_injection: true DD_TRACE_SAMPLE_RATE=0.1 in dd-trace 6.17.0 - it is not deprecated and Node honours it (but ddtrace 4.15.2 silently ignores it) instances: as a mapping datadog_dashboard flat query blocks

The AI bug catalog

Agent configuration & deployment

1. instances: written as a mapping, not a list

2. tags: env:prod — a scalar where the check wants a list

3. conf.d/redis.d/ — the integration is called redisdb

4. A right-shaped instance with a wrong-typed value

5. The phantom bug: “missing init_config: will break the check”

6. DD_SITE set to a URL

7. datadog/agent:latest in a manifest

8. Autodiscovery annotation on the Deployment (and in the v1 format)

dd-trace (Node.js)

9. log_injection — the snake_case option that does nothing

10. dd-trace initialised after the libraries it should instrument

11. Invented tracer options

12. DD_ENV=Prod in one service, prod in another

13. “Fixing” DD_TRACE_SAMPLE_RATE because it ‘is deprecated’ (it is not — but it is a no-op in Python)

ddtrace (Python)

14. ddtrace.config.logs_injection = … — writes succeed, reads raise

15. tracer.configure(hostname=…, port=…) — the 0.x/1.x API

RUM in the browser

16. allowedTracingOrigins, and RUM init with no allowedTracingUrls

Custom metrics

17. histogram where the requirement says “global percentile”

18. user_id as a metric tag

Terraform & monitors as code

19. datadog_dashboard request query written flat

20. Hallucinated arguments and blocks on datadog_monitor / datadog_dashboard

21. notify_no_data on a sparse metric

22. Role permission by name, not by ID

23. Exclusion filter written broader than the intent

Monitor, SLO, and catalog JSON

24. Burn-rate alert with the wrong .over() and no .short_window()

25. Composite monitor referencing IDs that are not IDs

26. Enum typos the constructor accepts and _unparsed catches

27. Service Catalog apiVersion: v2 with v3 fields

Spot the bug

Mental Model

Trade-offs

Common Misconceptions

1. `instances:` written as a mapping, not a list

2. `tags: env:prod` — a scalar where the check wants a list

3. `conf.d/redis.d/` — the integration is called `redisdb`

5. The phantom bug: “missing `init_config:` will break the check”

6. `DD_SITE` set to a URL

7. `datadog/agent:latest` in a manifest

9. `log_injection` — the snake_case option that does nothing

10. `dd-trace` initialised after the libraries it should instrument

12. `DD_ENV=Prod` in one service, `prod` in another

13. “Fixing” `DD_TRACE_SAMPLE_RATE` because it ‘is deprecated’ (it is not — but it is a no-op in Python)

14. `ddtrace.config.logs_injection = …` — writes succeed, reads raise

15. `tracer.configure(hostname=…, port=…)` — the 0.x/1.x API

16. `allowedTracingOrigins`, and RUM init with no `allowedTracingUrls`

17. `histogram` where the requirement says “global percentile”

18. `user_id` as a metric tag

19. `datadog_dashboard` request query written flat

20. Hallucinated arguments and blocks on `datadog_monitor` / `datadog_dashboard`

21. `notify_no_data` on a sparse metric

24. Burn-rate alert with the wrong `.over()` and no `.short_window()`

26. Enum typos the constructor accepts and `_unparsed` catches

27. Service Catalog `apiVersion: v2` with v3 fields