Worked review 1: a Node checkout service

The feature

The prompt was: “Add Datadog APM and log correlation to our Express checkout service. It uses pino. Service is checkout-api, environment production, send traces to the Datadog Agent.” What follows is the first draft an assistant returns for that prompt. It starts, it serves requests, it prints JSON logs — and nothing in it throws at startup. Every finding below was produced by running it.

How it was proven, and against what:

• dd-trace 6.17.0 (the harness pin in tools/probe/versions.json), express 5.2.1, pino 10.3.1, typescript 7.0.2, Node 24.13.0 on this Mac.
• A fake trace-agent: a 20-line Node HTTP server that accepts PUT /v0.4/traces, decodes the msgpack body, and prints one line per span. The service under review is pointed at it, so nothing is ever sent to Datadog and no API key exists anywhere.
• The tools/verify-snippets.mjs --options mode, which checks every tracer.init({...}) key and every DD_* name against the pinned .d.ts and the merged env-var reference.

The generated code

// @expect-error as generated -- every finding below comes from running this file
const express = require('express');
const pino = require('pino');
const tracer = require('dd-trace').init({
  service: 'checkout-api',
  env: 'Production',
  version: '1.0',
  log_injection: true,
  analytics: true,
  agentUrl: process.env.TRACE_URL || 'http://127.0.0.1:28126',
  traceSampleRate: 0.5,
  flushInterval: 200,
});

const logger = pino({ base: { service: 'checkout', env: 'prod' } });
const app = express();

app.get('/health', (req, res) => res.send('ok'));

app.post('/charge', async (req, res) => {
  const span = tracer.scope().active();
  try {
    logger.info('charging card');
    throw new Error('card declined');
  } catch (err) {
    span.setTag('error', true);
    logger.error({ msg: 'charge failed' });
    res.status(200).json({ ok: false });
  }
});

app.listen(process.env.PORT || 3100, () => logger.info('listening'));

And the Deployment fragment that came with it, so the container “sends traces to the Agent”:

# @expect-error DD_LOG_INJECTION is not a dd-trace variable; DD_AGENT_HOST=localhost is the pod itself, not the node Agent
env:
  - name: DD_LOG_INJECTION
    value: "true"
  - name: DD_AGENT_HOST
    value: localhost

The review walk

dd-trace initialised after express and pino were already loaded

The first two lines require the framework and the logger; only then does init() run. Run as generated, with the fake agent listening and two requests made (GET /health, POST /charge):

$ curl -s -XPOST localhost:3100/charge
TypeError: Cannot read properties of null (reading 'setTag')
    at .../bad/server.js:26:10
    at Layer.handleRequest (.../node_modules/router/lib/layer.js:152:17)

fake agent: (no spans received at all)

{"level":30,"time":1790736292381,"service":"checkout","env":"prod","msg":"charging card"}

Three failures from one cause. express was loaded before the tracer hooked require, so it was never instrumented: there is no express.request span, tracer.scope().active() is null (hence the TypeError on the first real request), the fake agent received nothing, and the pino line has no dd object — no trace id, so no log-to-trace link. The same code with only the two require lines moved below init() printed, for the same two requests:

DATADOG TRACER INTEGRATIONS LOADED - ["fs","net","dns","http","router","express","pino"]
{"level":30,...,"msg":"charging card","dd":{"trace_id":"6abc77b500000000453cf38eaa9ba509","span_id":"2662234485403838737",...}}

The docs say the same thing for the logger half: in docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/nodejs/ the snippet is commented “This line must come before importing the logger.”

🐛 Spot the bug

const express = require('express');
const pino = require('pino');
const tracer = require('dd-trace').init({ ... });

Show the answer

dd-trace patches modules as they are required, so it has to be initialised before anything it should instrument — the web framework, the HTTP client, the database driver and the logger. AI assistants put init() “after the imports” because that is where configuration usually goes. Two later findings below were invisible in this run until this one was fixed: with no active span, none of the tag or log problems could even execute. Fix: init() on the first line, or preload with node --require dd-trace/init app.js.

log_injection — a snake_case option that does nothing

The next block asks for log_injection: true. tsc --strict against the pinned typings, one key at a time:

$ tsc --noEmit --strict opts.ts
opts.ts(6,3): error TS2561: Object literal may only specify known properties, but 'log_injection' does not exist in type 'TracerOptions'. Did you mean to write 'logInjection'?

Plain JavaScript accepts the unknown key silently. This one is dangerous because it appears to work: the tracer default is on, so logs are injected anyway. Prove it with the opposite value — log_injection: false still leaves "log_injection_enabled":true in the startup config dump (the same result the bug catalog entry 9 recorded). The moment someone tries to turn it off, or the default changes, the line is a lie.

🐛 Spot the bug

log_injection: true,

Show the answer

The real option is logInjection (TracerOptions.logInjection, index.d.ts line 551 of the installed package). In a JavaScript file nothing rejects the wrong spelling; TypeScript and the harness --options mode both do. Fix: logInjection: true, or omit it entirely because it already defaults to true, or set DD_LOGS_INJECTION.

agentUrl — traces go to 127.0.0.1:8126, not where the code says

The service was told to send to http://127.0.0.1:28126. Two fake agents ran side by side, one on 28126 and one on the default 8126:

agent A (28126, the URL in code): 0 spans
agent B (default 8126):           6 spans

The tracer’s own startup line agrees: "agent_url":"http://127.0.0.1:8126/". agentUrl is not a dd-trace option (tsc: TS2353 ... 'agentUrl' does not exist in type 'TracerOptions'); the real key is url, or the environment variable DD_TRACE_AGENT_URL. Locally this works by coincidence, because 8126 is where a developer’s Agent listens. In a cluster, spans would flow to the pod’s own localhost:8126 and vanish.

🐛 Spot the bug

agentUrl: process.env.TRACE_URL || 'http://127.0.0.1:28126',

Show the answer

An invented option is a bug in the config, but the outage happens in the network: the spans are sent, somewhere else, with no error. Fix: delete the line and set DD_TRACE_AGENT_URL (or DD_AGENT_HOST) in the environment — see bug 11 for what value that should be in Kubernetes.

analytics and traceSampleRate — options that never existed

Same family, different consequence. tsc says TS2353 for analytics and TS2561 ... Did you mean to write 'sampleRate'? for traceSampleRate. The harness --options reports all of them at once, which tsc does not (it stops at the first excess property):

ERROR src/content/docs/en/x/bad.mdx:fence #2: tracer.init({...}) unknown key(s) [log_injection, agentUrl, traceSampleRate, analytics] ([email protected] TracerOptions)

The behavioural proof is a count. 40 requests through a tracer initialised with the invented key vs the real sampleRate: 0.5, reading _sampling_priority_v1 from the captured payloads:

{traceSampleRate: 0.5, analytics: true}  -> {priority 1: 40}
{sampleRate: 0.5}                        -> {priority 2: 15, priority -1: 25}

The invented key kept 100 percent of the traces; the real one produced a mix of “keep” (2) and “reject” (-1). Note that the rejected traces were still sent in this run to the fake agent with priority -1 — the tracer marks them, and the Agent and the ingestion controls decide what is dropped (docs.datadoghq.com/tracing/trace_pipeline/ingestion_mechanisms/). Whether a 50 percent head sample is a good idea is a separate question the trace sampling pipeline covers; the point here is that the line the author believed was cutting volume was doing nothing.

🐛 Spot the bug

analytics: true,
traceSampleRate: 0.5,

Show the answer

Neither key exists in TracerOptions. analytics is not in the pinned typings at all, and traceSampleRate is a plausible neighbour of sampleRate (the compiler suggests exactly that). Fix: delete analytics; use sampleRate or, better, DD_TRACE_SAMPLE_RATE / sampling rules, and only if the requirement really is head sampling.

env and version hard-coded — the Kubernetes env vars lose

The service is deployed with DD_ENV=prod and DD_VERSION=2.4.1 (from the pod labels, as unified service tagging prescribes). The code says env: 'Production', version: '1.0'. With those environment variables set, the tracer’s config dump and the spans both took the code values:

"env":"Production", "tags":{"service":"checkout-api","env":"Production","version":"1.0", ...}
express.request  POST /charge  env=Production version=1.0

Options passed to init() take precedence over DD_* variables — proven here, not assumed. So the deploy pipeline’s DD_ENV=prod and DD_VERSION=2.4.1 are silently discarded, every deploy reports version 1.0, and Production is a different env value from the prod used by every other service (the tracer forwards it verbatim; how the platform then normalises tags is a Datadog-side rule — see docs.datadoghq.com/getting_started/tagging/unified_service_tagging/).

🐛 Spot the bug

service: 'checkout-api',
env: 'Production',
version: '1.0',

Show the answer

Unified service tagging wants env, service and version set once, outside the code, so the same image runs in staging and prod and the deploy tooling owns the version. Putting them in init() freezes them into the image and overrides the environment. Fix: require('dd-trace').init() with no identity options, and DD_ENV, DD_SERVICE, DD_VERSION from the pod (bug 10).

The logger’s own service and env contradict the injected ones

pino({ base: { service: 'checkout', env: 'prod' } }) is the assistant’s attempt at “unified tagging in the logs”. Once injection works, one log line carries two answers:

{"level":30,...,"service":"checkout","env":"prod","msg":"charging card",
 "dd":{"trace_id":"6abc77b5...","service":"checkout-api","version":"1.0","env":"Production"}}

checkout vs checkout-api, prod vs Production. Datadog treats a JSON log’s top-level service as the log’s service and also recognises dd.service (docs.datadoghq.com/logs/log_configuration/pipelines/, “Service attribute” lists service, syslog.appname, dd.service). The docs I read do not say which wins when both are present in one line, so the honest statement is: the outcome depends on the org’s pipeline, and no offline tool can tell you. The review can still see that two different values were emitted by one process.

🐛 Spot the bug

const logger = pino({ base: { service: 'checkout', env: 'prod' } });

Show the answer

The injected dd block already carries service, env and version from the tracer, and DD_SERVICE/DD_ENV drive both. Hard-coding a second copy in the logger guarantees they drift. Fix: pino() with no base identity fields — one source of truth (the environment), read by the tracer, which stamps the logs.

setTag('error', true) — on the wrong span, with no message

With init fixed, the handler ran. What the fake agent received for POST /charge:

express.request    POST /charge   error=0 http.status_code=200
router.middleware  handle         error=0
router.middleware  <anonymous>    error=1

tracer.scope().active() inside a route handler is not the request’s root span; it is the innermost span — here the Express router.middleware layer. The error:1 landed there. The root express.request span, the one APM’s service error rate and the service page are computed from, has error=0. And because the value was the boolean true, there is no error.message, error.type or error.stack on any span. In the fixed run, the same failure produces:

express.request  POST /charge  error=1 http.status_code=502
payment.charge   payment.charge error=1 error.type=Error error.message=card declined

🐛 Spot the bug

const span = tracer.scope().active();
...
span.setTag('error', true);

Show the answer

Two mistakes in three lines. active() returns whichever span is current, not the root. And setTag('error', true) flags a span but records no cause; passing the Error object (setTag('error', err)) records error.message, error.type and error.stack. The clean idiom is to wrap the work in your own span with tracer.trace('payment.charge', fn) — an exception thrown (or a promise rejected) inside fn marks that span as an error, with the message, automatically.

A failed charge answered with 200 OK

res.status(200).json({ ok: false }). The express.request span above shows http.status_code=200, error=0. dd-trace’s HTTP server plugin decides “error” by status: validateStatus defaults to code => code < 500 (index.d.ts, HttpServer.validateStatus). A declined card returned as 200 is a success as far as every APM monitor, the service error rate and any availability SLO are concerned. With 502, the same run marks the root span error=1.

🐛 Spot the bug

res.status(200).json({ ok: false });

Show the answer

That is an application design choice made invisible to observability. Return a status that matches the outcome (a genuine server-side or upstream failure is 5xx; a client-caused decline may legitimately be 4xx, which the default does not count as an error — set validateStatus if your definition of an error differs). Do not choose the status code to satisfy a dashboard, but do not hide a failure from one either.

/health is traced on every probe

The first fake-agent listing above contains express.request GET /health — a full trace, plus two router.middleware spans, for each liveness and readiness probe. One probe every 10 seconds is 8,640 probe traces per pod per day of pure noise, and it dilutes latency and error statistics for the service. With tracer.use('http', { server: { blocklist: ['/health'] } }) (Http.blocklist in the typings; type-checks with tsc), the same two requests produced only the /charge trace: the /health request did not appear in the fake agent at all.

🐛 Spot the bug

app.get('/health', (req, res) => res.send('ok'));

Show the answer

Nothing is wrong with the route; the bug is what is missing next to it. Probe traffic should be excluded from tracing with the http plugin’s blocklist (or filter), placed after init() and before the server starts. The ingestion-side alternative — a sampling rule that drops the resource — works too but still costs the tracer work; the docs for both are under docs.datadoghq.com/tracing/.

DD_LOG_INJECTION — a variable that does not exist

In the Deployment, injection was “enabled” with DD_LOG_INJECTION=true. The harness, on a copy of that fence:

ERROR src/content/docs/en/x/bad.mdx:fence #1: unknown env var DD_LOG_INJECTION

and the runtime proof, same code, only the variable differs, reading the config dump and one log line:

DD_LOG_INJECTION=false  -> "log_injection_enabled":true   {"msg":"hi","dd":{"service":"x","version":"1.0.0"}}
DD_LOGS_INJECTION=false -> "log_injection_enabled":false  {"msg":"hi"}

The real name is DD_LOGS_INJECTION (plural). The unknown one is ignored, which is the same “silently accepted” failure as the snake_case option, one layer further out.

🐛 Spot the bug

env:
- name: DD_LOG_INJECTION
  value: "true"

Show the answer

Environment variable names are matched literally and unknown ones are not reported by the tracer. The pinned harness checks every DD_* name in a collected fence against dd-trace’s own supported-configuration list. Fix: DD_LOGS_INJECTION — or delete it, since the Node tracer’s default is already on.

DD_AGENT_HOST=localhost — the pod, not the node Agent

localhost inside a pod is that pod. The tracer honours it exactly as written:

DD_AGENT_HOST=localhost    -> "agent_url":"http://localhost:8126/"
DD_AGENT_HOST=10.20.30.40  -> "agent_url":"http://10.20.30.40:8126/"

In Kubernetes the Datadog Agent normally runs as a DaemonSet, so the address is the node’s IP. From the docs (docs.datadoghq.com/containers/kubernetes/apm/): “The application container needs the DD_AGENT_HOST environment variable that points to status.hostIP”, with a fieldRef on status.hostIP, or use the Admission Controller. Whether the node Agent accepts the traffic is a separate Agent-side setting (hostPort or the host-network / Unix socket option) — that part is the subject of the next worked review.

🐛 Spot the bug

- name: DD_AGENT_HOST
value: localhost

Show the answer

localhost works on a laptop with a local Agent, which is exactly why an assistant writes it. In a pod it points at a port nothing listens on, and the traces are dropped without any error the application can see. Fix: valueFrom.fieldRef.fieldPath: status.hostIP.

The corrected code

src/reading-datadog/worked-review-1/server.js

const tracer = require('dd-trace').init({ logInjection: true });
tracer.use('http', { server: { blocklist: ['/health'] } });

const express = require('express');
const pino = require('pino');

const logger = pino();
const app = express();

app.get('/health', (req, res) => res.send('ok'));

app.post('/charge', async (req, res) => {
  try {
    await tracer.trace('payment.charge', async () => {
      logger.info('charging card');
      throw new Error('card declined');
    });
  } catch (err) {
    logger.error({ err }, 'charge failed');
    res.status(502).json({ ok: false, error: 'payment_failed' }); // choose the status that matches what happened
  }
});

app.listen(process.env.PORT || 3100, () => logger.info('listening'));

k8s/checkout-api.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  labels:
    tags.datadoghq.com/env: prod
    tags.datadoghq.com/service: checkout-api
    tags.datadoghq.com/version: 2.4.1
spec:
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
        tags.datadoghq.com/env: prod
        tags.datadoghq.com/service: checkout-api
        tags.datadoghq.com/version: 2.4.1
    spec:
      containers:
        - name: checkout-api
          image: registry.example.com/checkout-api:2.4.1
          env:
            - name: DD_ENV
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/env']
            - name: DD_SERVICE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/service']
            - name: DD_VERSION
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/version']
            - name: DD_AGENT_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: DD_LOGS_INJECTION
              value: "true"

Run for real (fake agent on the default port, DD_ENV=prod DD_SERVICE=checkout-api DD_VERSION=2.4.1, one GET /health and one POST /charge). The fake agent printed exactly these four spans and no /health trace:

express.request    POST /charge     error=1 env=prod version=2.4.1 http.status_code=502
router.middleware  handle           error=0 env=prod version=2.4.1
router.middleware  <anonymous>      error=0 env=prod version=2.4.1
payment.charge     payment.charge   error=1 env=prod version=2.4.1 error.type=Error error.message=card declined

and the log lines now carry a single, consistent identity:

{"level":30,...,"msg":"charging card","dd":{"trace_id":"6abc77ea0000000033867be986a3a994","span_id":"598578184621079437","service":"checkout-api","version":"2.4.1","env":"prod"}}
{"level":50,...,"err":{"type":"Error","message":"card declined",...},"msg":"charge failed","dd":{...same trace_id...}}

The YAML was not applied to a cluster (no kind cluster exists for the course harness; --k8s mode reports and exits clean). What was checked is every DD_* name in it, by the harness --options mode.

What changed and why

#	Bug	Fix	Proven by
1	init() after express/pino are loaded	init() on line 1	no spans, null span, no dd in logs vs all three present
2	log_injection (snake_case)	logInjection or omit	tsc TS2561; config dump
3	agentUrl invented	DD_TRACE_AGENT_URL / DD_AGENT_HOST	agent on 28126 got 0 spans, 8126 got 6
4	analytics, traceSampleRate invented	delete; real sampleRate only if intended	40 of 40 priority 1 vs 15/25
5	env/version hard-coded in init()	identity from DD_ENV/DD_SERVICE/DD_VERSION	spans show Production/1.0 despite DD_ENV=prod
6	service/env also hard-coded in the pino base	one source; injected dd block	two different values in one line
7	setTag('error', true) on scope().active()	tracer.trace(...) wrapping the work	error=1 on router.middleware, 0 on root
8	Failure answered with 200	status that matches the outcome	root span error=0, status 200 vs error=1, 502
9	/health traced	tracer.use('http', { server: { blocklist } })	probe trace present vs absent
10	DD_LOG_INJECTION	DD_LOGS_INJECTION (or omit)	harness “unknown env var”; config dump
11	DD_AGENT_HOST=localhost in a pod	fieldRef: status.hostIP	config dump; docs URL

What the reviewer writes in the PR

Not approvable as-is — 11 things, in the order they bite:

1. init() must be the first line. Everything required before it is un-instrumented: today there are no Express spans, scope().active() is null (the first /charge throws) and the logs have no trace id.
2. Four invented options (log_injection, agentUrl, analytics, traceSampleRate) — none exist in TracerOptions; run tsc or npm run verify. agentUrl in particular means traces go to :8126, not where the code says.
3. Do not put env, version or service in code, or in the pino base. They override the pod’s DD_* values (proven) and disagree with the injected dd block. One source: the environment.
4. Error handling: setTag('error', true) marks a router layer, not the root, with no message. Wrap in tracer.trace(...). And a declined charge must not be 200.
5. Block /health in the http plugin.
6. Deployment: DD_LOG_INJECTION is not a variable (DD_LOGS_INJECTION is, and it is the default anyway), and DD_AGENT_HOST must come from status.hostIP, not localhost.

Mental Model

flowchart TD
  A["require order: express + pino first"] -->|"1: nothing instrumented"| B["tracer.init options"]
  B -->|"2, 3, 4: invented keys ignored"| C["tracer config"]
  B -->|"5: code beats DD_ENV / DD_VERSION"| C
  C -->|"3, 11: wrong agent address"| D["Agent"]
  E["pino base service + env"] -->|"6: contradicts dd block"| F["log line"]
  G["route handler"] -->|"7: error on inner span"| H["spans"]
  G -->|"8: 200 on failure"| H
  I["/health probe"] -->|"9: traced"| H
  J["pod env"] -->|"10: unknown variable"| C

Where each bug sits in the Node service

Four of the eleven bugs (2, 3, 4, 10) are names nobody checks: JavaScript and the tracer accept any key, and the environment accepts any variable. tsc and the harness --options mode close that gap for free. Bug 1 is an ordering error no option checker sees. The other six (5, 6, 7, 8, 9, 11) type-check and start; only a captured payload, a config dump or a log line shows them. That is the reason this review used a fake agent and not just a linter — and why bug 1 goes first: it hid the rest.

Trade-offs

• Config in code vs environment. Code-level init({ service, env, version }) is easy to read and wrong for a multi-environment image; the environment is invisible in the source but is the only thing the deploy tooling controls. Choose the environment for identity; keep code options for behaviour (logInjection, plugin settings).
• blocklist vs an ingestion rule. Blocking /health in the tracer removes the work and the data at the source; a Datadog-side sampling rule can be changed without a deploy but the spans are still created and sent.
• Fake agent vs real Agent. A local receiver shows exactly what the tracer sent and nothing about what Datadog does with it. Normalisation, ingestion sampling and pipeline behaviour stay docs-only claims in this review, labelled as such.

Common Misconceptions

• “If init() accepted my options object, the options are valid.” Proven above: all four invented keys changed nothing, with no warning at any log level. Only tsc, the harness or the config dump tell you.
• “tracer.scope().active() is the request span.” Proven above: inside an Express route it is the innermost router.middleware span; tagging it left the root at error=0.
• “Setting env in code and also DD_ENV on the pod is belt and braces.” Proven above: the code value wins, silently. Two sources mean the wrong one wins.
• “An HTTP 200 with ok: false is fine as long as the log says error.” Every APM error-rate monitor is computed from spans, not from log text.

🐶 The one-line takeaway

In an AI-written tracer setup, the code that starts without an error is the code to distrust: check every option name against the typings, put init() first, keep identity in DD_* variables, and look at one captured span before believing any of it works.

The service under review requires express and pino on lines 1-2 and calls dd-trace's init() on line 3. What does the run show for POST /charge?

Everything works; init() patches modules already in memory No spans reach the agent, tracer.scope().active() is null so the handler throws TypeError, and the log line has no dd object Spans arrive but without the env tag Only the log correlation is affected

tracer.init({ log_injection: true, agentUrl: 'http://127.0.0.1:28126' }) starts without error. What is true?

Both keys work; the tracer converts snake_case and camelCase Neither key exists in TracerOptions: tsc reports TS2561 / TS2353, injection is on only because it is the default, and traces went to the default :8126 agentUrl works, log_injection does not The tracer refuses to start when a key is unknown

The pod sets DD_ENV=prod and DD_VERSION=2.4.1; the code calls init({ env: 'Production', version: '1.0' }). Which values appear on the spans?

prod and 2.4.1, because environment variables always win Production and 1.0 - options passed to init() take precedence over DD_* variables Both values, as a list The tracer refuses to start

A handler catches a payment error, runs tracer.scope().active().setTag('error', true) and returns HTTP 200. What did the captured spans show?

error=1 on the root express.request span with the message error=1 on an inner router.middleware span, error=0 on the root, http.status_code=200, and no error.message anywhere error=0 on every span Two root spans

A manifest sets DD_LOG_INJECTION=false to disable log injection. What happens?

Injection is disabled Nothing: the variable does not exist, the config dump still says log_injection_enabled:true and the log line still has the dd block; the real name is DD_LOGS_INJECTION The tracer fails to start Only the trace id is removed

Which claim in this review is a documentation quote rather than something proven by a local run?

Spans go to port 8126 when agentUrl is set In Kubernetes the app container's DD_AGENT_HOST should come from status.hostIP log_injection is not a TracerOptions key The root span has error=0 after setTag on the active span

Submit

Mark as complete