The Trace Sampling Pipeline

The idea in one sentence

A trace passes five gates in a fixed order — the tracer’s head sampler, the tracer’s rate limiter, the Agent’s samplers (which mostly advise rather than enforce), the ingestion controls on the backend, and finally retention filters plus intelligent retention — and each gate has its own knob, its own evidence tag on the span, and its own way of being silently misconfigured.

🐶 How every proof in this lesson was made

Two counterparties, both local. (1) A fake trace-agent: a 40-line Python HTTP server that answers GET /info, accepts PUT /v0.4/traces, decodes the msgpack body and prints each root span’s _sampling_priority_v1, _dd.p.dm, _dd.rule_psr, _dd.limit_psr and _dd.agent_psr, and can reply with a chosen rate_by_service. dd-trace 6.17.0 (the harness pin in tools/probe/node) emits traces to it via DD_TRACE_AGENT_URL=http://127.0.0.1:18227. (2) A real datadog/agent:7 (Agent 7.83.3, container dd-internals) with a fake key and its APM intake pointed at a local fake intake, fed by a script posting msgpack traces to its real :8126 receiver, so the Agent’s real responses and real forwarded payloads can be counted. ddtrace 4.15.2 was introspected from the harness venv. Nothing goes to Datadog. Timings and counts are from this Mac, one run each; rates over 5-second windows are approximate. The Phase 3 lesson Ingestion Sampling vs. Retention Filters introduces the two-layer idea and _dd.rule_psr; this lesson walks the whole pipeline and adds the parts it did not run.

The pipeline, once

flowchart TB
  S["Root span starts"] --> HS{"1. Head sampler in the tracer: DD_TRACE_SAMPLING_RULES, else DD_TRACE_SAMPLE_RATE, else Agent rate_by_service"}
  HS -->|"priority 2 or 1: keep"| RL{"2. Tracer rate limiter: DD_TRACE_RATE_LIMIT"}
  HS -->|"priority -1 or 0: reject, still sent unless discard"| AG
  RL -->|"over the limit: priority -1"| AG
  RL -->|"within the limit"| AG["3. Agent: samplers and feedback: DD_APM_TARGET_TPS, DD_APM_ERROR_TPS, DD_APM_PROBABILISTIC_SAMPLER_*"]
  AG -->|"kept traces"| IC["4. Ingestion controls: Ingestion Control page, remote configuration"]
  AG -.->|"rate_by_service back to tracers"| HS
  IC --> INGESTED["Ingested spans: APM metrics come from all traffic"]
  INGESTED --> RF["5a. Retention filters: UI or API, first match wins"]
  INGESTED --> IR["5b. Intelligent retention: diversity plus 1 percent flat"]
  RF --> IDX["Indexed spans searchable for the retention period"]
  IR --> IDX

From span to searchable trace

#	Gate	Where it runs	Control	Evidence left on the data
1	Head sampler	tracer, at the root span	`DD_TRACE_SAMPLING_RULES`, `DD_TRACE_SAMPLE_RATE`, Agent rates	`_sampling_priority_v1`, `_dd.p.dm`, `_dd.rule_psr`, `_dd.agent_psr`
2	Rate limiter	tracer	`DD_TRACE_RATE_LIMIT`	`_dd.limit_psr`
3	Agent samplers	Agent (`trace-agent`)	`DD_APM_TARGET_TPS`, `DD_APM_ERROR_TPS`, `DD_APM_PROBABILISTIC_SAMPLER_ENABLED`	`rate_by_service` response, `_dd.p.dm` `-9`
4	Ingestion controls	Datadog backend + remote config to SDK	Ingestion Control page	`_dd.p.dm` `-11` / `-12` in the tracer constants
5	Retention	Datadog backend	retention filters, intelligent retention	none on the wire (docs only)

Gate 1: the head sampler, decided once at the root

The decision is made at the first span and travels with the trace. The docs: “The decision to keep or drop a trace is made at the start of the root span and then propagated to other services as part of their request context” (ingestion mechanisms). The values come from the pinned tracer (node_modules/dd-trace/ext/priority.js): USER_REJECT: -1, AUTO_REJECT: 0, AUTO_KEEP: 1, USER_KEEP: 2, and the mechanism codes from packages/dd-trace/src/constants.js: DEFAULT: 0, AGENT: 1, RULE: 3, MANUAL: 4, APPSEC: 5, SPAN: 8, REMOTE_USER: 11, REMOTE_DYNAMIC: 12, AI_GUARD: 13. The _dd.p.dm tag is - plus that code.

What the fake trace-agent printed for 40 to 200 root spans per configuration (DD_SERVICE=sampling-lab):

== default, no rules (40 traces)
  x40 priority=1 dm=-0 rule_psr=None limit_psr=None agent_psr=1
== rule sampleRate 1.0        DD_TRACE_SAMPLING_RULES='[{"service":"sampling-lab","sample_rate":1.0}]'
  x40 priority=2 dm=-3 rule_psr=1 limit_psr=1 agent_psr=None
== rule sampleRate 0
  x40 priority=-1 dm=None rule_psr=0 limit_psr=None agent_psr=None
== rule 0.5 (200 traces)
  x107 priority=-1 dm=None rule_psr=0.5
  x93  priority=2  dm=-3   rule_psr=0.5 limit_psr=1
== DD_TRACE_SAMPLE_RATE=0.25 (200 traces)
  x154 priority=-1 dm=None rule_psr=0.25
  x46  priority=2  dm=-3   rule_psr=0.25 limit_psr=1
== rule with "discard": true
  received root spans: 0

Five facts are visible. (1) With nothing configured, the tracer uses the default mechanism (dm=-0) and priority 1, and _dd.agent_psr=1 because no Agent has told it otherwise. (2) A matching rule switches the mechanism to -3 and the priority to 2 (user keep) for kept traces; the rule’s own rate is stamped as _dd.rule_psr. (3) The global DD_TRACE_SAMPLE_RATE=0.25 is implemented as a rule — it leaves _dd.rule_psr=0.25 and dm=-3, and 46 of 200 (23%) survived. (4) A rejected trace (priority=-1) is still sent to the Agent: 40 rejected traces arrived at the fake agent. That is deliberate; the Agent uses them for the trace metrics. (5) Only "discard": true in the rule (in the pinned .d.ts: “a trace chunk rejected by this rule is fully dropped: it is excluded from client-side stats and never sent to the Agent”) stops the send: 0 root spans arrived.

The third source of a rate is the Agent. I reran with the fake agent replying {"rate_by_service":{"service:sampling-lab,env:lab":0.25}} to every flush, emitting 400 traces in 20-trace steps every 500 ms:

  x20  priority=1 dm=-0 agent_psr=1       # before the first response arrives
  x99  priority=1 dm=-1 agent_psr=0.25    # kept under the Agent's rate
  x281 priority=0 dm=None agent_psr=0.25  # rejected under the Agent's rate

The first 20 traces went out at the default; after the first response the tracer adopted the Agent’s rate (dm=-1, AGENT), and 99 of 380 (26%) were kept. The precedence is therefore rule, then Agent rate, then default.

Gate 2: the tracer’s rate limiter

DD_TRACE_RATE_LIMIT caps how many traces per second the head sampler may keep, applied after the rate. The docs: “If no value is set, a limit of 100 traces per second is applied.” Rule sample_rate: 1.0 with DD_TRACE_RATE_LIMIT=5, 60 traces emitted as fast as a loop can go:

  x5  priority=2  dm=-3  rule_psr=1 limit_psr=1
  x1  priority=-1 dm=None rule_psr=1 limit_psr=0.8333333333333334
  x1  priority=-1 dm=None rule_psr=1 limit_psr=0.7142857142857143
  ...
  x1  priority=-1 dm=None rule_psr=1 limit_psr=0.08333333333333333

The first 5 pass with _dd.limit_psr=1. Every later trace in that second is rejected (priority=-1) and stamped with the limiter’s effective rate, which decays as 5 / n (0.833 = 5/6, 0.714 = 5/7, … 0.0833 = 5/60). _dd.limit_psr is the evidence that gate 2, not the rule, dropped a trace — _dd.rule_psr=1 next to priority=-1 is the signature of a rate limiter at work. The Node tracer also supports a per-rule cap (maxPerSecond in the .d.ts, max_per_second in the env JSON); a rule {"sample_rate":1.0,"max_per_second":2} kept exactly 2 of 60. Python’s ddtrace 4.15.2 does not: the same env value makes the process fail at import.

$ DD_TRACE_SAMPLING_RULES='[{"service":"sampling-lab","sample_rate":0.5,"max_per_second":3}]' python -c "import ddtrace; ddtrace.tracer"
  File ".../ddtrace/_trace/sampler.py", line 156, in set_sampling_rules
    sampling_rules.append(SamplingRule(**rule))
TypeError: SamplingRule.__init__() got an unexpected keyword argument 'max_per_second'

The introspected Python signature is SamplingRule(sample_rate, service, name, resource, tags, provenance, discard) — no max_per_second. The accepted form printed DatadogSampler(agent_rates={}, limiter=RateLimiter(rate_limit=7, ...), rules=[SamplingRule(sample_rate=0.5, service=GlobMatcher(pattern=sampling-lab), ..., discard=False)]).

Gate 3: the Agent — mostly advice, with a few enforcers

This is the gate most people get wrong, so it was measured against the real Agent. The Agent’s target_traces_per_second (DD_APM_TARGET_TPS, default 10) does not drop head-sampled traces; it computes a rate and tells the tracers. The docs say the same: “The Datadog Agent continuously sends sampling rates to SDKs to apply at the root of traces. The Agent adjusts rates to achieve a target of ten traces per second overall.” I posted 100 root spans per second (priority 1, one service) to the real receiver and printed the rate_by_service field of each response, which changes only when the rate does:

t= 0.0s sent 100 traces -> {"service:,env:": 1}
t= 5.1s sent 100 traces -> {"service:,env:": 0.125, "service:sampling-lab,env:lab": 0.125}
t=10.1s sent 100 traces -> {"service:,env:": 0.1,   "service:sampling-lab,env:lab": 0.1}

Target 10 per second against 100 offered converges on 0.1 after about 10 seconds. But the Agent forwarded everything I sent: 720 priority-1 traces in, 720 chunks out at the fake intake (chunks x720 priority=1). The rate is only a request to the tracer. A tracer that ignores it (a custom client, or a fixed DD_TRACE_SAMPLING_RULES) still gets everything through. The Agent’s own shipped template says what 0 does (read from /etc/datadog-agent/datadog.yaml.example): “As of 7.35.0 sampling cannot be disabled and setting ‘max_traces_per_second’ to 0 no longer disables sampling, but instead sends no traces to the intake.” With DD_APM_MAX_TPS=0, rate_by_service went to 0, agent status printed Priority sampling rate for 'service:sampling-lab,env:lab': 0.0%, and the Writer line read Traces: 0 payloads, 0 traces after 1,000 traces had been received. The stats writer still sent payloads (Stats: 2 payloads, 2 stats buckets): trace metrics are computed on everything the Agent receives, not on what it forwards.

Three Agent-side samplers do enforce.

== error sampler: 500 traces, priority 0 + error=1, 50/s for 10 s
   chunks x105 priority=0 arrived at the intake
== priority 0, no error, same rate
   total chunks 0
== probabilistic sampler (DD_APM_PROBABILISTIC_SAMPLER_ENABLED=true, ..._SAMPLING_PERCENTAGE=20): 1000 priority-1 traces
   chunks x210 priority=1 _dd.p.dm=-9        # status: "Probabilistic sampling percentage: 20%"

The error sampler rescued 105 of 500 rejected-but-erroring traces — about errors_per_second: 10 for 10 seconds, which the docs describe as catching “pieces of traces that contain error spans not caught by head-based sampling, at a rate of up to 10 traces per second per Agent.” A rejected trace without an error was dropped completely (0 chunks). The rare sampler is off by default (enable_rare_sampler: false in agent config --all; the docs: default false from 7.41.0). The probabilistic sampler keeps a fixed percentage by hashing: 210 of 1,000 (21%) at a 20% setting, tagged _dd.p.dm=-9, a mechanism that only the Agent produces (it is not among the tracer’s constants above). With it enabled the response body was an empty {}: the Agent stops sending rates when it does the sampling itself.

Gate 4: ingestion controls

Everything above ends at the Datadog intake. What happens next is backend configuration, so it is docs-sourced, not run. The Ingestion Control page provides “visibility into the ingestion configuration of your applications and services” and lets you “Adjust trace sampling rates for high throughput services or endpoints”; the change reaches tracers through Remote Configuration, requiring “Datadog Agent 7.41.1 or higher” and SDK versions such as Python v2.9.0 and Node.js v5.16.0, and “you do not need to redeploy the service for the change to take effect” (ingestion controls). The pinned tracer reflects this: SAMPLING_MECHANISM_REMOTE_USER: 11 and SAMPLING_MECHANISM_REMOTE_DYNAMIC: 12 exist in its constants, so a trace sampled by a UI-driven rate carries _dd.p.dm=-11 or -12 instead of -3. And the caveat that saves most incident reviews, from the same page: “APM Metrics are always calculated based on all traces, and are not impacted by ingestion controls.”

Gate 5: retention filters and intelligent retention

Also backend, also docs-sourced (trace retention). Retention filters index ingested spans for 15 days (“kept for 15 days according to the retention filters that are set up on your account”), and order matters: “If a span matches a retention filter early in the list, the span is either kept or dropped. Any matching custom retention filter lower on the list does not catch the already-processed span.” Crucially, “Retention filters do not affect what traces are collected by the Agent and sent to Datadog (‘ingested’).” Intelligent retention is the always-on baseline: diversity sampling “retains at least one span (and the associated trace) for each combination of environment, service, operation, and resource,” together with a “uniform 1% sample of ingested spans.” Nothing about gates 4 and 5 is visible in a span on the wire, which is why the only honest proof available offline is the docs quote.

Mental Model

Read the pipeline as a funnel with two feedback valves and one hard floor. The head sampler decides per trace at the source and is the only gate that saves network and CPU. The rate limiter is a safety valve on that same gate. The Agent looks at total traffic and asks tracers to change their rate — a control loop with a lag of several seconds — and enforces only for the special cases (errors, probabilistic, MAX_TPS=0). Everything after ingestion is a storage decision about spans that already arrived and were already counted in metrics. When a trace is missing, find which gate dropped it by its fingerprint: priority=-1 plus _dd.rule_psr is a rule, plus _dd.limit_psr < 1 is the limiter, priority=0 plus _dd.agent_psr < 1 is the Agent rate.

Trade-offs

Choice	Benefit	Cost
Tracer rules (`DD_TRACE_SAMPLING_RULES`)	Decide per service/resource/tag; saves CPU and bandwidth at the source	Static per deploy unless remote configuration is used; syntax differs by language (see `max_per_second`)
Agent-driven rates (default)	No config; adapts to traffic toward a 10 traces/s target	A lag of seconds after a traffic jump; not enforced against a tracer that ignores it
`discard: true` in a rule	Drops the trace and its stats contribution	Trace metrics under-count; the request rate on the service page is no longer the real rate
Agent probabilistic sampler	Uniform percentage across every SDK and OTel source, enforced	The decision is made at the Agent, after the tracer already paid to create and send every trace (1,000 received, 210 forwarded in the run above)
Error sampler at the default 10/s	Errors survive head sampling of healthy traffic	Above 10 errors/s per Agent, error traces are dropped too
Ingestion controls (UI)	Change rates without redeploy	Needs Agent and SDK at or above the remote-config minimums; ownership of “who set this rate” moves out of git
Retention filters	Keep exactly what you search, for the retention period	They cannot bring back a trace that was not ingested

Common Misconceptions

“DD_APM_TARGET_TPS=10 means the Agent forwards at most 10 traces per second.” Measured: 720 priority-1 traces in, 720 out. It is a rate advertised to tracers.
“max_traces_per_second: 0 turns sampling off.” It sends no traces to the intake (measured: Traces: 0 payloads, 0 traces after 1,000 received).
“A sampled-out trace never leaves the app.” A rejected trace (priority=-1 or 0) is still sent to the Agent unless the rule says discard; that is how metrics stay complete.
“_dd.rule_psr on a span means it was kept.” It means a rule matched. priority=-1 next to it means the rule (or the limiter behind it) rejected it.
“One DD_TRACE_SAMPLING_RULES value works in every language.” Node accepts max_per_second; Python 4.15.2 raises TypeError at import.
“Retention filters reduce ingestion cost.” They act after ingestion; docs: they “do not affect what traces are collected by the Agent and sent to Datadog.”

Reading AI-generated code

A model asked for “sample 10% of checkout traces, but never more than 5 a second” writes the per-rule cap it saw in a Node example.

🐛 Spot the bug

# scripts/checkout-sampling.sh   (Python service, ddtrace)
# Sample 10% of checkout traces and cap the rule at 5 traces per second
export DD_SERVICE=checkout
export DD_TRACE_SAMPLING_RULES='[{"service":"checkout","sample_rate":0.1,"max_per_second":5}]'

Show the answer

max_per_second inside a rule is a Node feature. In the pinned Python tracer (ddtrace 4.15.2) SamplingRule.__init__ has no such keyword, and the real run above ended with TypeError: SamplingRule.__init__() got an unexpected keyword argument 'max_per_second' — the service fails at startup instead of sampling. (Node, by contrast, honoured max_per_second:2 and kept exactly 2 of 60.) For Python use the global limiter, which caps traces that matched a sampling rule:

export DD_SERVICE=checkout
export DD_TRACE_SAMPLING_RULES='[{"service":"checkout","sample_rate":0.1}]'
export DD_TRACE_RATE_LIMIT=5

A second model-written config disables sampling in the Agent:

🐛 Spot the bug

api_key: <your-api-key>
apm_config:
enabled: true
# keep every trace: 0 means unlimited
max_traces_per_second: 0

Show the answer

The Agent’s own shipped template (datadog.yaml.example in the container) says the opposite: “As of 7.35.0 sampling cannot be disabled and setting ‘max_traces_per_second’ to 0 no longer disables sampling, but instead sends no traces to the intake.” Measured with DD_APM_MAX_TPS=0: rate_by_service returned 0, agent status printed Priority sampling rate ... 0.0%, and 1,000 received traces produced Traces: 0 payloads, 0 traces in the Writer line. The key is also deprecated in favour of target_traces_per_second. To keep more, raise the target above your peak traffic:

api_key: <your-api-key>
apm_config:
  enabled: true
  target_traces_per_second: 200
  errors_per_second: 10