Ingestion Sampling vs. Retention Filters

The idea in one sentence

“Trace sampling” is actually two separate, stacked controls — one decides what even gets sent to Datadog, the other decides what stays searchable once it’s already there — and treating them as one setting is the single most common APM sampling mistake.

Layer 1: ingestion-time sampling

This layer lives in the tracer (and sometimes the Agent), and it answers one question: how many of the traces your application generates are even sent to Datadog in the first place. It runs before any trace ever leaves your infrastructure. A rate configured here of, say, 0.2 means 80% of traces are simply never transmitted — they don’t exist in Datadog in any form, searchable or not:

# Tracer-side config — controls what even reaches Datadog at all
# DD_TRACE_SAMPLE_RATE is not deprecated, but ddtrace 4.15.2 (Python) ignores it; rules work in both
DD_TRACE_SAMPLING_RULES='[{"sample_rate": 0.2}]'   # sample 20% of traces at ingestion time

Agent-side ingestion controls exist too (sampling rules applied as traces pass through the Agent before upstream submission), but the concept is the same regardless of exactly where the decision is made: anything not ingested never reaches Datadog at all, full stop. There is no later step that can recover a trace this layer dropped.

Layer 2: APM Retention Filters

Everything that survives layer 1 lands in Datadog as an ingested trace. What happens next is a completely separate decision, made server-side, by APM Retention Filters (filter_type: spans-sampling-processor). A retention filter decides which already-ingested traces remain searchable and inspectable long-term — ingestion and retention are not the same gate.

A retention filter has two independent knobs:

• rate — the fraction of matching spans to retain.
• trace_rate — a separate fraction of the whole trace to retain, for any trace that contains at least one matching span.

{
  "data": {
    "attributes": {
      "name": "keep-checkout-errors",
      "enabled": true,
      "filter_type": "spans-sampling-processor",
      "filter": { "query": "service:checkout status:error" },
      "rate": 1.0,
      "trace_rate": 1.0
    }
  }
}

This example says: for any span matching service:checkout status:error, keep 100% of those spans (rate: 1.0), and also keep 100% of the complete traces that contain one of those spans (trace_rate: 1.0) — so you don’t just retain the error span in isolation, you retain the whole surrounding trace to see how the request got there.

On top of anything you configure yourself, Datadog also runs an always-on Intelligent Retention Filter — it automatically keeps a representative proportion of traces across your traffic for general application-health visibility, independent of and in addition to any custom retention filters you define.

The practical implication: two gates, not one

If you only think about “my sampling rate,” you’re only thinking about layer 1. That leaves two failure modes hiding in plain sight:

• You can ingest 100% of traces and still lose the ones you actually care about later, if no retention filter is configured to keep them — a high ingestion rate is not the same thing as guaranteed retention.
• You can ingest a small fraction of overall traffic and still fully retain the traces that matter — for example, a targeted retention filter that keeps every error trace — because retention filters operate on whatever made it through layer 1, and a rate/trace_rate of 1.0 on a narrow query is a completely different guarantee than your overall ingestion percentage.

The two layers are independent by design: ingestion controls cost and volume at the front door, retention filters control long-term searchability of whatever came through that door.

flowchart LR
  App[Instrumented app generates traces] --> Layer1[Layer 1: ingestion-time sampling in tracer/Agent]
  Layer1 -->|dropped: never reaches Datadog| Gone[Not ingested]
  Layer1 -->|sampled subset ingested| Backend[Datadog APM backend: all ingested traces]
  Backend --> Layer2[Layer 2: APM Retention Filters rate + trace_rate]
  Backend --> IRF[Intelligent Retention Filter: always-on baseline]
  Layer2 -->|kept| Searchable[Searchable, inspectable traces]
  IRF --> Searchable

Two stacked sampling layers

🐶 The one-line takeaway

Ingestion-time sampling (tracer/Agent) decides what is even sent to Datadog, and APM Retention Filters (server-side, with independent rate and trace_rate) decide what stays searchable afterward — they are two stacked, independent gates, so a high ingestion rate does not guarantee retention, and a low ingestion rate does not prevent full retention of the traces that matter.

The full pipeline, one diagram

“Layer 1” above is actually three cooperating mechanisms, not one, and there is a third kind of sampling this lesson hasn’t covered yet — single-span sampling — that keeps individual spans even when the trace-level sampler drops the whole trace around them. One diagram for the full path a span travels, tracer to searchable:

flowchart LR
  App["App code creates a span"] --> Sampler["Tracer sampler: DD_TRACE_SAMPLING_RULES + DD_TRACE_RATE_LIMIT"]
  Sampler -->|trace dropped| Gone1["Trace never sent"]
  Sampler -->|trace kept| AgentRL["Agent rate limiter: apm_config.target_traces_per_second"]
  SingleSpan["DD_SPAN_SAMPLING_RULES: single-span sampling"] -.keeps this one span even if its trace was dropped.-> AgentRL
  AgentRL -->|over target, dropped| Gone2["Excess traces dropped at the Agent"]
  AgentRL -->|kept| Intake["Datadog intake: trace ingested"]
  Intake --> RF["APM Retention Filters: rate + trace_rate"]
  Intake --> IRF["Intelligent Retention Filter: always-on baseline"]
  RF -->|kept| Searchable["Searchable, inspectable traces"]
  IRF --> Searchable

Ingestion sampling, Agent rate limiting, and retention filters, end to end

Per docs.datadoghq.com/tracing/trace_pipeline/ingestion_mechanisms/: sampling starts with head-based sampling — a decision “made at the start of the root span” and propagated through the whole trace — configured via DD_TRACE_SAMPLING_RULES (rule-based) or single-span sampling via DD_SPAN_SAMPLING_RULES. The Agent then “continuously sends sampling rates to SDKs to apply at the root of traces,” adjusting toward a target throughput; separately, the Agent runs two more samplers of its own on top of head-based sampling: an error sampler (catches error spans at a configurable rate, DD_APM_ERROR_TPS) and a rare sampler (catches otherwise-unseen span combinations). Every ingested span gets an ingestion_reason tag recording which of these mechanisms is why it made it through. The docs draw the same layer split this lesson does, in two separate pages: Ingestion Controls (everything before intake) and Trace Retention (everything after).

Single-span sampling: keeping one span when the whole trace is dropped

DD_SPAN_SAMPLING_RULES is a mechanism this lesson hasn’t covered yet, and it answers a real gap in the two-layer story above: what if the trace gets dropped by ingestion sampling, but one specific span inside it — a slow database call, a call to a particularly important downstream service — is worth keeping anyway? Single-span sampling ingests that one span independently of the trace-level decision. Proven with a real dd-trace config dump — the rule is genuinely parsed and applied, not a no-op key:

src/span-sampling-config.js

require('dd-trace').init({
  service: 'checkout-web',
  logInjection: false,
});

$ DD_SPAN_SAMPLING_RULES='[{"service":"checkout-web","name":"checkout.apply_discount","sample_rate":1.0,"max_per_second":50}]' \
    node -e "const t=require('dd-trace').init({service:'probe'}); console.log(JSON.stringify(t._tracer._config.spanSamplingRules))"
[{"service":"checkout-web","name":"checkout.apply_discount","sampleRate":1,"maxPerSecond":50}]

And the Python equivalent, real introspection of the parsed rule object (ddtrace.internal.sampling.get_span_sampling_rules()), confirming both fields actually landed:

$ DD_SPAN_SAMPLING_RULES='[{"service":"checkout-web","name":"checkout.apply_discount","sample_rate":1.0,"max_per_second":50}]' \
    python -c "from ddtrace.internal.sampling import get_span_sampling_rules as g; r=g()[0]; print(r._sample_rate, r._max_per_second)"
1.0 50

max_per_second matters here specifically because single-span sampling exists to rescue interesting spans from a dropped trace — without its own cap, a rule matching a high-volume operation would defeat the whole point of the ingestion sampler it’s working around.

Proving the sampling env vars are actually read

Four different env vars claim to control sampling in some way. Rather than trust the names, dumped each tracer’s real, merged config after setting exactly one at a time — this is the same technique (and the same harness mode, --options, checking every key against the pinned dd-trace .d.ts) that caught the logInjection/log_injection bug in the correlating-signals module:

$ DD_TRACE_SAMPLE_RATE=0.3 node -e "const t=require('dd-trace').init({service:'probe'}); console.log(JSON.stringify(t._tracer._config.sampler))"
{"rules":[],"rateLimit":100,"sampleRate":0.3}

$ DD_TRACE_SAMPLING_RULES='[{"service":"checkout-web","sample_rate":0.5}]' node -e "const t=require('dd-trace').init({service:'probe'}); console.log(JSON.stringify(t._tracer._config.sampler))"
{"rules":[{"service":"checkout-web","sampleRate":0.5}],"rateLimit":100}

$ DD_TRACE_RATE_LIMIT=25 node -e "const t=require('dd-trace').init({service:'probe'}); console.log(JSON.stringify(t._tracer._config.sampler))"
{"rules":[],"rateLimit":25}

All three are genuinely read and merged into the tracer’s own sampler config in dd-trace (Node). The default rateLimit: 100 confirms the review’s point that a rate limiter is present even when nothing about it is configured explicitly — more on that below.

🐶 A real cross-language surprise: DD_TRACE_SAMPLE_RATE does nothing in the pinned ddtrace (Python)

The same test against ddtrace==4.15.2 tells a different story. grep -rn "DD_TRACE_SAMPLE_RATE" ddtrace/ across the entire installed package finds it in exactly one place — the SUPPORTED_CONFIGURATIONS telemetry list — and nowhere else. Setting it has zero effect on the real sampler:

$ DD_TRACE_SAMPLE_RATE=0.4 python -c "import ddtrace; print(ddtrace.tracer._sampler.rules)"
[]

Compare DD_TRACE_SAMPLING_RULES, which genuinely populates the same object:

$ DD_TRACE_SAMPLING_RULES='[{"service":"checkout-web","sample_rate":0.5}]' \
    python -c "import ddtrace; print(ddtrace.tracer._sampler.rules)"
[SamplingRule(sample_rate=0.5, service=GlobMatcher(pattern=checkout-web), name=None, resource=None, tags={}, provenance=default, discard=False)]

This is stronger than “deprecated” — the docs page for the Node.js tracer configuration does not label DD_TRACE_SAMPLE_RATE deprecated either (it describes it plainly as controlling “the ingestion sample rate… between the Agent and the backend,” still functional). What’s actually true, proven above: DD_TRACE_SAMPLE_RATE works in dd-trace (Node) and is a complete no-op in the pinned ddtrace (Python) client. A cross-language codebase that sets it via a shared env var and expects the same behavior in both languages will get silently inconsistent sampling — Node samples at the configured rate, Python samples at its unconfigured default (100%, capped only by the rate limiter). DD_TRACE_SAMPLING_RULES is the one mechanism proven to work identically in both.

The rate limiter is a second, independent cap

Setting a sampling rate to 1.0 (or omitting sampling rules entirely, which also means “keep everything by default”) does not mean every trace actually gets sent — DD_TRACE_RATE_LIMIT/rateLimit caps the absolute number of traces per second the sampler will let through, on top of whatever the rate says. ddtrace (Python) actually prints a real warning explaining this relationship when the two are set inconsistently:

$ DD_TRACE_RATE_LIMIT=25 python -c "import ddtrace; print(ddtrace.tracer._sampler.limiter)"
DD_TRACE_RATE_LIMIT is set to 25 and DD_TRACE_SAMPLING_RULES is not set. Tracer rate limiting is only applied to spans that match tracer sampling rules. All other spans will be rate limited by the Datadog Agent via DD_APM_MAX_TPS.
RateLimiter(rate_limit=25, tokens=25, last_update_ns=..., effective_rate=1.0)

That warning is the two-layer story from this lesson’s opening, stated by the tracer itself: without a sampling rule to attach the tracer-side rate limit to, the Agent’s own rate limiter (apm_config.target_traces_per_second, see below) is the thing actually capping throughput. Set both a rule and a rate limit together, and the tracer-side cap applies to spans matching that rule specifically:

$ DD_TRACE_RATE_LIMIT=25 DD_TRACE_SAMPLING_RULES='[{"service":"checkout-web","sample_rate":1.0}]' \
    python -c "import ddtrace; print(ddtrace.tracer._sampler.limiter)"
RateLimiter(rate_limit=25, tokens=25, last_update_ns=..., effective_rate=1.0)

_dd.rule_psr and _sampling_priority_v1 on real captured spans

Every span carries the evidence of the sampling decision that let it through, as metrics. Captured via the same local fake-trace-agent technique proven in Traces, Spans, and the Service Map (a plain HTTP listener decoding the real msgpack payload — nothing sent to Datadog): with no sampling rule configured, a real root span’s metrics carries only the baseline priority tag:

"metrics": { "_dd.measured": 1, "_sampling_priority_v1": 1 }

With DD_TRACE_SAMPLING_RULES='[{"service":"checkout-web","sample_rate":1.0}]' set, the same request produces a real span whose metrics gained a new field and whose priority value changed:

"metrics": { "_dd.measured": 1, "_dd.rule_psr": 1, "_sampling_priority_v1": 2 }

_dd.rule_psr (“rule priority sample rate”) only appears once a sampling rule actually matched — it is the rule’s own configured rate (1.0 here), stamped onto the span as proof of which rate applied. _sampling_priority_v1 moved from 1 (AUTO_KEEP, the tracer’s own default decision) to 2 (USER_KEEP, a rule explicitly said keep) — a real, visible difference between “nothing told the tracer to keep this” and “a rule you wrote told it to.”

Agent-side: target_traces_per_second, not the deprecated max_traces_per_second

The spec for this lesson pointed at apm_config.max_traces_per_second as the Agent-level cap to prove via configcheck — running a real Agent 7.83.3 container and reading its own shipped config template (datadog.yaml.example) turned up something the spec got wrong:

$ docker run --rm datadog/agent:7 sh -c \
    "grep -A5 'max_traces_per_second - float' /etc/datadog-agent/datadog.yaml.example"
#   # @param max_traces_per_second - float - optional - default: 10
#   # @env DD_APM_MAX_TPS - float - optional - default: 10
#   # @env DD_MAX_TPS - float - optional - default: 10
#   # DEPRECATED - please use `target_traces_per_second` instead.

apm_config.max_traces_per_second is real, still works, and defaults to 10 (not 100) — but the Agent’s own shipped template marks it deprecated in favor of apm_config.target_traces_per_second (identical meaning, identical default). A datadog.yaml using the modern key, agent configcheck’d against a real, freshly-pulled datadog/agent:7 container (DD_API_KEY fake, DD_DD_URL pointed at a closed local port, nothing sent anywhere):

conf.d/../datadog.yaml

apm_config:
  enabled: true
  target_traces_per_second: 20
  errors_per_second: 10

$ docker run -d --name dd-apm -e DD_API_KEY=00000000000000000000000000000fa \
    -e DD_DD_URL=http://127.0.0.1:1 -e DD_HOSTNAME=dd-apm-lesson \
    -v $PWD/datadog.yaml:/etc/datadog-agent/datadog.yaml:ro datadog/agent:7
$ docker exec dd-apm agent configcheck
=== cloud_hostinfo check ===
...

configcheck completed with every default check listed and zero errors or warnings about apm_config — the Agent silently accepts the deprecated key too (it just prefers the new one). The Agent’s real trace-agent process was not reachable within this sandboxed run’s window (agent status’s APM Agent section reported “Not running or unreachable on localhost:5012” — expected, since DD_DD_URL points at a closed port and no real network path exists here); configcheck accepting the config without complaint is the proof this lesson can honestly make.

The retention filter JSON model, validated — and a real name correction

The spec for this lesson also expected a model called ApmRetentionFilterCreateRequest. The real, installed datadog_api_client==2.61.0 has no such module. Enumerating every retention model it actually ships:

$ python -c "import pkgutil, datadog_api_client.v2.model as m; \
    print([n for _, n, _ in pkgutil.iter_modules(m.__path__) if n.startswith('retention_filter')])"
['retention_filter', 'retention_filter_all', ..., 'retention_filter_create_attributes',
 'retention_filter_create_data', 'retention_filter_create_request', 'retention_filter_create_response', ...]

The real class is RetentionFilterCreateRequest (no Apm prefix — this client’s naming groups APM and RUM retention filters differently: rum_retention_filter_create_request is the RUM-specific sibling). Validating this lesson’s earlier JSON example against the real model:

$ python -c "
from datadog_api_client.v2.model.retention_filter_create_request import RetentionFilterCreateRequest
payload = {'data': {'type': 'apm_retention_filter', 'attributes': {
    'name': 'keep-checkout-errors', 'enabled': True,
    'filter_type': 'spans-sampling-processor',
    'filter': {'query': 'service:checkout status:error'},
    'rate': 1.0, 'trace_rate': 1.0}}}
print(RetentionFilterCreateRequest(**payload))
"
validated OK

A broken version, missing the required filter_type field, produces a real error naming exactly what’s missing:

# @expect-error RetentionFilterCreateAttributes.__init__() missing 1 required positional argument: 'filter_type'
from datadog_api_client.v2.model.retention_filter_create_request import RetentionFilterCreateRequest

payload = {"data": {"type": "apm_retention_filter", "attributes": {
    "name": "broken-filter", "enabled": True,
    "filter": {"query": "service:checkout"}, "rate": 1.0
}}}
RetentionFilterCreateRequest(**payload)

🐶 A documented gap in this course's own harness

Retention filter JSON isn’t in this course’s harness’s supported-JSON directory list (monitors/, dashboards/, slos/, service-catalog/, logs-pipelines/) — npm run verify’s strict --json mode has no path convention for it, so the fence above is validated manually (as shown) rather than automatically on every build. Noted here rather than silently claimed as harness-covered.

Reading AI-generated code

🐛 Spot the bug

app/configure_sampling.py

# AI comment: "Sample 20% of traces to control APM costs"
import os

os.environ["DD_TRACE_SAMPLE_RATE"] = "0.2"

# ... ddtrace-run picks this up before the app starts ...

Show the answer

This is exactly the pattern shown in the Node.js example earlier in this lesson, ported to Python — and it silently does nothing. Proven above: grep -rn "DD_TRACE_SAMPLE_RATE" ddtrace/ across the entire pinned ddtrace==4.15.2 package finds it only in the telemetry-reporting SUPPORTED_CONFIGURATIONS list, never read into the real sampler (ddtrace.tracer._sampler.rules stayed [] with this var set). The service keeps sampling at its unconfigured default — effectively 100%, capped only by the Agent’s own rate limiter — while the team believes costs are being controlled at 20%. The fix that is proven above to actually work in this pinned version: DD_TRACE_SAMPLING_RULES='[{"sample_rate": 0.2}]'.

Mental Model

Think of the whole pipeline as a series of doors, each one only able to make things smaller, never bring back what an earlier door already closed. The tracer’s sampler is the first door — proven above, its decision (_sampling_priority_v1, _dd.rule_psr) is stamped onto the span before it ever leaves the process. The Agent’s rate limiter is a second door, enforcing a hard cap regardless of what the first door decided. Retention filters are a third, completely separate set of doors, but they open on ingested traffic only — they can never see, let alone rescue, anything the first two doors already closed. Single-span sampling is the one exception to “doors only close smaller” in this pipeline: it is a side door that can keep one span even after the main door (trace-level sampling) has already closed on the trace around it.

Trade-offs

• DD_TRACE_SAMPLING_RULES vs. DD_TRACE_SAMPLE_RATE — sampling rules are more precise (proven above: per-service, per-name, per-resource) and, proven above, are the only mechanism confirmed to work identically in both Node and Python; a flat sample rate is simpler to reason about but (proven above) is silently inert in the pinned Python client.
• Tracer-side rate limit vs. Agent-side rate limit — proven above via ddtrace’s own warning: the tracer-side DD_TRACE_RATE_LIMIT only applies to spans matching a sampling rule; everything else is capped later, by the Agent’s target_traces_per_second. Relying on only one of the two leaves a real gap for whichever traffic doesn’t match your rules.
• Single-span sampling vs. a broader trace-level rule — proven above to genuinely rescue one important span from an otherwise-dropped trace, at the cost of losing everything else in that trace’s context; a trace-level rule keeps the whole story but costs proportionally more ingested volume.

Common Misconceptions

• “A sample rate of 1.0 (or no rules at all) guarantees every trace is ingested.” No — proven above: the Agent’s target_traces_per_second (default 10, real Agent 7.83.3 config template) still caps throughput regardless of the tracer-side decision.
• “DD_TRACE_SAMPLE_RATE behaves the same in every Datadog tracing library.” No — proven above: it is read into the real sampler config in dd-trace (Node) but has zero effect in the pinned ddtrace (Python) — confirmed by grepping the entire installed package for the string.
• “apm_config.max_traces_per_second is the current, recommended Agent setting.” No — proven above from the real Agent 7.83.3 config template: it is explicitly marked DEPRECATED - please use target_traces_per_second instead, though it still works and defaults to 10.
• “Single-span sampling is just trace-level sampling with a smaller scope.” No — proven above: it is a genuinely separate mechanism that can keep one span even when the trace-level sampler already dropped the entire surrounding trace, which a smaller trace-level rate could never do.

What does ingestion-time sampling in the tracer/Agent control?

Which already-ingested traces stay searchable long-term How many traces are even sent to Datadog in the first place The color coding of the Service Map How DDSketch percentiles are computed

In an APM Retention Filter, what is the difference between rate and trace_rate?

rate and trace_rate are aliases for the same setting rate only applies to error traces, trace_rate only applies to non-error traces rate is the fraction of matching spans to retain, trace_rate is the fraction of the whole trace containing a matching span to retain — two independent controls trace_rate controls ingestion, rate controls retention

A trace is dropped by ingestion-time sampling before it reaches Datadog. What happens next?

The Intelligent Retention Filter can still recover it A retention filter with trace_rate: 1.0 can still recover it Nothing can recover it — it was never ingested, so no retention filter ever sees it It is stored in a low-cost archive for later retrieval

A team ingests only 5% of overall traffic, but has a retention filter matching status:error with rate: 1.0 and trace_rate: 1.0. What is true?

No error traces will be retained since ingestion is only 5% Every error trace that made it through the 5% ingestion sample will be fully retained and searchable The retention filter is ignored because ingestion sampling is below 100% The Intelligent Retention Filter overrides the custom filter

Real test: DD_TRACE_SAMPLE_RATE=0.4 set, then ddtrace.tracer._sampler.rules printed on the pinned ddtrace==4.15.2 (Python). What does it show, and what does the same test show on dd-trace (Node)?

Both print a populated sampling rule at 0.4 — the env var works identically in both Python prints [] (the var is a genuine no-op, confirmed by grepping the whole package) while Node's config.sampler shows sampleRate: 0.4 actually applied Both raise an error because DD_TRACE_SAMPLE_RATE was removed in these versions Node ignores it too — only DD_TRACE_SAMPLING_RULES works in any language

What does apm_config.target_traces_per_second in a real Agent 7.83.3 datadog.yaml.example replace, and why?

It replaces errors_per_second, which was removed entirely It replaces apm_config.max_traces_per_second, which the Agent's own shipped config template marks "DEPRECATED - please use target_traces_per_second instead" (both default to 10) It replaces DD_TRACE_RATE_LIMIT, which only exists client-side Nothing — they are two unrelated settings that must both be configured

Submit

Mark as complete