Ingestion Sampling vs. Retention Filters

The idea in one sentence

“Trace sampling” is actually two separate, stacked controls — one decides what even gets sent to Datadog, the other decides what stays searchable once it’s already there — and treating them as one setting is the single most common APM sampling mistake.

Layer 1: ingestion-time sampling

This layer lives in the tracer (and sometimes the Agent), and it answers one question: how many of the traces your application generates are even sent to Datadog in the first place. It runs before any trace ever leaves your infrastructure. A rate configured here of, say, 0.2 means 80% of traces are simply never transmitted — they don’t exist in Datadog in any form, searchable or not:

# Tracer-side config — controls what even reaches Datadog at all
DD_TRACE_SAMPLE_RATE=0.2   # sample 20% of traces at ingestion time

Agent-side ingestion controls exist too (sampling rules applied as traces pass through the Agent before upstream submission), but the concept is the same regardless of exactly where the decision is made: anything not ingested never reaches Datadog at all, full stop. There is no later step that can recover a trace this layer dropped.

Layer 2: APM Retention Filters

Everything that survives layer 1 lands in Datadog as an ingested trace. What happens next is a completely separate decision, made server-side, by APM Retention Filters (filter_type: spans-sampling-processor). A retention filter decides which already-ingested traces remain searchable and inspectable long-term — ingestion and retention are not the same gate.

A retention filter has two independent knobs:

rate — the fraction of matching spans to retain.
trace_rate — a separate fraction of the whole trace to retain, for any trace that contains at least one matching span.

{
  "data": {
    "attributes": {
      "name": "keep-checkout-errors",
      "enabled": true,
      "filter_type": "spans-sampling-processor",
      "filter": { "query": "service:checkout status:error" },
      "rate": 1.0,
      "trace_rate": 1.0
    }
  }
}

This example says: for any span matching service:checkout status:error, keep 100% of those spans (rate: 1.0), and also keep 100% of the complete traces that contain one of those spans (trace_rate: 1.0) — so you don’t just retain the error span in isolation, you retain the whole surrounding trace to see how the request got there.

On top of anything you configure yourself, Datadog also runs an always-on Intelligent Retention Filter — it automatically keeps a representative proportion of traces across your traffic for general application-health visibility, independent of and in addition to any custom retention filters you define.

The practical implication: two gates, not one

If you only think about “my sampling rate,” you’re only thinking about layer 1. That leaves two failure modes hiding in plain sight:

You can ingest 100% of traces and still lose the ones you actually care about later, if no retention filter is configured to keep them — a high ingestion rate is not the same thing as guaranteed retention.
You can ingest a small fraction of overall traffic and still fully retain the traces that matter — for example, a targeted retention filter that keeps every error trace — because retention filters operate on whatever made it through layer 1, and a rate/trace_rate of 1.0 on a narrow query is a completely different guarantee than your overall ingestion percentage.

The two layers are independent by design: ingestion controls cost and volume at the front door, retention filters control long-term searchability of whatever came through that door.

flowchart LR
  App[Instrumented app generates traces] --> Layer1[Layer 1: ingestion-time sampling in tracer/Agent]
  Layer1 -->|dropped: never reaches Datadog| Gone[Not ingested]
  Layer1 -->|sampled subset ingested| Backend[Datadog APM backend: all ingested traces]
  Backend --> Layer2[Layer 2: APM Retention Filters rate + trace_rate]
  Backend --> IRF[Intelligent Retention Filter: always-on baseline]
  Layer2 -->|kept| Searchable[Searchable, inspectable traces]
  IRF --> Searchable

Two stacked sampling layers