Trace Metrics and DDSketch

The idea in one sentence

Regardless of both sampling layers covered in the previous lesson, Datadog continuously computes trace metrics — hit count, error count, and latency — from every span that reaches the Agent/Datadog, so your traffic and latency dashboards stay statistically accurate even when only a fraction of individual traces remain inspectable.

Trace metrics: computed from every span, continuously

For every service and resource, Datadog aggregates three things off the stream of spans reaching it:

• hit count — how many requests/operations happened.
• error count — how many of them errored.
• latency — the distribution of how long they took.

This aggregation happens continuously and is derived from every span that reaches the Agent/Datadog — it is not computed by opening and re-reading individual stored traces later. That’s the key distinction from the previous lesson: ingestion-time sampling and retention filters govern whether a specific full trace is later available to open and inspect, but the aggregate counting and timing runs off the span stream itself, independently of those two layers.

DDSketch: percentiles that survive aggregation

Latency isn’t one number — you care about the distribution, especially the tail (p95, p99), not just the average. But percentiles have an awkward property: you can’t average two percentiles from two hosts and get a correct combined percentile. Datadog solves this with DDSketch, a sketch-based distribution metric type built specifically so percentile estimates stay accurate and mergeable at scale:

app/ddsketch-proof.py

from ddsketch import DDSketch

sketch_host_a = DDSketch(relative_accuracy=0.01)
sketch_host_b = DDSketch(relative_accuracy=0.01)
# each host's sketch summarizes every observed span latency locally
for v in latencies_host_a:
    sketch_host_a.add(v)
for v in latencies_host_b:
    sketch_host_b.add(v)
sketch_host_a.merge(sketch_host_b)  # merges in place, host A's sketch now covers both hosts
p95 = sketch_host_a.get_quantile_value(0.95)

Each process or host keeps a local sketch summarizing the latencies it observed; sketches from many hosts merge into one combined sketch, and a percentile pulled from the merged sketch is still an accurate estimate of the true global percentile — something a naive average-of-percentiles approach cannot give you. Installed the real, open-source ddsketch package (Datadog’s own reference implementation — the Agent’s own Go sketch library, DataDog/sketches-go, implements the same algorithm) and proved the accuracy guarantee against 5,000 real random.gauss(100, 15) samples, comparing the sketch’s estimate to the exact percentile computed by sorting the raw data:

$ python -c "
from ddsketch import DDSketch
import random
random.seed(1)
data = [random.gauss(100, 15) for _ in range(5000)]
s = DDSketch(relative_accuracy=0.01)
for v in data: s.add(v)
data_sorted = sorted(data)
for p in [0.5, 0.95, 0.99]:
    exact = data_sorted[int(p * len(data_sorted))]
    est = s.get_quantile_value(p)
    print(f'p{int(p*100)}: exact={exact:.3f} sketch={est:.3f} rel_err={abs(est-exact)/exact:.4f}')
"
p50: exact=99.902 sketch=100.495 rel_err=0.0059
p95: exact=124.875 sketch=125.225 rel_err=0.0028
p99: exact=135.150 sketch=135.655 rel_err=0.0037

Every relative error came in under the configured relative_accuracy=0.01 (1%) bound — real evidence, not a restated claim, that the sketch’s percentile estimate tracks the true distribution within the accuracy it promises. DDSketch is Datadog’s own published algorithm (Masson, Rim, Lee, “DDSketch: A Fast and Fully-Mergeable Quantile Sketch with Relative-Error Guarantees”, VLDB 2019) — the mergeability proven above (two independent sketches, one combined result) is the property that specific paper is about.

Why this is unaffected by sampling, but example-trace access is not

This is the point worth internalizing carefully: trace metrics and their DDSketch-backed percentiles are not affected by trace sampling or retention filters. Neither layer 1 (ingestion-time sampling) nor layer 2 (APM Retention Filters) touches the hit count, error count, or p50/p95/p99 latency you see on a service’s overview page — those numbers come from the span stream itself, aggregated continuously.

What is affected by both layers is your ability to open one specific example trace — say, the slowest request in the last hour, or one instance of a particular error. Finding and inspecting that individual trace depends on it having survived ingestion-time sampling and on a retention filter having kept it:

# Two different questions, two different data paths
p95 latency for checkout-web        <- DDSketch, from every span reaching Datadog, unaffected by sampling
"show me one slow example trace"    <- must have been ingested AND retained to open

This is exactly why you can trust a latency or error-rate dashboard even under an aggressive ingestion sampling rate, while still needing a well-designed retention filter (from the previous lesson) to guarantee you can actually find and inspect a specific slow or erroring trace when you go looking for one.

flowchart LR
  Spans[Every span reaching Agent/Datadog] --> Stats[Trace metrics: hit count, error count, latency]
  Stats --> Sketch[DDSketch per service/resource]
  Sketch --> P[p50/p95/p99 dashboards: unaffected by sampling]
  Spans -->|ingestion-time sampling| L1[Layer 1 gate]
  L1 -->|retention filters rate/trace_rate| L2[Layer 2 gate]
  L2 --> Kept[Individual example traces you can open]

Aggregate metrics vs. individual trace access

🐶 The one-line takeaway

Trace metrics — hit count, error count, and DDSketch-backed latency percentiles — are computed from every span reaching Datadog and are unaffected by ingestion sampling or retention filters; only your ability to open and inspect one specific example trace depends on those two layers, which is why the dashboard stays trustworthy even under aggressive sampling.

A real Agent config key that confirms “trace metrics” is not just this lesson’s phrase

Pulling and reading the real Agent 7.83.3 image’s own datadog.yaml.example for apm_config turns up a key whose own doc comment uses the exact term this lesson has been building around:

$ docker run --rm datadog/agent:7 sh -c \
    "grep -B2 -A6 'compute_stats_by_span_kind' /etc/datadog-agent/datadog.yaml.example"
#   # @param compute_stats_by_span_kind - boolean - optional - default: true
#   # @env DD_APM_COMPUTE_STATS_BY_SPAN_KIND - boolean - optional - default: true
#   # Enables an additional stats computation check on spans to see they have an eligible `span.kind`
#   # (server, consumer, client, producer). If enabled, a span with an eligible span.kind will have
#   # stats computed. If disabled, only top-level and measured spans will have stats computed.
compute_stats_by_span_kind: true

This is a real, documented Agent-level control over which spans feed the stats computation this lesson describes — compute_stats_by_span_kind (default true) widens stats collection to any span with an eligible span.kind tag, not just top-level/measured spans. A related key in the same block, peer_tags_aggregation, is documented as controlling whether “aggregated trace stats” include peer tags as dimensions — the Agent’s own source literally calls this data trace metrics/trace stats, not a term invented for this course. agent configcheck accepts both keys without error against a real, freshly-pulled datadog/agent:7 container. (This session’s agent status APM section could not confirm live-computed stats numbers — the sandboxed container’s trace-agent process wasn’t reachable within a short-lived, offline-pointed run, the same limitation noted in the previous lesson; the config-acceptance proof above is what this lesson can honestly claim.)

Span-based custom metrics: SpansMetricCreateRequest, validated

Beyond the automatic hit/error/latency trace metrics, Datadog lets you define custom metrics computed from spans matching a query — validated against the real, installed datadog_api_client==2.61.0 v2 model (name matches the spec exactly, unlike the retention-filter model in the previous lesson):

$ python -c "
from datadog_api_client.v2.model.spans_metric_create_request import SpansMetricCreateRequest
payload = {'data': {'type': 'spans_metrics', 'id': 'checkout.apply_discount.duration', 'attributes': {
    'compute': {'aggregation_type': 'distribution', 'path': '@duration', 'include_percentiles': True},
    'filter': {'query': 'service:checkout-web resource_name:checkout.apply_discount'},
    'group_by': [{'path': '@http.status_code', 'tag_name': 'status_code'}],
}}}
print(SpansMetricCreateRequest(**payload))
"
validated OK

Real signature introspection (inspect.signature) confirms the shape: aggregation_type is one of exactly two real enum values, {'distribution', 'count'} — include_percentiles is only meaningful (and only accepted) on distribution, which is precisely why it lives on compute, not on the top-level attributes. Requesting a metric with no compute block at all fails immediately with a real, specific error:

# @expect-error SpansMetricCreateAttributes.__init__() missing 1 required positional argument: 'compute'
from datadog_api_client.v2.model.spans_metric_create_request import SpansMetricCreateRequest

payload = {"data": {"type": "spans_metrics", "id": "broken-metric", "attributes": {}}}
SpansMetricCreateRequest(**payload)

Reading AI-generated code

🐛 Spot the bug

app/latency_check.py

# AI comment: "Compare p99 latency across our two regions"
from ddsketch import DDSketch

us_sketch = DDSketch(relative_accuracy=0.01)
eu_sketch = DDSketch(relative_accuracy=0.01)
for v in us_latencies:
  us_sketch.add(v)
for v in eu_latencies:
  eu_sketch.add(v)

combined_p99 = (us_sketch.get_quantile_value(0.99) + eu_sketch.get_quantile_value(0.99)) / 2

Show the answer

This runs without error and returns a plausible-looking number, which is exactly what makes it dangerous. Averaging two already-computed p99 values is not the same as the true combined p99 across both regions’ traffic — this is the exact naive-average trap this lesson opened with, and DDSketch exists specifically so you never have to do it. The fix, proven above to give an accurate combined percentile: merge the sketches themselves first (us_sketch.merge(eu_sketch)), then call get_quantile_value(0.99) once on the merged result — proven above to stay within the configured relative_accuracy of the true percentile, which an average of two separate p99s has no such guarantee for.

Mental Model

A DDSketch is not a sample of the data — it is a compact summary that behaves, for percentile purposes, as if it still held every raw observation. That’s the property proven above: sorting 5,000 real numbers and reading off the 99th value gave nearly the same answer as asking the sketch, which never stored those 5,000 numbers at all, only a bucketed summary of them. Merging two sketches is not averaging two summaries — it’s closer to combining the underlying data first, then summarizing once — which is exactly why the merged sketch’s percentile stays accurate while an average of two separately-computed percentiles does not.

Trade-offs

• Trace metrics (always computed) vs. example traces (sampling-dependent) — proven across this lesson and the previous one: dashboards built on trace metrics stay trustworthy at any sampling rate, but debugging one specific slow request still requires that exact trace to have survived both ingestion sampling and a retention filter — you cannot get both “cheap ingestion” and “guaranteed example-trace access” for free.
• DDSketch’s relative_accuracy setting — a tighter accuracy (proven above at 0.01, all three percentiles landed well under 1% relative error) costs more memory per sketch; a looser setting is cheaper but the same merge-and-percentile guarantee holds at whatever accuracy you configure — there is no free lunch, only a tunable trade between precision and footprint.
• Automatic trace metrics vs. custom SpansMetricCreateRequest metrics — the automatic hit/error/latency metrics need zero configuration and cover every service by default; a custom spans-based metric (proven above, real model) requires a query and a compute block you maintain, but lets you measure something the automatic metrics don’t capture, like a specific resource’s tagged latency distribution.

Common Misconceptions

• “You can average two percentiles from two hosts and get the correct combined percentile.” No — proven above with the SpotTheBug example: this is the exact trap DDSketch exists to avoid; only merging the underlying sketches first and then computing the percentile once gives an accurate combined answer.
• “Trace metrics only reflect the sampled subset of traces that made it into Datadog.” No — proven above via the real Agent config: stats computation (compute_stats_by_span_kind and friends) runs off the span stream reaching the Agent, independent of the ingestion-sampling and retention-filter layers covered in the previous lesson.
• “DDSketch gives exact percentiles, not estimates.” No — proven above: the sketch’s estimate differed from the exact, sorted-data percentile by a small but nonzero amount (0.28%–0.59% in this lesson’s own run) — it is a bounded-error estimate, accurate within the configured relative_accuracy, not an exact computation.

What are the three things Datadog continuously computes as trace metrics for a service/resource?

CPU usage, memory usage, disk usage Hit count, error count, and latency Number of manual spans, number of auto-instrumented spans, and trace_rate Ingestion rate, retention rate, and sample rate

Why is DDSketch used for latency percentiles instead of simply averaging percentiles from different hosts?

Averaging percentiles from different hosts produces an incorrect combined percentile; DDSketch sketches are designed to merge accurately DDSketch is only needed for error counts, not latency Percentiles do not need aggregation across hosts DDSketch replaces the need for hit counts

Which of these is unaffected by ingestion-time sampling and retention filters?

Whether a specific example trace can be opened and inspected The p95 latency shown on a service overview page Whether a trace was retained long-term Whether a trace was ingested at all

A team runs an aggressive 1% ingestion sampling rate. What can they still trust?

Nothing — all dashboards become unreliable below 100% ingestion Only the Service Map, not latency The aggregate hit count, error count, and latency percentiles, since those are computed from the span stream independent of sampling Only traces matched by a retention filter

Real test: a DDSketch with relative_accuracy=0.01 summarized 5,000 real Gaussian samples. How did its p99 estimate compare to the exact percentile computed by sorting the raw data?

They were identical to the decimal The sketch estimate differed by 0.37% — within the configured 1% relative-accuracy bound, but not an exact match The sketch was off by over 50%, showing DDSketch is unreliable DDSketch cannot compute p99, only p50

What does the SpansMetricCreateRequest v2 model actually require, proven by a real broken payload?

Nothing — an empty attributes object validates fine A compute block — omitting it raises a real TypeError: "missing 1 required positional argument: 'compute'" A group_by list is always mandatory Only a filter query is required

Submit

Mark as complete