DogStatsD and Aggregation

The idea in one sentence

A custom metric is aggregated up to three times between your statsd.increment() call and a graph — optionally in the client library, always in the Agent’s DogStatsD server in 10-second buckets, and for DISTRIBUTION a third time in the backend by merging DDSketch structures — and which layer does the math decides whether a percentile is trustworthy.

🐶 How every proof in this lesson was made

Same datadog/agent:7 container as Agent Architecture (Agent 7.83.3), but with DD_DD_URL pointed at a small local fake intake (a 30-line Python HTTP server on host.docker.internal:18999 that decompresses the zstd body and saves it) so the exact payload the Agent flushes can be read — nothing is ever sent to Datadog, DD_API_KEY is a fake. Packets came from nc -u and from the datadog Python package 0.54.0; the sketch demo used ddsketch 3.0.1. DD_DOGSTATSD_METRICS_STATS_ENABLE=true turns on agent dogstatsd-stats. The intake’s series/sketch bodies are protobuf; I decoded them with a small hand-written wire decoder using the field numbers of Datadog’s public agent_payload.proto. Timings are from this Mac. The Phase 3 lesson Custom Metrics with DogStatsD covers the datagram format, the metric types and the cardinality trap; this lesson is about what happens after the datagram arrives.

Terminology (shared with the Phase 3 lesson): Agent-side aggregation happens in the Agent’s DogStatsD server, client-library (client-side) aggregation happens in the app’s client before a datagram is sent, and server-side means the Datadog backend.

Layer 1: the client library (optional)

The application-side library can buffer and aggregate before touching the socket. The docs for it (Dogstatsd high throughput): “Client libraries can aggregate metrics on the client side, which reduces the number of messages submitted to the Agent,” and “Client-side aggregation is available for gauges, counters and sets.” Here is the effect measured on the wire: 1,000 increment calls and 1,000 gauge calls (values 0..999) from datadog 0.54.0, received by a plain UDP socket that counts datagrams.

no aggregation, no buffering : datagrams=2000 bytes=72890
   first datagram: b'internals.page.views:1|c|#route:home\n'
no aggregation, buffering    : datagrams=52   bytes=72890
   first datagram: b'internals.page.views:1|c|#route:home\ninternals.page.views:1|c|#route:home\n...'
aggregation + buffering      : datagrams=1    bytes=76
   first datagram: b'internals.page.views:1000|c|#route:home\ninternals.queue.depth:999|g|#q:jobs\n'

Three modes, same 2,000 calls. Buffering packs many lines into one datagram (2000 to 52) but sends every line. Aggregation collapses the calls first: one counter line with the sum (1000|c) and one gauge line with the last value (999|g) — 76 bytes total. Note datadog 0.54.0’s DogStatsd(disable_aggregation=...) defaults to True (I read the signature), so a stock Python client does not aggregate; you must opt in. Aggregation is also type-selective. Repeating the run with distribution, histogram and set calls (1,000 each, aggregation on):

50 datagrams  Counter({'internals.latency.dist': 1000, 'internals.latency': 1000, 'internals.users': 7})

The 1,000 set samples (values i % 7) became 7 lines — one per unique member — but the 1,000 distribution and 1,000 histogram samples went out untouched, exactly the “gauges, counters and sets” the docs list. A DISTRIBUTION cannot be pre-summed without losing the shape that percentiles need.

Layer 2: the Agent’s DogStatsD server

Packets sent with nc -u -w0 127.0.0.1 18125: 5 counter increments, 2 gauges, 3 histogram values, 4 distribution values, 3 set members, a @0.1 sampled counter and a timer. I sent that set twice — the first time (02:17:03) to read the receive-side view, agent dogstatsd-stats (only the lesson’s metrics shown), the second (02:18:59) to capture the flushed payloads on a fresh fake intake:

Metric                 | Tags        | Count | Last Seen
-----------------------|-------------|-------|-----------------
internals.page.views   | route:home  | 5     | 2026-09-30 02:17:03 ...
internals.latency.dist | svc:api     | 4     | ...
internals.latency      | svc:api     | 3     | ...
internals.users        |             | 3     | ...
internals.queue.depth  | q:jobs      | 2     | ...
internals.timer        | svc:api     | 1     | ...
internals.sampled      |             | 1     | ...

Count is samples received per name+tags, before any aggregation: page.views is 5 because five lines arrived. The sampled counter is 1, not 10 — the @0.1 is applied later. (In this container the Agent’s own Trace Agent also sends its datadog.trace_agent.* telemetry to DogStatsD, so the unfiltered command is noisy; filter on your prefix.) The first send also included one malformed line, internals.bad:notanumber|c. A malformed line is counted and logged, not silently dropped: Metric Parse Errors: 1 in agent status and, in agent.log:

CORE | ERROR | (comp/dogstatsd/server/impl/server.go:693 in errLog) | Dogstatsd: error parsing metric message '"internals.bad:notanumber|c"': could not parse dogstatsd metric values: strconv.ParseFloat: parsing "notanumber": invalid syntax

Now the send-side view — the payload the Agent actually flushed to the fake intake, decoded (/api/v2/series, application/x-protobuf, zstd, interval: 10 on every series):

internals.page.views          rate   0.5   tags [route:home]     # 5 increments / 10 s
internals.sampled             rate   1.0                         # 1 sample at @0.1 = 10 events / 10 s
internals.queue.depth         gauge  42.0  tags [q:jobs]         # 10 then 42: last value wins
internals.users               gauge  2.0                         # a, b, a: unique members
internals.timer.count         rate   0.1   internals.timer.avg/median/max/95percentile  gauge  250.0
internals.latency.count       rate   0.3   .avg 20.0  .median 20.0  .max 30.0  .95percentile 30.0

Everything the Phase 3 lesson said about metric types is visible here as a wire fact. A counter becomes a rate (count divided by the 10-second bucket: 5 / 10 = 0.5), which is why a counter’s stored unit is per-second; the @0.1 sample rate is undone by the Agent (1 sample / 0.1 / 10 s = 1.0). A gauge keeps the last value; a set collapses to a unique count; a histogram (and |ms timer) fans out into five series. The first decoded series carry timestamps on 10-second boundaries (1790734730), and the payload reached the intake about 10 seconds after the packets were sent — the bucket has to close before it can be flushed, and the aggregator’s own flush tick (about every 15 seconds, see Agent Architecture) adds up to that much again. That is the honest floor on “real-time” for a custom metric.

Two settings from agent config --all frame this layer: dogstatsd_context_expiry_seconds: 20 and dogstatsd_expiry_seconds: 300. The first one is visible in the captured payloads too: after the burst, the intake kept receiving

internals.page.views  rate  points [(0.0, 1790734740), (0.0, 1790734750)]
internals.page.views  rate  points [(0.0, 1790734760)]

— zeros for a counter that stopped — for roughly the 20-second context expiry, then nothing. The Agent keeps reporting 0 for a known counter context until it expires so that graphs show a drop to zero instead of a gap. The batching settings are dogstatsd_buffer_size: 8192, dogstatsd_queue_size: 1024 and dogstatsd_packet_buffer_flush_timeout: 100ms, and dogstatsd_no_aggregation_pipeline: true (default) is the escape hatch for metrics that carry their own timestamp — they skip the bucket and go through as-is.

Layer 3: DISTRIBUTION and the sketch

The same fake intake also received a different endpoint for the distribution: /api/beta/sketches, not /api/v2/series. Decoded, that payload for four values 10, 20, 30, 1000:

metric internals.latency.dist  tags [svc:api]
  ts 1790734730  cnt 4  min 10.0  max 1000.0  avg 265.0  sum 1060.0
  k = [1487, 1531, 1557, 1784]   n = [1, 1, 1, 1]

There is no .p95 series and no raw values. The Agent sent a DDSketch: a count, exact min/max/sum, and a sparse list of bucket keys k with counts n. This is what makes a percentile computable later, across every host and any tag grouping, on the backend. The docs say it directly (Distributions): “all raw data collected during the flush interval and the aggregation occurs server-side using Datadog’s DDSketch data structure,” giving “globally accurate percentile aggregations (p50, p75, p90, p95, p99 or any percentile of your choosing with up to two decimal points).”

How the keys work is in the paper, Masson, Rim and Lee, “DDSketch: A Fast and Fully-Mergeable Quantile Sketch with Relative-Error Guarantees”, PVLDB 12(12): 2195-2205, 2019 (doi:10.14778/3352063.3352135). Its abstract: “rank error guarantees do not preclude arbitrarily large relative errors, and this often occurs in practice when the data is heavily skewed,” and “several combined sketches must be as accurate as a single sketch of the same data. We present the first fully-mergeable, relative-error quantile sketching algorithm with formal guarantees.” Definition 1: “x̃q is an α-accurate q-quantile if |x̃q − xq| ≤ α·xq.” The construction: “Let γ := (1+α)/(1−α)” and a value x is assigned to “the bucket indexed by ⌈log_γ(x)⌉,” so bucket i covers γ^(i-1) < x ≤ γ^i and any value in it is reported as 2γ^i/(γ+1), within α of every value it could be.

The captured keys are consistent with that. Spacing between 10 and 20 is 44 keys; ln(2)/ln(γ) with α = 1/128 (γ = 1.01575) is 44.4. 20 to 30 is 26 keys against 25.9. 30 to 1000 is 227 against 224.6 — two keys off, because the Agent uses a fast approximate logarithm rather than an exact ln, and its key offset differs from ddsketch’s; I did not reproduce the exact Agent mapping, and α = 1/128 is my inference from the spacing, not a documented value. The sketch has a bounded number of buckets (binLimit collapses the lowest ones, per the paper), so its size depends on the data’s dynamic range, not on how many samples were added.

The reason all this machinery exists, run with the reference implementation (ddsketch 3.0.1, α = 1/128) on three simulated hosts of 20,000 latencies each:

true p99 of all 60000 values : 1651.166
average of per-host p99      : 889.898  rel.err 46.1%
merged DDSketch p99          : 1659.353  rel.err 0.496%  (alpha=0.7812%)
worst rel.err over p1..p99.9 : 0.563%
bins in merged sketch: 640  count 60000.0

Averaging the three hosts’ p99 values is off by 46%. Merging the three sketches (sketch.merge(other), which adds bucket counts) is off by 0.5%, inside the guarantee, at every quantile checked. That is what “fully mergeable” buys: the backend can merge sketches from thousands of Agents and any grouping and still answer p99:internals.latency.dist{*}.

flowchart LR
  APP["App code: statsd.increment / distribution"] --> CL["Client library (optional aggregation and buffering)"]
  CL -->|"UDP or UDS datagrams"| SRV["Agent DogStatsD server (10 s buckets)"]
  SRV -->|"counters, gauges, sets, histogram stats: /api/v2/series"| FW["Forwarder"]
  SRV -->|"distribution: DDSketch: /api/beta/sketches"| FW
  FW --> BE["Backend"]
  BE -->|"sketch merge: any tag grouping"| Q["p99 across all hosts"]

Where a custom metric is aggregated

Mental Model

Every aggregator in the chain answers the same question — what may I throw away? — with a different answer. The client library may throw away repeats of the same counter/gauge/set value. The Agent may throw away everything except the bucket summary: sum/10 s for counters, last value for gauges, five derived stats for histograms. Only the sketch keeps enough shape (bucket keys with counts) that the backend can still compute any percentile over any grouping later. Once you have discarded the shape, you cannot get it back: a per-host p99 gauge can never be turned into a fleet p99.

Trade-offs

Choice	Benefit	Cost
Client-side aggregation on	2000 calls become 1 datagram of 76 bytes; far less UDP and syscall pressure	Opt-in in some clients; the flush interval adds latency; counters/gauges/sets only
Client-side buffering only	52 datagrams instead of 2000 with no change in semantics	Every sample still crosses the socket; a lost datagram loses many lines
HISTOGRAM (h)	Cheap: five series per context regardless of sample count	Percentiles exist per host only; averaging them across hosts is wrong (46% off in the demo)
DISTRIBUTION (d)	Fleet-wide p99 within the sketch’s relative error, any tag grouping, chosen after the fact	A sketch payload per context, heavier than a gauge; the Phase 3 lesson covers how that interacts with cardinality and custom-metric volume
Sample rate @0.1	10x fewer packets	Noisier estimate; the Agent scales the count back up, but a rare event can vanish
dogstatsd_no_aggregation_pipeline (timestamped metrics)	You control the timestamp	Bypasses the bucket, so rate semantics differ; use only when you mean it

Common Misconceptions

• “dogstatsd-stats shows what was sent to Datadog.” It shows samples received per name+tags. Proven above: page.views = 5 received, but the Agent flushed one series point 0.5.
• “A counter’s value is what I incremented.” On the wire it is a rate: 5 increments in a 10-second bucket is stored as 0.5, and @0.1 is scaled back up (1.0).
• “The Agent sends a value the instant I call gauge().” Proven above: the bucket must close and the aggregator must flush; expect about 10 to 20 seconds, on this Mac.
• “Averaging p99 across hosts gives the fleet p99.” Measured: 46.1% relative error, versus 0.496% for a merged sketch.
• “Client-side aggregation is on by default.” In datadog 0.54.0 disable_aggregation defaults to True. Check your client.
• “A DISTRIBUTION is stored as raw values.” The payload above is a count, min/max/sum and bucket keys — a sketch. There are no raw points to replay.

Reading AI-generated code

🐛 Spot the bug

app/checkout_metrics.py

from datadog import DogStatsd

statsd = DogStatsd(host="127.0.0.1", port=8125)

def record_checkout(duration_ms: float, region: str) -> None:
  # histogram() gives us p50/p95/p99 across the whole fleet,
  # so we can alert on the global p99 in every region.
  statsd.histogram("checkout.duration", duration_ms, tags=[f"region:{region}"])

Show the answer

histogram() is aggregated by each host’s Agent. The intake capture above shows what it produces: internals.latency.count, .avg, .median, .max and .95percentile as per-host gauges, computed only from the samples that one Agent saw. Nothing on the backend can merge them, and the simulation shows why that matters: averaging three hosts’ p99 was 46.1% off the true fleet p99. A global-p99 alert built on this is measuring “the mean of per-host p99s”, not the p99 of user experience. Use distribution(), which ships a DDSketch (/api/beta/sketches) that the backend merges (0.496% error in the same simulation). The custom-metric lesson in Phase 3 is the place for the cost side of that choice.

app/checkout_metrics.py

from datadog import DogStatsd

statsd = DogStatsd(host="127.0.0.1", port=8125)

def record_checkout(duration_ms: float, region: str) -> None:
    statsd.distribution("checkout.duration", duration_ms, tags=[f"region:{region}"])

A Python client with disable_aggregation=False and buffering on sent 1,000 increments and 1,000 gauge values (0..999) in the real capture. What arrived at the UDP socket?

2000 datagrams, one per call 52 datagrams containing every line One 76-byte datagram: a counter line with the sum (1000|c) and a gauge line with the last value (999|g) Nothing -- the client waits for the Agent to poll it

In the same capture, 1,000 distribution() and 1,000 histogram() calls with aggregation enabled produced how many lines?

1 each, summed 1,000 each, untouched -- client-side aggregation covers only counters, gauges and sets 0, they are rejected 7 each

agent dogstatsd-stats showed internals.page.views Count 5 after five increments, while the decoded /api/v2/series payload carried one rate point of 0.5. Why?

A bug that halves counters dogstatsd-stats counts samples received; the Agent then aggregates a 10-second bucket and flushes it as a rate (5 / 10 s = 0.5) The Agent lost half the packets Rates are always half the count

Which of these is true of the decoded /api/beta/sketches payload for the values 10, 20, 30 and 1000?

It contains the four raw values It contains p50/p95/p99 already computed on the host It contains count, min, max, sum and sparse bucket keys with counts, from which the backend can compute any percentile after merging It is identical to the histogram payload

In the simulation of three hosts (60,000 values), averaging each host's p99 versus merging their DDSketches gave what relative error against the true p99?

46.1% for the average, 0.496% for the merged sketch 0.5% for both 46.1% for the merged sketch, 0.496% for the average Both were exact

After the counter burst, the fake intake kept receiving internals.page.views points with value 0.0 for about 20 seconds. Which setting explains this?

dogstatsd_buffer_size dogstatsd_context_expiry_seconds (20): the Agent keeps reporting a known counter context as 0 until the context expires forwarder_retry_queue_payloads_max_size check_runners

Submit

Mark as complete