Custom Metrics with DogStatsD

The idea in one sentence

DogStatsD is a small daemon bundled inside the Agent that lets your application code submit custom metrics by firing off a lightweight datagram over UDP or a Unix domain socket, without writing a check at all — this lesson sends real datagrams at a real Agent and reads its real internal counters to prove exactly how they’re received and aggregated.

Why DogStatsD exists

Checks (from the previous lesson) are great for infrastructure and known integrations, but they don’t help when you want to measure something specific to your own application code — “how many checkout attempts happened,” “how long did this function take,” “how many unique users hit this endpoint.” Writing a full Agent check for that would be overkill. DogStatsD solves this by listening for tiny datagrams that any process on the host can fire at it — over UDP (8125 by default) or, proven below, a Unix domain socket. The Agent picks those datagrams up, aggregates them, and forwards them upstream — your application never talks to Datadog’s backend directly.

The datagram format

A DogStatsD submission is a single line of text sent over UDP or UDS:

<METRIC_NAME>:<VALUE>|<TYPE>|@<SAMPLE_RATE>|#<TAG_KEY_1>:<TAG_VALUE_1>,<TAG_2>

For example, submitting a counter for checkout attempts, tagged by environment and payment provider:

checkout.attempts:1|c|@1|#env:prod,provider:stripe

checkout.attempts — the metric name
1 — the value being submitted
c — the type (count, in this case)
@1 — the sample rate (here, 1 means “no sampling,” i.e. every event is sent)
#env:prod,provider:stripe — comma-separated tags

Metric types

DogStatsD supports a fixed set of type codes:

COUNT (c) — increments a counter by the given value; use for “how many times did X happen.”
GAUGE (g) — sets the metric to an exact value at this instant; use for “what is the current value of X.”
HISTOGRAM (h) — records the statistical distribution of a value aggregated Agent-side, per host (by the Agent’s DogStatsD server).
DISTRIBUTION (d) — records the statistical distribution of a value, aggregated globally, server-side (the Datadog backend merges sketches).
SET (s) — counts the number of unique values submitted for a metric, e.g. unique user IDs.
TIMER (ms) — an alias for HISTOGRAM, conventionally used for timing measurements in milliseconds; it behaves exactly like h under the hood.

Proof: real HISTOGRAM and DISTRIBUTION datagrams, read back with `agent dogstatsd-stats`

Rather than a client library (this lesson’s harness-pinned Python venv only carries ddtrace and datadog_api_client, not datadog/hot-shots — proving the raw wire protocol directly is a strictly stronger proof anyway, since it shows exactly what a client library would generate under the hood), this lesson sends the documented datagram format straight over UDP with Python’s built-in socket module, at a real datadog/agent:7 container with its DogStatsD UDP port published to the host and dogstatsd_metrics_stats_enable turned on so agent dogstatsd-stats has something to show:

$ docker run -d --name dd-dsd \
    -e DD_API_KEY=<fake> -e DD_DD_URL=http://127.0.0.1:1 \
    -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
    -e DD_DOGSTATSD_METRICS_STATS_ENABLE=true \
    -p 18125:8125/udp \
    datadog/agent:7

# 20 HISTOGRAM + 20 DISTRIBUTION samples, one Python socket.sendto() loop
$ python3 -c "
import socket, time
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
addr = ('127.0.0.1', 18125)
for i in range(20):
    sock.sendto(f'lesson.histogram.duration:{10+i}|h|#env:lesson'.encode(), addr)
    sock.sendto(f'lesson.distribution.duration:{10+i}|d|#env:lesson'.encode(), addr)
    time.sleep(0.05)
"

$ docker exec dd-dsd agent dogstatsd-stats
lesson.distribution.duration | env:lesson | 20 | 2026-09-29 14:02:17...
lesson.histogram.duration    | env:lesson | 20 | 2026-09-29 14:02:17...

Both types were received and counted identically at the ingestion layer — 20 raw datagrams each, real proof the wire protocol and reception path treat h and d the same way going in. Where they genuinely diverge is what the Agent does with those raw values next, and that part connects directly back to the previous lesson’s check-API proof: a self.histogram(...) call in a check produced five real derived metrics (.avg, .max, .median, .95percentile, .count) computed Agent-side, on that one Agent — DogStatsD’s HISTOGRAM type runs through the exact same Agent-side aggregation code path, so the same five-series fan-out and the same “only accurate for this one host” limitation apply. DISTRIBUTION, by contrast, ships the raw values upstream for aggregation across every host and container submitting that metric name — a genuinely backend-side capability this lesson cannot observe locally without a real Datadog org (no account exists here, by design), which is exactly why the official guidance is to prefer DISTRIBUTION whenever an accurate global percentile (fleet-wide p99 latency, for example) is what you actually need.

The cardinality trap, proven with an exploding metric count

The previous foundations-module lesson covered cardinality in the abstract; here it is a real, measured explosion. Thirty COUNT submissions of the same metric name, each with a genuinely unique request_id tag value (the textbook AI-generated-code mistake — “tag every request with its ID for traceability”):

$ python3 -c "
import socket, uuid
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
addr = ('127.0.0.1', 18125)
for i in range(30):
    uid = uuid.uuid4().hex[:8]
    sock.sendto(f'lesson.requests.count:1|c|#request_id:{uid}'.encode(), addr)
"

$ docker exec dd-dsd agent dogstatsd-stats | grep lesson.requests.count | wc -l
30
$ docker exec dd-dsd agent dogstatsd-stats | grep lesson.requests.count | head -3
lesson.requests.count | request_id:ef2343e3 | 1 | 2026-09-29 14:02:42...
lesson.requests.count | request_id:056a91e3 | 1 | 2026-09-29 14:02:42...
lesson.requests.count | request_id:f02cdb4b | 1 | 2026-09-29 14:02:42...

One metric name, thirty submissions, thirty distinct rows in dogstatsd-stats — because dogstatsd-stats tracks unique metric-name-plus-tag-combination contexts, exactly the same unit Datadog bills custom metrics against. In a real production system firing this pattern continuously, every single request creates a permanent, never-reused time series — this is the local, hands-on version of the “custom-metrics bill explosion” the Metrics without Limits lesson covers next.

UDP vs. Unix domain socket (UDS)

DogStatsD listens on both transports simultaneously by default — real proof, the same container’s dsd.socket file exists out of the box, no extra config:

$ docker exec dd-dsd ls -la /var/run/datadog/
srwx-w--w- 1 root root 0 ... apm.socket
srwx-w--w- 1 root root 0 ... dsd.socket

Sending 5 more datagrams over the Unix socket specifically (from inside the container, AF_UNIX/SOCK_DGRAM connected to /var/run/datadog/dsd.socket) and re-checking agent status shows the Agent tracking the two transports as genuinely separate counters:

$ docker exec dd-dsd agent status | grep -A2 -B2 "Udp Packets\|Uds Packets"
  Udp Packets: 74      # unchanged -- these 5 went over UDS, not UDP
  Uds Bytes: 140
  Uds Packets: 6        # was 1, now 6 -- exactly the 5 new UDS datagrams

UDP is simpler (any language’s plain socket API works, no filesystem path to mount/share) but connectionless and unauthenticated — anything that can reach the port can submit metrics, including from a different container or namespace if dogstatsd_non_local_traffic is on, as this lesson’s own setup deliberately enabled to reach the container from the host at all. A Unix domain socket is filesystem-scoped (only processes that can see the same mount can write to it) and is what makes origin detection possible at all: per the real config template in this Agent’s own version, dogstatsd_origin_detection’s doc comment is explicit that it only works “When using Unix Socket” — DogStatsD can tag metrics with container metadata (image, container ID) automatically, but that enrichment has no UDP equivalent, because a UDP packet carries no reliable sender-identity information the Agent could use to look up which container it came from.

flowchart LR
  A[App process] -->|UDP or UDS datagram| B[DogStatsD in Agent]
  B -->|HISTOGRAM: aggregate per host| C[.avg/.max/.median/.95percentile as GAUGE, .count as RATE]
  B -->|DISTRIBUTION: send raw values| D[Datadog backend]
  D -->|aggregate globally| E[Accurate percentiles across all hosts]
  C --> F[Datadog backend]

DogStatsD path vs. HISTOGRAM/DISTRIBUTION aggregation

Reading AI-generated code

🐛 Spot the bug

# AI comment: "Track checkout latency per request for detailed debugging"
import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

def track_checkout(request_id: str, duration_ms: float):
  tag = f"request_id:{request_id}"
  sock.sendto(f"checkout.duration:{duration_ms}|h|#{tag}".encode(), ("127.0.0.1", 8125))

Show the answer

This looks like reasonable, well-intentioned instrumentation — a real datagram format, a real HISTOGRAM type code, a tag that seems useful for “detailed debugging.” The bug, proven above with a real Agent and 30 real submissions: tagging a metric by request_id means every single call creates a brand-new, permanent time-series context — proven above, dogstatsd-stats showed exactly one row per unique tag value, with no reuse. This is precisely the custom-metrics cardinality explosion the next lesson (Metrics without Limits) exists to let you recover from after the fact, but the right fix here is not to submit the high-cardinality tag as a metric tag at all — put request_id in a log line or a trace span attribute (searchable, not billed per unique value the way indexed metric tags are), and keep the metric tagged only by things worth graphing and grouping by, like route or provider.

Mental Model

Think of DogStatsD as a mail slot with two openings — UDP is the outdoor slot anyone walking by can post through (fast, no ID check, and proven above needs dogstatsd_non_local_traffic explicitly opened to accept mail from outside the building at all); the Unix socket is the indoor slot, reachable only by people already inside the same building (filesystem-scoped), and because the building’s own staff can vouch for who’s dropping mail through it, the Agent can attach a “delivered by container X” sticky note (origin detection) — something the outdoor slot structurally cannot do. Once a value lands in either slot, HISTOGRAM is a clerk who does the math themselves, right there at that one desk (Agent-side, one host); DISTRIBUTION is a clerk who bundles the raw values and forwards them to head office, where math happens once with everyone’s numbers combined (server-side, global).

Trade-offs

UDP vs. Unix domain socket — UDP needs no shared filesystem path and works from any container/network namespace that can reach the port (at the cost of enabling dogstatsd_non_local_traffic, proven above, which trades some isolation for reachability); UDS is filesystem-scoped and enables origin detection, but only works when the sender and the Agent share a mount (same Pod, typically, in Kubernetes).
HISTOGRAM vs. DISTRIBUTION — proven above, both are received identically at the wire level; HISTOGRAM is cheaper (aggregated down to 5 series by the Agent before it ever leaves the host) but per-host-only accurate; DISTRIBUTION is more expensive (raw values shipped upstream) but gives a real, accurate global percentile — the right choice depends entirely on whether “accurate across the whole fleet” is a requirement or a nice-to-have.
Tagging liberally vs. tagging for cardinality — proven above, a single high-cardinality tag choice (request_id) turns one metric name into an unbounded number of billed, permanent time series; the fix covered in the very next lesson (Metrics without Limits) lets you keep submitting liberally and control cost at the indexing layer instead, but that’s a mitigation, not a reason to skip thinking about cardinality at instrumentation time.

Common Misconceptions

“DogStatsD only listens on UDP.” No — proven above: the same container’s dsd.socket Unix socket file exists by default, and real datagrams sent over it were counted by a separate, real Uds Packets counter in agent status.
“HISTOGRAM and DISTRIBUTION are just two names for the same thing, picked by taste.” No — proven above (and in the previous lesson’s check-API proof): HISTOGRAM’s five derived metrics are computed Agent-side, per host; DISTRIBUTION ships raw values for real global aggregation — a structurally different pipeline, not a naming preference.
“Origin detection works over UDP once you set DD_DOGSTATSD_ORIGIN_DETECTION=true.” No — quoted directly from this Agent version’s own config template: origin detection is documented as a Unix-socket-only feature; there is no UDP equivalent because a UDP packet carries no verifiable sender identity.
“A unique ID tag on a custom metric is a minor style choice, not a real cost problem.” No — proven above with 30 real submissions producing 30 permanent, distinct time-series contexts in a real Agent’s own stats: this is a direct, measurable driver of custom-metrics cost, not a hypothetical one.