How Datadog Works Internally

What this module is for

The earlier modules taught you to configure Datadog: install the Agent, send a custom metric, add a sampling rule, write a log pipeline. Each of those has a moment where it silently does something other than what the config seems to say — a dd_url that moves one of four writers, a histogram whose p99 cannot be averaged, an max_traces_per_second: 0 that ships nothing, a status remapper that turns SEVERE into OK, an OTLP config that configcheck accepts while the receiver is dead. This module opens the box for five subsystems and proves each mechanism from the outside, with the real tool, so that when you review a config someone (or an AI) wrote you can tell which knob is real and where it acts.

1. What is the Agent, physically, and where do the bytes go? (Agent Architecture — the supervised process tree, the check scheduler grid, the aggregator and the forwarder retry queue, agent status section by section)
2. Where is a custom metric aggregated, and why can’t you average a percentile? (DogStatsD and Aggregation — client, Agent and backend layers, decoded series and sketches payloads, DDSketch merge)
3. Which gate dropped my trace? (The Trace Sampling Pipeline — head sampler, rate limiter, the Agent’s rate_by_service feedback and its few enforcers, ingestion controls, retention)
4. What does a log pipeline really do to a line, and in what order? (Log Pipeline Processors — Agent-side rules run for real, Grok and remapper semantics from the docs, strict model validation of pipeline JSON)
5. How does OpenTelemetry data become Datadog data? (OTLP and Open Standards — the Agent receiver, the measured semantic mapping, the Collector paths validated with the real otelcol-contrib)

flowchart TD
  A["Agent Architecture: scheduler, aggregator, forwarder"] --> D["DogStatsD and Aggregation: what the aggregator flushes"]
  A --> T["Trace Sampling Pipeline: the Agent's samplers and feedback"]
  A --> L["Log Pipeline Processors: Agent-side rules, then backend"]
  A --> O["OTLP and Open Standards: the Agent as receiver"]
  D --> O
  T --> O

The five lessons and how they connect

How every proof in this module was made

• Real Agent. datadog/agent:7, resolved to Agent 7.83.3 (Go 1.26.7), started with a fake key and every intake redirected to a dead local port or to a small local fake intake that decompresses and saves each request, so nothing is sent to Datadog. Two of my early runs did contact real Datadog intakes with the fake key (the process endpoint, and once the database-monitoring event intake), both answered 403; those runs are identified in the lessons and the outputs shown come from later isolated runs.
• Decoded payloads. The series, sketch and trace bodies the Agent flushes are protobuf. A small hand-written wire decoder using the field numbers of Datadog’s public agent_payload.proto turned them into the values shown.
• Tracers. dd-trace 6.17.0 (tools/probe/node) and ddtrace 4.15.2 (tools/probe/venv) from the course harness, with a local fake trace-agent that prints each root span’s sampling tags and can reply with a chosen rate_by_service.
• Models. datadog-api-client 2.61.0 for pipeline JSON, plus a stricter per-processor check written for the log lesson.
• Collector. The locally present otel/opentelemetry-collector-contrib image, otelcol-contrib 0.161.0, with --network none.
• Docs. Anything that only exists in Datadog’s hosted backend (Grok, remappers, ingestion controls, retention filters, intelligent retention) is quoted from docs.datadoghq.com with the URL, fetched 2026-09-30. There is no local runtime for those and the lessons do not fake one.
• Timings and counts are from one Mac (Docker Desktop, linuxkit VM), one run each. They are evidence of a mechanism, not benchmarks.

🐶 One thing the module keeps proving

agent configcheck checks the checks. It exited 0 for a datadog.yaml whose OTLP receiver was dead, and it says nothing about dd_url, sampling or pipelines. When a lesson says a config “validates”, it says which tool ran and what that tool can and cannot see.

Reading order and what each lesson assumes

Lesson	Builds on (Phase 3)	Answers
Agent Architecture	The Datadog Agent and Integrations	What runs, what does the scheduler do, what does the forwarder do when the intake is down?
DogStatsD and Aggregation	Custom Metrics with DogStatsD	What leaves the client, what the Agent flushes, and why DISTRIBUTION uses a sketch
The Trace Sampling Pipeline	Ingestion Sampling vs. Retention Filters, Trace Metrics and DDSketch	Which gate decides, what tag does each leave, which gates enforce
Log Pipeline Processors	Log Collection and Processing Pipelines	What runs where, which duplicate wins, what the JSON model does not catch
OTLP and Open Standards	What Is Datadog	What the Agent and the Collector do to OTLP data

Mental Model

Every lesson applies one idea to a different subsystem: Datadog is a set of small pipelines with fixed stages, and each stage leaves evidence you can read. The Agent’s stages show up as agent status sections; a metric’s aggregation shows up as interval: 10 and type: rate in a payload; a sampling decision shows up as _sampling_priority_v1, _dd.p.dm and _dd.limit_psr on the span; an Agent-side log rule shows up in analyze-logs; an OTLP translation shows up as http.method and type=web. Where the hosted backend hides the evidence, the docs are the only source and the lesson says so.