Agent Architecture

The idea in one sentence

The Agent is a small supervised process tree around one core process whose internals are a straight pipeline — a check scheduler hands work to a fixed pool of runner workers, their samples land in the aggregator, the aggregator flushes to a serializer, and the forwarder owns every HTTP request and every retry — and agent status is a section-by-section dump of exactly those stages.

🐶 How every proof in this lesson was made

Real datadog/agent:7 container, resolved by agent version to Agent 7.83.3 (Go 1.26.7, linux/arm64), started with a fake key and every intake redirected to a dead local port so nothing leaves the machine: DD_API_KEY=<fake-key>, DD_DD_URL=http://127.0.0.1:1, DD_APM_DD_URL=http://127.0.0.1:1, DD_PROCESS_CONFIG_PROCESS_DD_URL=http://127.0.0.1:1, DD_LOGS_CONFIG_LOGS_DD_URL=127.0.0.1:1, DD_REMOTE_CONFIGURATION_ENABLED=false. Container name dd-internals, removed afterwards. Timing numbers are from this Mac (Docker Desktop, linuxkit VM), not a benchmark. The Phase 3 lesson The Datadog Agent and Integrations introduces the five components and conf.d/; this lesson goes one level below it.

The process tree

Phase 3 listed the components. Here is what a real container actually does with them. ps inside dd-internals, trimmed:

  PID COMMAND
    1 s6-svscan -t0 /var/run/s6/services
  424 s6-supervise agent
  426 s6-supervise trace
  427 s6-supervise process
  430 s6-supervise security
  429 s6-supervise sysprobe
  431 s6-supervise data-plane
  432 agent run
  434 /opt/datadog-agent/embedded/bin//trace-agent --config=/etc/datadog-agent/datadog.yaml
  435 /opt/datadog-agent/embedded/bin/system-probe-lite run --socket /opt/datadog-agent/run/sysprobe.sock ...

Asking s6-svstat for each supervised service:

agent: up (pid 432) 57 seconds
trace: up (pid 434) 57 seconds
process: down (exitcode 0) 57 seconds, normally up, ready 57 seconds
security: down (exitcode 0) 46 seconds, normally up, ready 46 seconds
sysprobe: up (pid 435) 57 seconds
data-plane: down (exitcode 0) 51 seconds, normally up, ready 51 seconds

Three things are visible here that a component diagram hides.

1. Every component is a separately supervised service, and each one decides at startup whether it has work. security exited 0 (“normally up” is s6’s wording for “would be restarted if it died with a non-zero code”) because compliance and runtime security are not enabled; that is a clean idle exit, not a crash.
2. The Trace Agent is its own binary, not a thread of the core Agent. Its log file, /var/log/datadog/trace-agent.log, begins with TRACE-LOADER | INFO | Listening to TCP receiver at port 8126... and Listening to unix receiver at path /var/run/datadog/apm.socket. In this build a small loader opens the listening sockets first, and the log says Listening to otlp port 5003 as well — the Trace Agent is also where the OTLP trace path ends up (see OTLP and Open Standards).
3. The Process Agent is no longer a separate running process on Linux in 7.83.3. process: down (exitcode 0) above, yet process collection is on — because the core Agent’s own log says Starting process-component with enabled checks=[process rtprocess] and Starting CheckSubmitter for host=internals-host, endpoints=[http://127.0.0.1:1]. The Process Agent section of agent status correspondingly reads Status: Not running or unreachable while the Process Component section right below it lists Enabled Checks: [process rtprocess]. If you were taught “Process Agent = its own PID”, check the version — the code moved into the core process, and the old binary now exits cleanly when it sees that.

The Cluster Agent is not in this picture at all because it is a different image (datadog/cluster-agent), deployed once per cluster instead of once per node — so it cannot be shown from a single datadog/agent:7 container, and this lesson does not pretend otherwise. What the docs say it is for (quoted from docs.datadoghq.com/containers/cluster_agent/): “By acting as a proxy between the API server and node-based Agents, the Cluster Agent helps to alleviate server load,” it “relays cluster level metadata to node-based Agents,” and it supports “Horizontal Pod Autoscaling (HPA) with custom Kubernetes metrics and external metrics.” The Kubernetes deployment shape is in the Phase 3 lesson linked above.

The pipeline inside the core Agent

flowchart LR
  AD["Autodiscovery + conf.d"] --> SCH["Check scheduler"]
  SCH -->|"every min_collection_interval"| RUN["Runner: 4 workers"]
  RUN --> AGG["Aggregator"]
  DSD["DogStatsD server"] --> AGG
  AGG --> SER["Serializer"]
  SER --> FWD["Forwarder + retry queue"]
  FWD -->|"HTTPS, one endpoint per payload type"| INT["Datadog intake"]
  TA["Trace Agent (own process)"] -.->|"own writer"| INT
  LA["Logs Agent"] -.->|"own pipeline"| INT

Core Agent data path

The dashed lines matter: the Trace Agent and the Logs Agent do not go through the core forwarder. They have their own writers and their own endpoint settings. The proof is two sections down.

Proof: the check scheduler and the runner

The core Agent log states the scheduler’s decisions as they happen:

2026-09-30 02:04:13 UTC | CORE | INFO | (pkg/collector/runner/runner.go:121 in ensureMinWorkers) | Runner 1 added 4 workers (total: 4)
2026-09-30 02:04:18 UTC | CORE | INFO | (pkg/collector/scheduler/scheduler.go:97 in Enter) | Scheduling check cpu with an interval of 15s
2026-09-30 02:04:18 UTC | CORE | INFO | (pkg/collector/scheduler/scheduler.go:97 in Enter) | Scheduling check disk:67cc0574430a16ba with an interval of 15s

Two facts fall out of these lines. The runner has 4 workers — that is the default of the check_runners setting, which agent config --all confirms (check_runners: 4) — so at most four check instances execute at the same moment, and a slow check occupies one worker for its whole run. And the scheduler works in intervals, not “when the previous run ended”: Scheduling check cpu with an interval of 15s. Reading Last Execution Date for cpu from three agent status calls 15 seconds apart (a sleep 16 between calls) gave 02:05:25.274, 02:05:55.274 and 02:06:10.274 — all on the same .274 millisecond offset and all multiples of 15 seconds after the first, which is the scheduler ticking on a fixed grid rather than drifting with execution time.

agent health exposes the same structure as named components, and the scheduler queues are literally in the list:

Agent health: PASS
=== 18 healthy components ===
ad-config-provider-do-query-actions, ad-processlistener, ad-scheduler-controller, ad-servicelistening,
aggregator, collector-queue-15s, collector-queue-60s, collector-queue-900s, dogstatsd-main, forwarder,
healthcheck, healthcheck, logs-agent, tagger-store, tagger-workloadmeta, workloadmeta-process,
workloadmeta-puller, workloadmeta-store

collector-queue-15s, -60s and -900s are the scheduler’s per-interval queues: a check whose min_collection_interval is 15 seconds sits in one queue, a 60-second check in another. agent check cpu --check-rate is the escape hatch that bypasses all of this: it runs one check instance once, synchronously, and prints the samples (system.cpu.idle.total … with "host": "internals-host") instead of handing them to the aggregator.

Reading agent status, section by section

Each header in a real agent status maps to one stage above. The real sections, in the order the Agent prints them, and what each one is telling you:

Section	What it reports	Read it when
Agent (v7.83.3)	version, PID, Config File, conf.d, checks.d paths	”which config file is actually loaded?”
Hostname	hostname: internals-host, hostname provider: configuration	tags/host mismatch — the provider tells you why this hostname won
Clocks	NTP offset, Intake offset	timestamps rejected or graphs shifted
Collector	every check instance with Total Runs, Metric Samples: Last Run/Total, Last Execution Date, Configuration Source	a check is silent — is it running, erroring, or not scheduled?
Aggregator	sample counts in, series flushed out	the pipeline is alive but nothing arrives
APM Agent	receiver ports, Endpoints, previous-minute receiver and writer	traces missing
DogStatsD	packet counters by transport, parse errors	custom metrics missing
Endpoints	the core forwarder’s URL and API key suffix	wrong site or proxy
Forwarder	transaction counters, errors by type, API key validity	data is being produced but not delivered
Logs Agent	transport, LogsProcessed/LogsSent, backpressure	logs missing or delayed
OTLP	Status: Enabled, Collector status: Running	OTLP ingest path
Process Component	enabled process checks, submission errors	Live Processes empty

Here are three of them from the real run, in full.

Aggregator (180 seconds after start — Status date 02:07:13, Agent start 02:04:13):

Aggregator
==========

  Checks Metric Sample: 3,042
  Dogstatsd Metric Sample: 1,369
  Event: 1
  Events Flushed: 1
  Number Of Flushes: 11
  Series Flushed: 2,400
  Service Check: 1
  Service Checks Flushed: 12

Checks Metric Sample and Dogstatsd Metric Sample are the two input doors into the aggregator — the check runner and the DogStatsD server — and Series Flushed is what left it. Number Of Flushes: 11 in 180 seconds is a flush roughly every 15 seconds (the first status of the earlier run, at 45 seconds, showed Number Of Flushes: 3 — same rate); that tick is one reason a value is never visible in Datadog the instant you produce it (the DogStatsD bucketing that adds to it is in DogStatsD and Aggregation).

Forwarder, with intake unreachable on purpose:

Forwarder
=========

  Transactions
  ============
    Dropped: 0
    Requeued: 572
    Retried: 16
    RetryQueueSize: 6

  Transaction Errors
  ==================
    Total number: 24
    Errors By Type:
      ConnectionErrors: 24

  On-disk storage
  ===============
    On-disk storage is disabled. Configure `forwarder_storage_max_size_in_bytes` to enable it.

  API Keys status
  ===============
    API key ending with ke00: Unable to reach the API Key validation endpoint

This is the forwarder doing its one job. Every payload becomes a transaction; a failed transaction is requeued (572 times here, because six payloads keep failing on retry) and sits in the in-memory RetryQueueSize: 6 with a backoff (capped by forwarder_backoff_max), not dropped (Dropped: 0). The log gives the per-payload-type URLs the forwarder posts to:

Error while processing transaction: error while sending transaction, rescheduling it: Post "http://127.0.0.1:1/api/v1/collector": dial tcp 127.0.0.1:1: connect: connection refused
... Post "http://127.0.0.1:1/api/v1/check_run": ...
... Post "http://127.0.0.1:1/api/v2/series": ...

/api/v2/series is metrics, /api/v1/check_run is service checks, /api/v1/collector is host metadata. The retry queue’s bound is forwarder_retry_queue_payloads_max_size: 15728640 (15 MiB, from agent config --all); forwarder_backoff_max: 64 (seconds) and forwarder_num_workers: 1 sit beside it. Payloads that do not fit in that bound are the ones counted under Dropped — the number to watch for “the Agent lost data” (this lesson’s run never filled the queue, so Dropped stayed 0; the overflow path is not reproduced here) — and On-disk storage is disabled means a restart empties the queue.

Endpoints and the two other writers — the proof that the core forwarder is not the only path. The first run of this container set only DD_DD_URL=http://127.0.0.1:1. The real status sections:

Endpoints
=========
  http://127.0.0.1:1 - API Key ending with:
      - ke00

APM Agent
=========
  ...
  Endpoints:
    https://trace.agent.datadoghq.com.

Logs Agent
==========
    Reliable: Sending uncompressed logs in SSL encrypted TCP to agent-intake.logs.datadoghq.com. on port 10516 ...

Process Endpoints
=================
    https://process.datadoghq.com. - API Key ending with:
        - ke00

dd_url moved one of four writers. The APM, logs and process endpoints each kept their default (and, in that first run, really attempted to reach them with the fake key: the Forwarder section showed HTTP Errors By Code: 403: 4, which is Datadog’s intake rejecting a fake key — the reason the run above redirects all four). Each has its own setting: apm_config.apm_dd_url, logs_config.logs_dd_url, process_config.process_dd_url.

Mental Model

Think of the Agent as one pipeline with fixed-size buffers between stages, plus sidecar processes that own their own pipelines. Checks are producers on a timer grid (scheduler + 4 runner workers), DogStatsD is a producer on a socket, both feed the aggregator, and the aggregator’s flush tick is the metronome of the whole metric path. The forwarder is the only stage that touches the network for that path, and it is a store-and-retry queue with a memory bound. Trace, logs and (in older builds) process data do not use that queue — they have their own endpoint and their own retry behaviour. When something is missing, you walk the stages left to right and each agent status section tells you which one stopped.

Trade-offs

Choice	Benefit	Cost
Separate supervised processes per component	A Trace Agent crash does not take metrics down; each can be enabled/disabled independently	More PIDs and memory; a component can be “down (exitcode 0)” and still look fine in a naive process check
Fixed runner pool (check_runners: 4)	Predictable resource use; a runaway check cannot spawn unlimited threads	One slow check (a hung network call) can pin a worker; raising the pool costs CPU
In-memory retry queue (15 MiB default)	Rides out intake blips with zero disk I/O	A restart during an outage loses the queue unless forwarder_storage_max_size_in_bytes is set
Cluster Agent as a separate deployment	Node Agents stop hitting the API server; one place for cluster checks and HPA metrics	One more thing to deploy, secure (token), and keep at a compatible version
Per-writer endpoint settings	A proxy or private link can be applied per data type	A “route everything through X” change needs every writer edited (see the SpotTheBug)

Common Misconceptions

• “The Agent is one process.” Proven above: ps shows agent run and a separate trace-agent, plus a supervisor with six services.
• “The Process Agent is always its own process.” In 7.83.3 the process checks run inside the core Agent (Starting process-component ...) and the process service exits 0; agent status reports both Process Agent: Not running or unreachable and Process Component: Enabled Checks: [process rtprocess]. Both lines are true at once.
• “dd_url sends all Agent traffic somewhere else.” Proven above — APM, logs and process endpoints kept their datadoghq defaults.
• “If the intake is down the Agent drops data immediately.” Proven above: 24 connection errors, Dropped: 0, six payloads waiting in RetryQueueSize. Loss starts only when the 15 MiB queue overflows or the process restarts with on-disk storage off.
• “Total Runs counting up means the check is healthy.” It only means the scheduler ran it. Metric Samples: Last Run: 0 on a check that should report is the real signal.

Reading AI-generated code

A model asked to “send all Agent traffic through our egress proxy” tends to write one setting and a confident comment.

🐛 Spot the bug

datadog.yaml

api_key: <your-api-key>
site: datadoghq.com

# Route ALL Agent traffic (metrics, traces, logs, processes) via the internal egress gateway
dd_url: https://egress.internal.example:3834

apm_config:
enabled: true
logs_enabled: true
process_config:
process_collection:
  enabled: true

Show the answer

dd_url only re-points the core forwarder — the one that posts to /api/v2/series, /api/v1/check_run and /api/v1/collector. The real agent status above, run with only DD_DD_URL set, still showed https://trace.agent.datadoghq.com. under APM Agent, agent-intake.logs.datadoghq.com. on port 10516 under Logs Agent and https://process.datadoghq.com. under Process Endpoints. Traces, logs and process data would keep trying to leave directly and be blocked (or, worse, quietly succeed and bypass the gateway you were auditing). To send everything through a proxy use the Agent’s proxy block, which applies to every writer, as in the proxy configuration docs:

datadog.yaml

api_key: <your-api-key>
site: datadoghq.com

proxy:
  https: http://egress.internal.example:3128
  no_proxy:
    - 169.254.169.254

apm_config:
  enabled: true
logs_enabled: true
process_config:
  process_collection:
    enabled: true

In the real run, s6-svstat showed security: down (exitcode 0) ... normally up. What does that tell you?

The Security Agent crashed and s6 will restart it in a loop The component started, found nothing enabled for it to do, and exited cleanly -- an idle exit, not a crash The Security Agent is missing from the image The exit code 0 means it is running normally

The log line Scheduling check cpu with an interval of 15s plus three agent status calls showing Last Execution Date 02:05:25.274, 02:05:55.274, 02:06:10.274 prove what?

The scheduler runs a check again 15 seconds after the previous run finished The scheduler ticks on a fixed 15-second grid regardless of how long each run took (same millisecond offset, multiples of 15s) Every check in the Agent runs every 15 seconds The cpu check runs on all four runner workers at once

With the intake unreachable, the real Forwarder section showed Requeued: 572, RetryQueueSize: 6, Dropped: 0, ConnectionErrors: 24. Which reading is correct?

Data was already lost -- 572 payloads were discarded Failed transactions are held in the in-memory retry queue and retried with backoff; nothing has been dropped yet The Agent stopped collecting after the first error The API key was rejected by Datadog

You set only dd_url to an internal gateway. In the real agent status, which writers still pointed at their datadoghq defaults?

None -- dd_url redirects all traffic Only the core forwarder APM (trace.agent.datadoghq.com), Logs (agent-intake.logs.datadoghq.com:10516) and Process (process.datadoghq.com) Only the Logs Agent

On Agent 7.83.3 Linux, agent status shows Process Agent: Status: Not running or unreachable and also Process Component: Enabled Checks: [process rtprocess]. Why is this not a contradiction?

One of the two sections is stale output The process checks run inside the core Agent (the "process-component"); the separate process-agent service exits 0 Live Processes is broken and needs a restart The Process Agent is running under the Trace Agent

Why can this lesson not show the Cluster Agent from the single datadog/agent:7 container?

The Cluster Agent is disabled by default in the Agent image It is a different image (datadog/cluster-agent) deployed once per cluster, so it is not part of the node Agent container It only runs on Windows It is the same process as the Trace Agent

Submit

Mark as complete