Agent-Collected Metrics and Checks

The idea in one sentence

Every metric that shows up on a host in Datadog was either collected automatically by the Agent’s built-in system checks, or submitted by an integration check running one of a small set of check API methods on a fixed collection interval.

Built-in system metrics

The moment the Agent starts, it runs a handful of core checks with no configuration required — cpu, memory, disk, network, io, and a few more. These produce the metrics you see on a host dashboard without installing anything: system.cpu.user, system.cpu.system, system.mem.used, system.mem.free, system.disk.used, system.net.bytes_rcvd, and so on. This is why a freshly installed Agent already has data flowing before you configure a single integration — the host-level picture is always there.

On top of the core checks, the Agent ships with integrations — Python-based checks for specific technologies (postgres, redis, nginx, kafka, and hundreds more). An integration check is only active if you drop a config file for it (typically in conf.d/<integration>.d/conf.yaml) and the Agent detects the config on the next check run. Enabling the postgres integration, for example, is what makes metrics like postgresql.connections or postgresql.rows_returned start appearing — there is no automatic discovery of “a postgres is running here” without that config.

The check API: gauge, count, rate, histogram

Every check — built-in or integration — is a small Python class that runs on a schedule and calls a handful of methods on self to submit values. These are the same four verbs you will use conceptually again in DogStatsD later:

from datadog_checks.base import AgentCheck

class MyServiceCheck(AgentCheck):
    def check(self, instance):
        # a point-in-time value, e.g. current connection count
        self.gauge('myservice.connections.active', 42, tags=['env:prod'])

        # an incrementing counter since the last check run
        self.count('myservice.requests.total', 137, tags=['env:prod'])

        # count normalized to "per second" by the Agent
        self.rate('myservice.errors.rate', 3, tags=['env:prod'])

        # distribution of a value observed during this check run
        self.histogram('myservice.query.duration', 0.045, tags=['env:prod'])

gauge — reports the current value of something at this instant, like a fuel gauge. Only the last value in the interval matters.
count — reports a raw number of occurrences during the check run. The Agent does not divide by time; you are submitting a count as-is.
rate — like count, but the Agent divides by the elapsed time between runs so the metric arrives in Datadog already normalized to a per-second rate.
histogram — reports a distribution of values seen during one check run, which the Agent aggregates client-side into a handful of derived metrics (average, max, percentiles, count) — the same client-side aggregation behavior you will see again with DogStatsD histograms.

The ~15 second collection interval

By default, every check the Agent runs is scheduled on a 15-second collection interval — the Agent wakes the check up, it calls self.gauge(...) / self.count(...) / etc. for whatever it observed since the last run, and the Agent flushes those values upstream. This interval is configurable per-check (min_collection_interval in the check’s config), but 15 seconds is the default you should assume unless a config says otherwise. This matters for rate and count in particular — the numbers those methods compute or report are only meaningful in the context of “since the last collection interval,” so changing the interval changes how those values are interpreted, not just how often they arrive.

flowchart LR
  A[Agent process] -->|runs every ~15s| B[Core checks: cpu, memory, disk, network]
  A -->|runs every ~15s| C[Integration checks: postgres, redis, nginx, ...]
  B -->|self.gauge / self.rate| D[system.* metrics]
  C -->|self.gauge / self.count / self.rate / self.histogram| E[integration.* metrics]
  D --> F[Datadog backend]
  E --> F

Where a host's metrics come from