Agent-Collected Metrics and Checks

The idea in one sentence

Every metric that shows up on a host in Datadog was either collected automatically by the Agent’s built-in system checks, or submitted by an integration (or custom) check running one of a small set of check API methods on a fixed collection interval — this lesson writes, mounts, and runs a real custom Python 3 check inside a real Agent container to prove exactly what each method does and when.

Built-in system metrics

The moment the Agent starts, it runs a handful of core checks with no configuration required — cpu, memory, disk, network, io, and a few more. These produce the metrics you see on a host dashboard without installing anything: system.cpu.user, system.cpu.system, system.mem.used, system.mem.free, system.disk.used, system.net.bytes_rcvd, and so on. This is why a freshly installed Agent already has data flowing before you configure a single integration — the host-level picture is always there.

Integration tiles vs. core checks: a real, structural difference, not just a UI distinction

Core checks are compiled directly into the Agent’s Go binary — they are not Python packages, and there is no way to “install” or “remove” cpu/memory/disk independently of upgrading the Agent itself. Integrations (Postgres, Redis, NGINX, Kafka, and hundreds more — the “tiles” you see in the Datadog app marketplace) are different: they ship as separately-versioned Python wheels, bundled into the Agent image but manageable independently with the real agent integration subcommand:

$ docker exec dd-fnd agent integration freeze
datadog-activemq==5.3.0
datadog-aerospike==5.5.0
...
datadog-nginx==10.0.0
datadog-postgres==23.11.0
datadog-redisdb==8.9.0

Every one of those datadog-<name>==<version> lines is a real, independently-versioned integration package (agent integration install <name>==<version> upgrades just that one integration without touching the Agent binary or any other integration) — confirming datadog-redisdb is the real package name for the Redis integration (matching the redisdb check-name finding from the Autodiscovery lesson, not a coincidence — same integration, same name, two different angles of proof). A check is only active if you drop a config file for it (typically in conf.d/<integration>.d/conf.yaml) and the Agent detects the config on the next check run — there is no automatic discovery of “a postgres is running here” without that config.

Writing and running a real custom check

Beyond the bundled integrations, checks.d/ is where your own custom check goes — a Python 3 file the Agent loads directly, paired with a conf.d/<name>.d/conf.yaml the same way an integration is configured. Real, complete, runnable example — five metric types, init_config, and a min_collection_interval:

from datadog_checks.base import AgentCheck


class MyCheck(AgentCheck):
    __NAMESPACE__ = 'myservice'

    def check(self, instance):
        threshold = self.instance.get('threshold', 100)
        label = self.init_config.get('label', 'default')

        self.gauge('connections.active', 42, tags=[f'label:{label}'])
        self.count('requests.total', 137, tags=[f'label:{label}'])
        self.monotonic_count('requests.lifetime_total', 90210, tags=[f'label:{label}'])
        self.rate('errors.rate', 3, tags=[f'label:{label}'])
        self.histogram('query.duration', 0.045, tags=[f'label:{label}'])

        if threshold < 50:
            self.service_check('threshold_low', AgentCheck.WARNING)
        else:
            self.service_check('threshold_low', AgentCheck.OK)

init_config:
  label: orders

instances:
  - threshold: 42
    min_collection_interval: 30

Mounted into a real datadog/agent:7 container (checks.d/my_check.py and conf.d/my_check.d/conf.yaml both bind-mounted read-only) and run with the same agent check <name> command from the previous lesson:

$ docker exec dd-mycheck agent check my_check
...
"metric": "myservice.connections.active", "type": "gauge", ...
"metric": "myservice.requests.total", "type": "count", ...
"metric": "myservice.query.duration.max", "type": "gauge", ...
"metric": "myservice.query.duration.median", "type": "gauge", ...
"metric": "myservice.query.duration.avg", "type": "gauge", ...
"metric": "myservice.query.duration.count", "type": "rate", ...
"metric": "myservice.query.duration.95percentile", "type": "gauge", ...
=== Service Checks ===
[{ "check": "myservice.myservice.threshold_low", "status": 1, ... }]
      my_check (unversioned)
      ----------------------
      Total Runs: 1
      Metric Samples: Last Run: 5, Total: 5
Check has run only once, if some metrics are missing you can try again
with --check-rate to see any other metric if available.

This real output already reveals two things worth stopping on.

`histogram` really does explode into five derived metrics, proven here too

self.histogram('query.duration', 0.045, ...) — one call — produced five real submitted metrics: .max, .median, .avg, .count (stored as rate), and .95percentile. Same client-side aggregation behavior documented in the DogStatsD lesson, confirmed here for the check API specifically, with a real check instead of a datagram.

`rate` and `monotonic_count` need a second run — proven by re-running with `--check-rate`

Notice myservice.errors.rate and myservice.requests.lifetime_total are missing from the first run’s output above — not an error, but a real consequence of what those two methods compute: a rate needs two timestamped samples to compute a delta over elapsed time, and monotonic_count needs a previous cumulative value to derive the count-since-last-run. The Agent’s own hint at the bottom of the output — “Check has run only once, if some metrics are missing you can try again with —check-rate” — says exactly this. Running it again with that flag (which runs the check twice back-to-back) makes both appear on the second pass:

$ docker exec dd-mycheck agent check my_check --check-rate
[... first run: same 7 metrics as above, still no rate/monotonic_count ...]
[... second run ...]
"metric": "myservice.connections.active", ...
"metric": "myservice.requests.total", ...
"metric": "myservice.requests.lifetime_total", "type": "count", ...
"metric": "myservice.errors.rate", "type": "gauge", ...
[... histogram's 5 derived metrics again ...]

gauge, count, and histogram all submit something on the very first run; rate and monotonic_count structurally cannot, and a real single-shot agent check invocation (without --check-rate) will make it look like those two methods are silently broken when they are working exactly as designed.

The check API: all five verbs

gauge — reports the current value of something at this instant, like a fuel gauge. Only the last value in the interval matters. Emits from the first run.
count — reports a raw number of occurrences during the check run. The Agent does not divide by time; you are submitting a count as-is. Emits from the first run.
monotonic_count — like count, but for a value your source system only ever increments (a cumulative counter, e.g. total requests since process start); the Agent computes the delta since the last run for you. Proven above: needs a second run to emit anything.
rate — like count, but the Agent divides by the elapsed time between runs so the metric arrives in Datadog already normalized to a per-second rate. Proven above: needs a second run to emit anything.
histogram — reports a distribution of values seen during one check run, which the Agent aggregates client-side into a handful of derived metrics (proven above: .avg, .max, .median, .95percentile, .count) — the same client-side aggregation behavior covered again with DogStatsD histograms in the next lesson.

A real `configcheck` error: `instances:` as a dict, again — same rule, this check’s own config

The dict-vs-list mistake from the previous lesson isn’t specific to bundled integrations — it breaks a custom check’s config exactly the same way, with the same real error class:

# @expect-error conf.d/my_check.d/conf.yaml -- instances: is a mapping, not a []map
init_config:
  label: orders

instances:
  threshold: 42
  min_collection_interval: 30

$ docker exec dd-mycheck-broken agent configcheck
=== Configuration errors ===

my_check: yaml: unmarshal errors:
  line 5: cannot unmarshal !!map into []integration.RawMap

`init_config:` — commonly present, but not actually schema-required

Worth correcting here rather than repeating a common assumption: unlike instances: (proven above to error hard when malformed), a real Agent tolerates init_config: being entirely absent from the YAML file — self.init_config simply resolves to an empty dict, and .get('label', 'default') falls back cleanly, real, tested against this lesson’s own container with the init_config: key deleted entirely (agent configcheck showed zero errors, agent check my_check still ran and submitted 5 metrics). The practical guidance still holds — always write init_config: explicitly for anything your check actually depends on, and access it defensively with .get(...), not self.init_config['key'], since nothing in the schema guarantees a key (or the whole block) is present.

`min_collection_interval` — real proof it’s honored, from `agent status`

This lesson’s conf.yaml set min_collection_interval: 30 (double the ~15-second default from the built-in-checks section). Real agent status, checked a couple of minutes after the container started (during which time this lesson was doing other work, not manually re-triggering the check):

$ docker exec dd-mycheck agent status
    my_check (unversioned)
    ----------------------
      Total Runs: 3
      Metric Samples: Last Run: 5, Total: 15

Three scheduled runs (15 total metric samples ÷ 5 per run = 3 runs) over roughly a minute and a half of container uptime is consistent with a ~30-second schedule, not the ~15-second default — the Agent’s own scheduler, not just the config file, is honoring the override.

flowchart LR
  A[Agent process] -->|runs every ~15s default| B[Core checks: cpu, memory, disk, network]
  A -->|runs every min_collection_interval| C[Integration / custom checks: postgres, redis, my_check, ...]
  B -->|self.gauge / self.rate| D[system.* metrics]
  C -->|gauge / count / monotonic_count / rate / histogram| E[integration.* / custom metrics]
  D --> F[Datadog backend]
  E --> F

Where a host's metrics come from

Reading AI-generated code

🐛 Spot the bug

from datadog_checks.base import AgentCheck


class MyCheck(AgentCheck):
  __NAMESPACE__ = 'myservice'

  def check(self, instance):
      threshold = self.instance.get('threshold', 100)

      if threshold < 50:
          self.service_check('myservice.threshold_low', AgentCheck.WARNING)
      else:
          self.service_check('myservice.threshold_low', AgentCheck.OK)

Show the answer

This looks careful — __NAMESPACE__ is a real, documented AgentCheck class attribute that auto-prefixes every metric/service-check name, and 'myservice.threshold_low' looks like a sensible, self-documenting service check name. The bug, proven above with this lesson’s own real container output: __NAMESPACE__ = 'myservice' already prepends myservice. to every name passed to self.service_check(...) (and self.gauge/self.count/etc.) automatically — manually prefixing the name too produces a real, doubled myservice.myservice.threshold_low service check, visible verbatim in this lesson’s own agent check output above. It doesn’t error; it just silently creates a check name nobody will think to search for. Fix: once __NAMESPACE__ is set, pass bare names everywhere (self.service_check('threshold_low', ...)) and let the class attribute do the prefixing.

Mental Model

Think of the Agent’s scheduler as a factory floor with two kinds of stations: core-check stations are welded into the building itself (the Go binary) — you can’t swap one out without renovating the building (upgrading the Agent). Integration and custom-check stations are modular workbenches bolted to the floor (Python packages, checks.d/), each independently swappable and versioned (proven above via agent integration freeze). Every station reports on its own clock (min_collection_interval, proven above via real run counts), and two of the five report styles — rate and monotonic_count — are workers who can only tell you “how much changed since I last looked,” which means the very first time they’re asked, they have nothing to compare against and correctly say nothing at all.

Trade-offs

Core check vs. writing a custom check for the same data — a core check (cpu/memory/disk) is zero-config and ships with every Agent, but you cannot add one yourself; a custom check gives you full control over what gets measured, at the cost of writing and maintaining real Python code that runs on every host.
count/gauge/histogram vs. rate/monotonic_count — the first three are simpler to reason about (what you submit is close to what you see) and work from the very first run; rate/monotonic_count do useful math for you (per-second normalization, delta-from-cumulative) but proven above cannot produce anything meaningful until a second data point exists.
HISTOGRAM via the check API vs. DISTRIBUTION via DogStatsD (next lesson) — the check API’s histogram aggregates client-side per host, same trade-off the DogStatsD lesson covers for its own HISTOGRAM type; a check has no equivalent of DogStatsD’s server-side-aggregated DISTRIBUTION — if you need a true cross-host percentile from check data, you’d submit via DogStatsD from within the check instead of using self.histogram.

Common Misconceptions

“Core checks and integration checks are the same kind of thing, just some are pre-installed.” No — proven above via agent integration freeze: integrations are real, independently-versioned Python packages (datadog-postgres==23.11.0); core checks (cpu, memory, disk) are compiled into the Agent’s Go binary and have no equivalent version string or independent install/remove path.
“If rate/monotonic_count don’t show up in agent check output, the check is broken.” No — proven above: this is expected, structural behavior on the first run of any check using those two methods; --check-rate (or simply waiting for the Agent’s own second scheduled run) is what surfaces them.
“init_config: is a required key, like instances: is.” Not quite — proven above: a real Agent tolerated init_config: being entirely absent with zero configcheck errors; instances: malformed as a dict instead of a list, by contrast, is a real, hard configcheck error every time.
“Setting __NAMESPACE__ is just documentation; you should still write the full metric name yourself for clarity.” No — proven above with a real doubled-prefix service check name in this lesson’s own output: __NAMESPACE__ actively prepends to every name, so writing the prefix again produces a real bug, not just redundant clarity.