Monitor Types and Thresholds

The idea in one sentence

A Datadog Monitor is a saved query plus a set of rules for turning that query’s result into a state — OK, Warn, Alert, or No Data — and among the many monitor types, the metric alert is the clearest place to learn how thresholds actually work.

Many monitor types, one mental model

Datadog ships quite a few monitor types. Checked the real, current enum straight from the installed datadog_api_client’s MonitorType.allowed_values rather than trusting a possibly-stale list from memory — it includes metric alert, log alert, trace-analytics alert (APM), composite, synthetics alert, slo alert, rum alert, process alert, service check, event alert/event-v2 alert, audit alert, ci-pipelines alert, ci-tests alert, cost alert, database-monitoring alert, data-jobs alert, network-performance alert, network-path alert, data-quality alert, error-tracking alert, and llm-observability alert.

All of them plug into the same underlying alerting and notification machinery — the same idea of states, thresholds, and messages. What differs is only where the data comes from. This lesson (and the rest of this module) uses the metric alert as the concrete example, because its threshold configuration is the shape every other monitor type reuses in spirit.

{
  "name": "Payments API 5xx errors spiking",
  "type": "log alert",
  "query": "logs(\"service:payments-api status:error\").index(\"main\").rollup(\"count\").last(\"5m\") > 50"
}

{
  "name": "payments-api p99 latency high",
  "type": "trace-analytics alert",
  "query": "trace-analytics(\"service:payments-api\").rollup(\"p99\", \"@duration\").last(\"5m\") > 2000000000"
}

Anatomy of a metric alert’s thresholds

A metric alert’s query already contains a comparison, and options.thresholds mirrors and extends it:

{
  "name": "High CPU usage on payments-api",
  "type": "metric alert",
  "query": "avg(last_5m):avg:system.cpu.user{service:payments-api} > 90",
  "options": {
    "thresholds": {
      "critical": 90,
      "warning": 80,
      "critical_recovery": 70,
      "warning_recovery": 50
    },
    "evaluation_delay": 300
  }
}

critical — crossing this value moves the monitor into the Alert state. It normally matches the comparison already written into the query string.
warning — an earlier, lower-severity line. Crossing it moves the monitor into the Warn state, which is useful for surfacing something drifting in the wrong direction before it becomes a page.

Both critical and warning are trigger thresholds — they answer “when do we consider this a problem.”

Recovery thresholds: a different pair of numbers on purpose

critical_recovery and warning_recovery are separate fields, and they answer a different question: “when do we consider this problem over.” If you don’t set them, Datadog defaults them to the same value as the trigger thresholds, but most teams override them deliberately.

Here’s why the gap matters. If the recovery threshold were equal to the trigger threshold, a metric oscillating right at 90 would alert, recover, alert, recover, over and over — this is called flapping, and it’s one of the fastest ways to make an on-call team stop trusting (and start ignoring) their pages.

Setting critical_recovery meaningfully below critical — 70 instead of 90 — creates a dead zone. Once the monitor has fired, the metric has to drop noticeably further before Datadog considers it actually resolved. A value bouncing between 88 and 92 stays in the Alert state the whole time, instead of flapping in and out of it.

`evaluation_delay`: don’t alert on data that hasn’t fully arrived

Some metrics arrive with a lag — cost/billing metrics and anything aggregated from a slow upstream pipeline are common examples. Without accounting for that lag, a monitor’s evaluation window can look artificially low or incomplete, and it can trigger (or flap) on data that simply hasn’t finished arriving yet.

evaluation_delay shifts the point in time the query is evaluated against, in seconds. With evaluation_delay: 300 (in the fence above), the monitor evaluates its window five minutes later than it otherwise would, giving late-arriving data time to show up before a decision is made.

Proof: the `notify_no_data` trap — it needs data to have shown up at least once

docs.datadoghq.com/monitors/configuration/ states this directly, and it is the single most common way a “we’ll get alerted if the heartbeat metric stops reporting” monitor quietly fails to do its job:

Note: The monitor must be able to evaluate data before alerting on missing data. For example, if you create a monitor for service:abc and data from that service is not reporting, the monitor does not send alerts.

In other words, notify_no_data protects against a metric that was reporting and then stopped — not against a metric that was misconfigured (wrong tag, wrong metric name, integration never installed) and therefore never reported in the first place. A monitor watching a metric that has zero history behaves identically whether that monitor is perfectly configured or completely broken: silence, no alert, either way.

{
  "name": "payments-api heartbeat metric missing",
  "type": "metric alert",
  "query": "avg(last_5m):avg:payments_api.heartbeat{*} < 1",
  "options": {
    "notify_no_data": true,
    "no_data_timeframe": 20
  }
}

The same docs page adds a second, independent trap on top: the default behavior for missing data depends on the query type, and notify_no_data/no_data_timeframe is only one of five selectable behaviors (Evaluate as zero, Show last known status, Show NO DATA, Show NO DATA and notify, Show OK) — quoting directly: “For queries with default_zero(), missing data is always evaluated as zero… For queries without default_zero(), Count queries default to Evaluate as zero, while other query types default to Show last known status.” An avg()-based query like the CPU monitor above defaults to Show last known status when data goes missing — meaning without an explicit notify_no_data: true, most metric alerts don’t even flip to a visible NO DATA state, they silently keep showing whatever status they last had, which could be a calm OK from hours ago.

Watchdog: zero-config anomaly detection, not a monitor you build

docs.datadoghq.com/watchdog/ describes Watchdog plainly: “Watchdog is Datadog’s AI engine, providing you with automated alerts, insights, and root cause analyses that draw from observability data across the entire Datadog platform… All Watchdog features come built-in, and do not require setup.” It “proactively computes a baseline of expected behavior for your systems, applications, and deployments… used to detect anomalous behavior.”

That “no setup” distinction matters for this lesson: every monitor JSON fence above is something a person (or an AI-generated Terraform/API call) deliberately authors — a query, a threshold, a type. Watchdog is the opposite: there is no type: "watchdog alert" in the real MonitorType enum checked above, and no JSON payload to author, because Watchdog alerts surface automatically from Datadog’s own baseline computation, not from a monitor definition anyone wrote. The closest thing you do configure yourself is an Anomaly Algorithm monitor (a query alert using an anomaly() function) — a related but distinct feature: an anomaly monitor is a query you write and tune, Watchdog is a background process that needs nothing written at all.

`renotify_interval`: a preview (full depth in the next lesson)

If a monitor stays triggered, renotify_interval (in minutes) makes it notify again periodically instead of notifying only once — the next lesson covers renotify_interval alongside escalation_message and composite monitors in full.

flowchart LR
  OK -->|value crosses warning e.g. 80| WARN[Warn]
  WARN -->|value crosses critical e.g. 90| ALERT[Alert]
  OK -->|value crosses critical directly| ALERT
  ALERT -->|value drops below critical_recovery e.g. 70| WARN
  WARN -->|value drops below warning_recovery e.g. 50| OK

Metric alert state transitions

Reading AI-generated code

🐛 Spot the bug

// AI comment: "Alert us if the payments-api heartbeat metric ever stops reporting"
{
"name": "payments-api heartbeat monitor",
"type": "metric alert",
"query": "avg(last_5m):avg:payments_api.heartbeat{*} < 1",
"options": {
  "thresholds": {
    "critical": 1
  }
}
}

Show the answer

This is valid JSON and passes Monitor(**data, _check_type=True) cleanly — proven above, the client model has no opinion on whether notify_no_data is present. The bug: there is no notify_no_data/no_data_timeframe (or on_missing_data) at all. If payments_api.heartbeat stops reporting entirely, the docs-quoted default for an avg() query applies — Show last known status — meaning the monitor silently keeps displaying whatever state it last had, with zero notification, for as long as the metric stays missing. The AI-generated comment’s stated goal (“alert us if it ever stops reporting”) is exactly the one scenario this configuration does not handle. The fix: add "options": {"notify_no_data": true, "no_data_timeframe": 20} alongside the existing thresholds — and remember the deeper trap proven above even after that fix: if the metric never reported even once (a brand-new service, a typo’d metric name), notify_no_data never fires either, because the monitor can’t evaluate missing data it never had a baseline for.

Mental Model

A metric alert’s five settings — critical, warning, critical_recovery, warning_recovery, evaluation_delay — are really answering three separate questions, and conflating them is where most misconfiguration comes from. “When is this bad?” is critical/warning. “When is this no longer bad?” is critical_recovery/warning_recovery — a genuinely separate question, not a mirror of the first, precisely because a dead zone between them is what stops flapping. “When should I even look?” is evaluation_delay — a question about the data’s own arrival time, unrelated to good or bad at all. notify_no_data answers a fourth, even more separate question — “what if there’s no data to judge” — and Watchdog isn’t answering any of these questions per-monitor at all; it’s a different system entirely, watching for anomalies nobody wrote a query for.

Trade-offs

Tight recovery thresholds (equal to trigger) vs. a deliberate dead zone — equal thresholds are simpler to reason about and match the query’s own comparison exactly, but proven above, they flap on any metric that hovers near the line; a dead zone stops flapping at the cost of the monitor staying in Alert state slightly longer than the metric’s own instantaneous value would suggest.
notify_no_data: true vs. leaving it unset — enabling it catches a metric that goes silent after reporting normally, which is genuinely useful for heartbeat-style checks; leaving it unset (the default for most query types, proven above from the docs) avoids noisy alerts on metrics that are expected to be sparse, at the cost of silent blind spots on metrics that should always report.
A hand-written Anomaly Algorithm monitor vs. relying on Watchdog — an anomaly monitor is tunable (algorithm choice, sensitivity, specific query) and shows up as an ordinary monitor in the same notification pipeline as everything else; Watchdog needs zero configuration and surfaces things nobody thought to write a query for, at the cost of not being tunable to a specific team’s notion of “anomalous” the way a hand-authored monitor is.

Common Misconceptions

“notify_no_data alerts on any metric that never shows up.” No — proven above by quoting the docs directly: the monitor must evaluate data at least once before it can alert on that data going missing. A metric that never reported at all produces silence, not an alert, notify_no_data or not.
“If I don’t configure no-data behavior, Datadog shows a visible NO DATA state by default.” No — proven above: for query types other than Count (and without default_zero()), the documented default is Show last known status, which silently keeps the monitor’s previous state rather than surfacing anything.
“Watchdog is a monitor type you create, like a metric alert or composite monitor.” No — proven above from docs.datadoghq.com/watchdog/ and the real MonitorType enum: there is no Watchdog monitor type, no JSON payload to author. Watchdog is a built-in, zero-configuration background system.
“Recovery thresholds are just documentation of the trigger thresholds, not something separately meaningful.” No — proven above: critical_recovery/warning_recovery are independent fields that, when set meaningfully below the trigger thresholds, are the entire mechanism that prevents flapping.