Composite Monitors and Notification Workflows

The idea in one sentence

A composite monitor combines two or more existing monitors with boolean logic so you can alert on a combination of conditions instead of any single monitor firing in isolation, and once something does need to notify someone, @-mentions and escalation_message control exactly who hears about it and when that widens.

Composite monitors: alert on combinations, not single signals

A composite monitor references the IDs of existing monitors and combines their trigger states with boolean operators — AND (&&), OR (||), NOT (!). The practical payoff is noise reduction: instead of paging on every monitor that fires, you page only when a specific combination of states is true.

{
  "name": "payments-api error rate (composite)",
  "type": "composite",
  "query": "111111 && !222222",
  "message": "Error rate is elevated on payments-api and the upstream-dependency monitor is healthy, so this looks like our own issue."
}

Here 111111 is the error-rate monitor’s ID and 222222 is the upstream-dependency monitor’s ID. The formula fires only when 111111 is alerting and 222222 is not alerting. If both are alerting at the same time, the composite stays quiet — the likely story is that the real problem is upstream, and the upstream monitor itself will already be paging whoever owns that dependency.

Routing notifications with `@`-mentions

A monitor’s message field carries @-mentions directly in its text: @username for an individual, @slack-channel-name for a connected Slack channel, @pagerduty-service-name for a connected PagerDuty service. Each of these has to be set up as an integration in Datadog first — the handle in the message is what routes the notification once that connection exists.

{
  "message": "@here Error rate is elevated on payments-api.\n@slack-payments-oncall @pagerduty-payments-critical"
}

`escalation_message`: a different message for re-notification

If a monitor stays triggered, renotify_interval (in minutes) makes it notify again periodically instead of notifying only once. escalation_message is a separate field that lets that re-notification carry different text than the original alert — commonly used to pull in a wider on-call escalation the longer the problem stays unresolved.

{
  "options": {
    "renotify_interval": 15
  },
  "message": "@slack-payments-oncall Error rate is elevated on payments-api.",
  "escalation_message": "@pagerduty-payments-escalation Still broken after 15 minutes, escalating to the secondary on-call."
}

The first page goes to the primary Slack channel. If nobody has resolved it fifteen minutes later, the next notification goes out with escalation_message’s text instead, pulling in PagerDuty’s secondary on-call.

Downtimes: muting without disabling evaluation

A Downtime (mute window) silences a monitor’s notifications for a known maintenance window without touching its evaluation. The monitor keeps running its query and keeps updating its internal state the whole time — only the outbound notification is suppressed. That matters because status history stays accurate, and notifications resume automatically the moment the Downtime window ends, with no manual re-enabling step.

flowchart LR
  A[Monitor A alerting] --> C{Composite formula}
  B[Monitor B alerting] --> C
  C -->|A AND NOT B is true| N[Notify via @-mentions]
  C -->|formula is false| Q[Stay quiet]
  N --> R{Still triggered after renotify_interval}
  R -->|yes| E[Send escalation_message]
  R -->|no| Done[Resolved]

Composite evaluation and escalation flow