Composite Monitors and Notification Workflows

The idea in one sentence

A composite monitor combines two or more existing monitors with boolean logic so you can alert on a combination of conditions instead of any single monitor firing in isolation, and once something does need to notify someone, @-mentions, template conditionals, priority, and restricted_roles control exactly who hears about it, how the message reads, and who is even allowed to touch it.

Composite monitors: alert on combinations, not single signals

A composite monitor references the IDs of existing monitors and combines their trigger states with boolean operators — AND (&&), OR (||), NOT (!). The practical payoff is noise reduction: instead of paging on every monitor that fires, you page only when a specific combination of states is true.

monitors/payments-api-composite.json

{
  "name": "payments-api error rate (composite)",
  "type": "composite",
  "query": "111111 && !222222",
  "priority": 2,
  "restricted_roles": ["role-a1b2c3d4-uuid"],
  "message": "@pagerduty-payments-critical {{#is_alert}}Error rate is elevated on payments-api and the upstream-dependency monitor is healthy.{{/is_alert}}",
  "options": {
    "renotify_interval": 15,
    "escalation_message": "@pagerduty-payments-escalation Still broken after 15 minutes, escalating to the secondary on-call."
  }
}

Here 111111 is the error-rate monitor’s ID and 222222 is the upstream-dependency monitor’s ID. The formula fires only when 111111 is alerting and 222222 is not alerting. If both are alerting at the same time, the composite stays quiet — the likely story is that the real problem is upstream, and the upstream monitor itself will already be paging whoever owns that dependency.

🐶 Proof: validates against the real Monitor model

Ran the fence above through Monitor(**data, _check_type=True) (pinned datadog-api-client==2.61.0): {"ok": true, "error": null}. priority (an int, docstring straight from the pinned client: “Integer from 1 (high) to 5 (low) indicating alert severity”) and restricted_roles (a List[str] of role IDs) are both real, current top-level Monitor fields.

Routing notifications with @-mentions

A monitor’s message field carries @-mentions directly in its text: @username for an individual, @slack-channel-name for a connected Slack channel, @pagerduty-service-name for a connected PagerDuty service. Each of these has to be set up as an integration in Datadog first — the handle in the message is what routes the notification once that connection exists.

Template conditionals: {{#is_alert}} and the rest of the state family

A monitor message is a template, and Datadog’s own template variables let one message body read differently depending on which state transition actually triggered it. docs.datadoghq.com/monitors/notify/variables/ documents the state-conditional family in full — quoting the real names and meanings:

• {{#is_alert}}/{{^is_alert}} — the monitor does/does not alert
• {{#is_warning}}/{{^is_warning}} — the monitor does/does not warn
• {{#is_recovery}}/{{^is_recovery}} — the monitor recovers from ALERT, WARNING, UNKNOWN, or NO DATA
• {{#is_no_data}}/{{^is_no_data}} — the monitor is/is not triggered for missing data
• {{#is_alert_recovery}} / {{#is_warning_recovery}} / {{#is_alert_to_warning}} — narrower recovery/transition variants
• {{#is_priority 'value'}} — the monitor has priority value, where value ranges from P1 to P5
• {{#is_renotify}}/{{^is_renotify}} — this specific notification is/is not a re-notification (the fence’s escalation_message above is effectively an {{#is_renotify}} block Datadog manages for you automatically)

The docs are explicit about a real footgun: “Conditional variables must have an opening and closing pair with the text and @-notifications in-between.” — an {{#is_alert}} with no matching {{/is_alert}} is malformed, not merely stylistic.

🐶 Proof: `priority` in the payload is an int, but the template conditional's argument is a string — two different representations of the same concept

{{#is_priority 'P2'}} in a message template takes a string ('P1' through 'P5'). The priority field in the JSON payload itself takes a plain int (1 through 5, no P prefix) — confirmed directly from the pinned client’s own docstring above. Tried passing the template’s string form into the payload field to see what happens: Monitor(name=..., type=..., query=..., priority="P2", _check_type=True) raises a real ApiTypeError: Invalid type for variable 'priority'. Required value type is one of [NoneType, int] and passed type was str at ['priority']. Two different syntaxes for the same concept, in two different parts of the same product, and copying one into the other’s slot is a real, provable type error.

escalation_message: a different message for re-notification, nested under options

If a monitor stays triggered, renotify_interval (in minutes) makes it notify again periodically instead of notifying only once. escalation_message is a separate field that lets that re-notification carry different text than the original alert — commonly used to pull in a wider on-call escalation the longer the problem stays unresolved. The composite fence above nests it correctly under options, matching the pinned client’s real MonitorOptions.attribute_map (which lists escalation_message alongside renotify_interval, not as a sibling of options).

🐶 Proof: a field at the wrong nesting level is NOT caught by --json's Monitor model check

An earlier draft of this lesson placed escalation_message at the top level of the monitor object, a sibling of options rather than nested inside it. Tried it directly against the real, pinned client: Monitor(name='x', type='metric alert', query='avg(last_5m):avg:a{*}>1', escalation_message='wrong place', _check_type=True) constructs with zero exception, and m.escalation_message reads back 'wrong place' cleanly. This client’s generated Monitor.__init__ accepts arbitrary unrecognized top-level keyword arguments through **kwargs with no whitelist check — confirmed by passing a nonsense key (totally_made_up_field='hello') through the same constructor, which is likewise accepted silently. This is a real, separate gap from the “won’t validate a malformed query string” caveat already documented in this course’s harness: --json’s Monitor model check validates fields it recognizes strictly (real ApiTypeError for priority: "P2", proven above), but says nothing at all about a field placed at the wrong nesting depth, or a field name that doesn’t exist anywhere in the real schema. The first draft’s misplaced escalation_message would have silently done nothing against the real API — this class of bug needs a real API call, or a human who has actually read MonitorOptions.attribute_map, to catch.

Downtimes: muting without disabling evaluation

A Downtime (mute window) silences a monitor’s notifications for a known maintenance window without touching its evaluation. The monitor keeps running its query and keeps updating its internal state the whole time — only the outbound notification is suppressed. That matters because status history stays accurate, and notifications resume automatically the moment the Downtime window ends, with no manual re-enabling step. The pinned client’s own Downtime model (datadog_api_client.v1.model.downtime) confirms the real field shape: scope (the tag-based scope to mute), monitor_id (mute one specific monitor instead of by scope), start/end, message, recurrence, and timezone are all real, current fields.

Workflow Automation: what happens after the notification

docs.datadoghq.com/service_management/workflows/ describes the product that picks up where a monitor notification leaves off: “Datadog Workflow Automation allows you to orchestrate and automate your end-to-end processes. Build workflows made up of actions that connect to your infrastructure and tools.” Directly relevant to this lesson: “Trigger your workflows from monitors, security signals, or dashboards, or trigger them manually.” A composite monitor’s @-mention can page a human; a Workflow triggered from that same monitor can instead (or additionally) run an automated remediation action — the docs’ own example is “Automate scaling of your AWS Auto Scaling Groups when monitors tracking critical metrics of these Auto Scaling Groups go into the alert state.” Workflow Automation is not something this lesson’s JSON fences model — it is a separate product surface (2000+ prebuilt actions, per the docs) — but it is the natural next stop after everything this lesson covers: a monitor decides when something is wrong, @-mentions and templates decide who hears about it and how it reads, and a Workflow can decide what happens automatically next.

flowchart LR
  A[Monitor A alerting] --> C{Composite formula}
  B[Monitor B alerting] --> C
  C -->|A AND NOT B is true| N[Notify via @-mentions + is_alert template]
  C -->|formula is false| Q[Stay quiet]
  N --> R{Still triggered after renotify_interval}
  R -->|yes| E[Send escalation_message]
  R -->|no| Done[Resolved]
  N -.can also trigger.-> W[Workflow Automation action]

Composite evaluation and escalation flow

🐶 The one-line takeaway

Composite monitors combine other monitors with &&/||/! to alert only on a meaningful combination of states (proven above against the real Monitor model, along with priority as an int and restricted_roles as a role-ID list); @-mentions route notifications and {{#is_alert}}-style template conditionals control what the message says per state; escalation_message (nested under options — proven above that a wrong-level placement is silently accepted, not an error) sends different text on re-notification; a Downtime mutes notifications without stopping evaluation; and Workflow Automation can turn any of these triggers into an automated remediation action instead of just a page.

Reading AI-generated code

🐛 Spot the bug

// AI comment: "Page P2 priority and escalate after 15 minutes"
{
"name": "payments-api error rate (composite)",
"type": "composite",
"query": "111111 && !222222",
"priority": "P2",
"message": "@pagerduty-payments-critical {{#is_alert}}Error rate is elevated.",
"escalation_message": "@pagerduty-payments-escalation Still broken, escalating."
}

Show the answer

Two real bugs here, both proven above against the pinned, real datadog_api_client. First: priority: "P2" copies the template conditional’s string syntax ({{#is_priority 'P2'}}) into the JSON payload’s priority field, which requires a plain int. Run through Monitor(**data, _check_type=True) this raises a real ApiTypeError: Invalid type for variable 'priority'. Required value type is one of [NoneType, int] and passed type was str — the fix is "priority": 2. Second: {{#is_alert}} has no matching {{/is_alert}} — an unclosed conditional block, which the docs state directly is malformed (“Conditional variables must have an opening and closing pair”). Third, more subtly: escalation_message sits at the top level instead of nested under options — proven above, this one does not raise an error at all; the client silently accepts it via **kwargs and it would do nothing against the real API. Only one of these three real bugs throws — the other two are silent, which is exactly why running the payload through the real model (and reading the real docs) matters more than trusting that clean-looking JSON is correct JSON.

Mental Model

A composite monitor’s query is a tiny boolean expression language whose only variables are other monitors’ current states — think of it as a logic gate wired to the outputs of other gates, not a new signal source of its own. Everything downstream of that gate firing — @-mentions, {{#is_alert}} conditionals, escalation_message, a Downtime, a Workflow trigger — is a separate, composable layer that doesn’t care whether the gate’s inputs came from a metric, a log, or another composite. That separation is exactly why the same notification/escalation mechanics apply identically to every monitor type in this module: the boolean logic (or lack of it, for a simple metric alert) is the only thing that changes.

Trade-offs

• A single composite monitor vs. paging on every underlying monitor separately — a composite cuts real noise (proven above: 111111 && !222222 stays quiet exactly when the likely cause is already being handled upstream), at the cost of a formula that itself needs to stay correct as underlying monitor IDs get recreated or renumbered.
• {{#is_alert}}-style per-state message templating vs. one static message for every state — a state-aware message tells the on-call engineer exactly what happened (alert vs. warn vs. recovery) in one glance, at the cost of a template that must respect the real open/close-pairing rule (proven above) or render incorrectly.
• A Downtime vs. disabling/deleting a monitor for planned maintenance — a Downtime keeps evaluation and status history intact and resumes notifying automatically when the window ends; deleting or disabling a monitor loses that history and requires someone to remember to turn it back on.

Common Misconceptions

• “priority in a monitor payload uses the same 'P1'–'P5' string format as the {{#is_priority}} template conditional.” No — proven above: the payload field is a plain int (1–5); passing the template’s string form raises a real ApiTypeError.
• “If a monitor JSON payload validates against the real Monitor model, every field in it is in the right place.” No — proven above: escalation_message at the top level (instead of nested under options) constructs with zero exception, because the model accepts unrecognized top-level keys via **kwargs. Passing strict type validation and being structurally correct are not the same guarantee.
• “{{#is_alert}} without a matching {{/is_alert}} still works, just renders the whole message unconditionally.” No — the docs state conditional variables must have a matching opening and closing pair; an unclosed block is malformed, not a harmless fallback.
• “A Downtime stops the monitor from evaluating until it ends.” No — a Downtime only suppresses outbound notifications; the monitor keeps running its query and updating its status the entire time, which is exactly what keeps status history accurate through a maintenance window.

What does a composite monitor formula like 111111 && !222222 mean?

Notify if either monitor 111111 or monitor 222222 is alerting Notify only if monitor 111111 is alerting and monitor 222222 is not alerting Notify only if both monitors are alerting at the same time Disable monitor 222222 whenever monitor 111111 fires

Why would a team use a composite monitor instead of just alerting on the error-rate monitor by itself?

To lower the query interval To avoid paging when the real cause is an upstream dependency that is already being handled Because single monitors cannot have thresholds To increase the number of notifications for the same incident

A monitor payload sets priority: "P2" (a string). What did the real, pinned datadog_api_client Monitor model prove about this?

It validates successfully -- priority accepts either a string or an int It raises a real ApiTypeError -- priority requires an int (1-5), while "P2" is the string format used only by the {{#is_priority}} template conditional It silently converts "P2" to the integer 2 It raises a JSONDecodeError

A monitor JSON payload places escalation_message at the TOP level (a sibling of options) instead of nested inside options. What did running this through the real Monitor(**data, _check_type=True) prove?

It raises an ApiTypeError because escalation_message is not a recognized top-level field It constructs with zero exception -- the client accepts unrecognized top-level keys via **kwargs with no whitelist check It raises an ApiValueError because the field is misplaced astro build fails because of the misplaced field

What is the real rule the docs state about {{#is_alert}}-style conditional template variables?

They are optional decorations that render as literal text if malformed They must have a matching opening and closing pair around the text and @-notifications in between They can only be used in error-tracking alerts They replace the need for an @-mention entirely

What happens to a monitor's evaluation while it is inside an active Downtime (mute window)?

The monitor stops evaluating its query entirely The monitor keeps evaluating and its state keeps updating, only outbound notifications are suppressed The monitor is permanently disabled until manually re-enabled The monitor automatically becomes a composite monitor

Per docs.datadoghq.com/service_management/workflows/, what can trigger a Datadog Workflow Automation workflow?

Only a manual button click in the UI, nothing else Monitors, security signals, or dashboards, or a manual trigger Only composite monitors specifically Workflows cannot be triggered by monitors at all

Submit

Mark as complete