Datadog as Code with Terraform

The idea in one sentence

The official Terraform Datadog provider lets you define monitors, dashboards, and SLOs as versioned, reviewable code, instead of clicking them together by hand in the UI, one environment at a time.

Why bother managing Datadog as code

Clicking a monitor together in the UI is fast the first time. The trouble shows up the second time: you need the same monitor in staging, the same dashboard in a new region, or you need to know exactly what changed to a threshold last month and why. UI-built configuration has no diff, no review step, and no single source of truth — two people can quietly disagree about what a monitor’s threshold “should” be, and nothing catches it.

Managing monitors, dashboards, and SLOs through the Terraform Datadog provider turns all of that into a code change: it lives in a repository, it gets reviewed in a pull request like any other change to production, and the same module can be instantiated for prod, staging, and every new service that comes along by changing a variable instead of re-clicking a wizard.

A real `datadog_monitor` resource

Here is a metric alert monitor expressed as Terraform:

resource "datadog_monitor" "high_cpu" {
  name    = "High CPU on checkout-service"
  type    = "metric alert"
  message = "CPU is high on checkout-service. Notify: @slack-sre-alerts"

  query = "avg(last_5m):avg:system.cpu.user{service:checkout-service} by {host} > 85"

  monitor_thresholds {
    warning  = 70
    critical = 85
  }

  tags = ["service:checkout-service", "team:checkout"]
}

type — metric alert is one of several monitor types (others include log alert, apm alert, and the cost alert type covered in the next lesson).
message — the notification body. The @slack-sre-alerts handle is a Datadog notification target, resolved the same way it would be if you typed it into the UI’s message box.
query — the actual alerting condition, evaluated as avg(last_5m):avg:system.cpu.user{service:checkout-service} by {host} > 85.
monitor_thresholds — warning and critical map directly onto the threshold sliders you’d otherwise drag in the UI.
tags — ordinary Datadog tags on the monitor object itself, useful for filtering the Manage Monitors page and for routing via Service Catalog ownership.

The important caveat: the query string in a datadog_monitor resource is not the same syntax you’d type into the Datadog UI’s monitor query builder. The UI builder is a guided widget that assembles a query for you; the Terraform query field is the raw API query syntax, and it has to be written by hand, correctly, the first time. By default, terraform plan sends that query to Datadog’s API for validation, so a malformed query fails at plan time rather than silently creating a broken monitor. If you genuinely need to skip that check — for example, referencing a metric that doesn’t exist yet in this environment — you can set validate = false on the resource, but that trades away your safety net, so it should be the exception, not the default.

Dashboards as code: `ordered` vs `free` layout

The dashboard lesson from the Infrastructure & Metrics module covered named queries, formulas, and template variables from the UI’s point of view. The same widget model shows up in Terraform via datadog_dashboard_v2, and the one extra decision you make in code is the layout_type.

With layout_type = "ordered", widgets simply stack top to bottom in the order you declare them — the simplest option, and a good default:

resource "datadog_dashboard_v2" "checkout_overview" {
  title       = "Checkout Service Overview"
  layout_type = "ordered"

  widget {
    timeseries_definition {
      request {
        query {
          name        = "query1"
          data_source = "metrics"
          query       = "avg:system.cpu.user{service:checkout-service} by {host}"
        }
        display_type = "line"
      }
    }
  }
}

With layout_type = "free", every widget needs its own widget_layout block, giving you exact pixel-grid placement with x, y, width, and height:

resource "datadog_dashboard_v2" "checkout_overview_free" {
  title       = "Checkout Service Overview (free layout)"
  layout_type = "free"

  widget {
    timeseries_definition {
      request {
        query {
          name        = "query1"
          data_source = "metrics"
          query       = "avg:system.cpu.user{service:checkout-service} by {host}"
        }
        display_type = "line"
      }
    }
    widget_layout {
      x      = 0
      y      = 0
      width  = 4
      height = 2
    }
  }
}

ordered is the right default for most operational dashboards — you rarely need to fight pixel positions for a dashboard that mostly gets scrolled top to bottom. free earns its keep for TV-wall dashboards or intentionally designed layouts where widgets need to sit at specific coordinates relative to each other.

flowchart LR
  A[Write datadog_monitor / datadog_dashboard_v2 in HCL] --> B[terraform plan]
  B --> C{Query validation}
  C -->|valid, or validate = false| D[Pull request review]
  C -->|invalid| E[Plan fails: fix query]
  D --> F[terraform apply]
  F --> G[Monitor / Dashboard / SLO created in Datadog]
  G --> H[Reused across prod, staging, new services via variables]

From HCL to live Datadog resources