Worked review 3: monitors, SLOs and dashboards as code

The feature

The prompt was: “Codify the checkout service’s observability in Terraform: a p95 latency monitor, a 99.9 percent availability SLO over 30 days, a burn-rate alert on that SLO, and a dashboard with the p95 latency. Also give me the JSON for the same objects in case we use the API directly.” Two artifacts came back: one .tf file and three JSON documents. Neither has a syntax error in the sense a text editor would show.

What was used, and how far each tool reaches:

• terraform validate with the pinned DataDog/datadog provider 4.22.0 on Terraform 1.16.4, in a scratch directory with a private copy of the plugin cache. The provider block uses fake keys and validate = false; terraform plan and apply were never run, and the AWS_* variables that exist on this machine were removed from the environment of every terraform process.
• datadog-api-client 2.61.0 models (Monitor, ServiceLevelObjective, Dashboard) two ways: the constructor with _check_type=True (what the course harness’s --json mode runs) and the client’s own deserializer, ApiClient(Configuration()).deserialize(json, (Model,), True), which sets _unparsed = True when an enum value or oneOf does not match. Nothing is sent anywhere.
• Docs, quoted with URL wherever a bug is beyond both tools — and that is the honest centre of this review: several of the bugs below pass every offline check.

The generated code

# @expect-error as generated -- five terraform validate errors at once, more behind them
resource "datadog_monitor" "checkout_latency" {
  name    = "Checkout p95 latency"
  type    = "metric alert"
  message = "p95 too high @pagerduty-checkout"
  query   = "avg(last_5m):p95:trace.http.request{service:checkout,env:prod} > 2"
  tags    = "team:payments"

  thresholds {
    critical = 2
    warning  = 1.5
  }
}

resource "datadog_service_level_objective" "checkout_availability" {
  name = "Checkout availability"
  type = "metric"

  query {
    numerator   = "sum:checkout.requests{env:prod,outcome:error}.as_count()"
    denominator = "sum:checkout.requests{env:prod}.as_count()"
  }

  thresholds {
    timeframe = "30days"
    target    = 99.9
    warning   = 99.95
  }
}

resource "datadog_monitor" "checkout_burn_rate" {
  name    = "Checkout SLO burn rate"
  type    = "slo alert"
  message = "Error budget is burning @pagerduty-checkout"
  query   = "burn_rate(\"${datadog_service_level_objective.checkout_availability.slo_id}\").over(\"1h\").long_window(\"6h\") > 10"
}

resource "datadog_dashboard" "checkout" {
  title = "Checkout overview"

  template_variables {
    name   = "env"
    prefix = "env"
  }

  widget {
    timeseries_definition {
      title = "p95 latency"
      request {
        query {
          metric_query {
            name        = "query1"
            data_source = "metrics"
            query       = "p95:trace.http.request{service:checkout,env:$env}"
          }
        }
        formula {
          formula_expression = "query2"
        }
        display_type = "line"
      }
    }
  }
}

And the API-shaped JSON that came with it:

// @expect-error monitor JSON as generated -- notify_no_data is a string; thresholds sits at the top level
{
  "name": "Checkout p95 latency",
  "type": "metric alert",
  "query": "avg(last_5m):p95:trace.http.request{service:checkout,env:prod} > 2",
  "message": "p95 too high @pagerduty-checkout",
  "thresholds": { "critical": 2, "warning": 1.5 },
  "options": { "notify_no_data": "true", "renotify_interval": 60 }
}

// @expect-error SLO JSON as generated -- type, timeframe and target are all wrong; the constructor accepts every one
{
  "name": "Checkout availability",
  "type": "metrics",
  "query": {
    "numerator": "sum:checkout.requests{env:prod,outcome:error}.as_count()",
    "denominator": "sum:checkout.requests{env:prod}.as_count()"
  },
  "thresholds": [{ "timeframe": "30days", "target": "99.9" }]
}

// @expect-error dashboard JSON as generated -- layout_type "grid" and display_type "lines" are not enum values
{
  "title": "Checkout overview",
  "layout_type": "grid",
  "widgets": [
    {
      "definition": {
        "type": "timeseries",
        "title": "p95 latency",
        "requests": [{ "q": "p95:trace.http.request{service:checkout}", "display_type": "lines" }]
      }
    }
  ]
}

The review walk

A reviewer does not run terraform validate once. Terraform reports structural errors first and only then evaluates references, so the file was reviewed in stages: the first run shows five errors, fixing them reveals a sixth, fixing that turns the run green — with three bugs left in the file.

tags = "team:payments" — a string where a set belongs

$ terraform validate
Error: Incorrect attribute value type
  on main.tf line 6, in resource "datadog_monitor" "checkout_latency":
   6:   tags    = "team:payments"
Inappropriate value for attribute "tags": set of string required, but have string.

🐛 Spot the bug

  tags    = "team:payments"

Show the answer

datadog_monitor.tags is a set of strings. AI writes the single-string form because that is how a tag is spelled in the UI and in DD_TAGS. Hard error, caught offline (the same string is also rejected by the API client model — see the JSON section). Fix: tags = ["service:checkout", "team:payments"].

thresholds { } — a block that does not exist

Error: Unsupported block type
  on main.tf line 8, in resource "datadog_monitor" "checkout_latency":
   8:   thresholds {
Blocks of type "thresholds" are not expected here.

The block is monitor_thresholds. thresholds exists on datadog_service_level_objective — the assistant carried the SLO spelling into the monitor. Compare bug 10: the JSON model accepts the misplaced field in silence.

🐛 Spot the bug

  thresholds {
  critical = 2
  warning  = 1.5
}

Show the answer

Two resources, two spellings: monitor_thresholds on datadog_monitor, thresholds on datadog_service_level_objective. Hard error offline. Fix: monitor_thresholds { critical = 2 warning = 1.5 }. A warning and a recovery threshold also matter operationally — docs: “If no recovery threshold is set, the monitor recovers whenever the alert conditions are no longer met” (docs.datadoghq.com/monitors/guide/recovery-thresholds/).

timeframe = "30days" — the provider does check this enum

Error: Invalid enum value
  with datadog_service_level_objective.checkout_availability,
  on main.tf line 24, in resource "datadog_service_level_objective" "checkout_availability":
  24:     timeframe = "30days"
`invalid value '30days' for SLOTimeframe: valid values are [7d 30d 90d custom]`

This is worth noting for later: the provider enforces SLOTimeframe at validate time. The Python client model in bug 11 does not.

🐛 Spot the bug

  thresholds {
  timeframe = "30days"
  target    = 99.9
  warning   = 99.95
}

Show the answer

The API vocabulary is 7d, 30d, 90d (and custom). The prose “30 days” in the prompt became 30days in the file. Hard error offline with the provider; fix timeframe = "30d".

The dashboard has no layout_type, and template_variables is plural

Error: Missing required argument
  on main.tf line 37, in resource "datadog_dashboard" "checkout":
  37: resource "datadog_dashboard" "checkout" {
The argument "layout_type" is required, but no definition was found.

Error: Unsupported block type
  on main.tf line 40, in resource "datadog_dashboard" "checkout":
  40:   template_variables {
Blocks of type "template_variables" are not expected here. Did you mean "template_variable"?

🐛 Spot the bug

resource "datadog_dashboard" "checkout" {
title = "Checkout overview"

template_variables {
  name   = "env"
  prefix = "env"
}

Show the answer

Two separate hard errors from the same 6 lines: layout_type (ordered or free) is required on datadog_dashboard, and the repeatable block is template_variable, singular. Both are the shape of error terraform validate exists for. Fix: add layout_type = "ordered", rename the block. (The bug catalog entry 20 records the same plural mistake.)

.slo_id — a reference that only fails in the second run

Fixing the six errors above and running again gives a new error that the first run could not show:

Error: Unsupported attribute
  on main.tf line 34, in resource "datadog_monitor" "checkout_burn_rate":
  34:   query   = "burn_rate(\"${datadog_service_level_objective.checkout_availability.slo_id}\").over(...
This object has no argument, nested block, or exported attribute named "slo_id".

Terraform stops evaluating references while structural errors remain. The resource’s identifier is the standard .id. (The docs page for burn-rate alerts calls the placeholder slo_id; the assistant turned the placeholder name into an attribute.)

🐛 Spot the bug

query = "burn_rate(\"${datadog_service_level_objective.checkout_availability.slo_id}\")...

Show the answer

The literal is "slo_id" in the docs’ template, where it means “replace with the alphanumeric ID of the SLO” (docs.datadoghq.com/service_management/service_level_objectives/burn_rate/). In Terraform, that ID is datadog_service_level_objective.checkout_availability.id. A useful review habit: a green validate after a red one is not yet review-complete, because a red run may have been hiding a class of error.

After those six fixes, this is the run:

$ terraform validate
Success! The configuration is valid.

It is still wrong in three places. Everything from here on is invisible to terraform validate.

The burn-rate query: wrong .over(), no .short_window(), and no threshold block

# @expect-error validates; the burn_rate grammar and the missing monitor_thresholds are only enforced by the Datadog API
resource "datadog_monitor" "checkout_burn_rate" {
  name    = "Checkout SLO burn rate"
  type    = "slo alert"
  message = "Error budget is burning @pagerduty-checkout"
  query   = "burn_rate(\"${datadog_service_level_objective.checkout_availability.id}\").over(\"1h\").long_window(\"6h\") > 10"
}

query is an opaque string to both the provider and the client model. The documented grammar (docs.datadoghq.com/service_management/service_level_objectives/burn_rate/, quoted from the page): burn_rate("slo_id").over("time_window").long_window("1h").short_window("5m") > 14.4. Three deviations. .over("1h"): the docs say “replace time_window with one of 7d, 30d or 90d - depending on which target is used to configure your SLO”, so this SLO’s value is 30d. No .short_window(): the docs pair a long and a short window and recommend “keeping the short window as 1/12 of the long window” (1h and 5m in their example). And no monitor_thresholds. The provider’s own docs page for this shows the threshold beside the query — monitor_thresholds { critical = 14.4 } for a > 14.4 query. The same query as a monitor JSON constructs OK and deserializes with _unparsed = False (bug 10’s tooling): the model checks that query is a str, nothing more. I did not call the API, so the exact rejection message is not shown; the claim rests on the docs.

🐛 Spot the bug

  query   = "burn_rate(\"${...checkout_availability.id}\").over(\"1h\").long_window(\"6h\") > 10"

Show the answer

Green in terraform validate, green in the model, green everywhere offline. .over names the SLO’s target window (7d, 30d or 90d), a burn-rate alert wants both a long and a short window (the docs recommend the short one at 1/12 of the long one), and the threshold in the query should be repeated in monitor_thresholds.critical. Fix: .over("30d").long_window("1h").short_window("5m") > 14.4 with monitor_thresholds { critical = 14.4 } — the docs’ worked example.

The SLO numerator counts the bad events

# @expect-error validates; numerator must be GOOD events - the SLI is numerator / denominator
query {
  numerator   = "sum:checkout.requests{env:prod,outcome:error}.as_count()"
  denominator = "sum:checkout.requests{env:prod}.as_count()"
}

The metric name checkout.requests and its outcome tag are stand-ins for whichever count metric the service really emits. The mistake is structural. The provider’s own example uses numerator = "sum:my.custom.count.metric{type:good_events}.as_count()" over denominator = "sum:my.custom.count.metric{*}.as_count()", and the SLO docs: “A metric query uses the sum of the good events divided by the sum of total events over time to calculate a Service Level Indicator (or SLI)” (docs.datadoghq.com/service_management/service_level_objectives/metric/). With errors in the numerator, the “availability” is the error rate — a healthy service scores 0.1 percent against a 99.9 target and the SLO is permanently red; a broken one scores higher and looks better.

🐛 Spot the bug

  query {
  numerator   = "sum:checkout.requests{env:prod,outcome:error}.as_count()"
  denominator = "sum:checkout.requests{env:prod}.as_count()"
}

Show the answer

This is the most common metric-SLO mistake AI makes and no offline tool sees it: both strings are valid metric queries, the arguments are both strings, and validate prints Success!. Only the reader who knows the SLI is good divided by total catches it. Fix: outcome:success in the numerator (or an equivalent good-event filter), and confirm in the Datadog SLO editor’s preview that the number moves the right way.

The dashboard’s formula names a query that does not exist

# @expect-error validates; formula_expression "query2" refers to a name no query defines
request {
  query {
    metric_query {
      name        = "query1"
      data_source = "metrics"
      query       = "p95:trace.http.request{service:checkout,env:$env}"
    }
  }
  formula {
    formula_expression = "query2"
  }
  display_type = "line"
}

formula_expression is a free string that is evaluated against the names of the queries in the same request. Terraform cannot resolve it and the Python model for the equivalent JSON ("formulas": [{"formula": "query2"}]) constructs and deserializes cleanly (_unparsed = False); both were run. The UI would show a widget with an error; the API is the only party that can reject it, and I did not call it.

🐛 Spot the bug

        formula {
        formula_expression = "query2"
      }

Show the answer

A dangling reference in a string. query1 is the only defined name. Fix: formula_expression = "query1". In review, every formula_expression deserves a finger-trace back to a name.

The JSON monitor: a misplaced field the model accepts, and a string where a bool goes

$ python monitor-bad.json                # constructor with _check_type=True
ApiTypeError: Invalid type for variable 'notify_no_data'. Required value type is bool and passed type was str at ['options']['notify_no_data']

That is the model doing its job: "notify_no_data": "true" is wrong, loudly. Now remove that one error and run the model again — the top-level thresholds object stays:

monitor thresholds at top level   constructor: OK (constructs)   deserialize: _unparsed = False
monitor, critical="2" (string)    constructor: ApiTypeError: Invalid type for variable 'critical'. Required value type is float and passed type was str at ['options']['thresholds']['critical']

Terraform rejected the same idea (bug 2) as an unsupported block; the JSON model accepts an unknown top-level field without complaint on both paths. The real location is options.thresholds. With additionalProperties open, this is a misplaced field the model has no opinion on; what the API does with an unrecognised top-level property is not shown here (untested, no API), but the intended warning threshold is certainly not where the API reads it.

🐛 Spot the bug

{
"type": "metric alert",
"thresholds": { "critical": 2, "warning": 1.5 },
"options": { "notify_no_data": "true" }
}

Show the answer

Two bugs of opposite visibility in one object: the string bool is a hard ApiTypeError, the misplaced thresholds is invisible to both the constructor and the deserializer. Fix: "options": { "thresholds": { "critical": 2, "warning": 1.5 }, "notify_no_data": false }.

The JSON SLO and dashboard: the constructor says fine, _unparsed says otherwise

slo   type="metrics"              constructor: OK (constructs)   deserialize: _unparsed = True
slo   timeframe="30days"          constructor: OK (constructs)   deserialize: _unparsed = True
slo   target="99.9" (string)      constructor: OK (constructs)   deserialize: _unparsed = True
slo   numerator=123               constructor: ApiTypeError: Invalid type for variable 'numerator'. Required value type is str and passed type was int
slo   no thresholds               constructor: TypeError: ServiceLevelObjective.__init__() missing 1 required positional argument: 'thresholds'
dash  layout_type="grid"          constructor: OK (constructs)   deserialize: _unparsed = True
dash  display_type="lines"        constructor: OK (constructs)   deserialize: _unparsed = True
dash  widget definition, no type  constructor: OK (constructs)   deserialize: _unparsed = True
dash  no layout_type              constructor: TypeError: Dashboard.__init__() missing 1 required positional argument: 'layout_type'

Read the first three lines together. The constructor — what the course harness’s --json mode uses — accepts an invalid enum (type), an invalid enum (timeframe) and a string where a float goes inside a nested list item (target), because — as measured here for thresholds[] — the items of a nested list are not type-checked on construction. Only the deserializer notices, and only as a boolean flag with no message. The check that closes the gap is six lines, and it is worth adding to a CI job for JSON-as-code:

app/unparsed_check.py

import json
import sys

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v1.model.dashboard import Dashboard
from datadog_api_client.v1.model.monitor import Monitor
from datadog_api_client.v1.model.service_level_objective import ServiceLevelObjective

MODELS = {"monitor": Monitor, "dashboard": Dashboard, "slo": ServiceLevelObjective}


def check(kind: str, path: str) -> bool:
    """Deserialize like the real client does; _unparsed is set on enum/oneOf mismatches."""
    with open(path) as f:
        raw = f.read()
    obj = ApiClient(Configuration()).deserialize(raw, (MODELS[kind],), True)
    print(f"{path}: _unparsed = {obj._unparsed}")
    return not obj._unparsed


if __name__ == "__main__":
    ok = check(sys.argv[1], sys.argv[2])
    sys.exit(0 if ok else 1)

Run against the three bad documents it prints _unparsed = True for the SLO and the dashboard; run against the three fixed documents below it prints _unparsed = False each time. (_unparsed does not say which field: bisect by hand, as the table above did.) It still cannot see the semantic bugs — a numerator counting errors, a dangling formula — which stay a job for a reviewer and, finally, the API.

🐛 Spot the bug

{
"type": "metrics",
"thresholds": [{ "timeframe": "30days", "target": "99.9" }]
}

Show the answer

Three enum/type mistakes in two lines, each of which passes ServiceLevelObjective(**data, _check_type=True). This is the reverse of bug 3: the Terraform provider rejects 30days at validate time, the Python client model does not. Fix: "type": "metric", "timeframe": "30d", "target": 99.9. Do not treat a green --json harness run as proof for an enum or for a nested field.

The corrected code

terraform/checkout-observability.tf

resource "datadog_monitor" "checkout_latency" {
  name    = "Checkout p95 latency"
  type    = "metric alert"
  message = "p95 too high @pagerduty-checkout"
  query   = "avg(last_5m):p95:trace.http.request{service:checkout,env:prod} > 2"
  tags    = ["service:checkout", "team:payments"]

  monitor_thresholds {
    critical = 2
    warning  = 1.5
  }
}

resource "datadog_service_level_objective" "checkout_availability" {
  name = "Checkout availability"
  type = "metric"
  tags = ["service:checkout", "team:payments"]

  query {
    numerator   = "sum:checkout.requests{env:prod,outcome:success}.as_count()"
    denominator = "sum:checkout.requests{env:prod}.as_count()"
  }

  thresholds {
    timeframe = "30d"
    target    = 99.9
    warning   = 99.95
  }
}

resource "datadog_monitor" "checkout_burn_rate" {
  name    = "Checkout SLO burn rate"
  type    = "slo alert"
  message = "Error budget is burning @pagerduty-checkout"
  query   = "burn_rate(\"${datadog_service_level_objective.checkout_availability.id}\").over(\"30d\").long_window(\"1h\").short_window(\"5m\") > 14.4"
  tags    = ["service:checkout", "team:payments"]

  monitor_thresholds {
    critical = 14.4
  }
}

resource "datadog_dashboard" "checkout" {
  title       = "Checkout overview"
  layout_type = "ordered"

  template_variable {
    name   = "env"
    prefix = "env"
  }

  widget {
    timeseries_definition {
      title = "p95 latency"
      request {
        query {
          metric_query {
            name        = "query1"
            data_source = "metrics"
            query       = "p95:trace.http.request{service:checkout,env:$env}"
          }
        }
        formula {
          formula_expression = "query1"
        }
        display_type = "line"
      }
    }
  }
}

$ terraform validate                      # provider datadog 4.22.0, no plan, no apply
Success! The configuration is valid.

The JSON twins, all four collected and run through the course harness’s --json mode (and through the _unparsed check above, each False):

monitors/wr3-checkout-latency.json

{
  "name": "Checkout p95 latency",
  "type": "metric alert",
  "query": "avg(last_5m):p95:trace.http.request{service:checkout,env:prod} > 2",
  "message": "p95 too high @pagerduty-checkout",
  "tags": ["service:checkout", "team:payments"],
  "options": {
    "thresholds": { "critical": 2, "warning": 1.5 },
    "notify_no_data": false,
    "renotify_interval": 60
  }
}

monitors/wr3-checkout-burn-rate.json

{
  "name": "Checkout SLO burn rate",
  "type": "slo alert",
  "query": "burn_rate(\"<slo_id>\").over(\"30d\").long_window(\"1h\").short_window(\"5m\") > 14.4",
  "message": "Error budget is burning @pagerduty-checkout",
  "tags": ["service:checkout", "team:payments"],
  "options": { "thresholds": { "critical": 14.4 } }
}

slos/wr3-checkout-availability.json

{
  "name": "Checkout availability",
  "type": "metric",
  "query": {
    "numerator": "sum:checkout.requests{env:prod,outcome:success}.as_count()",
    "denominator": "sum:checkout.requests{env:prod}.as_count()"
  },
  "thresholds": [{ "timeframe": "30d", "target": 99.9, "warning": 99.95 }],
  "tags": ["service:checkout", "team:payments"]
}

dashboards/wr3-checkout-overview.json

{
  "title": "Checkout overview",
  "layout_type": "ordered",
  "widgets": [
    {
      "definition": {
        "type": "timeseries",
        "title": "p95 latency",
        "requests": [
          {
            "response_format": "timeseries",
            "queries": [
              { "data_source": "metrics", "name": "query1", "query": "p95:trace.http.request{service:checkout,env:prod}" }
            ],
            "formulas": [{ "formula": "query1" }],
            "display_type": "line"
          }
        ]
      }
    }
  ]
}

"<slo_id>" in the burn-rate JSON is a deliberate placeholder for the real alphanumeric ID; the model validates the shape, not the ID.

What changed and why

#	Bug	Fix	Caught by
1	tags = "..." (string)	list of strings	terraform validate
2	thresholds block on a monitor	monitor_thresholds	terraform validate
3	timeframe = "30days"	"30d"	terraform validate (provider enforces the enum)
4	Dashboard: no layout_type	layout_type = "ordered"	terraform validate
5	Dashboard: template_variables	template_variable	terraform validate (“Did you mean”)
6	.slo_id attribute	.id	terraform validate — second run only
7	Burn-rate query: .over("1h"), no .short_window(), no threshold	.over("30d")...short_window("5m") > 14.4 + monitor_thresholds	docs only; the live API
8	SLO numerator counts errors	count good events	docs only; a human; the SLO preview
9	Dashboard formula names query2	query1	the live API / UI only
10	JSON monitor: notify_no_data: "true", top-level thresholds	bool; options.thresholds	model catches the bool; nothing offline catches the misplaced field
11	JSON SLO/dashboard enums and nested string target	valid enums, numeric target	_unparsed on deserialize — not the constructor / --json

What the reviewer writes in the PR

Eleven things, and only six of them are what terraform validate will tell you:

1. Terraform shape (six, staged). tags is a list; monitors use monitor_thresholds; SLO timeframe is 30d; the dashboard needs layout_type and template_variable (singular); and the burn-rate query references .id, not .slo_id. Note the last one only appears after the others are fixed — re-run after each.
2. The burn-rate query validates and is wrong: .over("30d"), .long_window("1h"), .short_window("5m") > 14.4, and put 14.4 in monitor_thresholds.critical. Grammar per the SLO burn-rate docs.
3. The SLO numerator must be the good events. As written, the SLI is the error rate. Check the number in the SLO editor before merge.
4. formula_expression = "query2" points at nothing. query1.
5. JSON: options.thresholds, not a top-level thresholds (the client model silently accepts it); notify_no_data is a bool; SLO type is metric, timeframe 30d, target a number; dashboard layout_type ordered, display_type line. The harness --json check passes several of these — please run the _unparsed check too.
6. Not verified: nothing here was sent to Datadog. The API’s own rejection messages for items 2, 3 and 4 are unknown to me; please confirm on a sandbox org before applying.

Mental Model

flowchart TD
  A["AI-written .tf and JSON"] --> B["terraform validate"]
  A --> C["client model: constructor"]
  A --> D["client model: deserializer"]
  A --> E["Datadog API and docs"]
  B -->|"1, 2, 3, 4, 5, 6: shape, enums, references"| F["caught"]
  C -->|"10: string bool"| F
  D -->|"11: _unparsed enums and nested types"| F
  E -->|"7, 8, 9: grammar, meaning, dangling names"| G["needs the API or a person"]
  C -->|"10: misplaced field passes"| G

Which layer can see which bug

Layer	Sees	Blind to
terraform validate (provider 4.22.0)	argument and block names, types, some enums (SLOTimeframe), references — once earlier errors are gone	the contents of a query string, names inside formula_expression, what a metric means
Model constructor (--json harness mode)	required fields, top-level and first-level types	enum values, nested list items, unknown fields, query grammar
Model deserializer (_unparsed)	enum and oneOf mismatches, nested types	which field; unknown fields; query grammar
Live Datadog API / a reviewer with the docs	grammar, existence of IDs and names, whether the SLI moves the right way	— (and none of it was run here)

The contrast between bug 3 and bug 11 is the sharpest lesson: the same wrong value, 30days, is a hard error in Terraform and a green light in the Python constructor. A review process that trusts one “validation passed” for both is wrong for half of them.

Trade-offs

• Terraform vs raw JSON/API. Terraform gives a typed schema, validate, references between objects (.id) and a plan/apply workflow; raw JSON gives you exactly the API’s shape and none of Terraform’s checks. If you keep JSON in git, add the _unparsed check to CI, because the harness’s constructor check is the weaker one.
• Provider strictness vs API truth. The provider rejects 30days, but only because its authors copied the enum into the schema; the burn-rate grammar is a string, so provider and API disagree about how much they know. Prefer the provider for shape, and a sandbox apply (outside this review) for meaning.
• Multi-window burn-rate alerts vs a simple budget alert. The long-and-short window pair (1h and 5m in the docs’ example) pages faster and resolves faster than a single window, at the cost of two thresholds to reason about; the docs page above walks the trade-off.

Common Misconceptions

• “terraform validate passed, so the monitor is correct.” Proven above: the stage-3 file printed Success! while the SLO counted errors, the burn rate had no short window and the formula pointed at nothing.
• “If the first validate shows five errors, there are five errors.” A sixth was behind them: Terraform reports references only after structure is valid.
• “The --json harness run is green, so the SLO JSON is valid.” The constructor accepted type: "metrics", timeframe: "30days" and target: "99.9"; only the deserializer’s _unparsed = True disagreed.
• “The model rejects unknown fields.” It accepted a top-level thresholds on the monitor with _unparsed = False on both paths.
• “An API-shaped JSON is safer than Terraform because it is what the API takes.” It is the same API, with fewer offline checks.

🐶 The one-line takeaway

Monitors, SLOs and dashboards as code fail in three layers: structure (Terraform catches it, staged), vocabulary (the provider catches enums the Python constructor does not, and _unparsed catches the rest), and meaning (numerator, burn-rate grammar, formula names) which no offline tool sees — so say out loud in the PR which layer each finding came from.

The first terraform validate on the AI file shows five errors. After fixing all five, a sixth error appears (This object has no argument, nested block, or exported attribute named "slo_id"). Why was it not in the first run?

The provider has a bug that hides it Terraform reports structural errors first and evaluates references only after the configuration is structurally valid, so a red run can hide a class of error The attribute only exists in later provider versions validate only reports the first five errors

timeframe = "30days" on datadog_service_level_objective versus "timeframe": "30days" in the SLO JSON. What do provider 4.22.0 and the Python client constructor do?

Both accept it The provider rejects it at validate (invalid value '30days' for SLOTimeframe: valid values are [7d 30d 90d custom]); the constructor accepts it and only the deserializer sets _unparsed = True Both reject it The constructor rejects it, the provider accepts it

A metric SLO has numerator = "sum:checkout.requests{env:prod,outcome:error}.as_count()" over a total-requests denominator. What does terraform validate say, and what is wrong?

It rejects the tag filter Success! - both are valid metric-query strings; the SLI is good events divided by total events, so this measures the error rate, and a healthy service scores far below a 99.9 target It reports that the numerator must come first It warns that .as_count() is deprecated

The burn-rate monitor query is burn_rate("<id>").over("1h").long_window("6h") > 10. Which statement is accurate?

terraform validate rejects it because .short_window() is missing Validate and the client model both accept it (the query is an opaque string); the documented form is .over(<SLO window>).long_window("1h").short_window("5m") > 14.4 with a matching monitor_thresholds.critical, and only the live API or a reviewer with the docs can tell The client model's deserializer reports _unparsed = True It is valid because .over takes the alert window

A monitor JSON has "thresholds": { "critical": 2 } at the top level instead of under options. What do the client model paths report?

ApiTypeError from the constructor Nothing: the constructor prints OK and the deserializer reports _unparsed = False - unknown top-level fields are accepted, unlike the same mistake in Terraform, which fails as an unsupported thresholds block _unparsed = True A TypeError for a missing argument

"thresholds": [{ "timeframe": "30d", "target": "99.9" }] (a string target) in an SLO JSON. What does the constructor with _check_type=True do?

Raises ApiTypeError for the string target Constructs fine, because items of a nested list are not type-checked on construction; only the deserializer's _unparsed = True disagrees, with no message about which field Coerces the string to a number Raises a TypeError for a missing argument

Submit

Mark as complete