Synthetics and Uptime Checks

The idea in one sentence

Synthetic Monitoring runs scripted checks proactively from Datadog’s own global network of locations instead of waiting for real user or backend traffic to reveal a problem, and once a test fails, it feeds into the exact same monitor and notification pipeline already covered earlier in this module.

API tests: single request-response checks

An API test is a single HTTP/TCP/DNS/ICMP request-response check — for example, hitting a health-check endpoint and asserting on the status code, response time, or body content. It runs repeatedly from whichever locations you choose.

{
  "name": "Checkout health check",
  "type": "api",
  "subtype": "http",
  "config": {
    "request": {
      "method": "GET",
      "url": "https://checkout.example.com/health"
    },
    "assertions": [
      { "type": "statusCode", "operator": "is", "target": 200 },
      { "type": "responseTime", "operator": "lessThan", "target": 300 }
    ]
  },
  "locations": ["aws:ap-southeast-1", "aws:eu-west-1"],
  "message": "@pagerduty-checkout-critical Checkout health check is failing.",
  "options": {
    "tick_every": 60,
    "retry": { "count": 2, "interval": 300 }
  },
  "status": "live"
}

🐶 Proof: this is the real SyntheticsAPITest model, checked field by field

Installed the pinned datadog-api-client==2.61.0 and listed the real, current SyntheticsAPITest.attribute_map directly from the installed package: config, locations, message, monitor_id, name, options, public_id, status, subtype, tags, type — every key in the fence above is real. Ran the fence through SyntheticsAPITest(**data, _check_type=True): {"ok": true, "error": null}.

Browser tests: full scripted user journeys

A Browser test runs a full scripted browser flow across multiple steps — load a page, click a login button, assert a dashboard element appears — to test a real user journey end to end. This catches things an API test physically cannot: a page can return 200 and still render a broken UI, or a flow can fail partway through even though every individual request behind it “succeeded.”

{
  "name": "Login flow",
  "type": "browser",
  "config": {
    "request": { "method": "GET", "url": "https://app.example.com/login" },
    "assertions": []
  },
  "steps": [
    { "type": "goToUrl", "params": { "value": "https://app.example.com/login" } },
    { "type": "click", "params": { "element": "#login-button" } },
    { "type": "assertElementPresent", "params": { "element": "#dashboard" } }
  ],
  "locations": ["aws:us-east-1"],
  "message": "@slack-app-oncall Login flow is failing.",
  "options": {
    "tick_every": 900,
    "retry": { "count": 1, "interval": 300 }
  },
  "status": "live"
}

🐶 Proof: SyntheticsBrowserTest, and a real error when message is left out

SyntheticsBrowserTest.attribute_map (same pinned client): config, locations, message, monitor_id, name, options, public_id, status, steps, tags, type — the fence above validates cleanly with _check_type=True. Tried the same payload with message omitted (an easy AI-generated oversight — nothing about “what does the page/click/assert flow look like” obviously implies a notification field is required): SyntheticsBrowserTest(**data, _check_type=True) raises a real TypeError: SyntheticsBrowserTest.__init__() missing 1 required positional argument: 'message'. message is required at construction time for both test types — there is no such thing as a Synthetic test with no one to notify, structurally, in the client itself.

Assertions: the real operator and type vocabulary

Both fences above use type/operator/target triples. Read the real, installed enums directly instead of guessing which operators exist — SyntheticsAssertionOperator.allowed_values: is, isNot, contains, doesNotContain, matches, doesNotMatch, lessThan, lessThanOrEqual, moreThan, moreThanOrEqual, isInLessThan, isInMoreThan, isUndefined, doesNotExist, validates. SyntheticsAssertionType.allowed_values includes statusCode, responseTime, header, body, certificate, tlsVersion, latency, packetLossPercentage, grpcHealthcheckStatus, and more — a much larger real vocabulary than the two types this lesson’s examples use, covering TCP/gRPC/certificate checks alongside plain HTTP.

Locations and private locations

The locations array (aws:ap-southeast-1, aws:eu-west-1, aws:us-east-1 above) references Datadog’s public managed location network. For internal-only endpoints, docs.datadoghq.com/synthetics/private_locations/ describes a separate mechanism: “Private locations allow you to monitor internal-facing applications or any private endpoints that aren’t accessible from the public internet.” Concretely, “Private locations come as Docker containers or Windows services that you can install inside of your private network… Your private location worker pulls your test configurations from Datadog’s servers using HTTPS, executes the test on a schedule or on-demand, and returns the test results to Datadog’s servers.” A private location’s ID goes into the exact same locations array as a public one — from the test definition’s point of view, a private location is just another string in that list, even though what runs the check is infrastructure you deployed yourself rather than Datadog’s own network.

Proof: the min_location_failed default trap

Running a test from multiple locations (both fences above use two or three) is often assumed to mean “we won’t get a false alarm from one flaky region.” The real SyntheticsTestOptions model has a field specifically for tuning that: min_location_failed. Datadog’s own Terraform provider documentation states its default plainly: “min_location_failed (Number) Minimum number of locations in failure required to trigger an alert. Defaults to 1.”

That default means a test configured to run from three locations — intended, in spirit, to average out one region’s transient network blip — alerts the instant any single one of those three locations fails, unless min_location_failed is explicitly raised. Multiple locations improve coverage (you find out if the endpoint is unreachable from one specific region) but do nothing for alert stability on their own — that second property depends entirely on min_location_failed, a separate field most people configuring “more locations for reliability” never touch. This is the same shape of trap as notify_no_data’s requirement to have reported once already, and Terraform’s own resource docs are where it’s stated most plainly, not the JSON API reference.

{
  "options": {
    "tick_every": 60,
    "retry": { "count": 2, "interval": 300 },
    "min_location_failed": 2
  }
}

Retry: absorbing a transient failure before it pages anyone

options.retry ({count, interval}, confirmed real on SyntheticsTestOptionsRetry.attribute_map) is a different mechanism from min_location_failed — it retries the same location’s check count more times, interval milliseconds apart, before considering that one location’s result a failure at all. The two compose: retry absorbs a single location’s momentary blip; min_location_failed decides how many locations, after their own retries are exhausted, must agree something is actually wrong before anyone is notified.

Same alerting pipeline, different signal source

A failing Synthetic test triggers a synthetics alert monitor — and a synthetics alert monitor uses the exact same threshold, notification, and escalation mechanics taught earlier in this module. Critical/warning thresholds, @-mentions, escalation_message, and Downtimes for planned maintenance all work identically; only the input feeding the monitor is different.

monitors/checkout-synthetics-alert.json

{
  "type": "synthetics alert",
  "query": "\"synthetics\" for test_public_id in (\"abc-def-ghi\") \"failure\" over \"5m\" last \"3\" check greater than 2",
  "options": {
    "thresholds": {
      "critical": 2,
      "warning": 1
    },
    "escalation_message": "@pagerduty-checkout-escalation Still failing, escalating."
  },
  "message": "@slack-checkout-oncall @pagerduty-checkout-critical Checkout health check is failing from multiple locations."
}

🐶 Proof: the synthetics alert monitor validates against the same real Monitor model as every other monitor type in this module

Ran the fence above through Monitor(**data, _check_type=True): {"ok": true, "error": null}. Same model, same shape as the metric alert, log alert, and composite monitor fences from earlier lessons — proving “same pipeline” is literally the same client-side type, not just similar-looking JSON.

Terraform: datadog_synthetics_test

terraform/synthetics-checkout-health.tf

resource "datadog_synthetics_test" "checkout_health" {
  name      = "Checkout health check"
  type      = "api"
  subtype   = "http"
  status    = "live"
  message   = "@pagerduty-checkout-critical Checkout health check is failing."
  locations = ["aws:ap-southeast-1", "aws:eu-west-1"]
  tags      = ["service:checkout-service", "env:production"]

  request_definition {
    method = "GET"
    url    = "https://checkout.example.com/health"
  }

  assertion {
    type     = "statusCode"
    operator = "is"
    target   = "200"
  }

  assertion {
    type     = "responseTime"
    operator = "lessThan"
    target   = "300"
  }

  options_list {
    tick_every         = 60
    min_location_failed = 2
    retry {
      count    = 2
      interval = 300
    }
  }
}

$ terraform init && terraform validate
Success! The configuration is valid.

Ran terraform init + terraform validate against the pinned datadog provider 4.22.0 (fake credentials, validate = false, no real API call): clean. Tried a deliberately broken type value to see a real error, since the HCL shape here (request_definition/assertion/options_list blocks, distinct from the JSON API’s nested config object) is different enough from the JSON payload shape that it is worth checking separately, not assumed to match:

# @expect-error terraform validate real error: Invalid enum value — 'invalid value "not-a-real-type" for SyntheticsTestDetailsType: valid values are [api browser mobile network]'
resource "datadog_synthetics_test" "broken_test" {
  name      = "Broken test"
  type      = "not-a-real-type"
  status    = "live"
  message   = "test"
  locations = ["aws:eu-central-1"]

  request_definition {
    method = "GET"
    url    = "https://example.org"
  }
}

The real error names the exact Go-side type (SyntheticsTestDetailsType) and lists the actual valid values — api, browser, mobile, network — confirming this Terraform resource covers the same four synthetic test families the JSON API’s type field does, from a completely independent, client-side schema check.

flowchart LR
  Loc1[Location: ap-southeast-1] --> Test[Synthetic test: API or Browser]
  Loc2[Location: eu-west-1] --> Test
  Test -->|retry absorbs one blip| Retried[Per-location result]
  Retried -->|min_location_failed locations agree| History[Test result history]
  History --> Mon[synthetics alert monitor]
  Mon -->|critical/warning thresholds| Notify[Same notification/escalation pipeline as metric and composite monitors]

From Synthetic test run to alert

🐶 The one-line takeaway

Synthetic Monitoring runs API tests (validated above against the real SyntheticsAPITest model) and Browser tests (SyntheticsBrowserTest, which requires message at construction — proven above) proactively from Datadog’s global or private locations, and a failing test triggers a synthetics alert monitor that validates against the exact same Monitor model as every other monitor type in this module. retry absorbs one location’s transient blip; min_location_failed (default 1, proven above from the docs) decides how many locations must actually agree before anyone is paged — running from more locations alone does not raise that bar.

Reading AI-generated code

🐛 Spot the bug

// AI comment: "Run from 3 locations for redundancy so one flaky region doesn't page us"
{
"name": "Checkout health check",
"type": "api",
"subtype": "http",
"config": {
  "request": { "method": "GET", "url": "https://checkout.example.com/health" },
  "assertions": [
    { "type": "statusCode", "operator": "is", "target": 200 }
  ]
},
"locations": ["aws:ap-southeast-1", "aws:eu-west-1", "aws:us-east-1"],
"message": "@pagerduty-checkout-critical Checkout health check is failing.",
"options": { "tick_every": 60 }
}

Show the answer

This validates cleanly against the real SyntheticsAPITest model — proven earlier in this lesson, three real location strings, a real assertion, nothing structurally wrong. The bug is exactly what the AI-generated comment claims to solve: the comment says three locations mean “one flaky region doesn’t page us,” but proven above from Datadog’s own Terraform provider docs, min_location_failed defaults to 1 — a single location failing, out of any number configured, triggers the alert. Adding locations here improved coverage (more vantage points) but did nothing for the exact alert-stability goal the comment states. The fix: add "min_location_failed": 2 (or higher) inside options to actually require more than one location to agree before paging — and separately, options.retry if the goal is also to absorb one location’s single transient blip before even counting it as a failure.

Mental Model

A Synthetic test’s result pipeline has three independent filters stacked in order, and conflating them is the whole trap this lesson proves. Retry is the first filter — “was this one location’s one check just a blip?” — applied per-location, before that location’s result is even finalized. min_location_failed is the second filter — “how many already-finalized location results have to agree it’s actually down?” — applied across locations, after retry. The synthetics alert monitor’s own thresholds are the third filter, identical in shape to every other monitor type in this module, applied to the test’s result history over time. Running from more locations only strengthens the first two filters if min_location_failed is tuned to match; by itself, more locations just means more independent chances for filter one to fail through to filter two.

Trade-offs

• API test vs. Browser test for the same endpoint — an API test is cheaper to run and simpler to reason about, but proven above, it cannot catch a 200-status page rendering a broken UI; a Browser test catches that class of failure at the cost of being slower, more expensive, and more fragile to unrelated UI changes breaking the script.
• More locations vs. a higher min_location_failed — more locations alone (proven above) widens coverage without changing alert stability; raising min_location_failed to match location count reduces false pages from one region’s transient issue, at the cost of a real regional outage needing to spread to more locations before anyone is paged.
• Public managed locations vs. private locations — public locations need zero infrastructure and cover the public internet’s vantage point; private locations (proven above, real Docker/Windows worker architecture) are required for internal-only endpoints, at the cost of running and maintaining that worker yourself.

Common Misconceptions

• “Running a test from more locations automatically makes alerts more stable against one flaky region.” No — proven above from Datadog’s own docs: min_location_failed defaults to 1, so any single location failing triggers the alert regardless of how many locations are configured; stability requires explicitly raising that field.
• “retry and min_location_failed are two names for the same thing.” No — proven above: retry re-checks the same location before finalizing its result; min_location_failed counts how many already-finalized locations must agree, after retries are done.
• “A Browser test’s JSON payload will construct successfully as long as steps is filled in correctly.” No — proven above: SyntheticsBrowserTest requires message at construction time regardless of how correct steps is; a payload missing it raises a real TypeError before steps is even considered.
• “A datadog_synthetics_test Terraform resource with an unsupported type value fails only at apply time, against the real API.” No — proven above: terraform validate catches an invalid type enum value entirely client-side, no API call involved, naming the real valid values directly in the error.

What is the key difference between an API test and a Browser test in Synthetic Monitoring?

API tests only run from one location, Browser tests run from many An API test is a single request-response check (HTTP/TCP/DNS/ICMP), a Browser test runs a full scripted browser flow across multiple steps Browser tests cannot make assertions on response content API tests require a monitor, Browser tests do not

A Synthetics API test JSON payload validates successfully against the pinned SyntheticsAPITest model. What did the equivalent test with the message field omitted prove?

It also validates successfully -- message is optional for both test types It raises a real TypeError -- message is a required positional argument for SyntheticsBrowserTest (and SyntheticsAPITest) construction It silently defaults message to an empty string It only fails for Browser tests, never for API tests

A test runs from 3 locations with no min_location_failed set. Proven above by quoting Datadog's own Terraform provider docs, what actually triggers the alert?

All 3 locations must fail before an alert fires A majority (2 of 3) locations must fail Any single one of the 3 locations failing triggers the alert, because min_location_failed defaults to 1 The test never alerts unless min_location_failed is explicitly set

How does options.retry differ from min_location_failed, proven above from the real model fields?

They are exactly the same setting under two different names retry re-checks the same location before finalizing its result; min_location_failed counts how many already-finalized locations must agree afterward retry only applies to Browser tests, min_location_failed only to API tests min_location_failed controls retry's count and interval

A datadog_synthetics_test Terraform resource is given type = "not-a-real-type". What did the real terraform validate run against the pinned provider 4.22.0 prove?

terraform plan succeeds but terraform apply fails against the real API terraform validate fails entirely client-side with a real error naming the valid values: api, browser, mobile, network Terraform silently ignores the invalid type and defaults to "api" terraform validate only checks required fields are present, not enum values

When a Synthetic test starts failing, how does it typically notify someone?

Synthetics has its own separate notification system unrelated to monitors Through a synthetics alert monitor, using the same thresholds/notification/escalation mechanics as any other monitor type It can only send an email, no Slack or PagerDuty integration It automatically creates a composite monitor

Submit

Mark as complete