SLOs, Error Budgets, and Burn Rate Alerts

The idea in one sentence

A Datadog SLO turns a target like “99.9% good over 30 days” into a trackable error budget, and Datadog gives you two different kinds of SLO alert because “how much budget is gone” and “how fast it’s going” are two different problems.

Three ways to define an SLO

Metric-based — a ratio of two metric queries, good events over total events:

slos/checkout-availability.json

{
  "name": "Checkout API availability",
  "type": "metric",
  "query": {
    "numerator": "sum:trace.http.request.hits{service:checkout-api,!http.status_code:5*}.as_count()",
    "denominator": "sum:trace.http.request.hits{service:checkout-api}.as_count()"
  },
  "thresholds": [
    { "timeframe": "30d", "target": 99.9 }
  ]
}

Monitor-based — built from the uptime history of one or more existing monitors. The SLO’s “good” time is however much of the window those monitors spent not triggered:

slos/checkout-monitor-based.json

{
  "name": "Checkout API monitor-based SLO",
  "type": "monitor",
  "monitor_ids": [111111, 222222],
  "thresholds": [
    { "timeframe": "30d", "target": 99.9 }
  ]
}

Time Slice — evaluates a query per fixed time slice across the window, for example “was p99 latency under 300ms in each 5-minute slice”:

slos/checkout-p99-latency.json

{
  "name": "Checkout API p99 latency SLO",
  "type": "time_slice",
  "sli_specification": {
    "time_slice": {
      "query": {
        "formula": "query1",
        "queries": ["p99:trace.http.request.duration{service:checkout-api}"]
      },
      "comparator": "<",
      "threshold": 300,
      "query_interval_seconds": 300
    }
  },
  "thresholds": [
    { "timeframe": "30d", "target": 99.9 }
  ]
}

🐶 Proof: all three validate against the real ServiceLevelObjective model

Installed datadog-api-client==2.61.0 (this course’s pinned version) and ran the exact validation the harness’s --json mode performs — ServiceLevelObjective(**data, _check_type=True) — against all three JSON fences above, byte-for-byte. All three: {"ok": true, "error": null}. thresholds is a required field on every SLO type in this client version (a metric/monitor/time_slice SLO missing it fails to construct at all), which is why all three fences above carry one even though the earlier draft of this lesson omitted it.

The error budget

A target like 99.9% over a 30-day window implies an allowed amount of “bad” time or events — that allowance is the error budget.

Target: 99.9% over 30d
30d = 43200 minutes
Allowed bad time (error budget) = 0.1% x 43200 = 43.2 minutes

Every bad event or minute of bad time consumes part of that 43.2-minute budget. Once the budget hits 0%, the window has already fallen below its 99.9% target.

Error budget alerts vs. burn rate alerts

This is the distinction that matters most in this lesson, and Datadog gives you two separate SLO alert types for it.

Error budget alert — threshold-based on the cumulative percentage of the budget consumed so far in the window. For example, alert once 75% of the budget is gone, and optionally warn earlier at 50%:

monitors/checkout-error-budget-alert.json

{
  "name": "checkout-api error budget consumption",
  "type": "slo alert",
  "query": "error_budget(\"<slo_id>\").over(\"30d\") > 75",
  "message": "@slack-checkout-oncall 75% of the 30d error budget for checkout-api is already consumed."
}

Burn rate alert — notifies when the rate of error-budget consumption exceeds a threshold and stays sustained over a period. The real query shape needs three pieces, not two: .over(), .long_window(), and .short_window():

monitors/checkout-burn-rate-alert.json

{
  "name": "checkout-api SLO burn rate (fast)",
  "type": "slo alert",
  "query": "burn_rate(\"<slo_id>\").over(\"30d\").long_window(\"6h\").short_window(\"30m\") > 10",
  "message": "@pagerduty-checkout-critical Burn rate is 10x normal - at this rate the entire 30d budget is gone in about 3 days."
}

🐶 Proof: the real burn_rate() shape, and what was wrong with the old one

docs.datadoghq.com/service_management/service_level_objectives/burn_rate/ gives this exact worked example: burn_rate("slo_id").over("time_window").long_window("1h").short_window("5m") > 14.4, and states .over() must be “replaced with one of 7d, 30d or 90d — depending on which target is used to configure your SLO” — .over() is the SLO’s own window, not an alerting window. A previous draft of this lesson wrote burn_rate("<slo_id>").over("1h").long_window("6h") > 10: "1h" in .over() (should have been "30d", matching the SLO defined above) and no .short_window() at all, which the same docs page requires. Neither mistake throws a JSON-shape error — --json’s ServiceLevelObjective/Monitor model validation checks required fields and field types, but query is just a str to the client, so a malformed Datadog query string inside it passes model validation either way (confirmed by running both the fixed and the old broken query string through the pinned Monitor model — both come back {"ok": true}). Catching this class of bug needs either a real API call or reading the docs’ query grammar directly — which is exactly why this lesson quotes the URL instead of only trusting model validation.

Choosing the burn rate threshold and window sizes

Datadog’s own docs give one fully worked, real example for a 30-day SLO target: burn rate 14.4, long_window("1h"), short_window("5m"), and label it “theoretical error budget consumed: 2%” over that long window. The same page gives the underlying formula (burn rate = (SLO target length in hours × % of budget consumed) / (long window in hours × 100)) and a second, equivalent way to think about it (length of SLO target / burn rate = time until the error budget is fully exhausted at that rate). Both formulas check out against this lesson’s own example, burn_rate(...).over("30d").long_window("6h") > 10:

Approach 1 (time to exhaustion):  30d / burn_rate(10)        = 3 days
Approach 2 (budget consumed over one long_window):
  (long_window_hours x burn_rate x 100) / (SLO_target_hours)
  = (6 x 10 x 100) / (30 x 24)  =  6000 / 720  =  8.33%

That 8.33%-per-6-hours figure is exactly why the message text in the fence above says “gone in about 3 days” — both formulas agree. docs.datadoghq.com’s own worked example follows the same math with different numbers (long_window=1h, burn_rate=14.4, over=30d → (1 x 14.4 x 100)/720 = 2%, matching the docs page’s own stated “2%” exactly).

Extending the same math to a small multi-window table (short window kept at Google’s recommended ~1/12 of the long window, a ratio the docs page states directly):

| Sensitivity | burn_rate | long_window | short_window (~1/12) | over | budget consumed / long_window | time to exhaust budget |
|-------------|-----------|-------------|-----------------------|------|--------------------------------|--------------------------|
| Fast page   | 14.4      | 1h          | 5m                    | 30d  | 2.00%  (docs' own example)      | 30d / 14.4 = ~2.08 days |
| This lesson | 10        | 6h          | 30m                   | 30d  | 8.33%                            | 30d / 10   = 3 days     |
| Slow ticket | 1         | 3d (72h)    | 6h                    | 30d  | 10.00%                           | 30d / 1    = 30 days    |

A larger burn rate paired with a shorter long_window catches a severe, fast-moving incident within minutes; a burn rate near 1 paired with a multi-day long_window is the “budget is on track to run out right on schedule” signal — worth a ticket, not a page. Running two or three of these side by side against the same SLO (one fast/paging, one slow/ticketing) is the standard pattern: a single burn-rate monitor can only tell you about the sensitivity it was tuned for.

The two alert types solve different problems. An error budget alert tells you “we’re running out” — it’s about the total consumed so far, useful for deciding whether it’s safe to ship a risky change this week. A burn rate alert tells you “we’re running out fast, right now” — it’s about velocity, not total. A sustained fast burn can exhaust an entire 30-day budget in hours, and a burn rate alert catches that within the first hour, well before the cumulative-consumption threshold that an error budget alert watches would ever cross.

flowchart LR
  SLO[SLO target 99.9% over 30d] --> Budget[Error budget]
  Budget --> EBA{Error budget alert: percent consumed above threshold}
  Budget --> BRA{Burn rate alert: consumption rate above threshold, sustained}
  EBA -->|e.g. 75 percent consumed| Notify1[Notify: budget running out]
  BRA -->|e.g. 10x burn rate for 1h| Notify2[Notify: fast incident right now]

Error budget alert vs. burn rate alert

🐶 The one-line takeaway

An error budget alert is threshold-based on the cumulative percentage of the error budget consumed so far in the window and tells you “we’re running out,” while a burn rate alert — real shape burn_rate("<slo_id>").over("<slo_window>").long_window("<h>").short_window("<h/12>"), with .over() always matching the SLO’s own 7d/30d/90d target — watches the sustained rate of consumption and tells you “we’re running out fast, right now.”

Reading AI-generated code

🐛 Spot the bug

// AI comment: "Page the on-call when the error budget is burning 10x too fast"
{
"name": "checkout-api SLO burn rate (fast)",
"type": "slo alert",
"query": "burn_rate(\"<slo_id>\").over(\"1h\").long_window(\"6h\") > 10",
"message": "@pagerduty-checkout-critical Burn rate is 10x normal."
}

Show the answer

This is valid JSON, and it passes Monitor(**data, _check_type=True) model validation cleanly — proven above, the client only checks that query is a str, not that its contents are a well-formed Datadog query grammar. Two real bugs hide inside that string. First, .over("1h") — .over() must be the SLO’s own configured window (7d/30d/90d), never an alerting window; "1h" isn’t even a valid SLO target length. Second, there is no .short_window() at all, which the burn-rate query grammar requires. Datadog’s docs (docs.datadoghq.com/service_management/service_level_objectives/burn_rate/) reject a query missing either piece. The fix, matching the SLO defined earlier in this lesson: burn_rate("<slo_id>").over("30d").long_window("6h").short_window("30m") > 10. Neither the AI-generated comment nor a JSON-shape validator would ever catch this — only reading the actual query grammar, or a real API call, does.

Mental Model

An SLO is an odometer with a fuel gauge bolted on. The odometer (the SLO’s own metric/monitor/time-slice query) just measures “good” vs “bad” continuously. The fuel gauge (the error budget) converts that into “how much bad time are we allowed, and how much is left.” An error budget alert reads the gauge itself — a threshold on the level. A burn rate alert watches how fast the needle is moving — a threshold on the rate of change, sustained over a window. A car can have a full tank (budget barely consumed) while burning fuel at a rate that would empty it in an hour if kept up, and a burn rate alert is the only one of the two that notices that in time to matter.

Trade-offs

• Metric-based vs. monitor-based vs. time-slice SLOs — metric-based is the most flexible (any ratio of two queries) but requires the “good”/“total” split to already exist as separate metric queries; monitor-based reuses monitors you already have but inherits whatever thresholds those monitors were tuned for; time-slice is the only one of the three that can express “must hold a threshold continuously” rather than “on average,” at the cost of a slightly more involved query shape (proven above: sli_specification.time_slice nests a formula/query pair plus a comparator/threshold).
• Error budget alerts vs. burn rate alerts — an error budget alert is simpler to reason about (one cumulative percentage) but only fires after real damage is already done; a burn rate alert catches trouble early but needs two numbers tuned correctly (long_window, short_window) and, proven above, a .over() that must exactly match the SLO’s own target window or the query doesn’t make sense.
• One burn-rate monitor vs. a fast+slow pair — a single burn-rate monitor tuned for fast incidents (proven above: 14.4/1h/5m, ~2% budget per window) misses a slow, steady leak that would exhaust the same budget over days; running a second, slower monitor (1/3d/6h in this lesson’s table) catches that case too, at the cost of maintaining and tuning two monitors per SLO instead of one.

Common Misconceptions

• “.over() in a burn rate query is the alerting window (how long the burn must be sustained).” No — proven above, quoting the docs directly: .over() is the SLO’s own configured target window (7d/30d/90d). The sustained-duration piece is .long_window().
• “A burn rate query without .short_window() is still valid, just less precise.” No — the docs’ worked example and every real burn-rate query require both .long_window() and .short_window(); a query missing .short_window() (proven above, this lesson’s old draft) is malformed, not merely imprecise.
• “JSON model validation (datadog_api_client) would catch a broken burn_rate() query string.” No — proven above by running both the fixed and the broken query text through the real, pinned Monitor model: both return {"ok": true}. The client validates field types (query must be a str), not the Datadog query grammar inside that string.
• “An SLO missing thresholds is still a usable SLO document.” No — proven above: ServiceLevelObjective(**data, _check_type=True) requires thresholds for every SLO type in this pinned client version; a document without it fails to construct.

What does a Metric-based SLO measure?

The uptime history of a single existing monitor A ratio between two metric queries, e.g. good events over total events Whether a query stayed under a threshold in every fixed time slice The number of Synthetic test locations that passed

A Time Slice SLO checking "was p99 latency under 300ms in each 5-minute window" evaluates its condition

Once at the end of the full 30d window Continuously in real time with no fixed boundaries Per fixed time slice across the window, e.g. once per 5-minute slice Only when a related monitor is in the Alert state

An SLO alert configured to notify when 75% of the 30d error budget has been consumed is an example of

A burn rate alert An error budget alert A composite monitor A Time Slice SLO

What is the correct, real shape of a burn rate alert query for an SLO defined over a 30-day window, alerting on a sustained 10x burn measured over 6 hours?

burn_rate("<slo_id>").over("1h").long_window("6h") > 10 burn_rate("<slo_id>").over("30d").long_window("6h").short_window("30m") > 10 error_budget("<slo_id>").over("30d").long_window("6h") > 10 burn_rate("<slo_id>").over("6h").short_window("30m") > 10

A monitor JSON with query set to the OLD, broken burn_rate("<slo_id>").over("1h").long_window("6h") > 10 (missing short_window, wrong .over()) is run through datadog_api_client's Monitor(**data, _check_type=True). What happens?

It raises an ApiValueError because the query string is malformed It raises an ApiTypeError because query must be an object, not a string It validates successfully -- the client only checks that query is a str, not the Datadog query grammar inside it It raises a JSONDecodeError

Why can a burn rate alert catch an incident earlier than an error budget alert would?

Burn rate alerts always use a shorter measurement window than error budget alerts Burn rate alerts measure how fast the budget is being consumed, so a sustained fast burn triggers it well before the cumulative consumed percentage crosses the error budget alert threshold Burn rate alerts do not depend on the SLO at all Error budget alerts only run once every 30 days

Submit

Mark as complete