Cost Governance and Cloud Cost Management

The idea in one sentence

Datadog cost is driven by what you choose to keep indexed and queryable — distinct custom-metric time series and indexed log events, per the billing docs — and Cloud Cost Management brings your actual cloud bill into the same query-and-monitor model.

Recap: two different cost levers, one theme

Two earlier modules each taught a cost lever:

Custom metrics (Infrastructure & Metrics, Metrics without Limits). The billing page (https://docs.datadoghq.com/account_management/billing/custom_metrics/) says: “A custom metric is uniquely identified by a combination of a metric name and tag values (including the host tag)”, and monthly billable usage “is calculated by taking the total of all distinct custom metrics (also known as timeseries) for each hour in a given month, and dividing it by the number of hours in the month to compute a monthly average value. Your billable usage is not impacted by data point submission frequency or the number of queries you run on your metrics.” So the driver is distinct time series (tag cardinality), not points per second. Two caveats from the same page: it describes the “cardinality-based Custom Metrics billing model” and points contracts that use Metric Name pricing SKUs elsewhere; and for metrics configured with Metrics without Limits, “you pay for ingested custom metrics and indexed custom metrics”, not just the indexed view.
Logs (Log Management, indexes and exclusion filters). The log billing page (https://docs.datadoghq.com/account_management/billing/log_management/) says: “At the end of the month, Datadog computes the total number of log events that have been indexed”. Below your commitment “your bill stays the same”; over it, “the committed amount is subtracted and on demand usage is charged with a 50% premium.” That page describes indexed-event billing only; it does not describe other log line items, so read your contract for anything else (ingestion, rehydration, and so on) rather than assuming they are free.

The common thread the docs support: the billing pages count retained, queryable things (distinct series, indexed events), not raw traffic. That is why an exclusion filter or a Metrics without Limits tag configuration is a cost decision. It does not support the stronger claim that Datadog never charges for anything that merely arrives; do not write that in a design doc.

Custom metrics: the allotments, quoted

Same billing page, with the numbers as the docs print them:

For metrics not configured with Metrics without Limits: “Account allotment: Pro: 100 indexed custom metrics per host; Enterprise: 200 indexed custom metrics per host”. Above the allotment, “for each 100 indexed custom metrics over the account allotment, you pay an amount that is specified in your current contract.”
For metrics configured with Metrics without Limits: ingested allotment is the same 100 (Pro) or 200 (Enterprise) per host, and “For each 100 ingested custom metrics over the account allotment, you pay $0.10.” Indexed overage again follows your contract.

The docs express allotments per host, so a fleet of many small hosts and a fleet of a few large ones get very different headroom for the same metric design. That is a good reason to estimate cardinality before go-live rather than reading the invoice afterwards.

Cloud Cost Management: your cloud bill as a metric source

The Cloud Cost Management page (https://docs.datadoghq.com/cloud_cost_management/) says: “Datadog ingests your cloud cost data and transforms it into metrics you can use in a search query on the Explorer page. If costs rise, you can correlate the increase with usage metrics to determine the root cause.” Setup pages exist for AWS, Azure, Google Cloud, Oracle, SaaS and AI costs and custom costs. Cost data is kept “with a retention period of 15 months” (same page).

For AWS (https://docs.datadoghq.com/cloud_cost_management/setup/aws/) the prerequisites are an AWS account with billing access, the AWS integration installed in Datadog, and a Cost and Usage Report; the docs also say “It may take between 48 and 72 hours for all available data to populate in your Datadog organization after a complete Cost and Usage Report is generated.” Datadog recommends the report come from the AWS management account, because “A delegated administrator account cannot discover other member accounts.”

Metric choice matters and the docs list them (same AWS page):

aws.cost.net.amortized.shared.resources.allocated “provides the most complete cost allocation for specific workloads and teams” and requires container cost allocation;
“If you do not have container cost allocation, use aws.cost.net.amortized”;
“If you do not have net amortized costs, use aws.cost.amortized.shared.resources.allocated or aws.cost.amortized”;
aws.cost.unblended “Matches the AWS invoice”; aws.cost.ondemand is “based on the list rate provided by AWS, excluding all savings plans, reservations, discounts, taxes, and fees”.

So a monitor on aws.cost.amortized and one on aws.cost.unblended will disagree by design; pick the one that matches the question (invoice reconciliation versus discounted cost allocation).

Datadog’s own bill is also queryable: the Datadog Costs page (https://docs.datadoghq.com/cloud_cost_management/datadog_costs/) says daily Datadog costs give “visibility into daily Datadog spending across dashboards, notebooks, cost monitors, Cloud Cost Explorer, reports, budgets, and anomalies”, that there is “no additional charge” for it, that viewing it “requires both the billing_read and usage_read permissions” (default role: Datadog Admin), and that “Datadog cost data has an expected data delay of 48 hours and is available for the past 15 months.”

A cost alert monitor, same mechanics as before

The fence below is the same datadog_monitor resource as the previous lessons with type = "cost alert". Provenance matters here: it is the configuration used by the provider’s own acceptance test TestAccDatadogMonitor_FormulaFunction_Cost at tag v4.22.0 (datadog/tests/resource_datadog_monitor_test.go), which is backed by a recorded API cassette. It validates against the pinned provider; it is not tuned guidance, and neither the query grammar nor the meaning of the 6 threshold is explained in the docs pages I could find.

resource "datadog_monitor" "cloud_cost_spike" {
  name    = "EC2 cost anomaly"
  type    = "cost alert"
  message = "Unusual EC2 spend detected"

  query = "formula(\"query1\").last(\"30d\").anomaly(direction=\"both\") > 6"

  variables {
    cloud_cost_query {
      data_source = "cloud_cost"
      name        = "query1"
      query       = "sum:aws.cost.amortized{servicename:ec2} by {account}"
      aggregator  = "sum"
    }
  }

  monitor_thresholds {
    critical = 6
  }
}

What is provable about it:

type = "cost alert" is one of the provider’s valid monitor types (provider docs, docs/resources/monitor.md at v4.22.0).
variables.cloud_cost_query is a real block (maximum 5 per monitor); its data_source accepts metrics, cloud_cost or datadog_usage, and aggregator accepts avg, sum, max, min, last, area, l2norm, percentile, stddev (same provider docs, and the validate error below).
The named query (query1) is then referenced from the top-level query formula.

What the Cloud Cost Monitor page (https://docs.datadoghq.com/monitors/types/cloud_cost/) adds about behaviour:

“Cloud Cost monitors use a 30-minute evaluation frequency and a 48-hour delayed evaluation window, since cost data may not be available until 48 hours after usage.”
Monitor types in the UI are Changes, Anomalies, Threshold, Forecast and Budget; “Anomaly monitors require at least 1 month of cloud cost data to evaluate since historical data is required to train the algorithm”, and anomalies use “the agile anomaly algorithm … with two bounds and monthly seasonality”.
“You can use custom metrics or observability metrics alongside a cost metric to monitor unit economics.”
The page lists Terraform as one way to create a cost monitor and links the datadog_monitor resource.

Consequence for the example: a cost monitor cannot page you about spend within the last 48 hours, and an anomaly monitor created in a fresh account has nothing to learn from. Cost alerts are for trends and budgets, not incident detection.

flowchart LR
  A["Custom metrics: distinct time series"] --> D[Cost governance decisions]
  B["Logs: indexed events"] --> D
  C["Cloud provider billing data, 48 to 72 h to populate"] --> E[Cloud Cost Management]
  E --> F["Queryable cost metric: aws.cost.amortized etc."]
  F --> G["Cost alert monitor: 30 min evaluation, 48 h delayed window"]
  G --> H[Same alert routing as any other monitor]
  D --> G

From cloud billing data to a cost alert

Reading AI-generated code

🐛 Spot the bug

# AI comment: "Alert on EC2 cost anomalies"
resource "datadog_monitor" "cloud_cost_spike" {
name    = "EC2 cost anomaly"
type    = "cost alert"
message = "Unusual EC2 spend detected"

query = "formula(\"query1\").last(\"30d\").anomaly(direction=\"both\") > 6"

variables {
  cloud_cost_query {
    data_source = "cloud_costs"
    name        = "query1"
    query       = "sum:aws.cost.amortized{servicename:ec2} by {account}"
    aggregator  = "total"
  }
}

monitor_thresholds {
  critical = 6
}
}

Show the answer

This reads like the monitor above and parses as HCL, but terraform validate against provider 4.22.0 rejects two values with the provider’s own error text:

Error: Invalid enum value
  on main.tf line 20, in resource "datadog_monitor" "cloud_cost_spike":
  20:       data_source = "cloud_costs"
`invalid value 'cloud_costs' for MonitorFormulaAndFunctionCostDataSource:
valid values are [metrics cloud_cost datadog_usage]`

Error: Invalid enum value
  on main.tf line 23, in resource "datadog_monitor" "cloud_cost_spike":
  23:       aggregator  = "total"
`invalid value 'total' for MonitorFormulaAndFunctionCostAggregator: valid
values are [avg sum max min last area l2norm percentile stddev]`

Both look natural (“cloud costs”, “total”), and neither appears in any doc a reviewer is likely to check from memory. The fix uses cloud_cost and sum (validated against the pinned provider):

resource "datadog_monitor" "cloud_cost_spike_fixed" {
  name    = "EC2 cost anomaly"
  type    = "cost alert"
  message = "Unusual EC2 spend detected"

  query = "formula(\"query1\").last(\"30d\").anomaly(direction=\"both\") > 6"

  variables {
    cloud_cost_query {
      data_source = "cloud_cost"
      name        = "query1"
      query       = "sum:aws.cost.amortized{servicename:ec2} by {account}"
      aggregator  = "sum"
    }
  }

  monitor_thresholds {
    critical = 6
  }
}

Validate only proves the enums and block shape. Whether the query string does what you want on your account (and whether aws.cost.amortized is the metric you should use, per the metric list above) is not something terraform validate can tell you.

Mental Model

Think of cost as a second telemetry stream with its own delay and its own units. Custom metrics and logs are cost sources inside Datadog, governed by the decisions you make about tags, indexes and exclusions; cloud cost is a cost source outside Datadog that Datadog imports (from a Cost and Usage Report, for AWS) and exposes as metrics. Both end up as things you can query, graph and monitor. The important physical fact is time: cloud cost arrives 48 to 72 hours after setup and lags by up to 48 hours afterwards, so it steers budgets, not incidents.

Trade-offs

Per-host allotments vs cardinality discipline — the docs express custom-metric headroom per host, so adding hosts adds allotment while a single high-cardinality tag can exceed it; the cheap fix is a tag decision (Metrics without Limits), which the docs say converts your billing into both ingested and indexed counts.
Committed log volume vs on demand — the docs describe a 50% premium on over-commitment, so under-committing is not free; over-committing pays for volume you may not use.
aws.cost.unblended vs amortized/net-amortized metrics — unblended “matches the AWS invoice”; amortized variants spread commitment discounts and support allocation to workloads. Reconciliation and attribution want different metrics.
Anomaly vs threshold vs forecast vs budget cost monitors — anomaly needs at least a month of data and uses monthly seasonality; threshold and change monitors work with less history but do not learn what normal is.

Common Misconceptions

“Custom metric cost depends on how many points I send.” The billing page says usage is “not impacted by data point submission frequency or the number of queries you run”.
“With Metrics without Limits I only pay for the indexed view.” The billing page says such metrics pay “for ingested custom metrics and indexed custom metrics”; ingested overage is quoted at $0.10 per 100 over allotment.
“Datadog never charges for anything it just receives.” The log billing page only describes indexed-event billing; this lesson makes no claim about other line items, and neither should you without reading your contract.
“A cost monitor will catch today’s spike.” The docs give a 48-hour delayed evaluation window.
“terraform validate passing means the cost monitor query is right.” It checks block shape and enums only (as the two errors above show); query semantics need your account.