Log-Based Metrics and Logging without Limits

The idea in one sentence

Logging without Limits is Datadog’s name for decoupling log ingestion from log indexing, and its most important consequence is that you can generate a log-based metric, or send a log to an Archive, from the entire ingest stream — even for logs an Exclusion Filter drops from every index.

Log-based metrics: measuring logs you don’t index

A log-based metric is computed from the full ingest stream — the same point in the pipeline the previous lesson’s Exclusion Filters act on — not from what’s already indexed. That distinction is the whole point: a log-based metric keeps working even for logs that never make it into any index.

Two shapes of log-based metric matter:

• A COUNT metric counts how many logs match a filter query, e.g. @status:error.
• A DISTRIBUTION metric aggregates a numeric log attribute, e.g. @duration, giving you percentiles and averages the way any distribution metric would.

{
  "data": {
    "id": "logs.page.load.count",
    "attributes": {
      "compute": { "aggregation_type": "distribution", "path": "@duration" },
      "filter": { "query": "service:web* AND @http.status_code:[200 TO 299]" },
      "group_by": [ { "path": "@http.status_code", "tag_name": "status_code" } ]
    }
  }
}

This example defines a distribution metric over the @duration attribute, scoped to successful requests on web* services, grouped by status code. Log-based metrics are computed at 10-second granularity and retained for 15 months — far longer than most index retention_days settings, and at a fraction of the cost, because the metric is a small aggregate, not the full log body. Source, quoted from https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics/ (fetched 2026-09-30): log-based metrics let you “visualize trends and anomalies over all of your log data at 10s granularity for 15 months”, and they are generated from the complete ingest stream even when exclusion filters limit what is indexed. This is docs-sourced, not run: no Datadog account exists in this course.

The practical payoff: you can set an index Exclusion Filter to drop level:debug logs entirely (saving on indexing cost), while still keeping a log-based COUNT metric of how many debug logs occurred per service, per minute, for over a year. You lose the ability to read individual debug log lines after they age out or get excluded, but you never lose the trend or anomaly signal.

Log Archives: the full stream, cheaply, for the long term

A Log Archive sends effectively the full, un-excluded log stream to storage you own — S3, GCS, or Azure Blob — regardless of what any index’s Exclusion Filters are doing. Archiving happens independently of indexing decisions, which is what makes it suitable for compliance and long-term retention: a log can be excluded from every index (cheap, no full-price search) and still land in the archive (cheap, long-term, not searchable without extra steps).

Archives are not searchable in the Logs Explorer as-is; per https://docs.datadoghq.com/logs/log_configuration/rehydrating/ (fetched 2026-09-30) there is a second, cheaper route than Rehydration: “Archive Search … streams results in real time directly from your archive without re-indexing, and charges only for the data scanned.” They are cold storage in your own cloud account, at your own cloud storage’s rates, which is precisely why they’re viable for retaining effectively everything for years when full index retention would not be.

Rehydration: temporarily re-indexing a slice of the past

When you actually need to investigate something inside an archive — a level:debug burst around an incident from four months ago, say — you use Rehydration: a time-scoped slice of the archive is re-indexed, typically into a temporary index, so it becomes searchable again in the Logs Explorer for as long as you need it. Rehydration is deliberately scoped by time range (and often by query) rather than replaying the whole archive, because re-indexing is what costs money — you’re paying to make a slice searchable again, not to store it, since it was already sitting safely in the archive.

flowchart LR
  A[Full ingest stream] --> B[Log-based metric\nCOUNT or DISTRIBUTION\n10s granularity, 15 months]
  A --> C[Log Archive\nS3 / GCS / Azure Blob]
  A --> D{Index Exclusion Filter}
  D -->|kept| E[Indexed: searchable,\nfull price, retention_days]
  D -->|excluded| F[Not in this index]
  C -.->|Rehydration: time-scoped| G[Temporary index\nsearchable again]

Ingestion vs. indexing: where each path leads

🐶 The one-line takeaway

Logging without Limits means ingestion and indexing are decoupled — a log-based metric (COUNT or DISTRIBUTION, 10-second granularity, 15-month retention) and a Log Archive (your own cheap cloud storage) both draw from the full ingest stream regardless of index Exclusion Filters, and Rehydration lets you temporarily re-index a time-scoped slice of an archive when you need to search it again.

LogsMetricCreateRequest, validated for real

Built the distribution metric sketched above as real JSON and validated it against datadog_api_client==2.61.0’s LogsMetricCreateRequest — the real name matches the spec exactly:

// (log-based metric config, not a harness-collected JSON dir -- see caveat below)
{
  "data": {
    "id": "logs.page.load.count",
    "type": "logs_metrics",
    "attributes": {
      "compute": { "aggregation_type": "distribution", "path": "@duration", "include_percentiles": true },
      "filter": { "query": "service:web* AND @http.status_code:[200 TO 299]" },
      "group_by": [ { "path": "@http.status_code", "tag_name": "status_code" } ]
    }
  }
}

$ python -c "
from datadog_api_client.v2.model.logs_metric_create_request import LogsMetricCreateRequest
import json
payload = json.load(open('metric.json'))
LogsMetricCreateRequest(**payload, _check_type=True)
print('validated OK')
"
validated OK

inspect.signature confirms aggregation_type has exactly two real values, {'count', 'distribution'}, and include_percentiles only lives on compute — the same shape this course’s own APM SpansMetricCreateRequest uses, a consistent pattern across the client, not a coincidence. Omitting compute entirely fails immediately with a real, specific error:

# @expect-error LogsMetricCreateAttributes.__init__() missing 1 required positional argument: 'compute'
from datadog_api_client.v2.model.logs_metric_create_request import LogsMetricCreateRequest
payload = {"data": {"type": "logs_metrics", "id": "broken", "attributes": {}}}
LogsMetricCreateRequest(**payload, _check_type=True)

Archives, validated — and a real field this lesson didn’t expect

Built a real S3 archive config and validated it against LogsArchiveCreateRequest:

// (log archive config, not a harness-collected JSON dir -- see caveat below)
{
  "data": {
    "type": "archives",
    "attributes": {
      "name": "payments-api archive",
      "query": "service:payments-api",
      "destination": {
        "type": "s3",
        "bucket": "my-log-archive-bucket",
        "path": "/payments-api",
        "integration": { "account_id": "<your-aws-account-id>", "role_name": "<your-role-name>" }
      },
      "rehydration_max_scan_size_in_gb": 100,
      "rehydration_tags": ["team:payments"]
    }
  }
}

$ python -c "
from datadog_api_client.v2.model.logs_archive_create_request import LogsArchiveCreateRequest
import json
payload = json.load(open('archive.json'))
LogsArchiveCreateRequest(**payload, _check_type=True)
print('validated OK')
"
validated OK

Two real fields this lesson didn’t know to expect until checking the model: rehydration_max_scan_size_in_gb and rehydration_tags live directly on the archive’s own attributes, not on a separate Rehydration-specific model. That means Rehydration is partly pre-configured at archive-creation time — a cap on how much data any rehydration job against this archive can scan, and default tags applied to rehydrated logs — not something invented fresh every time someone rehydrates a slice. A broken destination (missing the required bucket field) fails immediately with a real, specific error:

# @expect-error LogsArchiveDestinationS3.__init__() missing 1 required positional argument: 'bucket'
from datadog_api_client.v2.model.logs_archive_destination_s3 import LogsArchiveDestinationS3
LogsArchiveDestinationS3(type='s3', integration={'account_id': '<your-aws-account-id>', 'role_name': '<your-role-name>'})

What the client model does and does not enforce

Probed with _check_type=True against LogsMetricCreateRequest and LogsArchiveCreateRequest. aggregation_type advertises exactly {'distribution', 'count'} in allowed_values, yet the constructor accepts sum, and a distribution compute with no path passes as well. Wrong Python types and missing required arguments fail; wrong enum values and wrong semantics do not. The server is the real judge of those, and this course has no server.

$ python probe.py   # datadog_api_client 2.61.0, _check_type=True, run on this Mac
nofilter        VALID
bad agg 'sum'   VALID     <- enum not enforced at construction
dist, no path   VALID     <- distribution without a path also passes
s3 type 'ftp'   VALID     <- destination type not enforced either
scan size "big" ApiTypeError Invalid type for variable 'rehydration_max_scan_size_in_gb'.
                Required value type is one of [NoneType, int] and passed type was str
no name         TypeError LogsArchiveCreateRequestAttributes.__init__() missing 1 required positional argument: 'name'

🐶 Flex Logs and these two JSON dirs: a documented harness gap

Neither logs-metrics/ nor logs-archives/ is in this course’s harness’s supported --json directory list (only monitors/, dashboards/, slos/, service-catalog/, logs-pipelines/ are strict/collected) — the two JSON fences above were validated manually, as shown, not automatically by npm run verify. On Flex Logs specifically: the previous lesson found a real num_flex_logs_retention_days field on LogsIndex, confirming the feature is real and current in this pinned client version; this lesson has no further model-level surface for it (no archive- or metric-level Flex Logs field exists) and this lesson makes no pricing or retention-number claim for it: those are hosted-only facts, deliberately left to the Datadog Flex Logs docs rather than quoted from memory. Treat Flex Logs here as a real API-level fact, not a fully docs-cited product description.

Reading AI-generated code

🐛 Spot the bug

app/create_debug_metric.py

# AI comment: "Track debug log volume per service without indexing debug logs"
from datadog_api_client.v2.model.logs_metric_create_request import LogsMetricCreateRequest

payload = {
  "data": {
      "type": "logs_metrics",
      "id": "logs.debug.count",
      "attributes": {
          "compute": {"aggregation_type": "count"},
          "group_by": [{"path": "service"}],
      },
  }
}
LogsMetricCreateRequest(**payload, _check_type=True)

Show the answer

This validates cleanly — compute is present with a real aggregation_type, group_by is well-formed. The bug is semantic, the same shape as the exclusion-filter trap from the previous lesson: there is no filter block at all, so this metric counts every log reaching the ingest stream, grouped by service — not specifically debug logs, despite the AI comment’s stated intent. The fix is adding the missing scope: "filter": {"query": "level:debug"}. Nothing here throws, because an absent filter is valid (it just means “match everything”), which is exactly why this class of bug survives model validation — proven above, the model checks shape, not scope.

Mental Model

Logging without Limits works because ingestion produces one stream that multiple, independent consumers can each draw their own thing from — a log-based metric (proven above, its own LogsMetricCreateRequest) and a Log Archive (proven above, its own LogsArchiveCreateRequest) are not “views” into an index; they are separate destinations fed from the same upstream point, before any index Exclusion Filter gets a vote. That’s the entire mechanical reason excluding level:debug from an index doesn’t affect either of them: exclusion filters live downstream of the fork this stream takes, not upstream of it.

Trade-offs

• Log-based metric vs. Log Archive vs. index — a log-based metric (proven above, cheap, 10s granularity, 15 months) gives you a pre-defined aggregate forever, at the cost of losing the individual log lines; an archive (proven above, cheap, your own cloud storage) keeps everything but isn’t searchable without Rehydration; an index (full price) is searchable immediately but costs the most and is the only one with a retention ceiling you pay for directly.
• A tight rehydration_max_scan_size_in_gb vs. a loose one — proven above, this is a real, archive-level field: a tight cap protects against an accidentally huge, expensive rehydration job, at the cost of a legitimate large investigation potentially needing the cap raised first.
• Defining a log-based metric vs. relying on Log Analytics over a relaxed Exclusion Filter — a metric (proven above) is cheap and permanent but locked into whatever compute/group_by you defined upfront; relaxing an Exclusion Filter gets you full ad-hoc Log Analytics, at full indexing cost, for as long as the relaxation stays in place.

Common Misconceptions

• “A log-based metric without a filter only counts the logs you meant it to.” No — proven above in the SpotTheBug example: an absent filter block is valid and means “match everything,” a real, silent scope bug that passes model validation cleanly.
• “Rehydration settings are configured fresh every time you rehydrate.” No — proven above: rehydration_max_scan_size_in_gb and rehydration_tags are real fields on the archive’s own attributes, set at archive-creation time, not invented per rehydration job.
• “Log-based metric and archive JSON get the same automatic model validation as monitors/dashboards/pipelines.” Not in this course’s own harness — proven above: neither logs-metrics/ nor logs-archives/ is in the harness’s collected directory list, so both fences here were validated by hand, not by npm run verify.
• “_check_type=True rejects an invalid aggregation_type or archive destination type.” No — proven above: sum and ftp both construct cleanly. It rejects wrong Python types ("big" for an int) and missing required arguments, nothing more.

A log-based metric is computed from which stream?

Only logs that made it into an index The full ingest stream, before any index Exclusion Filter is applied Only logs stored in a Log Archive Only logs from the last 24 hours regardless of index settings

At what granularity and retention are log-based metrics kept?

1-minute granularity, 30 days retention 10-second granularity, 15 months retention 1-hour granularity, 7 days retention Same granularity and retention as the source index

A Log Archive receives which logs?

Only logs that survive every index's Exclusion Filters Effectively the full, un-excluded log stream, independent of index Exclusion Filters Only logs matching a log-based metric filter Only logs dropped by Agent-side exclude_at_match

What is Rehydration?

Permanently restoring an entire archive into the main index A way to increase an index's daily_limit Temporarily re-indexing a time-scoped slice of an archive so it becomes searchable again A Processing Pipeline processor that re-parses old logs

A log-based metric JSON has a compute block but no filter block at all. Does the real, installed LogsMetricCreateRequest model reject this?

Yes, filter is a required field No — proven above, it validates cleanly. An absent filter means "match every log reaching the ingest stream," which can silently make the metric count far more than intended Yes, but only for COUNT aggregation type, not DISTRIBUTION No, but it emits a warning at construction time

What two real fields does the LogsArchiveCreateRequest model expose directly on an archive's attributes, related to Rehydration?

archive_ttl_days and rehydration_priority rehydration_max_scan_size_in_gb and rehydration_tags — meaning some Rehydration behavior is pre-configured at archive-creation time, not invented fresh per rehydration job rehydration_enabled and rehydration_index_name There are no Rehydration-related fields on the archive model at all

You probe LogsMetricCreateRequest with aggregation_type "sum" and _check_type=True. What happens with the pinned datadog_api_client 2.61.0?

ApiValueError, because only count and distribution are allowed It constructs cleanly — enum values are not enforced at construction, so the server (not this harness) is what would reject it TypeError for a missing argument It is silently rewritten to count

Submit

Mark as complete