Indexes, Exclusion Filters, and Retention

The idea in one sentence

A log can be dropped or filtered at three genuinely different stages — on the host before it ever leaves, or at an index’s daily quota, or by an index’s own Exclusion Filter — and each of those has different cost and recoverability implications, so they must not be conflated.

Stage 1: Agent-side `exclude_at_match`

Before a log ever leaves the host, the Agent itself can drop it via a log_processing_rules entry with type exclude_at_match, matching a regex pattern against the raw log line.

logs:
  - type: file
    path: /my/app/file.log
    service: payments-api
    source: java
    log_processing_rules:
      - type: exclude_at_match
        name: exclude_healthcheck_lines
        pattern: 'GET /healthz'

This is the cheapest possible place to filter, because a dropped line never consumes network, intake, or storage — but it is also final. A log excluded here is gone forever: it is not archived, it does not reach a Processing Pipeline, and it cannot be recovered later. Only use this for genuinely worthless lines (health-check noise, debug spam you never want in any form), because there is no undo.

Stage 2: Log Indexes — `retention_days` and `daily_limit`

Logs that survive the Agent and pass through intake and a Processing Pipeline arrive at one or more Log Indexes. An index is where full-price, searchable retention actually happens, and each index configures two numbers that matter for cost and behavior:

{
  "name": "main",
  "daily_limit": 100,
  "retention_days": 30,
  "exclusion_filters": [
    { "name": "exclude-debug", "query": "level:debug" }
  ]
}

retention_days — how long logs written into this index stay searchable in the Logs Explorer at full price.
daily_limit — a GB/day quota for the index; once the quota is hit for the day, the index either stops accepting further logs or (depending on configuration) starts sampling, so a badly sized daily_limit can silently cap what’s searchable regardless of anything else in your config.

An organization typically has more than one index (e.g. main for production services, a shorter-retention index for noisy or low-value logs), and which index a log lands in is itself controlled by index-level filter criteria — but the key fact for this lesson is that retention_days and daily_limit are properties of the index, not of the Agent or the pipeline.

Stage 3: Exclusion Filters — the main lever for indexed volume

An Exclusion Filter lives on a specific index and uses a Lucene-style query — the same query syntax you’d type into the Logs Explorer search bar — to sample out or fully drop matching logs from being written into that index. In the config above, { "name": "exclude-debug", "query": "level:debug" } means logs matching level:debug do not get written into the main index.

This is the main day-to-day lever for controlling indexed, searchable, full-price volume, and it is deliberately more flexible and less final than Agent-side dropping:

A log excluded from one index by its Exclusion Filter can still be routed into a different index (with different retention or cost characteristics), if your index routing sends it there.
A log excluded from every index can still reach a Log Archive, if archiving is configured — the next lesson covers this.
Because the exclusion happens at the index, not on the host, it can be changed centrally in Datadog without touching a single Agent config file anywhere.

Putting the order together

The full path a log takes, end to end, is: Agent collects it → an optional Agent-side exclude_at_match rule may drop some logs before they ever leave the host → surviving logs reach intake → a Processing Pipeline parses and enriches every log that matches its filter → the log reaches a Log Index, whose own Exclusion Filters decide what actually gets indexed, searchable, and billed at full price.

flowchart LR
  A[Agent collects log] --> B{exclude_at_match\nmatches?}
  B -->|yes, dropped forever| Z[Gone: no archive, no index]
  B -->|no| C[Intake]
  C --> D[Processing Pipeline\nparse + enrich]
  D --> E{Index Exclusion Filter\ne.g. level:debug}
  E -->|excluded| F[Not written to this index\nmay still reach archive/other index]
  E -->|kept| G["Indexed: searchable\nfor retention_days, billed at full price"]

Where a log can be dropped or filtered

Index JSON, validated — real field names, one real surprise

Built the index config sketched above as real JSON and validated it against datadog_api_client==2.61.0’s LogsIndex model. The real field name for retention is num_retention_days, not retention_days — confirmed by inspect.signature, worth calling out since the spec for this lesson and this lesson’s own earlier JSON example both used the shorter name:

// (index config, not a harness-collected JSON dir -- see caveat below)
{
  "name": "main",
  "filter": { "query": "*" },
  "daily_limit": 100,
  "num_retention_days": 30,
  "exclusion_filters": [
    { "name": "exclude-debug", "is_enabled": true, "filter": { "query": "level:debug", "sample_rate": 0.0 } }
  ]
}

$ python -c "
from datadog_api_client.v1.model.logs_index import LogsIndex
import json
payload = json.load(open('index.json'))
m = LogsIndex(**payload, _check_type=True)
print('validated OK')
"
validated OK

A field this lesson did not know to expect until checking the real model: num_flex_logs_retention_days — a second, separate retention-days field on the very same LogsIndex model, confirming Flex Logs (a cheaper, longer-retention storage tier for indexed logs) is a real, current concept in this pinned client version, not a renamed or retired feature. Precise pricing, exact retention windows, and current product naming for Flex Logs are UI/billing-plane facts this lesson cannot verify without a docs URL — noted, not invented.

The over-broad exclusion filter trap: the model happily accepts it

The classic mistake — an Exclusion Filter with query: "*" and sample_rate: 1.0, which drops 100% of every log that reaches this index — validates perfectly cleanly. There is nothing structurally wrong with it; it’s just a filter that matches everything and keeps none of it:

{
  "name": "drop-everything-by-accident",
  "is_enabled": true,
  "filter": { "query": "*", "sample_rate": 1.0 }
}

$ python -c "
from datadog_api_client.v1.model.logs_index import LogsIndex
payload = {'name': 'main', 'filter': {'query': '*'}, 'num_retention_days': 30, 'daily_limit': 100,
  'exclusion_filters': [{'name': 'drop-everything-by-accident', 'is_enabled': True,
    'filter': {'query': '*', 'sample_rate': 1.0}}]}
m = LogsIndex(**payload, _check_type=True)
print('validated OK -- model accepts a filter that drops every single log matching *')
"
validated OK -- model accepts a filter that drops every single log matching *

This is purely semantic — query, sample_rate, and every other field are the right types, so LogsIndex has no basis to reject it. The only way to catch “this exclusion filter is over-broad” is reading the query and rate together and reasoning about what they actually match, the same class of gap the previous two lessons already proved (order-dependent pipelines, Union-typed processor fallback) — a validating model is not the same thing as a correct config.

`LogsIndexesOrder`: first matching index wins

An org typically routes logs across more than one index, and LogsIndexesOrder is the real, separate model (confirmed in the client) controlling the order indexes are evaluated in — a log is written into the first index in that order whose own filter matches it, not into every index whose filter happens to match:

$ python -c "
from datadog_api_client.v1.model.logs_indexes_order import LogsIndexesOrder
m = LogsIndexesOrder(index_names=['main', 'low-value', 'debug-index'])
print('validated OK', m)
"
validated OK {'index_names': ['main', 'low-value', 'debug-index']}

The practical consequence: an index’s own filter being broad and appearing first in this order can silently absorb logs a later, more specific index was meant to catch — the later index’s filter is never even evaluated for a log the earlier one already claimed. This ordering behavior is UI/backend routing logic this lesson could not independently exercise without a real org (the LogsIndexesOrder model only proves the shape of the ordering list, not the runtime routing decision) — stated here as documented product behavior, not re-derived from a local tool.

Reading AI-generated code

🐛 Spot the bug

// (index config)
// AI comment: "Drop noisy debug logs, keep everything else searchable as before"
{
"name": "main",
"filter": { "query": "*" },
"num_retention_days": 30,
"daily_limit": 100,
"exclusion_filters": [
  { "name": "reduce-debug-noise", "is_enabled": true,
    "filter": { "query": "*", "sample_rate": 0.9 } }
]
}

Show the answer

This validates cleanly against LogsIndex — every field is well-typed, and nothing here is structurally wrong. The bug, proven above by reading what query: "*" actually matches: it matches every log, not just debug-level ones. Combined with sample_rate: 0.9, this drops 90% of every single log reaching this index, not 90% of debug noise — the AI comment’s intent (“drop noisy debug logs”) required a query like level:debug, and the * wildcard silently expanded the blast radius from “one noisy log level” to “the entire index.” The fix is the query, not the rate: "query": "level:debug" scopes the exclusion to what was actually intended.

Mental Model

An index’s Exclusion Filter is a query with a consequence, not a safety-checked control — proven above twice, LogsIndex validates the JSON shape and nothing about what the query actually matches or how much of your traffic that represents. Think of query and sample_rate as two independent multipliers on “how much of this index’s incoming traffic disappears”: a narrow query times a high rate drops a lot of a small slice; a broad query (* being the broadest possible) times any nonzero rate drops that same fraction of everything. Reading an exclusion filter safely means reading the query’s actual scope first, the rate second — reversing that order is exactly how “reduce debug noise” becomes “silently lose 90% of the index.”

Trade-offs

num_retention_days vs. num_flex_logs_retention_days — proven above, both are real fields on the same LogsIndex: full-price index retention is faster to search and simpler to reason about, at higher cost; Flex Logs retention (confirmed to exist, not independently priced or benchmarked here) trades some of that for a cheaper, longer window — the two are not the same guarantee even though both are “retention.”
A broad, catch-all index filter (query: "*") vs. narrow, purpose-built indexes — one broad index is simple to reason about and route into, but (proven above with LogsIndexesOrder) if it sits first in index order, it can silently swallow logs a more specific, later index was meant to receive; narrow indexes avoid that trap at the cost of more indexes to maintain and route between.
Structural JSON validation vs. reading the actual query scope — this lesson’s SpotTheBug is the clearest version of a pattern from the last two lessons: a model validating cleanly is necessary but not sufficient proof a config does what you intended; reading query and sample_rate together, by hand, is still required.

Common Misconceptions

“An Exclusion Filter with a high sample_rate only drops what its query narrowly describes.” No — proven above: the query’s scope, not the rate, determines the blast radius; a broad query with any nonzero rate can drop far more than intended, and LogsIndex validates such a filter with zero complaint.
“retention_days is the real field name on a LogsIndex.” No — proven above via inspect.signature: the real field is num_retention_days. There is a second, separate num_flex_logs_retention_days field for Flex Logs retention on the same model.
“A log matching more than one index’s filter gets written to all of them.” Not by default — proven above via the real LogsIndexesOrder model: index evaluation order matters, and the first matching index in that order is where an unrouted log lands, not every matching index.