Structured Logging and Log Facets

The idea in one sentence

Structured (JSON) logs let a Processing Pipeline extract typed attributes reliably instead of relying on fragile Grok parsing of free text, and those attributes become Facets you can filter, group, and aggregate on in the Logs Explorer — but only for logs that actually reach an index.

Why structure beats Grok on free text

A Grok Parser can pull attributes out of an unstructured line like 2024-01-15 ERROR user=42 duration=812ms checkout failed, but every pattern is a bet on the log staying in that exact shape. Add a word, reorder a field, or have one service log slightly differently than another, and the Grok pattern silently stops matching — or worse, matches wrong.

A structured log sidesteps that bet entirely, because the fields already have names and types before Datadog sees them:

{
  "timestamp": "2024-01-15T10:22:31Z",
  "level": "error",
  "service": "checkout-api",
  "user": { "id": 42 },
  "duration_ms": 812,
  "message": "checkout failed"
}

Here a Processing Pipeline doesn’t need a Grok Parser at all for the fields that are already named keys — level, service, user.id, duration_ms arrive as typed attributes automatically. Grok is still useful for the parts of a message that remain free text (the message field itself, or logs from systems you don’t control the format of), but the reliable path for anything you do control is to log JSON in the first place.

Attributes become Facets

Once a log has structured attributes — whether pulled out by Grok or native JSON fields — Datadog can turn a given attribute into a Facet: a field the Logs Explorer treats as filterable, groupable, and chartable, the same way you’d facet search results on any site. Faceting on status_code lets you filter to status_code:500 or group a search by status_code to see a breakdown; faceting on service lets you slice any query by which service produced each log.

Consistent structure across services is what makes faceting actually useful at the org level. If one service calls it status_code, another calls it httpStatus, and a third buries it in an unparsed message string, you cannot facet across all three consistently. This is why remapping matters as part of getting logs into a consistent shape:

• Status Remapper — maps whatever severity field a service uses (level, severity, log.level, …) onto Datadog’s standard status facet, so “show me all errors across every service” is one query instead of one query per service.
• Message Remapper — designates which attribute is the log’s primary human-readable message, so the Logs Explorer shows something sensible in the message column regardless of what the source service happened to call that field.

Log Analytics only sees indexed logs

This is the fact that ties directly back to the previous lesson: Log Analytics — group-by breakdowns, computed measures, and other aggregations over a facet — only operates over logs that are actually indexed. An Exclusion Filter that keeps level:debug logs out of an index doesn’t just hide them from raw search; it also means those logs cannot appear in any Log Analytics aggregation, because there is nothing indexed to aggregate over.

If you need to slice-and-dice a field on logs an Exclusion Filter normally drops, you have exactly the same three options the previous lesson already gave you, applied to this specific need:

1. Temporarily disable or narrow the Exclusion Filter so the logs you need to analyze actually get indexed.
2. Generate a log-based metric instead — a COUNT or DISTRIBUTION computed off the full ingest stream gives you a trend without needing the logs indexed at all.
3. Rehydrate the relevant time range from a Log Archive into a temporary index, run the analysis, then let the temporary index expire.

Want to group-by @http.status_code on logs currently excluded by level:debug filter?
-> Option 1: relax the Exclusion Filter (costs indexing $$, gets full Log Analytics)
-> Option 2: log-based metric on @http.status_code (cheap, but only pre-defined aggregations)
-> Option 3: Rehydrate the time range (temporary cost, full Log Analytics for that window)

flowchart LR
  A["JSON log\nlevel, service, duration_ms"] --> B[Processing Pipeline\nStatus Remapper, Message Remapper]
  B --> C[Typed attributes]
  C --> D[Facets in Logs Explorer\nfilter / group / chart]
  D --> E{Log actually indexed?}
  E -->|yes| F[Log Analytics: group-by,\ncomputed measures work]
  E -->|no, excluded| G[No Log Analytics\nuse log-based metric or Rehydration]

From structured field to facet to analytics

🐶 The one-line takeaway

Structured (JSON) logs make attribute extraction reliable instead of Grok-fragile, consistent attributes (via Status Remapper, Message Remapper, and friends) become Facets you can filter and group on, and Log Analytics aggregations only work over indexed logs — so a field hidden by an Exclusion Filter needs a relaxed filter, a log-based metric, or a Rehydration before you can group-by it.

Real structured logs, with trace correlation fields injected

Structured logging and trace correlation are the same mechanism from two different angles — Connecting Logs and Traces proves the dd.* injection fields in depth; this lesson’s own proof, using pino (Node) with the pinned dd-trace, shows the same dd.* block arriving as native, already-typed JSON keys, not something a Grok pattern has to extract:

src/pino-structured-log.js

const tracer = require('dd-trace').init({ service: 'checkout-api', logInjection: true });
const pino = require('pino')();

tracer.trace('checkout.request', () => {
  pino.error({ user: { id: 42 }, duration_ms: 812 }, 'checkout failed');
});

$ node src/pino-structured-log.js
{"level":50,"time":1790690518779,"pid":18822,"hostname":"...","user":{"id":42},"duration_ms":812,
"msg":"checkout failed","dd":{"trace_id":"6abbc4d6000000007c4f928caf80a6a5",
"span_id":"8957539316800726693","service":"checkout-api"}}

user.id and duration_ms are real, typed JSON fields — proven above, no Grok Parser needed to extract them; they arrive as a nested object and a number, respectively, exactly the reliability advantage this lesson opened with. The Python equivalent (ddtrace’s DD_LOGS_INJECTION, feeding a hand-written JSON formatter instead of %-style log lines) produces the same dd.* shape, real output:

$ DD_LOGS_INJECTION=true DD_SERVICE=checkout-api DD_ENV=production DD_VERSION=2.4.1 python json_log_test.py
{"level": "ERROR", "message": "checkout failed",
 "dd": {"trace_id": "6abbc4dd000000009bb491e69f700f86", "span_id": "13978525130697882112",
        "service": "checkout-api", "env": "production", "version": "2.4.1"}}

Reserved attributes: names Datadog already treats specially

A handful of attribute names are reserved — Datadog’s UI and processing already know what to do with them the moment they show up in a log, structured or not: host (which machine/container produced the log), service (unified service tagging, same as traces/metrics), status (severity, what Status Remapper writes into), message (the primary human-readable text, what Message Remapper writes into), timestamp (the log’s official time, what Date Remapper writes into), and trace_id/dd.trace_id (the correlation pivot proven above). Because these are reserved, a structured log that names its own fields host, service, status, message, or timestamp gets those values picked up automatically, with no remapper needed at all — the remappers from the previous lesson exist specifically for logs that don’t already use these exact names.

🐶 Facets vs. Measures — not verified via a fresh docs fetch this session

Datadog’s Logs Explorer distinguishes a Facet (any attribute you filter/group by — string or numeric) from a Measure (specifically a numeric attribute enabled for range-based filtering and numeric aggregation, like duration_ms). This lesson could not WebFetch docs.datadoghq.com to pull the exact current wording for that distinction this session (a prior request in this same session hit a rate limit) — stated here as commonly-documented product terminology, not confirmed by a fresh fetch. Treat the Facet/Measure split as directionally correct, not URL-cited.

Facet creation: a real gap — this is UI-only, no public API model

Checked the real, installed datadog_api_client==2.61.0 for a model to create a log Facet on an attribute — there isn’t one. Every facet model that exists in this client is for a different product’s search facets (Service Catalog search, Incident search, CSM host facets, Model Lab) or for reading facet metadata (facet_info_request/facet_info_response), not for defining a new log Facet:

$ python -c "
import pkgutil, datadog_api_client.v2.model as m
print([n for _, n, _ in pkgutil.iter_modules(m.__path__) if 'facet' in n])
" | tr ',' '\n' | grep -i "logs\|create"
# (no output -- no logs-specific facet CREATE model exists in either v1 or v2)

Creating a log Facet is a UI-only operation in Datadog (Logs Explorer → attribute → “Create facet”), not something this pinned API client exposes a model for — consistent with the spec’s own expectation that this would likely be UI-only, now confirmed by exhaustively checking the client’s real module list rather than assuming.

source and automatic pipeline install

Tying back to the previous lesson’s source: java/source: nginx field: setting a recognized source value on a logs: block is also what triggers Datadog to auto-install the matching integration’s built-in Processing Pipeline — you don’t create or attach it yourself. This is the mechanism, not a new one: the same source tag that labels a log for filtering is the same value Datadog’s pipeline-install logic reads to decide which built-in Grok rules to wire up automatically the first time a log with that source arrives.

Reading AI-generated code

🐛 Spot the bug

src/log-user-object.js

// AI comment: "Log full user context as structured data for facets"
const pino = require('pino')();

function logCheckoutFailure(user, durationMs) {
pino.error({
  user: user,  // the whole user object -- id, email, address, payment_method, ...
  duration_ms: durationMs,
}, 'checkout failed');
}

Show the answer

This produces genuinely structured JSON — proven above, pino output is real, typed, faceting-ready data, so nothing here throws or silently fails the way earlier bugs in this course did. The problem is a design one, not a crash: logging the entire user object nests every one of its fields (email, address, payment_method, whatever else that object happens to carry) into the log line as attributes, which (a) can quietly leak PII into a log index most engineers assume only has id and duration_ms in it, and (b) creates one facet-eligible attribute per field the object happens to have today, with no review step — a field added to the user object next month becomes a new facet-eligible log attribute automatically, without anyone deciding that was a good idea. The fix is the same discipline as any structured-logging code review: log the specific fields you actually want faceted (user.id), not an object you don’t fully control the shape of.

Mental Model

A reserved attribute name is a contract, not a convention — proven above, service/status/message/timestamp get picked up with zero remapper configuration the moment a structured log uses those exact names, because Datadog’s own processing already watches for them. Everything else you log is just data until something — a Grok Parser’s extraction, or JSON’s own key names — gives it a name Datadog’s Facet system can hang a filter/group/chart on. Structuring your logs well is, in a real sense, pre-deciding your own remapper work: log service/status/message directly, and two of the three most common processors in the previous lesson’s pipeline become unnecessary for that log source.

Trade-offs

• Native JSON keys vs. Grok-extracted attributes — proven above (user.id, duration_ms arriving pre-typed): JSON needs no pattern maintenance and never silently stops matching; Grok remains necessary for logs from systems you don’t control the format of, at the cost of being one framework-version change away from silently breaking.
• Logging a whole object vs. specific fields — proven above in the SpotTheBug example: logging an entire user object is less code today, but creates unreviewed, drifting facet surface area and real PII-leak risk; logging named fields is more code today and a stable, auditable attribute set tomorrow.
• Reserved attributes vs. custom-named ones — using Datadog’s reserved names (service, status, message, timestamp) means zero remapper configuration, at the cost of needing every service’s logging code to agree on those exact names; custom names are more natural per-team but need a Status/Message/Date Remapper to normalize centrally.

Common Misconceptions

• “Creating a Facet is something you can do via the public API, like a monitor or dashboard.” No — proven above by exhaustively checking the installed client: no datadog_api_client model exists for creating a log Facet in either v1 or v2; it is a Logs Explorer UI action.
• “Reserved attributes like service and status need a remapper to work.” No — proven by definition above: a structured log that already names its fields service/status/message/timestamp gets them recognized automatically; remappers exist for logs that use different field names.
• “Logging a whole object is the same as logging its useful fields.” No — proven above: every field the object happens to carry becomes log data and a facet-eligible attribute, whether or not that was the intent, which is a real, silent PII and facet-sprawl risk, not just a style preference.

Why is a native JSON log field more reliable than a Grok-parsed field from free text?

JSON logs are always smaller in size The field already has a name and type before Datadog sees it, so it does not depend on a pattern matching an exact text shape Grok Parsers cannot run on JSON logs at all JSON logs skip the Processing Pipeline entirely

What is a Facet in the Logs Explorer?

A raw, unparsed log line A field derived from a log attribute that can be filtered, grouped, and charted on A synonym for a Log Archive A type of Agent processing rule

What does the Status Remapper accomplish across multiple services?

It deletes logs below a certain severity It maps each service's own severity field naming onto Datadog's standard status facet, so error queries work consistently across services It changes an index's retention_days It converts logs into JSON automatically

A field is excluded from an index by an Exclusion Filter. What happens to Log Analytics group-by on that field?

It still works normally because Log Analytics reads the full ingest stream It stops working for those logs, because Log Analytics aggregations only operate over indexed logs It automatically falls back to a log-based metric with no changes needed It only affects DISTRIBUTION metrics, not group-by

Does the real, installed datadog_api_client (v1 or v2) expose a model for creating a log Facet on an attribute?

Yes, LogsFacetCreateRequest in v2 No — exhaustively checking every "facet" model in both v1 and v2 finds only search/read facet models for other products (Service Catalog, Incidents, CSM, Model Lab); creating a log Facet is a Logs Explorer UI action Yes, but only in v1, not v2 Facets do not exist as a concept in the API client at all, in any form

A structured log names its severity field exactly status instead of level or severity. What happens without any Status Remapper configured?

Nothing — a remapper is always required regardless of field naming It is recognized automatically, because status is a reserved attribute name Datadog already treats specially It gets rejected as an invalid field name It only works if the value is numeric

Submit

Mark as complete