Tags and Unified Service Tagging

The idea in one sentence

A tag is just key:value, but three specific tags — env, service, and version — applied the same way across metrics, logs, and traces are what let you pivot from one signal to another for the same deployable unit; this lesson proves the syntax rules, where a tag can come from, and exactly where cardinality gets expensive, all against a real Agent instead of from memory.

Tagging fundamentals

Every metric, log, and trace in Datadog can carry tags, written as key:value — env:prod, region:ap-southeast-1, team:payments. Tags come from a few places:

Host tags — attached automatically by the Agent (hostname, OS, cloud provider/instance metadata) or configured once in datadog.yaml’s tags: list.
DD_TAGS — the environment-variable equivalent of tags:, read by both the Agent and (for a subset) the tracer libraries.
Container labels — via DD_CONTAINER_LABELS_AS_TAGS, proven below, an explicit opt-in mapping rather than an automatic one.
Integration tags — added by a check’s config, like the service:orders-db tag in the Postgres example from the previous lesson.
Custom tags — anything you add yourself, on a host, a container, or directly on a metric submission.

Tags are what make Datadog’s UI usable at scale: instead of hunting through a list of 4,000 hosts, you filter by env:prod AND team:payments and get exactly the slice you need. This works identically whether you’re looking at Infrastructure, Logs, or APM — same tags, same filtering syntax, everywhere.

Reserved tags

A handful of tag keys are treated specially by Datadog’s backend, not just conventionally: host (the reporting host, usually set automatically, not something you normally override by hand), and the unified-service-tagging trio env, service, version (covered in depth below). Datadog’s UI, APM service pages, and Software Catalog all assume these keys mean what the platform expects them to mean — reusing host or service for something unrelated to their reserved meaning breaks features that read them, not just your own dashboards.

Tag syntax, and a real surprise: normalization happens server-side, not on the Agent

Per docs.datadoghq.com/getting_started/tagging/#defining-tags: a tag may contain letters (any Unicode letter), numbers, underscores, minuses, colons, periods, and forward slashes after the leading letter; “all other characters (including commas, emoji, backslashes, and spaces) are converted to underscores”; tags are limited to 200 characters total (key + : + value); and — the surprising part — “Span tags and metric tags are normalized to lowercase… This normalization occurs at Datadog’s backend, not the Agent level.”

Proof: a real datadog/agent:7 container configured with an intentionally messy tags: list —

api_key: "<DD_API_KEY>"
tags:
  - "Team:Payments API"
  - "env:PROD"
  - "region:ap-southeast-1"

— and real agent status output shows the tags completely unchanged, uppercase and internal space included:

$ docker exec dd-tags agent status
  host tags:
    Team:Payments API
    env:PROD
    region:ap-southeast-1

This matches the docs exactly: normalization (lowercasing, space-to-underscore) is a Datadog-backend behavior, not something the Agent does before shipping. agent status shows you the tag as you configured it, not as Datadog will ultimately index it — a real, non-obvious gap between “what I see locally” and “what I’ll be able to filter by.”

`DD_TAGS` has a sharper trap than `tags:` — it silently splits on whitespace

docs.datadoghq.com/getting_started/tagging/#defining-tags also states: “Whitespace in DD_TAGS values is not converted to underscores” — unlike a tags: YAML list item (each item is one whole string), DD_TAGS is a space-separated list, so a space inside what you intended as one tag value splits it into two separate tags instead. Real proof, a container configured with DD_TAGS="team:payments api region:ap-southeast-1" (intending one tag team:payments api, plus region:ap-southeast-1):

$ docker exec dd-ddtags agent status
  host tags:
    api
    region:ap-southeast-1
    team:payments

The intended team:payments api tag silently became two host tags: team:payments (truncated) and a bare, keyless api tag. Nothing errors — this is exactly the kind of AI-generated-config mistake that ships clean and only shows up as a weird extra tag in the UI weeks later.

Container labels: `DD_CONTAINER_LABELS_AS_TAGS`, opt-in, not automatic

Docker/container labels do not automatically become Datadog tags — you have to explicitly map them with DD_CONTAINER_LABELS_AS_TAGS, a JSON object of {"<label name>": "<tag key>"}. Real proof this env var is actually read (not just accepted silently), via the Agent’s own agent config diagnostic command, which dumps the fully-resolved runtime config:

$ docker run -d --name dd-labeltags \
    -e DD_API_KEY=<fake> \
    -e DD_CONTAINER_LABELS_AS_TAGS='{"com.docker.compose.service":"service_name"}' \
    datadog/agent:7
$ docker exec dd-labeltags agent config | grep container_labels_as_tags
container_labels_as_tags: '{"com.docker.compose.service":"service_name"}'

This is a different mechanism from the Autodiscovery com.datadoghq.ad.* labels covered in the previous lesson — those configure which check runs; DD_CONTAINER_LABELS_AS_TAGS configures which existing container labels become tags on whatever telemetry that container produces. Neither implies the other.

Cardinality: the tag you should not add

Not every piece of data should become a tag value. Datadog tracks time series (and indexed log facets) per unique combination of tag values, so a tag whose value is different for basically every event — a raw user ID, a request ID, a full URL with query parameters — creates a new time series or facet value for almost every data point. This is called high cardinality, and it’s expensive in two ways: it inflates your custom-metrics bill (each unique tag combination on a metric is its own billed series), and it makes the UI less useful, because a tag filter dropdown with a million distinct values stops being something a human can actually use to narrow things down.

The rule of thumb: tag by things you’d actually want to filter or group by — env, service, version, region, team, endpoint (as a normalized route pattern, not a raw path) — and put genuinely unique identifiers (user ID, request ID, order ID) in the log message or a trace attribute instead of a tag.

`DD_CHECKS_TAG_CARDINALITY`: an explicit control, not just a guideline

Beyond “don’t tag high-cardinality data yourself,” the Agent has an explicit cardinality dial for the tags it itself attaches to check metrics. Straight from Agent 7.83.3’s own pkg/config/config_template.yaml on GitHub:

## @param checks_tag_cardinality - string - optional - default: "low"
## @env DD_CHECKS_TAG_CARDINALITY - string - optional - default: "low"
## Configure the level of granularity of tags to send for checks metrics and events. Choices are:
##   * low: add tags about low-cardinality objects (clusters, hosts, deployments, container images, ...)
##   * orchestrator: add tags about pod (in Kubernetes), or task (in ECS/Mesos) -level of cardinality
##   * high: add tags about high-cardinality objects (individual containers, user IDs in requests, ...)

Real proof the env var is actually read, again via agent config:

$ docker run -d --name dd-card -e DD_API_KEY=<fake> \
    -e DD_CHECKS_TAG_CARDINALITY=orchestrator datadog/agent:7
$ docker exec dd-card agent config | grep checks_tag_cardinality
checks_tag_cardinality: orchestrator

The sibling setting DD_DOGSTATSD_TAG_CARDINALITY controls the same three-level choice for DogStatsD-submitted custom metrics specifically — covered in the custom-metrics lesson in the next module. Raising cardinality from low to high is a deliberate, explicit trade — not something that happens by accident the way an unbounded custom tag value does.

Unified service tagging

This is the tagging convention the rest of this course depends on. Three tag keys are reserved and treated specially across every Datadog product:

env — which environment this is running in (prod, staging, dev).
service — the name of the deployable unit (orders-api, checkout-worker).
version — the specific build/release of that service (1.4.2, a git SHA, whatever your release process produces).

The reason these three exist as a dedicated convention rather than just being “tags you could add yourself” is correlation: when a host’s Agent, an application’s tracer, and a log shipper all apply the exact same env, service, and version values to their respective telemetry, Datadog can join them. You look at a spike in orders-api’s error-rate metric for env:prod, version:1.4.2, jump straight to the traces carrying those same three tags, and from there to the exact log lines emitted during those requests — without ever hand-correlating timestamps or hostnames yourself. That pivot only works because the tag values, not just the keys, match exactly across all three telemetry types — a typo like Prod vs prod silently breaks the join (and, proven above, that specific mismatch would not be caught for you by any Agent-side normalization).

flowchart LR
    subgraph Tags["env:prod  service:orders-api  version:1.4.2"]
      M["Metrics"]
      T["Traces"]
      L["Logs"]
    end
    M -.same 3 tags.-> T
    T -.same 3 tags.-> L
    L -.same 3 tags.-> M

Unified service tagging ties telemetry together

Setting it consistently: env vars and Kubernetes labels

On a plain host or a single container, unified service tagging is set through three environment variables that the Agent and tracer both read:

export DD_ENV="prod"
export DD_SERVICE="orders-api"
export DD_VERSION="1.4.2"

In Kubernetes, the equivalent is a set of labels on the Deployment and Pod template — tags.datadoghq.com/env, tags.datadoghq.com/service, tags.datadoghq.com/version — and those same values are then re-injected as DD_ENV/DD_SERVICE/DD_VERSION environment variables using the Kubernetes downward API, so the tracer inside the container sees them too:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    tags.datadoghq.com/env: "prod"
    tags.datadoghq.com/service: "orders-api"
    tags.datadoghq.com/version: "1.4.2"
spec:
  template:
    metadata:
      labels:
        tags.datadoghq.com/env: "prod"
        tags.datadoghq.com/service: "orders-api"
        tags.datadoghq.com/version: "1.4.2"
    spec:
      containers:
        - name: my-app
          env:
            - name: DD_ENV
              valueFrom: { fieldRef: { fieldPath: "metadata.labels['tags.datadoghq.com/env']" } }
            - name: DD_SERVICE
              valueFrom: { fieldRef: { fieldPath: "metadata.labels['tags.datadoghq.com/service']" } }
            - name: DD_VERSION
              valueFrom: { fieldRef: { fieldPath: "metadata.labels['tags.datadoghq.com/version']" } }

The label lives in two places (Deployment metadata and Pod template metadata) because the Agent’s Autodiscovery reads pod labels directly for host/container-level tagging, while the downward API projection is what gets the same values into the process’s own environment for the tracer to pick up — one source of truth, two consumers.

Reading AI-generated code

🐛 Spot the bug

# AI comment: "Tag this host with team and region for easier filtering"
export DD_API_KEY="<DD_API_KEY>"
export DD_TAGS="team:payments api region:ap-southeast-1 tier:critical"

Show the answer

This looks like a normal, valid DD_TAGS assignment — three key:value pairs, space-separated, which is genuinely the documented format for separate tags. The bug, proven above with a real Agent’s agent status output: whoever wrote this meant the tag value to be payments api (a two-word service name with a space), but DD_TAGS splits unconditionally on whitespace with no quoting mechanism to escape it. The real result is four host tags, not three: team:payments, a bare keyless api tag, region:ap-southeast-1, and tier:critical — silently wrong, no error anywhere. The fix: either eliminate the space (team:payments-api or team:payments_api), or move this specific tag into the YAML tags: list instead, where each list item is one untouched string and a space inside it does not split.

Mental Model

Think of the Agent as a shipping clerk who writes the label exactly as handed to them, not a copyeditor. Whatever case, spacing, or format you give a tag, the Agent puts it on the package unchanged — agent status shows you literally that label. The warehouse (Datadog’s backend) is where lowercase-normalization and indexing actually happen, out of the Agent’s sight entirely. Meanwhile DD_TAGS is a clerk who was told “separate items with a space” and takes that instruction completely literally, even when a space was meant to be inside one item’s description — there’s no way to tell the clerk “this space is different,” which is exactly why the YAML tags: list (one string per line, no ambiguity) is the safer form whenever a tag value might contain a space.

Trade-offs

tags: (YAML list) vs. DD_TAGS (env var) — DD_TAGS is more convenient for anything already parameterized by environment variables (container orchestrators, CI-injected values) and requires no file mount; tags: in datadog.yaml is unambiguous for any value containing a space, proven above to be exactly the case DD_TAGS gets wrong silently.
DD_CHECKS_TAG_CARDINALITY: high vs. the low default — high gives you per-container or per-user-ID granularity when you genuinely need to debug at that resolution; proven above it’s real and reads correctly, but the same high-cardinality cost this lesson warns about for custom tags applies here too — it’s an explicit, config-level version of the same trade-off, not a free upgrade.
DD_CONTAINER_LABELS_AS_TAGS (opt-in mapping) vs. Autodiscovery com.datadoghq.ad.* labels — the labels-as-tags mapping is a blunt, host-wide “map this label to this tag on everything,” simple to reason about; Autodiscovery labels are check-specific and richer (full instance config, not just a tag), but only affect the one check they’re attached to. They solve different problems and are frequently both present on the same container without conflicting.

Common Misconceptions

“agent status shows me the tag exactly as Datadog will index and let me filter on it.” No — proven above: a tag configured as Team:Payments API shows up in agent status completely unmodified, uppercase and space included; lowercase and space-to-underscore normalization happens at Datadog’s backend, per docs, not on the Agent.
“DD_TAGS="key:value with spaces" produces one tag with an underscored value, the same as a tags: YAML entry would.” No — proven above: DD_TAGS splits on every whitespace character with no escaping, silently producing multiple tags (including a bare, keyless one) instead of the one intended tag.
“Docker/container labels automatically become Datadog tags.” No — proven above via agent config: they only become tags if you explicitly map them with DD_CONTAINER_LABELS_AS_TAGS; an unmapped label is invisible to Datadog’s tagging.
“Raising tag cardinality is just a UI setting with no cost implication.” No — DD_CHECKS_TAG_CARDINALITY: high (proven above to be real and Agent-read) trades the exact same custom-metrics billing and dropdown-usability cost this lesson describes for manually-added high-cardinality tags; it’s the same trade-off through a config knob, not an exception to it.