Unified Service Tagging in Practice

The idea in one sentence

Set env, service, and version once on a deployable unit, and that same triple is what lets a metrics dashboard, an APM service page, and the Logs Explorer all agree on exactly which service you are looking at.

One deployable unit, tagged once

Take a single deployable unit, checkout-service, and tag it with DD_ENV=production, DD_SERVICE=checkout-service, and DD_VERSION=2.4.1. Rather than hardcoding those three values in several places, the same Kubernetes labels/downward-API pattern from Foundations sets them once on the pod and reflects them into environment variables.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-service
spec:
  template:
    metadata:
      labels:
        tags.datadoghq.com/env: "production"
        tags.datadoghq.com/service: "checkout-service"
        tags.datadoghq.com/version: "2.4.1"
    spec:
      containers:
        - name: checkout-service
          image: registry.example.com/checkout-service:2.4.1
          env:
            - name: DD_ENV
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/env']
            - name: DD_SERVICE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/service']
            - name: DD_VERSION
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/version']

The labels are the single source of truth. The downward API reflects them into DD_ENV/DD_SERVICE/DD_VERSION, so those environment variables always match what Kubernetes — and the Agent’s autodiscovery — already see on the pod. Change the label once, and every consumer downstream picks up the new value on the next rollout.

Three consumers, one identity

The Agent’s infrastructure/container metrics — autodiscovery tags container and infra metrics straight from the pod’s labels, so CPU, memory, and request-rate metrics for checkout-service pods carry env, service, and version without any separate configuration.
The tracer initialization for APM — dd-trace reads DD_ENV, DD_SERVICE, and DD_VERSION from the environment automatically, so every span it creates is tagged with the same triple.

// dd-trace picks up DD_ENV, DD_SERVICE, and DD_VERSION from the environment automatically
require('dd-trace').init({
  logInjection: true
});

The log integration from the previous lesson — because log injection reads the same active service configuration the tracer uses, the dd.env, dd.service, and dd.version fields stamped onto every log line are that identical triple too.

Proof: case mismatch silently breaks the join

The pivot only works because the string is identical across all three products — and it turns out nothing in the client library normalizes that string for you. Ran dd-trace with DD_ENV=Prod (capital P) and dumped the real, resolved tracer config instead of assuming it gets lowercased:

$ DD_ENV=Prod DD_SERVICE=checkout-Service DD_VERSION=2.4.1 node -e "
    const t = require('dd-trace').init({ startupLogs: false });
    console.log('tags:', JSON.stringify(t._tracer._config.tags));
  "
tags: {"service":"checkout-Service","env":"Prod","version":"2.4.1", ...}

Prod and checkout-Service go out on the wire exactly as typed — dd-trace does not lowercase or otherwise normalize them client-side. Datadog’s own tagging docs (docs.datadoghq.com/getting_started/tagging/) state plainly that “a single tag key can have multiple values if it’s configured differently across sources” — the platform does not merge env:Prod and env:production into one value; they exist side by side as two different tag values under the same key. So if the Kubernetes label in the manifest above is tags.datadoghq.com/env: "Prod" while a teammate’s Terraform-managed service next to it sets DD_ENV=production (lowercase), the two services do not fail loudly — they just end up on two different env facets. A dashboard scoped to env:production silently shows one service and not the other; the APM service page for env:production doesn’t list the mismatched service at all. This is the same class of bug as the logInjection typo from the previous lesson: no exception, just a filter that quietly excludes data nobody meant to exclude. Datadog’s tagging docs do separately note that tag keys are normalized to lowercase (“Span tags and metric tags are normalized to lowercase, so avoid using camel case in tag keys”) — but that rule is about the key (env, service), not the value, which is exactly the gap this proof exposes.

Agent-level tags: `datadog.yaml`’s own `tags:` block

Unified service tagging is per-deployable-unit; the Agent also has its own, host-wide tags: list in datadog.yaml, applied to every metric/check the Agent itself collects on that host, independent of any single service’s DD_ENV/DD_SERVICE/DD_VERSION:

api_key: "<DD_API_KEY>"
hostname: probe-host
tags:
  - team:checkout
  - datacenter:local-probe

Ran a real datadog/agent:7 container (docker run --name dd-corr, DD_API_KEY an all-zeros fake, DD_DD_URL pointed at a closed local port so nothing was ever sent out) with this exact file mounted, then agent status:

$ docker exec dd-corr agent status
===============
Agent (v7.83.3)
===============
...
Hostname
========
  hostname: probe-host
  host tags:
    datacenter:local-probe
    team:checkout

Real Agent, real output: the two tags: entries in datadog.yaml show up verbatim as “host tags” — applied host-wide, on top of (not instead of) whatever env/service/version an individual service on that host sets for itself.

Container labels: `com.datadoghq.tags.` and Kubernetes `tags.datadoghq.com/`

The Kubernetes labels used earlier in this lesson have a direct Docker Compose/plain-Docker equivalent — com.datadoghq.tags.env/com.datadoghq.tags.service/com.datadoghq.tags.version container labels (docs.datadoghq.com/getting_started/tagging/unified_service_tagging/). Started a plain container with those labels and pointed the same dd-corr Agent’s docker.sock-based Autodiscovery at it — no config file, no annotation, just labels on docker run:

$ docker run -d --name labeled-app \
    --label com.datadoghq.tags.env=production \
    --label com.datadoghq.tags.service=checkout-service \
    --label com.datadoghq.tags.version=2.4.1 \
    alpine:3 sleep 300

$ docker exec dd-corr agent tagger-list | grep -A2 'container_name:labeled-app'
Tags: [container_id:e0f42036... container_name:labeled-app docker_image:alpine:3
env:production ... service:checkout-service ... version:2.4.1 ...]

agent tagger-list (the real command that dumps everything the Agent’s tagger has resolved for every entity it sees) shows env:production, service:checkout-service, and version:2.4.1 attached to the container purely from its Docker labels — the same three-key convention as the Kubernetes labels, translated for a plain-Docker or Compose deployment that has no Kubernetes downward API to reflect labels into env vars.

`service` override on a single span

Unified service tagging sets the default for an entire process, but a single span can still declare a different service — useful for a database client span that should roll up under the database’s own service identity instead of the calling app’s:

$ DD_SERVICE=checkout-service DD_ENV=production node -e "
    const tracer = require('dd-trace').init({ startupLogs: false });
    tracer.trace('pg.query', { service: 'checkout-postgres' }, (span) => {
      console.log(JSON.stringify(span.context()._tags));
    });
  "
{"service.name":"checkout-postgres","service":"checkout-service","env":"production", ...}

The span’s own service.name tag is checkout-postgres (the override passed to tracer.trace()’s options), while the process-wide service tag stays checkout-service underneath it — a targeted exception to the “set once” rule above, not a contradiction of it: most spans in the process still default to the process-wide identity, and only the one span that opted out carries a different service.

The pivot: one triple, three products, zero cross-referencing

With the triple flowing into all three products, a single incident walk looks like this: start on a metrics dashboard scoped to service:checkout-service env:production, and see CPU, latency, and request rate climbing. Jump to the APM service page for the same service, env, and version, and see trace latency, error rate, and the service’s neighbors on the service map. Jump again to the Logs Explorer, pre-filtered to that same triple, and see the exact log lines — including any dd.trace_id-tagged errors from the previous lesson — for that version, in that environment.

Compare that with the before state: metrics tagged service:checkout, a Java tracer tagged service:checkout-svc, and logs tagged application:checkout. Nothing lines up, and whoever is debugging has to remember three different names while moving between products. After tagging consistently, the string is identical everywhere, so any of the three products’ UI can pre-filter to the same identity, and a saved view in one product describes the exact same slice of the world in another.

flowchart LR
  T["env=production\nservice=checkout-service\nversion=2.4.1"] --> Agent[Agent autodiscovery]
  T --> Tracer[Tracer init]
  T --> LogInj[Log injection]
  Agent --> M[Metrics dashboard]
  Tracer --> A[APM service page]
  LogInj --> L[Logs Explorer]
  M -.pivot.-> A
  A -.pivot.-> L
  L -.pivot.-> M

One triple, three products

Reading AI-generated code

🐛 Spot the bug

metadata:
labels:
  tags.datadoghq.com/env: "Production"
  tags.datadoghq.com/service: "checkout-service"
  tags.datadoghq.com/version: "2.4.1"
---
# k8s/checkout-worker.yaml  (AI comment: "same env as checkout-service")
metadata:
labels:
  tags.datadoghq.com/env: "production"
  tags.datadoghq.com/service: "checkout-worker"
  tags.datadoghq.com/version: "2.4.1"

Show the answer

Both manifests look consistent at a glance, and both are perfectly valid YAML — nothing here throws an error or fails a kubectl apply. The bug, proven above with a real tracer config dump: "Production" and "production" are not the same tag value. checkout-service and checkout-worker will end up on two different env facets in Datadog. A dashboard, monitor, or SLO scoped to env:production silently includes checkout-worker and excludes checkout-service — the exact service someone most likely meant to include. The fix is purely mechanical: pick one casing convention (lowercase is the safe default, matching how Datadog treats tag keys) and apply it identically everywhere tags.datadoghq.com/env (or DD_ENV) is set, across every manifest, Terraform file, and Dockerfile label in the org.

Mental Model

Think of the env/service/version triple as a shared password across three different systems that never talk to each other directly. Metrics, APM, and Logs are three separate products with three separate storage and query engines — the only reason a dashboard in one can pre-filter to “the same thing” as a service page in another is that all three happen to have been handed the exact same string. Nothing enforces that sameness centrally; there is no registry that says “these are all one service.” It works purely by convention — the same convention this lesson shows flowing through Kubernetes labels, Docker labels, and environment variables — and it breaks exactly where that convention is applied inconsistently, silently, with no error anywhere in the pipeline.

Trade-offs

Kubernetes labels + downward API vs. Docker labels (com.datadoghq.tags.*) — the downward API pattern gives you one source of truth (the pod label) reflected automatically into env vars for any container in the pod; Docker/Compose labels (proven above via agent tagger-list) need no downward-API-equivalent machinery at all, but only work where the Agent has Docker socket access for Autodiscovery, and don’t reflect into the app’s own environment automatically.
Process-wide identity vs. a per-span service override — leaving every span on the process-wide service keeps the service map simple, one node per deployable unit; overriding service on selected spans (proven above, a pg.query span tagged checkout-postgres) gives a more accurate service map (the database shows up as its own node) at the cost of every reader needing to know that convention exists before the service map makes sense.
Enforcing lowercase tag values by convention vs. by tooling — a documented lowercase-only rule costs nothing to write down but nothing stops a new Terraform module or a copy-pasted YAML file from breaking it (proven above); a CI lint step that rejects any tags.datadoghq.com/env/DD_ENV/DD_SERVICE/DD_VERSION value containing an uppercase character catches it before merge, at the cost of one more check every deploy has to pass.

Common Misconceptions

“Datadog automatically merges env:Prod and env:production into one value.” No — proven above with a real tracer config dump and Datadog’s own tagging docs: mismatched casing produces two separate, coexisting tag values under the same key, not one merged value.
“Tag values are lowercased the same way tag keys are.” No — Datadog’s docs state that tag keys are normalized to lowercase; the docs make no such claim about values, and the client-side proof above shows dd-trace sends env/service/version values exactly as configured, capital letters included.
“Setting service on one span changes the service identity for the whole process.” No — proven above: overriding service in tracer.trace()’s options changes only that one span’s service.name tag; the process-wide DD_SERVICE value is untouched and still applies to every other span.
“Container labels are Kubernetes-only; plain Docker/Compose has no equivalent.” No — proven above: com.datadoghq.tags.env/service/version Docker labels are picked up by the Agent’s docker.sock-based Autodiscovery exactly like Kubernetes pod labels are, just via a different mechanism (no downward API needed).