Production Rollout Checklist

The idea in one sentence

Going to production with Datadog is a checklist, not a single switch, and every item on it traces back to a decision this course already taught you how to make — from Foundations tagging, through Infrastructure & Metrics, Log Management, APM & Distributed Tracing, Monitors/Alerting/SLOs, and Correlating Signals.

🐶 Proof: what was run for this lesson, and what was not

Run locally, nothing sent to Datadog (fake keys, DD_DD_URL=http://127.0.0.1:1, Mac): Datadog Agent 7.83.3 (agent version from datadog/agent:7) reading container environment variables through a read-only Docker socket and printing them with agent tagger-list; helm template of the datadog/datadog chart 3.250.3; the DatadogAgent JSON schema published in the DataDog/datadog-operator repo at tag v1.30.0 (the Operator chart datadog/datadog-operator 2.27.0 ships app version 1.30.0) used as a JSON Schema validator; the upstream Kubernetes deployment-apps-v1 JSON schema (v1.33 standalone-strict) for the Deployment fences. Not run, docs only: a real Kubernetes cluster (the harness --k8s mode needs a kind-verify-k8s cluster that this session did not have), Bits AI, and anything that needs a Datadog org.

Tagging and ownership, first because everything else depends on it

• Unified service tagging is consistent everywhere. env, service, and version are applied the same way across Agent configuration, APM tracer setup, and every logs: source — this is the Foundations-module tagging model that lets a log line, a trace, and a metric correlate back to one deployable unit. Fixing this after go-live means every dashboard and monitor built in the meantime silently missed some of your telemetry.
• Service Catalog entries exist and are accurate. Every service that will page someone has a Service Catalog entry with real ownership: which team, which escalation path, which links to runbooks. This is what the Correlating Signals module set up, and it’s the difference between an alert reaching the on-call engineer and an alert reaching a stale mailing list.

The unified tagging checklist, item by item

From https://docs.datadoghq.com/getting_started/tagging/unified_service_tagging/:

1. The three reserved tags are env, service and version, mapped to DD_ENV, DD_SERVICE and DD_VERSION.
2. “In containerized environments, env, service, and version are set through the service’s environment variables or labels (for example, Kubernetes deployment and pod labels, Docker container labels). The Datadog Agent detects this tagging configuration and applies it to the data it collects from containers.”
3. On Kubernetes, “To get the full range of unified service tagging in Kubernetes, add the following labels to both the parent workload and the pod template”: tags.datadoghq.com/env, tags.datadoghq.com/service, tags.datadoghq.com/version. The pod-level-metrics variant says to add the labels “to the pod spec, as well as optionally to the parent workload”.
4. DD_ENV/DD_SERVICE/DD_VERSION on the application are “only required if you are using APM in your applications”.
5. With Docker, “make sure the Agent can access your container’s Docker socket”.
6. “On ECS Fargate, you must add these tags to your application container, not the Datadog Agent container.”
7. version “is expected to change with each new application deployment”.
8. A global env tag on the Agent plus a different env on a pod means “that pod’s data contains both env tags” (same page). Pick one owner for env.

Point 2 is the one that can be proven without a cluster. I started an Agent with the Docker socket mounted read-only (-e DD_ENV=prod -e DD_TAGS=team:checkout on the Agent itself) and two throwaway containers, one with all three variables and one where DD_SERVICE was misspelled DD_SERVCE:

$ docker exec dd-production agent status | grep -A3 "host tags"
  host tags:
    env:prod
    team:checkout

$ docker exec dd-production agent tagger-list     # container_id shortened
Tags: [container_id:<id> container_name:dd-production-good docker_image:busybox env:prod image_name:busybox service:checkout-api short_image:busybox version:1.4.2]
Tags: [container_id:<id> container_name:dd-production-partial docker_image:busybox env:prod image_name:busybox short_image:busybox]

Two facts fall out. First, the Agent turned DD_ENV/DD_SERVICE/DD_VERSION inside the container into env:/service:/version: tags without any tracer being involved. Second, the misspelled DD_SERVCE produced no error, no warning and no service: tag at all; the container just shows up half-tagged. The Agent’s own DD_ENV became a host-level tag, but there is no equivalent Agent-level DD_SERVICE host tag in agent status — service and version belong to the workload, not the host.

On Kubernetes the same information arrives through pod labels and the container env. A Deployment that satisfies the docs’ “full configuration” (this validates against the upstream Kubernetes schema):

k8s/checkout-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  labels:
    tags.datadoghq.com/env: "prod"
    tags.datadoghq.com/service: "checkout-api"
    tags.datadoghq.com/version: "1.4.2"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
        tags.datadoghq.com/env: "prod"
        tags.datadoghq.com/service: "checkout-api"
        tags.datadoghq.com/version: "1.4.2"
    spec:
      containers:
        - name: checkout-api
          image: registry.example.com/checkout-api:1.4.2
          env:
            - name: DD_ENV
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/env']
            - name: DD_SERVICE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/service']
            - name: DD_VERSION
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/version']

The fieldRef env vars read from the pod’s own labels, so there is one place to change a value. This is the docs’ own pattern; the pod-level reading of those labels by the Agent needs a live cluster and was not run here.

Kubernetes install: Datadog Operator or Helm

The install page (https://docs.datadoghq.com/containers/kubernetes/installation/) lists three methods and marks one: “Datadog Operator (recommended): a Kubernetes operator that you can use to deploy the Datadog Agent on Kubernetes and OpenShift. It reports deployment status, health, and errors in its Custom Resource status, and it limits the risk of misconfiguration thanks to higher-level configuration options.” The other two are Helm and manual DaemonSet installation. Note that the Operator’s own install commands use Helm (helm install datadog-operator datadog/datadog-operator) — the choice is between managing the Agent with a DatadogAgent resource and managing it with the datadog/datadog chart’s values, not between “Helm” and “no Helm”. The docs give no feature-parity table between the two and this lesson does not invent one.

Same intent, both ways. The Operator form (a DatadogAgent, collected as a k8s fence):

k8s/datadog-agent.yaml

apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
spec:
  global:
    clusterName: checkout-prod
    site: datadoghq.com
    credentials:
      apiSecret:
        secretName: datadog-secret
        keyName: api-key
    tags:
      - "env:prod"
    podLabelsAsTags:
      app.kubernetes.io/team: team
  features:
    apm:
      enabled: true
    logCollection:
      enabled: true
      containerCollectAll: true

The Helm form (values for the datadog/datadog chart, not a Kubernetes object, so it is deliberately not a collected k8s/ fence):

# helm values for datadog/datadog (not collected by the k8s harness)
datadog:
  apiKeyExistingSecret: datadog-secret
  clusterName: checkout-prod
  site: datadoghq.com
  tags:
    - "env:prod"
  podLabelsAsTags:
    app.kubernetes.io/team: team
  apm:
    portEnabled: true
  logs:
    enabled: true
    containerCollectAll: true

helm template datadog datadog/datadog --version 3.250.3 -f values.yaml exits 0 and the rendered Agent DaemonSet carries DD_CLUSTER_NAME=checkout-prod, DD_TAGS=["env:prod"], DD_KUBERNETES_POD_LABELS_AS_TAGS={"app.kubernetes.io/team":"team"}, DD_APM_ENABLED=true and DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true. The DatadogAgent fence validates against the v2alpha1 schema from the Operator repo at v1.30.0 (the schema sets additionalProperties: false, so unknown keys are rejected).

Now the difference that matters for review. Misspell two keys in each (clusterName to cluster_name, containerCollectAll to collectAll):

# DatadogAgent, validated against the v2alpha1 schema
ERR spec/features/logCollection - Additional properties are not allowed ('collectAll' was unexpected)
ERR spec/global - Additional properties are not allowed ('cluster_name' was unexpected)

# Helm values, helm template datadog/datadog 3.250.3
exit code 0
DD_CLUSTER_NAME occurrences in rendered output: 0
DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL: "false"

The Operator side rejects both typos. The Helm side accepted both, dropped the cluster name entirely, and left container log collection at the chart default "false" while logs.enabled stayed true — an Agent that looks like it is collecting logs and collects none. This is a property of these two artifacts at these versions (the chart’s values.schema.json did not reject unknown keys), not a general verdict, but it is exactly the kind of bug an AI-written values file carries.

Trade-off summary: the Operator gives a typed, status-reporting resource and rejects unknown fields at admission; Helm has the larger install base and plain-values workflow that most existing GitOps repos already use. Whichever you pick, run the rendered or applied manifest through a schema check in CI. The docs also note that as of Operator chart version 2.19.0 the Agent Helm chart’s registryMigrationMode applies to Agent images managed by the Operator, so image-registry settings can flow between the two.

Log and metric cost, decided deliberately, not by default

• Log indexes are sized on purpose. Retention and daily quota per index reflect the actual value of that log stream, not “index everything and figure it out later” — the Log Management module’s whole point.
• Exclusion filters exist for high-volume, low-value logs before volume scales up in production, not retrofitted after the first surprising invoice.
• Log-based metrics are set up for anything you plan to exclude. If you still need counts or trends from a log stream you’re about to filter out, configure log-based metrics first — excluding the raw logs should never silently blind you to a trend you cared about.
• Custom metric tags are reviewed with cardinality in mind. High-cardinality tags like pod_name can still be submitted; Metrics without Limits controls which ones are actually indexed and queryable, and that choice should be deliberate before go-live, not discovered from a cost spike.

APM sampling and retention reviewed as two separate layers

• Ingestion sampling is set intentionally. This is the layer that decides how much of total trace volume reaches Datadog at all — it should be a deliberate rate, not whatever the default happened to be.
• Retention filters are reviewed separately from sampling. Confirm which traces get retained long-term regardless of sampling — for example, everything with an error, or everything above a latency threshold — and that nobody on the team is conflating “we sample less” with “we retain less.” They are two different knobs from the APM & Distributed Tracing module, and go-live is the time to check both, not just one.

Alerting wired to what actually matters

• SLOs are defined for the handful of services the business actually depends on — not a vanity metric copied onto every service just because it was easy to add.
• Burn-rate alerts are wired to those SLOs, so a team gets notified while there’s still error budget left to react, not after it’s already gone. This is the Monitors, Alerting, and SLOs module’s core mechanism, and it only works if it’s actually connected before go-live.
• Every monitor’s notification target is checked against the Service Catalog owner from the tagging step above, so a page reaches the team that can actually act on it.

Manage it all as code, review it like code

• Monitors, dashboards, and SLOs are expressed as Terraform resources — datadog_monitor, datadog_dashboard, datadog_service_level_objective — rather than clicked together by hand, as covered earlier in this module. Validate every HCL block with terraform validate on the pinned provider in CI.
• terraform plan runs with query validation on, as part of ordinary pull request review, before any monitor or dashboard change reaches production — the same review bar as any other infrastructure change.
• RBAC is checked — see the previous lesson: logs_modify_indexes is global, can delegate Logs Read Index Data and Logs Write Exclusion Filters, and is held by the managed Standard role by default, so “only the platform team” is something you have to make true with custom roles.
• A Cloud Cost Management cost alert monitor exists for the services or accounts this rollout affects, so a cost spike gets the same alerting treatment as a latency spike, not a surprise at the end of the month.

Where Bits AI fits (docs-quoted, nothing was run)

Everything in this section is from the docs; no Bits AI feature was used, and Datadog’s pricing and rollout status can change, so read the linked pages at go-live time.

• What it is: “Bits AI is your agentic teammate in Datadog, built to automate development, security, and operational workflows. You can chat and collaborate with Bits in real time, or delegate full tasks—like alert investigations, code fixes, or security triage—and let it take care of the details.” (https://docs.datadoghq.com/bits_ai/)
• The docs list seven features there: Bits Investigation, Bits Code, Bits Security Analyst, Bits Chat, Bits Data Analysis, Bits Detection and Bits Remediation. The page for Bits Investigation lives at the older path bits_ai/bits_ai_sre and the blog links still say “Bits AI SRE” — a naming trap when you read older posts.
• Bits Investigation: “an autonomous AI agent that investigates production issues end to end. It iteratively forms hypotheses, gathers relevant telemetry, and uses data-based reasoning to help on-call engineers pinpoint root causes.” (https://docs.datadoghq.com/bits_ai/bits_ai_sre/)
• Availability: the docs show a warning that “This product is not supported for your selected Datadog site” for app.ddog-gov.com and us2.ddog-gov.com.
• Cost: “Learn about Bits AI pricing on AI Credits” (https://docs.datadoghq.com/account_management/billing/ai_credits/). No price is quoted here.
• Wiring: its integrations page says it integrates with GitHub, Grafana, Dynatrace, Splunk, Sentry and ServiceNow, that GitHub setup includes “Tag your APM telemetry with Git information”, and that for monitor alerts, if the monitor already has @slack, @case or @oncall notifications, “Bits automatically posts its findings to those destinations” (https://docs.datadoghq.com/bits_ai/bits_investigation/configure/). Microsoft Teams support is labelled Preview there.

What this means for the checklist: Bits AI reads the telemetry you already send, so it is downstream of every item above. Do not put it on the critical path for go-live; decide, per feature, whether the AI Credits cost and its access to your code and telemetry fit your data policy, and only after tagging, ownership and monitor routing are correct — those are what an investigation has to work with.

Go-Live Checklist
1. Unified tagging (env/service/version) verified across Agent, APM, logs; pod template labels present
2. Service Catalog ownership accurate for every paging service
3. Kubernetes install method chosen (Operator or Helm) and the manifest/values schema-checked in CI
4. Log indexes and exclusion filters sized deliberately; log-based metrics set up first
5. Custom metric tag cardinality reviewed via Metrics without Limits
6. APM ingestion sampling and retention filters reviewed as two separate layers
7. SLOs and burn-rate alerts wired to business-critical services only
8. Monitor notification targets checked against Service Catalog owners
9. Monitors/dashboards/SLOs as Terraform code, reviewed via PR with query validation on
10. RBAC checked: logs_modify_indexes and exclusion/archive rights on custom roles, read access broad
11. Cloud Cost Management cost alert in place for the affected services/accounts
12. Optional: Bits AI features evaluated against cost (AI Credits) and data policy

flowchart LR
  A[Tagging and ownership] --> K["Kubernetes install: Operator or Helm"]
  K --> B[Log and metric cost decisions]
  B --> C[APM sampling and retention review]
  C --> D[SLOs and burn-rate alerting]
  D --> E[Infrastructure as code review]
  E --> F[Go-live]
  F -.optional.-> I["Bits AI evaluation"]
  A -.Service Catalog owner.-> D
  B -.cost alert monitor.-> E

Production rollout checklist, phase by phase

🐶 The one-line takeaway

A production rollout checklist for Datadog is not one switch to flip — it’s unified tagging (labels on the pod template, misspelled variables fail silently) and Service Catalog ownership set first, the Operator or Helm install schema-checked, log/metric cost decided deliberately, APM sampling and retention filters reviewed as two separate layers, SLOs and burn-rate alerts wired to business-critical services, all of it managed as reviewed Terraform code with RBAC and cost alerting checked before go-live, and Bits AI treated as an optional layer on top.

Reading AI-generated code

🐛 Spot the bug

# AI comment: "Unified service tagging for checkout-api"
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
labels:
  tags.datadoghq.com/env: "prod"
  tags.datadoghq.com/service: "checkout-api"
  tags.datadoghq.com/version: "1.4.2"
spec:
replicas: 2
selector:
  matchLabels:
    app: checkout-api
template:
  metadata:
    labels:
      app: checkout-api
  spec:
    containers:
      - name: checkout-api
        image: registry.example.com/checkout-api:1.4.2

Show the answer

This is a valid Deployment — it passes the Kubernetes schema — and it looks tagged. The labels are only on the Deployment’s own metadata. The Datadog docs ask for them “on both the parent workload and the pod template” for full configuration, and for the pod-level path they are required on the pod spec while the parent workload is optional. Labels on metadata.labels of a Deployment are not copied onto its pods, so the pods that the Agent actually sees carry no tags.datadoghq.com/* labels (this is standard Kubernetes behaviour, not something this lesson ran against a cluster). The fix is the manifest shown earlier in this lesson: the same three labels under spec.template.metadata.labels, plus DD_ENV/DD_SERVICE/DD_VERSION from fieldRef if the app uses APM.

k8s/checkout-deployment-fixed.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  labels:
    tags.datadoghq.com/env: "prod"
    tags.datadoghq.com/service: "checkout-api"
    tags.datadoghq.com/version: "1.4.2"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
        tags.datadoghq.com/env: "prod"
        tags.datadoghq.com/service: "checkout-api"
        tags.datadoghq.com/version: "1.4.2"
    spec:
      containers:
        - name: checkout-api
          image: registry.example.com/checkout-api:1.4.2

Mental Model

A go-live checklist is a dependency graph, not a to-do list. Tags are the join key for logs, traces and metrics, so they come first; ownership is the join key for alerts and people; the install method decides how the Agent gets those tags; cost and sampling decide what you keep; SLOs and alerts decide who is woken; Terraform and RBAC decide who can change any of it. Bits AI sits on top: it consumes what the lower layers produce, so it inherits their quality and their gaps.

Trade-offs

• Operator vs Helm — the docs recommend the Operator and it rejected unknown fields in the check above; Helm is the more common existing workflow and, at chart 3.250.3, silently ignored unknown keys. Switching costs an install-path migration, so pick before go-live, not after.
• Labels on both workload and pod template vs pod template only — both is the docs’ “full configuration”; pod-only is the docs’ minimum for pod-level metrics. Extra labels cost a few lines and keep the Deployment queryable by the same tags.
• Adopting Bits AI early vs later — earlier gives on-call help sooner but adds an AI Credits cost line and grants access to code and telemetry (per its integration docs); later lets you fix tagging and routing first. The docs quote no price, so the cost side is unknown until you check your account.
• Schema-checking manifests in CI vs trusting the install command — a schema check costs a pipeline step and catches the silent-typo class that both the Helm run and the misspelled DD_SERVCE container demonstrated.

Common Misconceptions

• “If the Deployment has the labels, the pods are tagged.” Docs require the pod template; a Deployment’s own labels are not pod labels.
• “Helm will complain if I mistype a values key.” At chart 3.250.3 helm template exited 0 and dropped the mistyped keys; the Operator’s schema rejected them.
• “The Operator replaces Helm.” The documented Operator install itself runs helm install datadog-operator datadog/datadog-operator; it replaces the datadog/datadog chart’s values with a DatadogAgent resource.
• “Setting DD_ENV on the Agent tags everything correctly.” It becomes a host-level tag (agent status shows env:prod), and the docs warn that a different env on a pod yields both tags on that pod’s data.
• “Bits AI SRE is a separate product from Bits Investigation.” The docs page titled Bits Investigation sits at the bits_ai_sre path; older posts use the earlier name.

Why should ingestion sampling and retention filters both be reviewed separately before go-live?

They are the same setting under two different names, so reviewing one covers the other They are two separate layers — sampling controls how much trace volume arrives at all, retention filters control what is kept long-term regardless of sampling Retention filters only matter for logs, not traces Ingestion sampling is deprecated and no longer configurable

Before excluding a high-volume, low-value log stream, what should be set up first?

Nothing — exclusion filters cannot affect visibility into trends A log-based metric, if you still need counts or trends from that stream A new Service Catalog entry A cost alert monitor specifically for that log stream

A Deployment has tags.datadoghq.com/env|service|version labels only under its own metadata.labels. According to the unified service tagging docs, what is missing?

Nothing — labels on the Deployment propagate to pods The same labels on the pod template, which the docs require for full configuration (both parent workload and pod template) A DD_HOSTNAME variable A Service Catalog entry

A container is started with DD_ENV=prod and a misspelled DD_SERVCE=checkout-api. What did agent tagger-list show for it in this lesson?

An error naming the unknown variable env:prod but no service: tag — no error or warning was produced service:checkout-api, because the Agent corrects typos No tags at all

You mistype containerCollectAll as collectAll in a DatadogAgent manifest and, separately, in a Helm values file for chart 3.250.3. What did the checks in this lesson show?

Both fail loudly The DatadogAgent schema check rejected it ("Additional properties are not allowed"); helm template exited 0 and left DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL at "false" Both silently succeed The Helm run rejected it, the Operator run accepted it

What do the Bits AI docs say about pricing, and how does this lesson treat it?

A fixed monthly price is stated and quoted They point to the AI Credits page; the lesson quotes no number and tells you to check your account Bits AI is free for all sites Pricing is only available for the gov sites

Submit

Mark as complete