OTLP and Open Standards

The idea in one sentence

OTLP is a wire format, not a Datadog feature, so two translators sit between an OpenTelemetry SDK and a Datadog view — the Agent’s OTLP receiver and the Collector’s Datadog exporter (or its newer OTLP HTTP path) — and each one applies the same semantic-convention mapping (service.name to service, http.request.method to http.method, span kind and db.system to an operation name and span type), so the useful question is never “does it accept OTLP” but “what did it turn my attributes into”.

🐶 How every proof in this lesson was made

Real Agent 7.83.3 (datadog/agent:7, container dd-internals, fake key), its intake pointed at a local fake HTTP intake, with OTLP/HTTP JSON posted by curl and the Agent’s forwarded payloads decoded (the same protobuf wire decoder as DogStatsD and Aggregation). Collector configs were checked with the real otelcol-contrib 0.161.0 (--version output) from a locally present otel/opentelemetry-collector-contrib image, using validate and, for one wiring bug, a few seconds of a real start, always with --network none so nothing could leave the host. Two of my early Agent runs did contact real Datadog intakes with the fake key (the process endpoint, and once the database-monitoring event intake, both answered 403); the runs whose output appears below either point every intake at the local fake or use --network none. Docs are quoted with URLs (fetched 2026-09-30). The Phase 3 lesson What Is Datadog already shows the Agent receiving OTLP and a Collector validate; this lesson goes into what the translation does. Timings and counts: this Mac.

The two paths, once

flowchart LR
  SDK["App with OpenTelemetry SDK"] -->|"OTLP grpc :4317 or http :4318"| AG["Agent OTLP receiver: otlp_config"]
  SDK -->|"OTLP"| COL["OpenTelemetry Collector"]
  AG --> MAP1["Semantic mapping and sampling in the Agent"]
  MAP1 --> INT["Datadog intake"]
  COL -->|"datadog exporter, or otlp_http exporter"| MAP2["Semantic mapping in exporter or intake"]
  MAP2 --> INT

From an OpenTelemetry SDK to Datadog

OTLP ingest in the Agent

It is off until you turn it on

The docs say it plainly: “OTLP ingestion is off by default” (OTLP ingestion by the Datadog Agent). A default container:

$ docker exec dd-internals agent status
OTLP
====
  Status: Not enabled
  Collector status: Not running

$ curl -X POST -H 'Content-Type: application/json' --data @otlp_trace.json http://127.0.0.1:14318/v1/traces
curl: (52) Empty reply from server        # HTTP 000: nothing is listening behind the published port

Nothing listens, so an OTel SDK pointed at the Agent fails on export (SDKs log and drop by design). Turning it on gives two Status lines with different meanings, and the difference is the whole story of the next section.

The keys, the defaults and the two lies of configcheck

agent config --all in the running container shows the settings that matter and their defaults:

otlp_config:
  metrics:
    enabled: true
    histograms:
      mode: distributions
    sums:
      cumulative_monotonic_mode: to_delta
    resource_attributes_as_tags: false
  logs:
    enabled: false
  traces:
    enabled: true
    internal_port: 5003
    span_name_as_resource_name: false
    probabilistic_sampler:
      sampling_percentage: 100
  receiver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

(Trimmed: the batch, keepalive and CORS blocks are left out. The two endpoints read 0.0.0.0 only because this container’s env set them; the receiver has no endpoint until you do.) The docs give the two knobs that turn it on: “For gRPC (localhost:4317): DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT; for HTTP (localhost:4318): DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT”, and a sentence that is easy to miss: “These must be passed to both the core Agent and trace Agent processes.” Logs are off by default “to prevent unexpected logs billing.”

Now the proof that configcheck does not check this. A datadog.yaml with one wrong word, protocol for protocols:

# @expect-error otlpreceiver.Config has invalid keys: protocol
api_key: <your-api-key>
otlp_config:
  receiver:
    protocol:
      http:
        endpoint: 0.0.0.0:4318

Started with that file (container --network none), then the three commands people run to check it:

$ agent configcheck ; echo "configcheck exit $?"
configcheck exit 0

$ agent status
OTLP
====
  Status: Enabled
  Collector status: Closed

$ grep "OTLP ingest" /var/log/datadog/agent.log
CORE | ERROR | (comp/otelcol/collector/impl-pipeline/pipeline.go:116 in func1) | Error running the OTLP ingest pipeline: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):
'receivers' error reading configuration for "otlp": decoding failed due to the following error(s):
'otlpreceiver.Config' has invalid keys: protocol

agent configcheck lists resolved check configurations from conf.d/; it never looks at datadog.yaml’s otlp_config and exited 0. agent status said Status: Enabled (the setting is present) but Collector status: Closed (the pipeline died on startup); only the last line of the log names the bad key. So two different lines mean two different things, and the harness’s --agent gate, which runs configcheck over a lesson’s # datadog.yaml fences, proves the Agent starts with the file, not that OTLP works. Read Collector status and grep the log. With protocols spelled right (the fix, collected by the harness):

datadog.yaml

api_key: <your-api-key>
otlp_config:
  receiver:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
  logs:
    enabled: false

the same three commands gave configcheck exit 0, Status: Enabled, Collector status: Running.

localhost versus 0.0.0.0 is a network boundary

The docs’ default env values are localhost:4317 and localhost:4318, and their container guidance is: “If running in a containerized environment, use 0.0.0.0 instead of localhost to ensure the server is available on non-local interfaces.” I set gRPC to localhost:4317 and HTTP to 0.0.0.0:4318 in one container and read the listening sockets from /proc/net/tcp, then connected from a second container:

listening (inside the Agent container):   127.0.0.1:4317     [::]:4318
from another container, to the Agent's IP:  4317 ConnectionRefusedError [Errno 111] Connection refused
                                            4318 open

An Agent in its own pod or container with the default bind is unreachable by its neighbours. The docs add a second constraint that no config can fix: telemetry from collectors or instrumented apps on one host cannot be sent to an Agent on a different host (“The supported setup is an ingesting Agent deployed on every OpenTelemetry-data generating host”) — the supported topology is one Agent per host.

What the Agent does to an OTLP span

One trace, two spans, posted as OTLP/HTTP JSON. The resource carried service.name=checkout-otel, service.version=1.4.2, deployment.environment.name=lab; the server span was GET /orders/{id} with http.request.method=GET, http.route=/orders/{id}, http.response.status_code=500, status Error (“upstream failed”); the child was a client span with db.system=postgresql and db.statement. What arrived at the fake intake (/api/v0.2/traces, decoded):

tracer_version 'otlp-'  env lab  hostname internals-host  app_version 1.4.2
chunk priority 1  tags {_dd.p.dm: -9, _dd.otlp_sr: 1.00}
 span service=checkout-otel  name=http.server.request  resource="GET /orders/{id}"  type=web  error=1  duration 50 ms
    meta  http.method=GET  http.route=/orders/{id}  span.kind=server  env=lab  version=1.4.2
          error.msg="upstream failed"  otel.status_code=Error  otel.status_description="upstream failed"
          otel.scope.name=lab.instrumentation  otel.trace_id=5b8efff798038103d269b633813fc60c
    metric http.status_code=500.0  _top_level=1.0
 span service=checkout-otel  name=postgresql.query  resource="SELECT * FROM orders WHERE id = ?"  type=sql  duration 30 ms
    meta  db.system=postgresql  db.statement="SELECT * FROM orders WHERE id = $1"  sql.query="SELECT * FROM orders WHERE id = ?"

Each translation is a rule, and every one agrees with the docs table (semantic mapping): service.name to service, service.version to version, deployment.environment.name to env, http.request.method to http.method, http.response.status_code to http.status_code (and, as measured, it becomes a numeric metric not a string tag). The rest are inferences the Agent makes and the docs describe as span-type logic in order of precedence — explicit span.type, then “Span kind server” to web — which is why the server span is web and the postgresql client span is sql. The operation name http.server.request and the resource GET /orders/{id} are derived, not copied: the resource is the method plus http.route (the old-convention run below keeps the same OTel span name yet gets the resource GET), and the DB span’s resource is the statement with its $1 obfuscated to ?, not its OTel span name SELECT orders. The trace keeps its OTel origin as otel.* tags.

Three variations, each run:

old semantic conventions (http.method, http.url, http.status_code, deployment.environment):
   span name=http.server.request  resource="GET"        env=lab   http.status_code=500.0
   # no http.route, so the resource is only the method; deployment.environment still maps to env

otlp_config.traces.span_name_as_resource_name=true:
   span name=GET_orders_id  resource="GET /orders/{id}"
   span name=SELECT_orders  resource="SELECT orders"

otlp_config.traces.probabilistic_sampler.sampling_percentage=25, 400 traces posted:
   chunks forwarded 98   _dd.p.dm=-9  _dd.otlp_sr=0.25

Old and new attribute names both reach env/http.method on 7.83.3 (the docs footnote says deployment.environment.name “Requires Datadog Agent 7.58.0+, Datadog Exporter v0.110.0+, or Datadog OTLP intake”). Without http.route a server span’s resource collapses to the bare method — the classic cause of one giant GET resource in the service page. The sampler is the link back to The Trace Sampling Pipeline: an OTLP trace never went through a Datadog tracer’s head sampler, so the Agent’s otlp_config.traces.probabilistic_sampler is what samples it (98 of 400, 24.5% at a 25% setting; mechanism -9, the same one as the Agent’s probabilistic sampler there, plus _dd.otlp_sr recording the rate). The docs list a limit that follows from this: “Data from applications instrumented with OpenTelemetry SDK cannot be used in some Datadog proprietary products, such as App and API Protection, Continuous Profiler, and Ingestion Rules.” OTLP through the Agent has no DD_TRACE_SAMPLING_RULES and no Ingestion Control page rates.

What the Agent does to OTLP metrics

Three OTLP metric types posted, then a second export ten seconds later, with the resource attributes service.name and deployment.environment.name:

OTLP gauge lab.queue.depth = 42 {queue=jobs}
   -> series gauge   42.0   tags [queue:jobs, service:checkout-otel, env:lab, instrumentation_scope:lab.meter, instrumentation_scope_version:n/a]

OTLP cumulative monotonic sum lab.requests = 10, then 25 {route=home}
   -> first export: no point at all
   -> second export: series count  15.0   (a delta, 25 - 10)

OTLP delta histogram lab.latency  count 4, sum 1060, min 10, max 1000, bounds [50, 500], buckets [2,1,1]
   -> /api/beta/sketches  cnt 4  min 10.0  max 1000.0  sum 1060.0  k=[1590, 1739]  n=[3, 1]

That is cumulative_monotonic_mode: to_delta and histograms.mode: distributions at work. Resource attributes were promoted to unified-service tags (service, env) even though resource_attributes_as_tags is false (that setting governs the other resource attributes), and instrumentation_scope is added as a tag by default. Two consequences worth knowing before you build a dashboard. A cumulative counter’s first value is a baseline and is dropped, so the first interval after a restart is empty. And the histogram arrives as a DDSketch built from the explicit buckets — only two distinct sketch keys for three OTel buckets, so a percentile from it is only as fine as your explicitBounds (see DogStatsD and Aggregation for how a sketch answers percentiles).

The Collector path

The docs currently recommend two different Collector setups, and old AI-written configs mix them up. From Set Up the OpenTelemetry Collector: a configuration “tested with the OpenTelemetry Collector Contrib distribution v0.154.0” that uses the OTLP HTTP exporter (“Sends telemetry to Datadog’s OTLP intake endpoints”, endpoint https://otlp.<YOUR_DD_SITE>), a span_metrics connector for RED metrics, and a Datadog extension. From Configure the Datadog Exporter and Connector: “For new OpenTelemetry Collector configurations, Datadog recommends the OTLP HTTP exporter and span_metrics connector setup. The Datadog Exporter and Datadog Connector remain supported, and existing configurations do not need to migrate.” So the datadog exporter is legacy-but-supported, and it is the one the Phase 3 lesson and most existing configs use. Its shape, validated for real:

otel/collector-datadog-exporter.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318

processors:
  batch:
    send_batch_max_size: 100
    send_batch_size: 10
    timeout: 10s

connectors:
  datadog/connector:

exporters:
  datadog/exporter:
    api:
      site: datadoghq.com
      key: ${env:DD_API_KEY}

service:
  pipelines:
    metrics:
      receivers: [otlp, datadog/connector]
      processors: [batch]
      exporters: [datadog/exporter]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog/connector, datadog/exporter]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog/exporter]

otelcol-contrib validate on this file printed only a debug line and exited 0. Three broken variants, same command:

api.key renamed to api_key            exit 1   Error: exporters::datadog/exporter: api.key is not set
exporter id "datadogexporter"         exit 1   Error: failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):
                                               'exporters' unknown type: "datadogexporter" for id: "datadogexporter" (valid values: [logzio bmchelix clickhouse ...
connector removed from traces exporters
  (still a receiver in metrics)       exit 1   Error: failed to build pipelines: connector "datadog/connector" used as receiver in [metrics] pipeline but not used in any supported exporter pipeline

Unlike the Agent’s configcheck, the Collector’s own validate does build the pipeline graph, so it catches wiring mistakes — that last one is caught because validate and a real start both build the pipelines (a container started with the same file had exited 1 with the same message when I looked ten seconds later). The connector is deliberately an exporter of the traces pipeline and a receiver of the metrics pipeline: it turns spans into trace metrics. With the legacy exporter the start-up log of 0.161.0 says “Trace metrics are now disabled in the Datadog Exporter by default. To continue receiving Trace Metrics, configure the Datadog Connector or disable the feature gate.” — the connector is what produces APM trace metrics from your spans.

Mental Model

Think of OTLP as the same wire, two consumers with different feature sets. The Agent consumer turns OTLP into Datadog’s native span and metric shapes and then behaves like any Agent: it has its own sampler, its own tags, its own forwarder, and it runs on the host. The Collector consumer lives in your infrastructure and its Datadog components translate on the way out. In both, the important object is the mapping table: attributes that match a convention become first-class (service, env, http.method), attributes that do not stay as raw tags, and anything the mapping needs but the SDK did not send (http.route, deployment.environment.name) shows up later as a degenerate resource name or a missing env. Opening the payload, as above, is faster than reading any dashboard.

Trade-offs

Choice	Benefit	Cost
OTLP into the Agent	One local endpoint; the Agent’s tagging, sampler and forwarder; no extra Collector to run	Some Datadog products are unavailable for OTel-instrumented apps, per the docs; one Agent per host is the supported topology
Collector with datadog exporter	Central processing (filtering, tail sampling in a gateway) before Datadog; vendor-neutral pipeline	You own the Collector’s config, version and upgrades; the connector must be wired or trace metrics vanish
Collector with otlp_http exporter (docs’ current recommendation)	Uses Datadog’s OTLP intake; simpler exporter	Different config shape from most existing examples; check the intake limits the docs list
Datadog tracer instead of OTel SDK	Native features (Ingestion Rules, profiler, App and API Protection)	Vendor-specific instrumentation
0.0.0.0 bind	Reachable from other containers and pods	The docs themselves say localhost is the safer default outside containers
span_name_as_resource_name	Resource names follow your span names	Changes operation names too (GET_orders_id above); re-check monitors on operation names

Common Misconceptions

• “agent configcheck validated my OTLP config.” It exited 0 for a file whose OTLP pipeline was dead (Collector status: Closed). Check agent status and the log.
• “Status: Enabled means OTLP is working.” It means the setting is present. Collector status: Running is the working state.
• “OTLP is on when the Agent is on.” Off by default; a default container refused the connection.
• “OTLP spans are stored as sent.” They are mapped: http.request.method to http.method, span name to resource, kind plus attributes to span type and operation name.
• “A cumulative counter shows up on the first export.” The first value is a baseline and produced no point; the second produced a delta of 15.
• “OTLP traces obey DD_TRACE_SAMPLING_RULES.” No Datadog tracer is involved; the Agent’s OTLP probabilistic sampler applies (98 of 400 at 25%).
• “otelcol validate only checks YAML syntax.” In 0.161.0 it built the pipelines and rejected a miswired connector with exit 1.

Reading AI-generated code

🐛 Spot the bug

# datadog.yaml  -- "enable OTLP so our OpenTelemetry services can send traces"
api_key: <your-api-key>
otlp_config:
receiver:
  protocol:
    grpc:
      endpoint: 0.0.0.0:4317
    http:
      endpoint: 0.0.0.0:4318

Show the answer

protocol should be protocols. The real Agent accepts the file, agent configcheck exits 0, and agent status prints Status: Enabled — but the OTLP pipeline never starts (Collector status: Closed) and the only evidence is one line in agent.log: 'otlpreceiver.Config' has invalid keys: protocol. Every OTel SDK then fails to connect. The lesson to keep is which command to trust: Collector status: Running. The corrected block, exactly the one whose status was Running above:

# corrected datadog.yaml
api_key: <your-api-key>
otlp_config:
  receiver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

🐛 Spot the bug

otel/collector.yaml

receivers:
otlp:
  protocols:
    http:
      endpoint: localhost:4318
processors:
batch:
connectors:
datadog/connector:
exporters:
datadog/exporter:
  api:
    site: datadoghq.com
    key: ${env:DD_API_KEY}
service:
pipelines:
  metrics:
    receivers: [otlp, datadog/connector]
    processors: [batch]
    exporters: [datadog/exporter]
  traces:
    receivers: [otlp]
    processors: [batch]
    exporters: [datadog/exporter]

Show the answer

The connector is a receiver of the metrics pipeline but nothing feeds it: it is missing from the traces pipeline’s exporters. The real otelcol-contrib 0.161.0 refuses to start: Error: failed to build pipelines: connector "datadog/connector" used as receiver in [metrics] pipeline but not used in any supported exporter pipeline (exit 1, from both validate and a real start). The fix is to add datadog/connector to the traces exporters so span-derived trace metrics reach the metrics pipeline; the full working file is the collector-datadog-exporter.yaml above, which validated with exit 0. Do not “fix” the error by deleting the connector everywhere: the start-up log of this Collector version says “Trace metrics are now disabled in the Datadog Exporter by default”, so without the connector the exporter no longer produces them (I did not run that variant to observe the metrics side).

A default datadog/agent:7 container was sent an OTLP/HTTP trace with curl to the published port. What did agent status and curl show?

OTLP Status: Enabled, curl HTTP 200 OTLP Status: Not enabled, Collector status: Not running, curl got no HTTP response (exit 52) OTLP Status: Enabled, curl HTTP 404 The Agent enables OTLP automatically when it sees the first request

A datadog.yaml has otlp_config.receiver.protocol (singular) with an http endpoint. What did the real Agent show?

configcheck exit 1 with a schema error configcheck exit 0, OTLP Status: Enabled, but Collector status: Closed, and one ERROR line in agent.log naming the invalid key "protocol" Collector status: Running The Agent refused to start

One OTLP server span arrived with kind=server, http.request.method=GET, http.route=/orders/{id}, status Error. What did the Agent forward?

name=GET, resource=/orders/{id}, type=custom name=http.server.request, resource="GET /orders/{id}", type=web, error=1, and http.method=GET as a tag The span unchanged with OTel attribute names only The span is dropped because status Error is not supported

A cumulative monotonic OTLP sum was exported as 10 and ten seconds later as 25. What reached the intake with the default cumulative_monotonic_mode?

Two points, 10 and 25 No point for the first export, then a count of 15 (a delta) A single gauge of 25 A count of 35

Why does a Collector config that lists datadog/connector as a receiver of the metrics pipeline but not as an exporter of the traces pipeline fail, and what proves it?

YAML syntax error found by any linter The connector has no input: otelcol-contrib validate and a real start both exited 1 with "connector ... used as receiver in [metrics] pipeline but not used in any supported exporter pipeline" It only fails when a real API key is present It starts fine and only trace metrics are dropped

Which statement about sampling of OTLP traces sent to the Agent is supported by this lesson?

DD_TRACE_SAMPLING_RULES applies because the Agent is Datadog software The Agent's otlp_config.traces.probabilistic_sampler applies: 98 of 400 traces were forwarded at a 25% setting, tagged _dd.p.dm=-9 and _dd.otlp_sr=0.25 All OTLP traces are always dropped Sampling is decided by the OTel SDK only

Submit

Mark as complete