Traces, Spans, and the Service Map

The idea in one sentence

A trace is the whole end-to-end journey of one request, stitched together from spans that form a parent/child tree, and Datadog builds the Service Map by reading that tree’s structure directly off live traffic — not from a diagram anyone drew by hand.

Spans: the unit of a trace

A span represents one unit of work — an HTTP request, a database query, a function call. Every span carries a handful of fields that matter:

• name — the operation type, e.g. web.request or postgres.query.
• resource — the more specific thing being done, e.g. GET /checkout or SELECT * FROM orders.
• start and duration — when the span began and how long it took.
• parent/child relationship — a parent_id pointing at the span that caused this one to happen.

A span with no parent_id is the root span — it marks the start of the trace, usually the inbound request that kicked everything off. Every other span in the trace traces its ancestry back to that root, one parent at a time, which is what turns a flat pile of spans into a tree:

{
  "trace_id": "6141ee2f9a4d4b2e8f3c1a90",
  "span_id": "a1b2c3d4",
  "parent_id": null,
  "name": "web.request",
  "resource": "GET /checkout",
  "service": "checkout-web",
  "start": 1699999999000000000,
  "duration": 45000000
}

That parent_id: null is what makes this particular span the root — every span downstream of it (the database query it triggers, the call to a payments service, and so on) carries a parent_id that eventually points back here.

Auto-instrumentation vs. manual spans

Most of the spans in a trace show up without you writing any tracing code at all. A Datadog tracing library hooks into your framework, your HTTP client, and your database driver — when a request comes in, when an outbound call goes out, when a query runs — and creates a span automatically. This is auto-instrumentation, and it’s why installing a tracer against a typical web app already produces a reasonably complete trace on day one.

Auto-instrumentation stops at the boundary of your own business logic, though. If you want visibility into a specific block of code — a discount calculation, a batch job step, a cache lookup you wrote yourself — you wrap it in a manual span:

from ddtrace import tracer

def process_order(order):
    # auto-instrumentation already created a span for the inbound HTTP request;
    # this manual span adds visibility into a specific block of business logic
    with tracer.trace("checkout.apply_discount", resource="apply_discount") as span:
        span.set_tag("order.id", order.id)
        discount = calculate_discount(order)
        span.set_tag("discount.amount", discount)
    return discount

The manual span here becomes a child of whatever span was active when process_order ran — usually the root web.request span, or another manual span nested above it. Auto-instrumentation and manual spans are not competing techniques; they combine into the same tree.

The Service Map: built from the tree, not drawn by hand

Every span carries a service tag — checkout-web, payments-service, inventory-service. When one span’s child has a different service tag than its parent, that’s a real, observed dependency between two services, captured at the moment the request actually crossed that boundary.

The Service Map is Datadog reading exactly that signal across every trace flowing through your system: it collapses all those parent→child, service→service edges into a single live graph. Because it’s assembled from real span relationships rather than maintained by a person, it updates itself the moment traffic patterns change — a new downstream call shows up on the map as soon as it starts happening, and a decommissioned service drops off once nothing calls it anymore. Compare that to a hand-drawn architecture diagram, which is accurate on the day someone draws it and stale a month later.

flowchart LR
  A[Root span: web.request service checkout-web] --> B[Span: db.query service checkout-web]
  A --> C[Span: http.request service payments-service]
  C --> D[Span: db.query service payments-service]
  A --> E[Span: http.request service inventory-service]
  subgraph SM[Service Map derived from the span tree]
    S1[checkout-web] --> S2[payments-service]
    S1 --> S3[inventory-service]
  end
  A -.observed service boundary crossings become.-> SM

From span tree to Service Map

🐶 The one-line takeaway

A trace is a tree of spans rooted at the span with no parent, spans come from auto-instrumentation (framework/HTTP/DB hooks) plus any manual spans you add around custom logic, and the Service Map is just those spans’ parent→child and service-tag relationships rendered live — it updates itself as traffic changes instead of going stale like a hand-drawn diagram.

Real spans, captured off the wire

Every claim above about span fields is provable without a Datadog account at all: point a tracer’s DD_TRACE_AGENT_URL at a plain local HTTP server that decodes the msgpack payload the tracer actually POSTs, and print it. Nothing leaves the machine — the “agent” is a Node script using @msgpack/msgpack, listening on 127.0.0.1:

src/fake-trace-agent.js

const http = require('http');
const { decode } = require('@msgpack/msgpack');

const server = http.createServer((req, res) => {
  const chunks = [];
  req.on('data', (c) => chunks.push(c));
  req.on('end', () => {
    const body = Buffer.concat(chunks);
    if (req.url.startsWith('/v0.4/traces')) {
      const traces = decode(body); // array of traces, each an array of spans
      for (const trace of traces) for (const span of trace) console.log(JSON.stringify(span));
    }
    res.writeHead(200, { 'content-type': 'application/json' });
    res.end('{"rate_by_service":{"service:,env:":1}}');
  });
});
server.listen(18126, '127.0.0.1');

Against a real dd-trace-instrumented Express app (checkout-web, one root request span plus a manual child span for checkout.apply_discount), pointed at that listener with DD_TRACE_AGENT_URL=http://127.0.0.1:18126, the real decoded root span — trimmed to the fields this lesson called out above:

{
  "trace_id": 1716230160703035000,
  "span_id": 8607871282138202000,
  "parent_id": 1716230160703035000,
  "service": "checkout-web",
  "name": "express.request",
  "resource": "GET /checkout",
  "type": "web",
  "error": 0,
  "duration": 3979980,
  "meta": { "http.method": "GET", "http.route": "/checkout", "http.status_code": "200", "component": "express", "span.kind": "server" },
  "metrics": { "_dd.measured": 1, "_sampling_priority_v1": 1 }
}

And the manual checkout.apply_discount span, real output, a child several levels down the Express middleware chain:

{
  "trace_id": 1716230160703035000,
  "span_id": 8825565273850650000,
  "parent_id": 383229711255948900,
  "service": "checkout-web",
  "name": "checkout.apply_discount",
  "resource": "apply_discount",
  "error": 0,
  "duration": 19775,
  "meta": { "order.id": "48213", "_dd.integration": "opentracing" },
  "metrics": { "_sampling_priority_v1": 1 }
}

Every field this lesson described above is real, not illustrative: trace_id/span_id are real 64-bit values (msgpack encodes them as unsigned integers, not the zero-padded hex string shown earlier for readability), parent_id genuinely chains child → parent → root, duration is real nanoseconds measured on this Mac, and meta/metrics are the actual key/value bags a tracer fills — meta holds strings (tags), metrics holds numbers. Note this manual span has no type field at all — tracer.startSpan() without an explicit type option genuinely omits it; only spans from an auto-instrumentation integration (web, http, db, …) carry one by default.

Trace context propagation: continuing across a service boundary

The Service Map above is built from parent/child edges inside a trace, but a real distributed system continues the same trace across a network call to a different service — and that only works because both sides agree on a wire format for trace_id/span_id. That mechanism (DD_TRACE_PROPAGATION_STYLE, the x-datadog-trace-id/x-datadog-parent-id headers, W3C traceparent/tracestate, and what happens when two services share no format) is proven with real captured HTTP headers in Connecting Logs and Traces — not repeated here to avoid duplicating that proof, but it is the mechanism that makes the root span of this lesson’s traces the same root span a downstream payments-service would see, which is exactly what lets the Service Map draw a checkout-web → payments-service edge in the first place.

Two ways to set the service name — and which one wins

ddtrace-run (Python) reads DD_SERVICE from the environment and configures the global tracer with it before your module even executes. If your own code also sets a service name manually later in the same process, which one does a real captured span actually carry? Ran a Flask app under DD_SERVICE=env-configured-service ddtrace-run python app.py, with the app itself setting a different service name at import time, and captured the real spans via the same fake-agent technique above (ddtrace posts msgpack to /v0.5/traces, a more compact encoding than Node’s /v0.4/traces — a string table plus index-referenced spans — decoded the same way):

$ DD_TRACE_AGENT_URL=http://127.0.0.1:18130 DD_SERVICE=env-configured-service \
    ddtrace-run python app.py

Real captured span, service field on every one of the 19 spans this run produced:

{
  "service": "manual-configured-service",
  "name": "checkout.apply_discount",
  "resource": "apply_discount",
  "meta": { "_dd.base_service": "env-configured-service", "_dd.svc_src": "m", "order.id": "48213" }
}

The manual, later-executed override wins — every span carries service: "manual-configured-service", not the DD_SERVICE value ddtrace-run set first. ddtrace does not silently lose the original value, though: it tags every span with _dd.base_service (the service name the runtime infers, here still env-configured-service) and _dd.svc_src: "m" (source m = set manually), a genuine audit trail of exactly this kind of conflict, meant for exactly this kind of debugging session.

🐶 Proof: the AI-plausible fix does not even run

The obvious way to write “set the service manually” in Python is tracer.configure(service="...") — it reads naturally, and it is how several other Datadog tracers’ configure()/init() APIs work. Real output from the pinned ddtrace==4.15.2:

$ python -c "from ddtrace import tracer; tracer.configure(service='x')"
TypeError: Tracer.configure() got an unexpected keyword argument 'service'

Tracer.configure() in this version genuinely has no service parameter — inspect.signature(tracer.configure) confirms it accepts context_provider, compute_stats_enabled, appsec_enabled, trace_processors, and a handful of other keys, but not service. The real, working override point in this pinned version is the module-level ddtrace.config.service attribute (from ddtrace import config; config.service = "..."), which is what actually produced the captured spans above.

Error span semantics

A span records failure with three related pieces, all real fields, not conventions: a boolean-ish error flag (1/0 on the wire), and inside meta, error.type, error.message, and error.stack. Real captured output from the Python app above, deliberately raising inside the /checkout-error route:

{
  "name": "__main__.checkout_error",
  "resource": "/checkout-error",
  "error": 1,
  "meta": {
    "error.type": "builtins.ValueError",
    "error.message": "card declined",
    "error.stack": "Traceback (most recent call last):\n  File \".../app.py\", line 33, in checkout_error\n    raise ValueError(\"card declined\")\nValueError: card declined\n"
  }
}

Two things worth noting, both proven above: first, error.stack in a Python span is a real Python traceback string, not a JavaScript-style call stack — the shape of error.stack is language-native, only the field name is shared across tracers. Second, the error flag genuinely propagates upward: the real Node capture earlier in this lesson showed the deepest Express middleware span carrying error: 1 for a thrown error, but the root express.request GET /checkout-error span for that same request also carried error: 1 — a single failing span marks its whole request-level ancestor as an error too, which is why a service’s error rate (visible on the Service Map) reflects a failure anywhere in that request’s span tree, not just the exact frame that threw.

Reading AI-generated code

🐛 Spot the bug

app/set_service.py

# AI comment: "Override the service name for this one background worker,
# since it should show up separately from the main web service"
from ddtrace import tracer

tracer.configure(service="checkout-worker")

def process_job(job):
  with tracer.trace("worker.process_job") as span:
      span.set_tag("job.id", job.id)
      # ... do the work ...

Show the answer

This looks completely reasonable — configure() is the tracer’s own documented entry point for runtime settings, and service is a real concept the tracer understands everywhere else (DD_SERVICE, span tags, the Service Map). It does not run at all. Proven above with the pinned ddtrace==4.15.2: Tracer.configure() has no service parameter, so this raises TypeError: Tracer.configure() got an unexpected keyword argument 'service' at import time, before process_job ever executes — this is not a silently-ignored option like the logInjection bug in the correlating-signals module, it is a hard crash. The fix, proven above to actually take effect and show up on captured spans: from ddtrace import config; config.service = "checkout-worker".

Mental Model

A trace is not a single object anyone assembles — it is a claim, made independently by every span, about who its parent was. Nothing in the wire format has a “trace” container; decode()d msgpack above is literally just a flat array of spans, each one carrying a trace_id (which trace it claims to belong to) and a parent_id (which span it claims caused it). The tree, and the Service Map built from that tree, are both reconstructions Datadog performs after the fact from those individual claims — which is exactly why a propagation mismatch (two services that don’t share a header format) doesn’t throw an error anywhere: each side’s spans are still perfectly valid on their own, they just carry claims that no longer point at each other, so the reconstruction silently produces two smaller trees instead of one connected one.

Trade-offs

• Auto-instrumentation vs. manual spans — auto-instrumentation (proven above: express.request, express.middleware, tcp.connect, dns.lookup all appeared without a line of app code) gives broad coverage for free, but the resulting trace is noisy with framework internals (five express.middleware spans for one request in this lesson’s own capture); manual spans are precise and mean something to the reader, at the cost of someone having to remember to add them.
• ddtrace.config.service (Python) vs. DD_SERVICE — an in-code override, proven above, wins over the environment and is useful for a single process that genuinely needs a different identity (a background worker inside an otherwise web-service codebase), but it is easy to set accidentally and hard to spot in a code review, unlike an env var that shows up in the process’s own launch config.
• A local fake trace agent vs. the real Datadog Agent — decoding msgpack directly (as this lesson did) proves the tracer’s own output with zero infrastructure and zero risk of sending anything anywhere, but it skips everything the real Agent does next (its own rate limiting, stats computation, retry/buffering) — it proves what the tracer sends, not what Datadog eventually stores.

Common Misconceptions

• “A trace is a single message the tracer sends.” No — proven above: the wire payload is a flat array of independent spans, each claiming its own trace_id/parent_id; the “trace” is a reconstruction from those claims, not a container that exists on the wire.
• “tracer.configure(service=...) is how you override the service name in Python.” No — proven above with a real TypeError: that parameter does not exist in the pinned ddtrace==4.15.2. The real override point is ddtrace.config.service.
• “If DD_SERVICE and a manual override disagree, the original value is lost.” No — proven above: the manual value wins on the service field, but the original DD_SERVICE value survives on every span as _dd.base_service, with _dd.svc_src recording that it was manually overridden.
• “An error only marks the exact span that threw.” No — proven above: a Node request that failed deep in Express middleware showed error: 1 on both that middleware span and the root express.request span for the whole request.

What identifies a span as the root span of a trace?

It has the longest duration of any span in the trace It has no parent_id It is always named web.request It is the last span created chronologically

What is the main difference between auto-instrumentation and a manual span?

Manual spans cannot have a parent Auto-instrumentation only works in Python Auto-instrumentation is created automatically by hooks into your framework/HTTP client/DB driver, while a manual span is code you write to wrap a specific block of custom logic There is no difference — they produce identical data

What does the Service Map actually get built from?

A YAML file you maintain describing your architecture Parent/child span relationships and each span's service tag, observed from live traces A manually drawn diagram uploaded by an engineer DNS records collected by the Agent

Why does the Service Map stay accurate over time without anyone updating it?

Datadog support manually verifies it weekly It only shows services that were configured at setup time It is regenerated from real span/service relationships in live traffic, so new or removed dependencies show up as traffic actually changes It is cached once and never changes

A Python app runs DD_SERVICE=env-configured-service ddtrace-run python app.py, and the app itself sets ddtrace.config.service = "manual-configured-service" at import time. What service name do the real captured spans carry?

env-configured-service, because environment variables always win manual-configured-service — proven above on every one of 19 real captured spans, with _dd.base_service recording the original env-configured-service value Neither — the tracer refuses to start with conflicting service names It alternates randomly between the two on each span

What actually happens when AI-generated Python code calls tracer.configure(service="checkout-worker") against the pinned ddtrace==4.15.2?

It works exactly as intended, setting the service name It silently does nothing, like the logInjection/log_injection bug in the correlating-signals module It raises a real TypeError: Tracer.configure() got an unexpected keyword argument 'service' — the process crashes before any span is created It emits a deprecation warning but still sets the service name

Submit

Mark as complete