Continuous Profiler and Error Tracking

The idea in one sentence

Tracing tells you where something is slow or failing — which service, which span — while the Continuous Profiler and Error Tracking tell you why, down to the exact function burning CPU or the deduplicated root cause behind a pile of repeated stack traces.

Continuous Profiler: code-level visibility, always on

The Continuous Profiler runs constantly, in production, at low overhead, capturing CPU time, wall-time, and allocations at the code level — individual functions and call stacks, not just service-level aggregates. Because it’s always on rather than something you turn on to investigate one incident after the fact, the data is already there by the time you need it.

The profiler ties its samples back to the specific service, endpoint, and even the individual trace they occurred in. That link is what makes it useful for debugging a slow request: instead of guessing which part of checkout-web is expensive, you jump from a slow trace directly into a flame graph of exactly which function was running, and for how long, during that request:

app/profile-start.py

# ddtrace's profiler starts alongside the tracer — no separate instrumentation needed
from ddtrace.profiling import Profiler

prof = Profiler()
prof.start()

With both the tracer and profiler running, a slow span in a trace can point you straight at the function-level flame graph for that exact window of execution — tracing narrows down which span, the profiler narrows down which line of code.

Proving the profiler env vars are genuinely read — and a real local profile, no Agent needed

DD_PROFILING_ENABLED is a real, documented option in both pinned clients — dd-trace’s .d.ts declares profiling?: boolean with @env DD_PROFILING_ENABLED, and the installed package’s own supported-configurations.json lists the entire family: DD_PROFILING_CPU_ENABLED, DD_PROFILING_ALLOCATION_ENABLED, DD_PROFILING_HEAP_ENABLED, DD_PROFILING_CODEHOTSPOTS_ENABLED, DD_PROFILING_TIMELINE_ENABLED, DD_PROFILING_UPLOAD_PERIOD, and — the one worth testing directly — DD_PROFILING_EXPORTERS (default agent).

Does the profiler need a live Agent connection to prove it’s collecting real data? No — DD_PROFILING_EXPORTERS=file is a real, second exporter implementation (profiling/exporters/file.js, alongside exporters/agent.js in the installed package), and it writes profiles straight to local disk. Ran a CPU-burning function under it, zero Agent, zero network:

src/profile-test.js

const tracer = require('dd-trace').init({ service: 'checkout-web', profiling: true, logInjection: false });

function busy() {
  let x = 0;
  const end = Date.now() + 3000;
  while (Date.now() < end) x += Math.sqrt(x + 1);
  return x;
}
busy();
setTimeout(() => process.exit(0), 1000);

$ DD_PROFILING_ENABLED=true DD_PROFILING_EXPORTERS=file \
    DD_PROFILING_PPROF_PREFIX=./pprof-out/ DD_PROFILING_UPLOAD_PERIOD=2 \
    node src/profile-test.js
$ ls pprof-out/
events_worker_0_20260929T135419Z.pprof
space_worker_0_20260929T135419Z.pprof
wall_worker_0_20260929T135419Z.pprof

Real profile files, on disk, no Datadog account or Agent involved. The .pprof format is zstd-compressed protobuf — decompressing the wall-time profile and grepping the raw bytes for this script’s own function name confirms it captured genuine, code-level samples, not placeholder data:

$ zstd -d pprof-out/wall_worker_0_20260929T135419Z.pprof -o wall.decoded && strings wall.decoded | grep busy
(anonymous:L#1711:C#37)
/private/tmp/.../src/profile-test.js
busy

busy — this lesson’s own function — is really in there, exactly matching the claim that the profiler captures “individual functions and call stacks, not just service-level aggregates.”

The Python side works the same way, but the config accessor is not ddtrace.config.profiling — that raises a real AttributeError in the pinned ddtrace==4.15.2:

$ python -c "from ddtrace import config; print(config.profiling)"
AttributeError: <class '...Config'> object has no attribute profiling, profiling is not a valid configuration

The real accessor lives in a dedicated module, and the real env var for a local file export is DD_PROFILING_OUTPUT_PPROF (not DD_PROFILING_EXPORTERS=file — that’s Node-specific):

$ DD_PROFILING_ENABLED=true python -c \
    "from ddtrace.internal.settings.profiling import config as pconfig; print(pconfig.enabled, pconfig.output_pprof)"
True None

$ DD_PROFILING_ENABLED=true DD_PROFILING_OUTPUT_PPROF=./pprof-out-py/profile \
    python app/profile_test.py
$ ls pprof-out-py/
profile.<pid>.1.info.json
profile.<pid>.1.internal_metadata.json
profile.<pid>.1.pprof

info.json is itself proof the env var was read — it’s a full settings dump written by the profiler, and output_pprof in it reads back exactly the path that was set:

$ python -m json.tool pprof-out-py/profile.*.1.info.json | grep output_pprof
"output_pprof": "./pprof-out-py/profile"

🐶 What the profiler actually collects, and what could not be verified

Per the real files this lesson produced: the profiler collects wall-time samples, space/allocation samples, and event metadata — separate .pprof streams per type, exactly matching the “CPU time, wall-time, and allocations” claim above. One thing this lesson could not verify: error.fingerprint as a real, settable tracer-level tag. It does not appear anywhere in dd-trace’s .d.ts, and in the pinned ddtrace package the string fingerprint only turns up inside unrelated AppSec/ASM request-context code — not Error Tracking. Treat error.fingerprint as unconfirmed by this course’s tools, not as a documented tag, until a docs fetch confirms it.

Error Tracking: turning repeated stack traces into one issue

Error Tracking takes error events surfaced from traces and logs and automatically groups and deduplicates similar errors into a single issue, instead of leaving you to manually notice that the same NullPointerException has fired 482 times across a week of traces. Each issue carries a lifecycle you manage explicitly — typically new, triaged, and resolved — so a recurring error becomes one thing your team tracks to closure rather than hundreds of individual, unrelated-looking events:

{
  "issue": {
    "fingerprint": "[email protected]_discount",
    "status": "new",
    "occurrences": 482,
    "first_seen": "2026-07-01T10:03:00Z",
    "last_seen": "2026-07-08T09:12:00Z",
    "example_trace_ids": ["6141ee2f9a4d4b2e8f3c1a90", "9af23cd1b8e7f4a6c2d5e901"]
  }
}

Notice the example_trace_ids field — Error Tracking keeps a link back to actual example traces for the issue, so you can jump from “this issue has fired 482 times” straight into one concrete occurrence to see the full request context around it.

From “where” to “why”

Put the two together and they cover the gap tracing alone leaves open:

• Tracing — which service and span is slow or failing.
• Continuous Profiler — which function, at the code level, is consuming the CPU/wall-time/allocations during that slow span.
• Error Tracking — which deduplicated issue a failing span actually belongs to, and where that issue sits in its new/triaged/resolved lifecycle.

Neither tool replaces tracing — they both start from a trace or span and go one level deeper than “which service” can take you.

flowchart LR
  Trace[Slow or failing trace] --> Where[Tracing: which service, which span]
  Where --> Profiler[Continuous Profiler: which function, CPU/wall-time/allocations]
  Where --> ErrorTracking[Error Tracking: deduplicated issue, new/triaged/resolved]
  Profiler --> WhySlow[Why it is slow]
  ErrorTracking --> WhyFailing[Why it is failing]

From where to why

🐶 The one-line takeaway

Tracing narrows a problem down to which service and span; the always-on Continuous Profiler then narrows it down to which function burning CPU/wall-time/allocations, and Error Tracking automatically groups repeated failures into one issue with a new/triaged/resolved lifecycle — together they take you from where to why.

Reading AI-generated code

🐛 Spot the bug

app/profiling_setup.py

# AI comment: "Enable profiling and route it through the tracer's own config"
from ddtrace import config, tracer

config.profiling = True

def handle_request():
  with tracer.trace("checkout.handle_request"):
      ...

Show the answer

This looks like a natural extension of the tracer-config pattern used everywhere else in this course — set an attribute on ddtrace.config, done. It raises a real AttributeError the moment this module is imported: proven above, config.profiling does not exist in the pinned ddtrace==4.15.2. Profiling in this client is not a config.<x> attribute at all; it is a separate Profiler object (from ddtrace.profiling import Profiler; Profiler().start()) started independently of the tracer, configured through its own module’s env vars (DD_PROFILING_ENABLED, DD_PROFILING_OUTPUT_PPROF, and friends) rather than through ddtrace.config.

Mental Model

Tracing and profiling both sample the same running process, but at different resolutions and on different schedules. A span is a deliberate measurement — something in the code explicitly says “time this operation, tag it with this name.” A profiler sample is involuntary — proven above, it walked the real call stack of a plain while loop that never created a single span, and still captured busy by name. That’s why the two combine so well: tracing tells you when something was slow (a span with a long duration), and the profiler — sampling continuously, regardless of whether anything called tracer.trace() — was very possibly already capturing exactly what the CPU was doing during that window, whether or not the code inside it thought to measure itself.

Trade-offs

• Agent exporter vs. file exporter — proven above, the file exporter needs zero infrastructure and is perfect for local investigation or, as this lesson used it, for proving the profiler’s own output without an account; the Agent exporter is what a real deployment uses, batching and shipping profiles continuously so they show up next to traces in the UI.
• Always-on profiling vs. on-demand profiling — always-on (the profiler’s default posture) means the data already exists by the time you need it, at the cost of constant low background overhead; a profiler you only start during an incident has zero steady-state cost but cannot show you what was happening before you noticed the problem.
• Error Tracking’s automatic grouping vs. manual triage — deduplicating 482 occurrences into one issue (from the JSON example above) saves real attention, but grouping logic can occasionally merge errors that are actually distinct, or split ones that are actually the same root cause under a slightly different stack — worth spot-checking an issue’s example_trace_ids rather than trusting the count blindly.

Common Misconceptions

• “The Continuous Profiler needs a live Agent connection to prove it’s working.” No — proven above: DD_PROFILING_EXPORTERS=file (Node) and DD_PROFILING_OUTPUT_PPROF (Python) both write real, inspectable .pprof files straight to local disk, no Agent or account involved.
• “ddtrace.config.profiling is how you check or set profiling state in Python.” No — proven above with a real AttributeError: profiling lives in its own ddtrace.profiling.Profiler object and its own settings module, not on the shared ddtrace.config namespace.
• “A profiler sample only exists for code wrapped in a span.” No — proven above: a plain while loop with zero tracer calls still showed up by function name (busy) in the real captured wall-time profile.

What does the Continuous Profiler add on top of what tracing already shows you?

A list of which hosts exist in your infrastructure Code-level CPU/wall-time/allocation visibility tied back to the specific service, endpoint, and trace A replacement for the Service Map The sampling rate configured on your tracer

What does Error Tracking do with 482 occurrences of the same underlying error across many traces?

It creates 482 separate alerts It ignores duplicates entirely It automatically groups and deduplicates them into a single issue with a lifecycle you manage (e.g. new/triaged/resolved) It deletes all but the most recent occurrence

Why is the Continuous Profiler described as "always-on" rather than something you enable during an incident?

Because it only samples during deploys Because it runs constantly at low overhead, so code-level data is already available by the time you need to investigate a slow request Because it replaces the need for tracing entirely Because it is only available in a paid add-on tier with no runtime cost consideration

In one sentence, how do tracing and these two tools divide responsibility?

Tracing tells you why, profiler/Error Tracking tell you where Tracing tells you where (service/span), profiler and Error Tracking tell you why (code line / deduplicated root cause) They all do exactly the same thing for redundancy Tracing and Error Tracking are the same feature under two names

Does proving the Continuous Profiler works require a real Agent connection?

Yes — the profiler only exports to the Agent, there is no alternative No — proven above: DD_PROFILING_EXPORTERS=file (Node) and DD_PROFILING_OUTPUT_PPROF (Python) both write real, inspectable .pprof files straight to local disk Only in Python, not Node.js No — but the resulting files are placeholder data, not real samples

AI-generated Python code writes config.profiling = True (importing config from ddtrace) to enable profiling. What actually happens?

Profiling is enabled correctly — this is the documented API A real AttributeError is raised, because profiling.<x> access on ddtrace.config does not exist in the pinned ddtrace==4.15.2 — the real API is a separate Profiler object It silently does nothing, like the earlier log_injection bug It enables profiling but with the wrong sampling rate

Submit

Mark as complete