Observability at scale

What you’ll be able to do

• Explain why request-level logging fails for multi-step, multi-service agent workflows
• Design end-to-end tracing with a correlation ID propagated through model calls, tool calls, subagents and downstream services
• Attribute token cost, latency and errors to the specific step, tool, tenant or feature that caused them
• Choose sampling, retention and redaction strategies that keep observability affordable and compliant at volume

What you need to know

Why agent systems are harder to observe

A single user request to an agentic system can fan out into a dozen model calls, tool executions across several MCP servers, retrieval queries, subagent runs and calls into downstream services owned by other teams. The path isn't fixed: the model chooses it at runtime, so two identical-looking requests can take very different routes. Traditional per-request logging gives you a pile of disconnected events. You can see that something was slow or expensive, but not which decision caused it.

This lesson covers integration-level observability: following work across components and attributing cost and failure. Choosing evaluation metrics and general production monitoring dashboards belongs to domain P4.

Trace the whole workflow, not the model call

Treat every user-facing request (or every autonomous job) as a trace, and every step inside it as a span: each model call, each tool execution, each retrieval, each subagent, each downstream service call. The essentials:

• One correlation ID, propagated everywhere. Generate it at the entry point and pass it through the orchestrator, into tool handlers, across MCP servers and into downstream services' own logs. Without propagation, the trace ends at your boundary and the slow database call on the other side stays invisible.
• Parent-child structure. A subagent's spans nest under the orchestrator step that launched it, so you can see that a slow answer came from one subagent's retry loop.
• Standard instrumentation. Using an open standard such as OpenTelemetry lets agent spans sit alongside existing service traces instead of in a separate silo. Claude Code, for instance, can export telemetry through OpenTelemetry.
• Provider identifiers. Record the request ID the Claude API returns on each call, so you can line up your trace with the provider's view when you escalate an issue.

What each span should carry

• Model calls: model, the token usage the API returns (input, output and cache read/write counts), stop_reason, latency, prompt or template version.
• Tool calls: tool name, server, argument shape (redacted), result size, error category, retry count, duration.
• Retrieval: query, index version, document and chunk IDs returned, scores. This is the evidence trail when an answer turns out to be wrong.
• Business context: tenant, feature, workflow, user tier, as dimensions for attribution, not as raw personal data.

Attribution: cost and errors to the step that caused them

At scale, the question isn't "how much did we spend?" but "which feature, tenant, tool or step is driving spend, and is it worth it?" Aggregate token usage by span dimensions and you can find the tool that returns 40-page payloads into context, the tenant whose workflow loops, or the subagent with a cold cache because its prefix changes on every call. Watch cache hit rate per prompt template: a sudden drop usually means someone put dynamic content into what used to be a stable prefix.

Error attribution works the same way. Classify failures by origin: model (malformed tool call, refusal, hitting max_tokens), tool (timeout, validation error), downstream service (rate limit, outage) or orchestration (iteration limit hit, handoff dropped). An agent that "fails 3% of the time" is useless information. "62% of failures are the inventory service timing out during peak hours" is something someone can act on.

Semantic failures don’t throw errors

Many of the costliest agent failures come back as a clean HTTP 200: the model picked the wrong tool, retrieved stale chunks, looped through redundant calls, or confidently answered from a tool result that was empty. Instrument for them explicitly. Track tool-call counts per task and alert on outliers. Flag empty or truncated tool results that were followed by a confident answer. Track how often runs hit the iteration limit. Record which retrieved chunks backed the final answer. A correct-looking final answer doesn't prove the path was sound.

Making it affordable and compliant

• Tail-based or rule-based sampling. Keep every trace with an error, high cost, high latency or a flagged outcome, and sample the routine successes. Uniform head sampling throws away exactly the rare traces you need.
• Tiered payload capture. Keep metadata (tokens, timings, IDs, error classes) for all traces and full prompt and tool payloads only for sampled or flagged ones, with shorter retention.
• Redaction at capture. Scrub secrets and minimize personal data before it reaches the observability store. Traces are a common path for credentials and regulated data to escape, and retention rules (for example under GDPR) apply to logs too.
• Access control on traces. Full prompts can hold sensitive customer data. Restrict who can see payloads separately from who can see metrics.

Key concept

Make the workflow, not the model call, the unit of observation: one propagated trace across every tool and service, with cost and errors attributed to the step, tool and tenant that caused them.

Questions describe rising cost or intermittent failures that per-call logs can't explain. The best answer adds propagated tracing with step-level attribution and smart sampling. Distractors add more dashboards, log everything forever, or switch models before anyone knows where the problem is.

Practice scenario

Build exercise — Instrument one agent workflow end to end

Advanced · 60 min

What you’ll learn

• Propagating a correlation ID across model calls, tools and services
• Capturing span attributes that support cost and error attribution
• Designing a sampling and redaction policy for production volume

1. Generate a trace ID at the entry point of one agent workflow and propagate it through every model call, tool handler, subagent and at least one downstream service, recording a span for each step.

   • Why: Attribution is impossible if the trace breaks at a component boundary.
   • You should see: A single trace tree showing every step of a request, including the downstream service’s own span.

2. Add span attributes for model, token usage (including cache reads and writes), stop_reason, tool name, result size, error category and tenant. Then build one breakdown of cost by step and one of errors by origin.

   • Why: Aggregates by step and origin turn “costs went up” into a specific fix.
   • You should see: One or two steps or tools accounting for most of the spend or most of the failures.

3. Write a sampling and retention policy: which traces are always kept, which are sampled, which payload fields are redacted at capture, and who can view full payloads.

   • Why: Observability that keeps everything forever is expensive and a compliance liability. Uniform sampling drops the traces you need.
   • You should see: A policy that keeps every error and outlier, stores metadata for all traffic, and never stores secrets.

Exam traps

Logging each model call in isolation with no correlation across tools and services

You can see expensive or slow calls but not the decision chain that produced them. Propagate one trace ID end to end.

Monitoring only HTTP status codes when the costly failures are semantic

Wrong tool choices, stale retrieval and redundant loops all return success. Instrument tool-call counts, empty results and retrieval evidence.

Tracking total token spend with no attribution to tenant, feature or step

A total tells you there’s a problem but not where. Aggregate usage by span dimensions to find the step driving cost.

Storing full prompts and tool payloads indefinitely, including personal data and secrets

That creates cost and compliance exposure. Redact at capture, tier payload storage, and set retention.

Uniformly sampling traces so rare failures and expensive outliers are discarded

Keep every error, slow and high-cost trace, and sample only routine successes.

Treating a successful final answer as proof the intermediate tool calls behaved

Agents can reach a plausible answer through wasteful or unsound paths. Inspect the trace, not just the output.

Sources

• Claude Developer Platform Docs
• Model Context Protocol
• Anthropic, Claude Certified Architect – Professional Exam Guide v1.0 (July 2026)