Accuracy–latency trade-offs

What you’ll be able to do

• Express a latency requirement precisely: time to first token, end-to-end, interactive or asynchronous
• Map the main levers (model tier, extended thinking, retrieval depth, tool round-trips, verification passes) to their accuracy and latency effects
• Use routing and tiering so expensive configurations apply only to the requests that need them
• Remove latency that buys no accuracy, using caching, parallelism, streaming and batching
• Justify a configuration with measured evidence on representative traffic

What you need to know

Pin down which latency matters

"It needs to be fast" isn't a requirement. Before you trade anything, separate:

• Time to first token. What a chat user feels. Streaming makes a long answer feel responsive even when total generation time is unchanged.
• End-to-end completion. What matters when another system waits for a structured result, such as a fraud check in a payment flow or a classifier in an ingestion pipeline. Streaming doesn't help here.
• Throughput and turnaround. For overnight document processing, nobody is waiting on a single request. Asynchronous processing (such as the Message Batches API) trades immediacy for lower cost and higher volume.

Pair each latency target with the cost of an error on that path. A wrong autocomplete suggestion costs a keystroke. A wrong dosage extraction or an incorrect refund decision costs far more. That pairing, latency budget against error cost, is what justifies a configuration.

The levers and what they trade

• Model tier. Larger models generally handle harder reasoning better but respond more slowly and cost more. Smaller models are often good enough for classification, extraction and routing. Which is "good enough" is an empirical question for your task.
• Extended thinking. Giving the model room to reason before answering can lift accuracy on multi-step problems. It adds tokens and time, and buys little on simple lookups.
• Retrieval depth. More chunks, re-ranking and multi-query expansion can raise recall but add retrieval time and input tokens. Past a point, extra context dilutes relevance instead of adding it.
• Tool round-trips. Each agentic iteration is another model call plus tool execution. Fewer, better-designed tools, and tools that return exactly what's needed, reduce turns.
• Verification passes. A second model pass, a self-check or a deterministic validator catches errors but adds a step. Validators in code are cheap. A second model pass is not.
• Output length. Generation time grows with output tokens. Asking for concise structured output often cuts latency without hurting accuracy.

Route, don’t compromise globally

The weak design picks one configuration for all traffic and argues about whether it should be the fast one or the accurate one. The strong design routes. A cheap, fast classifier, or plain rules, sends routine requests to a fast configuration and hard or high-stakes ones to a stronger model with thinking enabled. Escalation on low confidence or failed validation works too: try the fast path, check the output deterministically, and retry with the strong configuration only when the check fails. Done well, most traffic gets fast responses and the hard tail gets the accuracy it needs.

Routing brings its own risks: misrouting hard requests to the weak path, and the classifier's own latency. Measure routing accuracy as its own metric, and lean toward the strong path when the cost of an error is high.

Remove latency that buys nothing

Some latency has no accuracy value at all, and removing it should come before any trade-off:

• Prompt caching. Put stable content (system prompt, tool definitions, policy documents) first and cache it, so repeated prefixes cost less and return sooner.
• Parallelism. Run independent retrievals, tool calls and subagents concurrently rather than one after another. Claude can request multiple tool calls in one turn when they're independent. Execute them in parallel.
• Precomputation. Move work that doesn't depend on the request (document summaries, embeddings, enrichment) to ingestion time.
• Streaming. For human-facing paths, streaming improves perceived latency at no accuracy cost.
• Batching. Move non-interactive workloads off the synchronous path entirely.

Justify with your own evidence

Stakeholders will ask why you didn't pick the fastest or the smartest option. The defensible answer is a small table built from your own evaluation set: for each candidate configuration, accuracy on representative and hard cases, latency at the percentiles your SLA cares about (tail latency, not just the average), and cost per request. Public benchmarks and vendor claims don't describe your data. Re-run the comparison when models, prompts or traffic change. Designing those evaluations belongs to domain P4. Here, the skill is turning their results into a configuration decision you can defend.

Key concept

Set the latency budget and error cost per path, remove latency that buys no accuracy, then spend the expensive levers only on the requests that need them, and prove the choice on your own traffic.

Questions offer "use the biggest model with maximum reasoning" and "use the smallest model everywhere" as distractors. The strong answer usually routes or tiers, applies caching, parallelism or batching first, and cites measured accuracy and tail latency on representative cases.

Practice scenario

Build exercise — Build a configuration decision table

Advanced · 50 min

What you’ll learn

• Translating a vague speed requirement into precise latency targets per path
• Measuring accuracy, tail latency and cost across candidate configurations
• Designing a routing rule with a safe escalation path

1. For one workload, write each request path’s latency target (time to first token, end-to-end, or asynchronous turnaround) and its cost of an error in business terms.

   • Why: Configurations are only justifiable against an explicit budget and stakes, not against “fast” or “accurate.”
   • You should see: At least two paths with different targets, such as interactive chat versus a structured decision consumed by another system.

2. Run three configurations (for example a fast model, a stronger model, and a stronger model with extended thinking) over an evaluation set split into routine and hard cases. Record accuracy per segment, p50/p95 latency and cost per request.

   • Why: Averages hide the hard tail. Segmenting shows where the expensive configuration actually earns its cost.
   • You should see: A small accuracy gap on routine cases and a meaningful one on hard cases.

3. Propose a routing rule (classifier or validation-triggered escalation), apply caching and parallelism to the shared steps, and re-measure the blended accuracy and p95 latency.

   • Why: Routing plus waste removal usually beats any single global configuration.
   • You should see: Blended latency inside the budget, with hard-case accuracy close to the strong configuration.

Exam traps

Choosing the most capable model and deepest reasoning for every request by default

It spends latency and cost on requests that don’t benefit. Route so the expensive configuration covers only the hard or high-stakes cases.

Downgrading the model for speed without measuring accuracy on the hard cases

Averages can look fine while the costly tail fails. Segment your evaluation before you downgrade.

Treating perceived latency and total completion time as the same requirement

Streaming fixes perceived latency for humans but does nothing for a system waiting on a complete structured result.

Running sequential tool calls or retrieval steps that could run in parallel

Serial independent steps add latency with no accuracy benefit. Parallelize before trading away quality.

Serving offline, non-urgent workloads through the synchronous interactive path

Batch or asynchronous processing fits workloads nobody is waiting on, and keeps capacity free for interactive traffic.

Justifying a configuration with benchmark scores instead of your own task evaluations

Benchmarks don’t describe your data or your error costs. Measure on representative traffic.

Sources

• Claude Developer Platform Docs
• Anthropic
• Anthropic, Claude Certified Architect – Professional Exam Guide v1.0 (July 2026)