Diagnosing system issues

What you’ll be able to do

• Triage a quality problem by layer (input and retrieval, prompt and context, model, output handling and tools) and by what changed
• Distinguish hallucination caused by missing context, by ignored context, and by an unanswerable question
• Diagnose prompt failures such as conflicting instructions, buried instructions and truncated output
• Recognize model mismatch in both directions and confirm it by running the same eval on another model
• Identify retrieval and indexing faults, especially after a document refresh or embedding change

What you need to know

Start with what changed

Most production regressions follow a change. Before forming theories, line up the onset of the symptom against the change log: prompt or code deploys, a re-index or document refresh, a new data source, an embedding-model change, a tool or API change upstream, a move to a different model snapshot, a shift in traffic mix (a new customer, a new language). If quality dropped the morning after a re-index while model, prompt and latency stayed flat, the re-index is the first suspect. This is why request logs must carry version identifiers for prompt, model, index and config (4.5); without them, you're guessing.

Then work through the layers

Work from the input towards the output, holding the other layers fixed:

1. Input and retrieval. Was the information needed to answer correctly actually in the context the model received? Look at the exact logged request, not a reconstruction.
2. Prompt and context. If it was present, did the instructions make clear how to use it, and did anything in the context contradict or bury them?
3. Model. If the context and instructions were right, is the task beyond this model, or has the model changed?
4. Output handling and tools. Was the response truncated, mis-parsed, or derailed by a tool error the agent misread?

Replay the failing request exactly as logged. If it reproduces, change one layer at a time (swap in the correct documents, simplify the prompt, try another model) and see which change fixes it. If it doesn't reproduce consistently, run it several times; intermittent failure points to sensitivity that a sturdier prompt or a code-side check should remove, not something to dismiss as randomness.

Hallucinations: three different causes

"Confident but wrong" covers three problems with different fixes:

• The right context was missing. Retrieval returned stale, irrelevant or partial chunks, and the model filled the gap plausibly. Fix retrieval; prompt edits won't supply facts that aren't there.
• The right context was present but not used. The answer was in chunk 14 of 20, or contradicted by an older document also retrieved. Fix ranking and context size, put the instruction to rely on sources near the relevant material, and require citations or quotes from the provided documents so unsupported claims become visible and checkable.
• The question had no answer in the sources. The system has no permitted way to say so. Explicitly allow "I can't find that in the provided documents," and add such cases to the eval set (4.2).

Telling these apart is a logging question: you need the retrieved chunks for the failing request. A groundedness check (does each claim trace to a retrieved passage) run as an online evaluator turns this from anecdote into a rate.

Retrieval and indexing faults

Retrieval failures are the most common cause of sudden, unexplained wrong answers in RAG systems, and they're invisible if you only look at the model's output. Typical causes: a refresh that failed partway, leaving a mix of old and new documents; query embeddings produced by a different embedding model or version than the stored vectors; a changed chunking or parsing step that splits tables or drops headers; metadata filters (date, product, tenant) that now exclude the right documents; duplicates of superseded documents outranking current ones. Check retrieval recall on a set of queries with known relevant documents before touching the prompt or model. Chunking and retrieval design live in domain P3; here the skill is recognizing the symptom.

Prompt failures

• Conflicting instructions, often from a system prompt that grew by accretion ("be concise" in one section, "explain your reasoning fully" in another). The model resolves the conflict differently on different inputs.
• Buried instructions. A critical rule sits in the middle of a very long context. Restructure, and keep long reference material clearly separated from instructions (prompt structure is domain P2).
• Regression from an edit that fixed one case and broke others, which is what the regression suite is for.
• Truncation. If stop_reason is "max_tokens", the output was cut off at the limit; the fix is a higher limit or a shorter required output, not a different prompt or model.
• Hard rules enforced only by prompt. If a rule must never be broken (never refund above a limit), a prompt failure there is a design failure. Enforce it in code.

Model mismatch, in both directions

A model can be under-powered for the task: multi-step reasoning, long-document synthesis or complex tool orchestration fails in ways no prompt fix resolves. Or it can be over-provisioned: a large model used for simple classification adds cost and latency with no measurable quality gain. Either way, confirm with evidence: run the same eval set on another model tier, with the prompt held constant. Mismatch also appears after a model change, since prompts tuned for one model don't always carry over. Model selection itself is domain P2; diagnosis is showing, with your eval, which model the task needs.

Key concept

Find what changed, then confirm whether the right information reached the model before blaming the prompt, and confirm the prompt before blaming the model; every step uses the exact logged request.

Questions here typically give a symptom plus a few facts that hold steady ("latency and model version unchanged," "after the nightly re-index"). Those facts rule layers out. The right answer is usually the first investigation step at the layer the facts point to, not a fix at a different layer.

Practice scenario

Build exercise — Run a layered diagnosis

Advanced · 45 min

What you’ll learn

• Building a change timeline for an incident
• Isolating the failing layer by holding others constant
• Telling missing-context hallucinations from ignored-context ones

1. Take five failing requests from logs (or construct them) and, for each, record the exact prompt, retrieved context, model snapshot, stop_reason and output, alongside a list of recent changes.

   • Why: Diagnosis needs the request as the model actually saw it, and onset usually lines up with a change.
   • You should see: For each failure, whether the needed information was present in context or not.

2. For failures where the information was missing, test retrieval on its own with the query and the known correct document; for failures where it was present, rerun with the instruction restructured or citations required.

   • Why: Missing-context and ignored-context failures need fixes in different layers.
   • You should see: Each failure tagged as a retrieval, prompt, or still-unexplained problem.

3. For anything still unexplained, run the same cases on another model tier with the prompt and context fixed, and record whether the failure persists.

   • Why: Model mismatch should be the conclusion of evidence, not the starting assumption.
   • You should see: A short diagnosis note per failure naming the layer, the evidence, and the fix to test.

Exam traps

Upgrading to a larger model before checking whether the right context reached the model

A bigger model can’t answer from documents it never received. Check retrieval and context first.

Adding “do not make things up” to the prompt as the fix for every hallucination

That doesn’t supply missing facts or repair broken retrieval. Identify which of the three causes applies, then fix that layer.

Debugging from a paraphrase of the request instead of the exact logged prompt and context

Reconstructed requests omit exactly the detail that caused the failure. Replay what was actually sent.

Ignoring the timing: not correlating onset with deploys, index refreshes or model changes

Most regressions follow a change. The change log narrows the search faster than any theory.

Treating output cut off by max_tokens as a model reasoning failure

A stop_reason of “max_tokens” means truncation. Raise the limit or shorten the required output.

Assuming a failure is model nondeterminism without rerunning it to see if it reproduces

Rerun it several times. A failure that reproduces has a cause; an intermittent one still signals a fragile prompt or a missing check.

Sources

• Claude Developer Platform Docs
• Anthropic
• Anthropic, Claude Certified Architect – Professional Exam Guide v1.0 (July 2026)