What you’ll be able to do
- Identify the characteristic failure modes of LLM systems: confabulation, stale knowledge, sycophancy, injection, compounding agent errors and silent degradation
- Distinguish limitations of the model from failures of the surrounding system (retrieval, tools, context assembly, integration)
- Build a risk register that ranks failure modes by impact, likelihood, reversibility and blast radius
- Pair each significant risk with a prevention, a detection and a containment control
What you need to know
Why LLM failures are different
Traditional software tends to fail loudly: an exception, a timeout, a 500. LLM systems often fail quietly and fluently. A wrong answer reads exactly like a right one, a missing fact gets filled with a plausible one, and an agent that took a wrong turn three steps ago keeps working confidently on a bad premise. That changes what risk management has to do. You can't rely on errors announcing themselves, so detection has to be designed in.
The failure modes to name in any design review
- Confabulation (hallucination). Plausible but unsupported content: invented citations, figures, API fields or policy clauses. It's most likely when the context lacks the answer and the prompt pressures the model to give one anyway.
- Stale or missing knowledge. The model's training has a cutoff and knows nothing about your private data. Anything time-sensitive or proprietary has to come from retrieval or tools, and if that layer serves stale content, the model will faithfully repeat it.
- Context problems. Key facts missing from the assembled context, buried in very long inputs, contradicted by other content, or lost when conversation history is summarized or truncated.
- Sycophancy and over-agreement. Going along with a user's incorrect premise, or reversing a correct answer under pushback.
- Instruction conflicts and injection. Conflicting instructions across system prompt, user and retrieved content, and deliberate manipulation (see 5.1).
- Compounding agent errors. In multi-step agents, small per-step error rates multiply across steps, and an early mistake becomes the premise for every later action. Tool misuse, loops and runaway cost belong here too.
- Unreliable self-assessment. Self-reported confidence and "I verified this" claims aren't calibrated evidence. Treat them as weak signals until you've measured them against labeled data.
- Non-determinism. The same input can produce different outputs, which complicates reproduction, auditing and regression testing.
- Over- and under-refusal. Declining legitimate requests in a sensitive domain, or complying where the business wanted a refusal.
- Silent degradation over time. Shifts in input distribution, a corpus refresh, a prompt edit, a model version change or a tool's API change can all move quality with no error raised.
- Operational and dependency risk. Provider outages, rate limiting, latency spikes and cost overruns, plus the risk that people over-rely on outputs (automation bias).
Model limitation or system failure?
Senior engineers lose time by reaching for "the model got worse" first. Most production incidents in Claude systems trace back to the system around the model: retrieval returning wrong or stale chunks, a tool returning an error the agent read as an empty result, a template change that dropped a section of the prompt, a truncation that cut off the policy. A quick triage:
- Did anything change recently, such as data, index, prompt, tool or model version? Start with the change that lines up with the symptom.
- Reproduce with the exact assembled context the model saw. If the correct answer isn't in that context, it's a context or retrieval failure, not a reasoning failure.
- If the context was right and the output still wrong, look at prompt clarity, conflicting instructions and task-model fit.
Logging the fully assembled request (with appropriate data controls) is what makes this triage possible at all.
A risk register built for LLM systems
A useful register ranks each failure mode on more than likelihood:
- Impact: who is harmed and how badly (financial, safety, legal, reputational).
- Reversibility: a bad draft a human edits is cheap; a sent payment, a deleted record or a published statement may not be recoverable.
- Blast radius: one user, one tenant or every user at once. A shared prompt or index bug fails at scale.
- Detectability: how quickly you'd know. A quiet failure needs active monitoring.
For each significant risk, record three controls. Prevention lowers the chance it happens (grounding with citations, scoped tools, explicit permission to say "I don't know"). Detection notices when it happens (output checks, sampled human review, eval suites run on every change, drift metrics). Containment limits the damage (approval gates, rate and spend limits, reversible actions, kill switches, fallbacks). Assign an owner and a review cadence so the register follows the system as it changes.
Limitations to state plainly to stakeholders
Part of the architect's job is saying clearly what the system can't promise. It won't be correct every time. Its outputs need to be checked in proportion to their consequences. Its behavior can shift when data or models change. And it shouldn't be the only control on a decision with legal or safety consequences. Writing these down in the design document sets realistic expectations and justifies the controls you're asking for.
Key concept
LLM systems fail quietly and fluently, so every significant failure mode needs a designed-in detection and a containment control, ranked by impact, reversibility and blast radius rather than by likelihood alone.
In questions, watch for symptoms that point to a specific layer (a corpus refresh, a prompt template change, a new tool version), and prefer answers that investigate or fix that layer over answers that blame or upgrade the model. When asked which risk to prioritize, choose the irreversible, wide-blast-radius one over the more frequent but cheap-to-correct one.
Practice scenario
Work it through, then open this
This is a system failure, not a model limitation. The agent can’t tell a timeout from “no exclusions found,” so it treats an access failure as a clean result, and an irreversible action (payment) follows from it. Fix the tool contract so access failures return a structured error that’s distinct from a valid empty result, and make auto-approval require a positive policy-status confirmation rather than the absence of a problem. In the risk register, auto-approval scores high on irreversibility and blast radius. Add detection (monitor the approval rate and the tool-failure rate against a baseline, and sample auto-approved claims) and containment (pause auto-approval automatically when lookups fail above a threshold).
Build exercise — Write an LLM risk register
Advanced · 45 min
What you’ll learn
- Enumerating failure modes specific to one Claude system rather than a generic list
- Scoring risks on impact, reversibility, blast radius and detectability
- Mapping each top risk to prevention, detection and containment controls
-
For one system you know, walk the request path (input, context assembly, retrieval, model, tools, output, downstream action) and list at least eight concrete failure modes, each tied to a specific stage.
- Why: Failure modes tied to a stage point straight at the control that addresses them. “Hallucination” in the abstract doesn’t.
- You should see: A list where more than half the items sit outside the model itself (retrieval, tools, templates, integration).
-
Score each one on impact, likelihood, reversibility, blast radius and detectability, and rank them.
- Why: Likelihood-only ranking puts frequent, harmless issues at the top and buries rare, irreversible ones.
- You should see: At least one low-likelihood item ranked near the top because it’s irreversible or affects every user.
-
For the top five, write one prevention, one detection and one containment control each, plus an owner.
- Why: Prevention alone assumes you’ll succeed every time; detection and containment are what make the remaining risk acceptable.
- You should see: No top risk relying only on a prompt instruction, and every quiet failure covered by an active detection.
Exam traps
Assuming a fluent, confident answer is a correct one
Fluency and correctness are separate. Confabulated content often reads as more confident than a correct but hedged answer.
Blaming the model first when the symptom points to retrieval, tools or context assembly
Most incidents trace back to the surrounding system. Check what changed and inspect the exact context the model received before suspecting the model.
Treating the model’s self-reported confidence as a reliable error detector
Self-assessment is a weak signal until it has been calibrated against labeled outcomes. Don’t build your only detection control on it.
Rating risks by likelihood alone and ignoring reversibility and blast radius
A rare failure that sends money or affects every tenant outranks a frequent one that a reviewer corrects in seconds.
Assuming a pre-launch evaluation keeps describing production behavior after data, prompts or models change
Inputs shift and components change. Re-run evals on every change and monitor production quality continuously.
Expecting a larger or newer model to remove a failure mode that comes from system design
A stronger model still can’t answer correctly from missing context or tell a mislabeled empty result from a real one. Fix the system.
Sources
- Claude Developer Platform Docs
- Anthropic
- Anthropic, Claude Certified Architect – Professional Exam Guide v1.0 (July 2026)