Curriculum › Eval, Testing & Debugging · 2.6% of the exam

Debugging and error handling

What you'll be able to do

  • Identify error types from their codes and respond appropriately
  • Attribute a failure to the integration layer or the model output
  • Trace a symptom back to where reality first diverged
  • Match recovery pattern to error class

What you’ll be able to do

  • Identify error types from their codes and respond appropriately
  • Attribute a failure to the integration layer or the model output
  • Trace a symptom back to where reality first diverged
  • Match recovery pattern to error class

What you need to know

Step 1 — identify the error type

  • 400 invalid_request_error — your request was malformed. A code bug. Retrying fails identically, forever. Audit the schema.
  • 401 authentication_error / 403 permission_error — credential problems, not transient.
  • 413 request_too_large — over the size ceiling (32 MB on Messages, 256 MB on Batch).
  • 429 rate_limit_erroryour usage hit a limit. Back off, respect retry-after, and ramp traffic gradually.
  • 500 api_error — an internal Anthropic failure. Retry with backoff.
  • 504 timeout_error — the request timed out while processing. Consider streaming for long-running requests.
  • 529 overloaded_error — aggregate load across all users. Nothing to do with your account. Retry with backoff.
 <p>The distinction the exam is most likely to reach for is <b>429 against 529</b>, because both are retryable and they mean opposite things. A 429 is about <i>you</i> — your rate, your tier, your ramp — and it carries a <code>retry-after</code> header telling you how long to wait. A 529 is about everyone; the queue is global and your own request rate is irrelevant to it. Same remedy in shape, different diagnosis, and only one of them is something you can fix by managing your own traffic.</p>

 <p>One trap inside the trap: a 429 with no <code>retry-after</code> header is usually a spend cap rather than a rate limit, and retrying will not clear it however long you wait.</p>
 <p>The distinction that gets tested is 400 against the other two. A 400 is yours to fix; 429 and 529 are transient conditions with standard recovery patterns. Retrying a 400 is the classic wasted cycle.</p>

Step 2 — attribute the failure

Two categories, completely different fixes:

  • Integration layer — a bad tool call, stale or cached data, a malformed request, missing context that was never passed.
  • Model output — a hallucination, a misread instruction, a task beyond the chosen tier.
ScenarioAn assistant confidently tells a patient their insurance is active. The eligibility tool silently returned a week-old cached value; coverage lapsed three days ago.

The model reasoned correctly over bad input. Every hour spent rewriting the prompt is wasted — the fix lives in the tool and data layer. The diagnostic question is always: given what it was told, was the reasoning sound? If yes, look upstream.

Step 3 — trace analysis

In multi-step systems the visible symptom usually surfaces several steps after the actual cause. A wrong final answer might trace back to a tool result four calls earlier that looked plausible enough to pass unnoticed.

Follow the recorded chain of calls, tool results, and intermediate outputs back to the first point where reality diverged from expectation — not the point where it became obvious. Those are rarely the same step, and debugging at the second one finds nothing.

Step 4 — match the recovery

  • Transient API error (429, 529) → retry with backoff
  • Malformed request (400) → fix the schema; do not retry
  • Malformed model output → schema validation plus corrective re-prompting, per lesson 6.3
  • Dependency down → explicit fallback behaviour, degraded but defined

Knowing that retries exist is not the skill. Matching the recovery to the error class is, and a mismatched recovery — retrying a 400, or re-prompting around a stale cache — burns the cycle without touching the cause.

Key concept

Identify the error type, attribute it to the integration layer or the model, trace back to the first divergence rather than the obvious symptom, then match the recovery to the class. Retrying a 400 fixes nothing.

Practice scenario

ScenarioAn assistant tells a patient their coverage is active. It isn't. The eligibility tool returned a cached value from last week.
Work it through, then open this

Attribute before you fix. The model reasoned correctly over bad input, so every hour spent on the prompt is wasted. The bug is in the tool and data layer. Ask the diagnostic question first: given what it was told, was the reasoning sound? If yes, look upstream.

Build exercise — Trace a failure to its first divergence

Intermediate · 30 min

What you’ll learn

  • Attributing a failure to the integration layer or the model
  • Reading a trace backwards to where reality first broke
  1. Find a real wrong answer your system produced, and write down exactly what the model was told at that moment — full prompt, tool results and all.

    • Why: The diagnostic question is whether the reasoning was sound given the input. You can’t answer it without the input.
    • You should see: Often the model reasoned correctly over something stale or malformed.
  2. Walk the trace backwards from the wrong output to the first step where a value differs from what you expected.

    • Why: In multi-step systems the symptom surfaces several steps after the cause. Debugging where it became obvious finds nothing.
    • You should see: A divergence point earlier than the one you’d have guessed.
  3. Classify the fix: schema correction, retry with backoff, validation plus re-prompting, or a defined fallback. Then check your code does that one and not a different one.

    • Why: Mismatched recovery — retrying a 400, re-prompting around a stale cache — burns the cycle without touching the cause.
    • You should see: Either a match, or a recovery path pointed at the wrong error class.

Exam traps

Retrying a 400

A malformed request fails identically every time. It is a code bug, not a transient condition.

Treating every wrong answer as a prompting problem

If the reasoning was sound given the input, the fault is upstream in the data or tool layer.

Debugging at the point the symptom appeared

In multi-step systems the cause is usually several steps earlier. Trace back to the first divergence.

Applying one recovery pattern to every error

Retry suits transient errors, schema fixes suit 400s, validation suits malformed output, fallbacks suit downed dependencies.

Treating 429 and 529 as the same condition

Both are retryable, but 429 is your own rate or spend limit and carries a retry-after header, while 529 is global load unrelated to your account. Only the first is fixed by managing your traffic.

Sources

Quick check

A production integration starts receiving 429 responses during a traffic spike. What's the appropriate handling?