What you’ll be able to do

Keep long model calls off synchronous request paths
Apply retry, timeout, and idempotency correctly around an LLM call
Recognise when the bug is plain engineering rather than prompting
Structure an integration so a model change is a small diff

What you need to know

Why ordinary engineering is on an AI exam

Because plenty of people can prompt well and still ship an integration that blocks a thread, retries a non-idempotent write, or has no path back when a dependency stalls. The blueprint includes this sub-skill precisely because the failure modes here get misdiagnosed as model problems.

Never block a request thread on a long call

Model calls are slow by web-request standards, and extended thinking or a long generation makes that worse. A synchronous handler that waits on one will hold its thread, and under load the pool exhausts and the service times out — for reasons that have nothing to do with Claude.

before — thread held for the whole generation

@app.post("/summarize") def summarize(req): result = client.messages.create(...) # blocks return result # pool exhausts under load

after — accept, queue, report

@app.post("/summarize") def summarize(req): job_id = queue.enqueue(run_summary, req) return {"job_id": job_id, "status": "queued"}

client polls /jobs/{id}, or you push a webhook on completion

Retries, timeouts, and the idempotency trap

Retry with backoff is right for transient conditions — a 429 or a 529. But wrap the whole operation carefully: if your handler calls the model and then writes a record, a naive retry of the combined unit can produce a duplicate write.

Make the downstream effect idempotent, or scope the retry to the model call alone. This is a plain distributed-systems concern that an LLM call doesn't exempt you from.

Structure so a model change is a small diff

Scatter the model name, prompt text, and parsing logic across twelve call sites and every upgrade becomes an archaeology exercise. Centralise them behind one module: the model reference in config, the prompt in a versioned file, and parsing in a single validated function.

The test is simple — if pinning to a new model version touches one line, you're structured correctly. If it touches twelve, that's the finding.

The diagnostic question

ScenarioAn intake service starts timing out during morning peak. Response quality, when responses do arrive, is fine. Latency per model call is unchanged from last month.

Nothing here points at the model. Quality is fine and per-call latency is flat — so this is concurrency, not cognition. The fix is an async path or a queue, and rewriting the prompt or switching tiers would burn a day without touching the cause.

Key concept

An LLM call is still a slow network call. Async paths, correct retry scope, idempotent downstream writes, and a single place where the model is referenced are what keep the integration layer from being blamed on the model.

Practice scenario

ScenarioAn intake service times out during morning peak. Response quality is fine. Per-call latency is identical to last month.

Work it through, then open this

Nothing points at the model — flat per-call latency with degraded throughput is thread exhaustion. Move the call off the synchronous request path onto a queue and return a job reference. Rewriting the prompt or changing tier would burn a day on the wrong layer.

Build exercise — Find the blocking call in your own service

Intermediate · 25 min

What you’ll learn

Telling a concurrency failure apart from a model failure
Scoping retries so they don’t duplicate writes

Find every place your code calls Claude inside a synchronous request handler. For each, ask what happens to the thread while it waits.
- Why: A slow model call on a request thread exhausts the pool under load, and the symptom looks nothing like its cause.
- You should see: At least one handler that blocks, or confirmation that everything already runs async.
Look at your retry logic and trace exactly what is inside the retried block. If a database write sits in there with the model call, a transient 429 will replay both.
- Why: Retry scope is a plain distributed-systems concern that an LLM call doesn’t exempt you from.
- You should see: Either an idempotent write, a retry scoped to the call alone, or a duplicate-record bug waiting to happen.
Grep for the model name string across the codebase.
- Why: If pinning to a new version means editing twelve files, upgrades get postponed — and postponed upgrades are how you end up on a rolling alias.
- You should see: Ideally one hit in a config module. Count what you actually find.

Exam traps

Diagnosing a concurrency failure as a model or prompt problem

Flat per-call latency with degraded throughput is a threading problem. No prompt change fixes a blocked pool.

Retrying a unit that includes a non-idempotent write

A transient API error retried at the wrong scope produces duplicate records. Scope the retry to the call, or make the write idempotent.

Spreading model references across many call sites

It turns a one-line version pin into a twelve-file change and makes upgrades risky enough to postpone.

Treating 429 and 400 the same way

429 is rate limiting and calls for backoff. 400 is your malformed request and will fail identically forever, however many times you retry.

Sources

Claude Developer Platform Docs

Software engineering foundations

What you'll be able to do