A/B testing and iterative improvement

What you’ll be able to do

• Run an improvement loop: error analysis, hypothesis, single change, offline regression, then online test
• Design an online A/B test with a pre-declared primary metric, guardrails, randomization unit and sample size
• Use shadow and canary releases to limit risk before full exposure
• Keep comparisons valid by pinning model versions and versioning prompts and datasets

What you need to know

The improvement loop

Iteration on an LLM system is easy to do badly: someone edits the prompt, eyeballs five outputs, and ships. A disciplined loop looks like this:

1. Error analysis. Read failing cases from the eval set and from production, and group them by cause (missing context, ignored instruction, wrong tool, format break). Fix the largest group first.
2. Hypothesis. "Adding the product's return policy to retrieved context will fix the 'invented return window' failures."
3. One change. Change a single variable so the result can be attributed. If you change the prompt and the model together and quality rises, you don't know which helped, or whether one helped and the other hurt.
4. Offline regression. Run the full suite (4.2), not just the cases you were fixing. Prompt changes frequently fix one behaviour and break another.
5. Online test, only for changes that pass offline. Then fold newly discovered failures back into the dataset.

Why offline isn’t enough, and online isn’t first

Offline evaluation is cheap, fast and safe, so it filters most ideas. But it can only measure what's in the dataset and what the graders score. Real users bring different inputs, and outcomes such as resolution, retention, escalation and conversion only show up in production. Online testing measures those, at the cost of exposing real users to a possibly worse variant. So the order is fixed: offline gate first, online confirmation second.

Staged exposure

• Shadow mode. The new variant receives a copy of real traffic and its output is logged and graded, but never shown to users. Good for catching failures, latency and cost surprises on real inputs with zero user risk. It can't measure user behaviour.
• Canary. A small share of traffic gets the new variant, with automatic rollback if a guardrail breaches.
• A/B test. A controlled split large enough to detect the effect you care about, run for a fixed period.

Designing the A/B test

• Declare the primary metric and guardrails before starting, using the structure from 4.1. Choosing the metric after seeing results lets noise pick the winner.
• Choose the randomization unit. For multi-turn assistants or anything with memory, assign per user or per account, not per request, so a conversation doesn't switch behaviour mid-stream and a user's later behaviour can be attributed to one variant.
• Size the test. Estimate the traffic needed to detect the smallest improvement worth shipping, and run until you reach it. Stopping the first time the dashboard shows a lead inflates false positives. Run across full weekly cycles if usage varies by day, and be wary of novelty effects in the first days.
• Check guardrails and segments before declaring a winner. A variant that lifts resolution overall but raises p95 latency past the SLA, or drops quality for one language, is not a clean win. That's a trade-off for stakeholders to decide, with the numbers (domain P6).

Keep the comparison valid

Both variants must differ only in the thing you're testing. Pin the model to a specific dated snapshot rather than a floating alias, so a model update doesn't land during the test. Version prompts, retrieval configuration and the eval dataset, and log the version identifiers with every request so every result is traceable (4.5). When you deliberately move to a new model, treat that as its own experiment: run the full offline suite on the new model before any traffic moves, because prompts tuned for one model don't always transfer.

Comparing open-ended output

When there's no single correct answer, pairwise comparison (which of two responses is better against the rubric) is often more reliable than absolute scores. Use a calibrated model judge or human raters, randomize the order of the two responses, and report the win rate with ties counted separately.

Key concept

Improve by single, attributable changes that pass the full offline regression suite, then confirm online with a pre-declared primary metric and guardrails, the right randomization unit, enough traffic, and a pinned model.

Exam options often describe a test that "worked" but was invalid: multiple changes at once, stopped early, randomized per request, or judged on the primary metric alone. Find the design flaw before evaluating the result.

Practice scenario

Build exercise — Write an experiment plan

Advanced · 35 min

What you’ll learn

• Writing a hypothesis tied to a failure cluster
• Specifying primary metric, guardrails, unit and duration up front
• Planning staged exposure with rollback triggers

1. From a set of failing cases, group failures by cause and write one hypothesis for the largest group, naming the single variable you will change.

   • Why: Error analysis points effort where it pays, and a single variable makes the result attributable.
   • You should see: A failure table with counts per cause and a one-line hypothesis.

2. Write the pre-registration: primary metric, guardrails with thresholds, randomization unit and why, minimum sample or duration, and the stopping rule.

   • Why: Decisions written before the data arrives can’t be bent to fit it.
   • You should see: A half-page plan someone else could execute without asking you anything.

3. Add the exposure stages (shadow, canary percentage, full test), the automatic rollback conditions at each stage, and which model snapshot and prompt version both variants use.

   • Why: Staged rollout limits damage; pinned versions keep the comparison clean.
   • You should see: Explicit version IDs for both arms and a rollback trigger tied to each guardrail.

Exam traps

Changing the prompt, model and retrieval settings in one release and crediting the result to one of them

Bundled changes can’t be attributed, and one may be hurting while another helps. Change one variable per experiment.

Skipping offline evaluation and testing every idea directly on production users

Online tests are slow and expose users to risk. The offline suite filters most ideas cheaply first.

Stopping the test as soon as the dashboard shows a lead

Early leads are often noise. Fix the sample size or duration in advance and run to it.

Randomizing per request in a multi-turn product, so a user sees both variants in one conversation

Assignment should follow the unit of experience, usually the user or account, so behaviour is consistent and attributable.

Declaring the winner on the primary metric without checking guardrails and segments

A lift that breaches latency or cost thresholds, or hurts a segment, is a trade-off to escalate, not a win.

Comparing variants while a floating model alias changes underneath both

Pin a dated model snapshot for the duration, and treat a model upgrade as its own experiment.

Sources

• Claude Developer Platform Docs
• Anthropic
• Anthropic, Claude Certified Architect – Professional Exam Guide v1.0 (July 2026)