Evaluation datasets and test frameworks

What you’ll be able to do

• Assemble an evaluation dataset from sampled production traffic, edge cases, adversarial cases and reviewed synthetic data, stratified by segment
• Choose code-graded, model-graded or human grading per criterion based on whether the answer is verifiable
• Calibrate a model-graded judge against human labels and control its known biases
• Layer tests into a framework: component checks, CI regression suite, end-to-end and adversarial suites, with held-out data

What you need to know

Where the cases come from

The dataset determines what your metrics (4.1) can see. A set written by the team that built the prompt mostly contains inputs the prompt already handles. A production-grade set mixes sources:

• Sampled real traffic, with personal data redacted or the sample handled under your data-governance rules. This is the only source that reflects actual phrasing, length and messiness.
• Edge cases: empty or very long inputs, multiple languages, contradictory documents, questions the knowledge base can't answer, inputs at the boundary between categories.
• Adversarial cases: prompt-injection attempts, requests designed to trigger disallowed output, attempts to make the agent use a tool it shouldn't. These feed the safety and security metrics.
• Synthetic cases generated with Claude to fill gaps in coverage (a rare document type, a new product). They are fast to produce but inherit the generator's blind spots, so a human reviews them before they enter the set.
• Every production failure that gets reported. Once fixed, it becomes a permanent regression case.

Stratify the set so every segment you report on has enough cases to produce a meaningful number; a rare but high-stakes segment may need deliberate oversampling. Version the dataset alongside prompts and code, so a score always refers to a known dataset version, and refresh it as production inputs drift.

Three ways to grade, and when each fits

• Code-graded. Exact or normalized match, regex, JSON schema validation, numeric tolerance, a unit test run against generated code, an assertion that a specific tool was or wasn't called. Fast, cheap, deterministic, and runnable on every commit. Use it for every criterion that has a verifiable answer.
• Model-graded (LLM-as-judge). A model scores output against a written rubric: grounded in the sources, answers the actual question, correct tone, no policy violation. It scales to open-ended output that code can't check. It is only as good as its rubric and its calibration.
• Human-graded. Domain experts judge quality, especially for subjective or high-stakes criteria. The most trustworthy and the most expensive; too slow to run on every change.

"Mixed methodology" means using all three in layers, each for what it does best. Code checks run first and cheaply filter format and correctness failures. A model judge scores the open-ended criteria at scale. Humans grade a smaller sample, which serves two purposes: it is the ground truth for high-stakes criteria, and it is how you check the judge.

Making a model judge trustworthy

• Specific rubrics. "Rate quality 1–10" produces noise. Break it into concrete criteria, each with a description of what passes and fails, ideally with an example of each.
• Reasoning before the verdict. Ask the judge to cite the evidence for its score, then give the score. It improves consistency and makes disagreements debuggable.
• Calibration against humans. Have humans grade a sample the judge also grades and measure agreement per criterion. Where agreement is low, fix the rubric or keep that criterion human-graded. Recheck when you change the judge's prompt or model.
• Known biases. In pairwise comparisons, judges can favour the first-listed answer, so run both orders. Judges can favour longer answers, so state in the rubric that length is not a merit. Grading your system's output with the same prompt that produced it invites self-agreement, so the judge has its own prompt and is often a separately chosen model.

From dataset to framework

A test framework organizes when each check runs:

• Component tests on prompt templates, tool schemas and output parsers, mostly code-graded.
• A regression suite in CI that runs on every prompt, model, tool or retrieval change and blocks merges that breach a guardrail.
• End-to-end and agent tests that check the final state an agent leaves behind and, as a diagnostic, its trajectory.
• An adversarial suite for safety and security, run on every release.

Keep a held-out portion of the dataset that is not used while iterating on prompts. If you tune against the same cases you use to decide whether to ship, scores rise while real quality may not. And because model output varies between runs, run important cases several times and report a pass rate rather than trusting a single pass.

Key concept

Grade each criterion with the cheapest method that can verify it: code for verifiable answers, a calibrated model judge for open-ended quality, humans for ground truth and for checking the judge, over a stratified, versioned dataset built from real traffic.

Questions here often offer a single grading method for everything (all human, or all LLM-judge). The better answer layers methods by criterion and includes a step that validates the model judge against human labels.

Practice scenario

Build exercise — Build a mixed-method eval harness

Advanced · 60 min

What you’ll learn

• Building a stratified dataset with a held-out split
• Assigning a grading method per criterion
• Measuring judge agreement against human labels

1. Collect 50–100 cases for one task from at least three sources (sampled real inputs, edge cases, adversarial cases), tag each with its segment, and hold out a fifth of them.

   • Why: Mixed sources expose failures a hand-written set never contains, and the held-out split guards against tuning to the test.
   • You should see: Every reporting segment represented, and a held-out file you agree not to look at while iterating.

2. For each success criterion, write down whether code can verify it; implement those checks as code, and write a criterion-by-criterion rubric for the rest, asking the judge for evidence before its verdict.

   • Why: Code checks are exact and free to rerun; a judge is only for what code can’t check.
   • You should see: A grading table where each criterion is labelled code, model or human, with the reason.

3. Grade 20–30 outputs yourself (or with a domain expert) on the model-graded criteria, compare with the judge, and revise the rubric where you disagree.

   • Why: An uncalibrated judge produces confident numbers that may not track what experts value.
   • You should see: An agreement figure per criterion, and at least one rubric line rewritten because of a disagreement.

Exam traps

Building the dataset only from happy-path examples the team wrote by hand

Hand-written sets reflect what the builders expected. Add sampled real traffic, edge cases and adversarial inputs.

Using an LLM judge without ever checking its agreement with human graders

Until it’s calibrated on a human-graded sample, a judge’s score is unverified. Measure agreement per criterion and recheck after changes.

Asking humans to grade everything, so the suite is too slow and expensive to run on each change

Human grading anchors the system, but it can’t gate every commit. Use it on samples and for calibration.

Using a model judge for criteria that code could verify exactly

Format, schema, exact values and tool-call assertions are deterministic. A judge adds cost and noise there.

Tuning prompts against the same examples used to decide whether to ship

Scores on a set you’ve optimized against overstate real quality. Keep a held-out split for the ship decision.

Running each test case once and treating a single pass as proof of reliability

Model output varies between runs. Repeat important cases and report a pass rate.

Letting the dataset go stale while production inputs drift

New products, phrasing and attack patterns appear. Refresh the set from production and add every fixed failure.

Sources

• Claude Developer Platform Docs
• Anthropic
• Anthropic, Claude Certified Architect – Professional Exam Guide v1.0 (July 2026)