Building an evaluation pipeline

What you’ll be able to do

• Build a dataset representative of real usage rather than hand-picked happy paths
• Choose exact-match, human, or model-graded evaluation for a given criterion
• Fix grading criteria before looking at outputs to avoid motivated reasoning
• Run the same suite on every change instead of eyeballing individual transcripts

What you need to know

What an eval actually is

Three parts, all required: a dataset that looks like real usage, a grading method fixed before anyone looks at an output, and a metric tracked across runs. Missing any one of the three isn't a smaller eval — it's not an eval. A single impressive transcript demonstrates that the system can work once. It says nothing about whether it works, and it's the failure mode this whole domain exists to correct: lesson 4.1 assumed a broken system in production; an eval suite is how you catch the break before it ships.

Where the dataset comes from

Hand-written examples cluster around what the author already expected to work. Production traffic — real user turns, real tool results, the malformed ones included — surfaces the cases nobody thought to write down. A representative set pulls from both, weighted toward production once there's any real traffic to draw from, and specifically includes edge cases the happy-path set would never generate: empty results, ambiguous requests, adversarial phrasing, multi-step tasks that fail on step three rather than step one.

Three ways to grade, and when each earns its cost

• Exact-match / code-based — a deterministic check: does the output parse as valid JSON, does the tool call use the right arguments, does a regex find the expected value. Cheap, fast, zero ambiguity. Use it for anything with a structurally correct answer.
• Human review — a person reads the output and judges it. The ground truth, and the most expensive per item. Reserve it for calibrating the other two methods and for spot-checking a sample, not for grading every run.
• Model-graded (LLM-as-judge) — a model call grades another model's output against a rubric. Scales to volumes human review can't reach, at the cost of the biases covered in lesson 4.3.

The trap is reaching for model grading by default. If a regex or a JSON schema check can grade it, that's strictly cheaper and has no bias to correct for. Save the judge call for criteria that are genuinely subjective — tone, helpfulness, whether an explanation actually addressed the question asked.

Decide the rubric before you see the outputs

Grading criteria set after looking at a batch of results tend to bend toward whatever the system already produced — a subtle, easy-to-miss form of grading the system on its own homework. Writing the pass/fail criteria against the task itself, before generation, keeps the bar fixed regardless of what comes out. This is the same discipline as a scientific pre-registration, for the same reason.

The suite runs on every change, not just once

An eval built once and never rerun catches nothing after the day it was built. The value is in rerunning the identical suite after every prompt edit, model swap, or tool change and comparing the metric — accuracy, task completion rate, cost, latency — against the last run. A prompt change that improves three example transcripts you happened to check by eye and silently regresses forty cases you didn't is exactly what a tracked metric catches and manual spot-checking doesn't.

Key concept

An eval is a representative dataset, a grading method fixed before you see outputs, and a metric tracked across every change — not a handful of examples that happened to work once.

Practice scenario

Build exercise — Turn a demo into a real eval

Intermediate · 30 min

What you’ll learn

• Building a dataset from real usage instead of hand-picked examples
• Fixing grading criteria before generation

1. Pull 15-20 real examples from actual usage or logs, deliberately including at least a few that are ambiguous, malformed, or edge-case rather than clean happy path.

   • Why: Hand-written examples cluster around what you already expected to work; real usage surfaces what you didn’t think to write down.
   • You should see: A set with at least a couple of cases you’re not confident the system handles well.

2. Before running anything, write down exactly what counts as a pass for each example, and pick exact-match, human, or model grading per example based on whether the answer is structurally checkable.

   • Why: Criteria set after seeing outputs tend to bend toward whatever the system already produced.
   • You should see: A rubric that would look the same whether the system passes or fails.

3. Run the suite, record the pass rate, then make one change and rerun the identical suite rather than re-reading the transcripts by eye.

   • Why: This is the actual regression check — the comparison a metric gives you that eyeballing individual outputs can’t.
   • You should see: A pass rate that moved, in a direction eyeballing five transcripts wouldn’t have reliably told you.

Exam traps

Testing against a handful of hand-written happy-path examples

That’s a demo, not an eval. It shows the system can work once, not that it works.

Deciding what counts as correct after seeing the outputs

Criteria fixed after the fact bend toward whatever the system already produced. Set the rubric before generation.

Reading transcripts by eye instead of rerunning the eval suite after a change

A tracked metric catches regressions in cases you didn’t happen to re-read. Manual spot-checking doesn’t.

Using model grading for something a deterministic check could grade for free

If a regex or schema check settles it, that’s cheaper and has no bias to correct for. Save judge calls for genuinely subjective criteria.

Sources

• Claude Developer Platform Docs