What you’ll be able to do
- Analyze accuracy by document type and field before reducing human review
- Have the model output field-level confidence and calibrate review thresholds against a labeled validation set
- Use stratified random sampling of high-confidence extractions to measure ongoing error rates and catch novel error patterns
- Route low-confidence and ambiguous or contradictory extractions to human review, prioritizing limited reviewer capacity
What you need to know
Aggregate accuracy can hide a failing segment
An extraction pipeline reports 97% accuracy and the team wants to stop human review. That number is an average, and averages hide concentrations. If 90% of volume is standard invoices extracted at 99%, the remaining 10%, say handwritten receipts or multi-currency invoices, could be at 80% and the aggregate would still look excellent. The same is true across fields: vendor name and total may be near-perfect while tax_amount or due_date is wrong one time in six.
Before automating anything, analyze accuracy by document type and by field, and confirm performance is consistent across all segments. Reduce review only for the segments that have shown they can do without it.
Field-level confidence, calibrated on labeled data
To decide what a human should look at, the pipeline needs a signal per extraction. Have the model output a confidence score for each field, not one score for the whole document, since errors tend to concentrate in particular fields. A document with a confident total and an uncertain due date should send just that field for review.
A raw model-reported confidence isn't something to threshold directly. Its numbers aren't guaranteed to mean what they say: "0.9" might be right 70% of the time for one field and 99% for another. Calibrate the thresholds using a labeled validation set: run the pipeline on documents with known correct values, measure actual accuracy at each confidence level for each field, and set the review threshold where measured accuracy meets your requirement. Calibration turns the score into a routing signal you can rely on.
Route review where it matters
Reviewer time is limited, so spend it where errors are likely:
- Low model confidence on a field, below its calibrated threshold.
- Ambiguous or contradictory source documents, such as a total that doesn't match the line items or two different dates for the same event. Even a confident extraction from a contradictory source needs a human judgment.
Prioritize those queues rather than spreading reviewers evenly across all output. Uniform review wastes most of its effort on extractions that were already right.
Routing is also per field. When only due_date falls below its threshold, the reviewer checks that field against the source rather than re-keying the whole document, which stretches the same reviewer hours across many more documents.
Keep measuring what you automate: stratified sampling
Once high-confidence extractions skip review, you lose visibility into whether they're still right. Inputs change: a new vendor template, a new scan quality. A pattern the model has never seen can produce errors that are confident. The remedy is ongoing stratified random sampling of high-confidence extractions:
- Random, so the sample is representative and you get an unbiased error-rate estimate for automated output.
- Stratified by document type (and other relevant segments), so small segments are sampled enough to measure. A plain uniform sample of mostly standard invoices would include very few multi-currency documents.
- Reviewed by humans, which both measures the error rate over time and surfaces novel error patterns that confidence scores alone would never flag.
How the pieces fit
Validate per segment before automating. Calibrate field-level confidence on labeled data to set thresholds. Route low-confidence and contradictory cases to people, most important first. Keep a stratified sample of the automated remainder under review. Each step closes a gap the others leave open: segmentation catches hidden weak spots, calibration makes the routing signal trustworthy, and sampling catches the errors that confidence can't see.
Key concept
Don’t automate on an aggregate. Check accuracy by document type and field, calibrate field-level confidence thresholds against labeled data, send low-confidence and contradictory extractions to reviewers first, and keep a stratified random sample of high-confidence output under human review.
When a scenario cites one impressive overall accuracy number as the reason to remove human review, the expected answer is to break it down by segment first.
Practice scenario
Work it through, then open this
Three gaps. The 96% hasn’t been broken down by document type or field, and a uniform sample probably included few handwritten supplements, so their accuracy is largely unknown. The 0.85 threshold is a raw model score that hasn’t been calibrated against labeled data per field. And “ending review” leaves no way to detect drift or new error patterns. First, measure accuracy by document type and field. Then calibrate field-level thresholds on a labeled validation set, automate only the segments that meet the bar, route low-confidence and contradictory claims to reviewers first, and keep a stratified random sample of auto-approved extractions under ongoing review.
Build exercise — Calibrate a review threshold
Intermediate · 35 min
What you’ll learn
- Breaking an aggregate accuracy figure into segments
- Calibrating field-level confidence against labeled data
- Designing an ongoing stratified sample for automated output
-
Take a labeled set of extraction results (or construct one with at least three document types) and compute accuracy per document type and per field, alongside the overall figure.
- Why: The overall number decides nothing until you know whether any segment is far below it.
- You should see: At least one segment or field markedly below the aggregate.
-
For one field, bucket the model’s confidence scores and compute actual accuracy in each bucket. Choose the threshold where measured accuracy meets your target.
- Why: Raw confidence values aren’t guaranteed to match observed accuracy. Calibration makes the threshold mean something.
- You should see: A calibrated threshold that differs from the one you’d have guessed, and possibly a different threshold for each field.
-
Design the review flow: a priority queue for low-confidence and contradictory extractions, plus a stratified random sample of auto-accepted extractions by document type, with a fixed weekly review volume.
- Why: Limited reviewers go where errors are likely, and sampling keeps measuring the part no one else looks at.
- You should see: Every document type represented in the weekly sample, including the rare ones.
Exam traps
Automating extraction because aggregate accuracy is high, without checking accuracy per document type and field
Averages hide weak segments. Validate each document type and field before reducing review.
Using the model’s raw confidence scores as review thresholds without calibrating them on labeled data
Uncalibrated scores don’t reliably match observed accuracy. Set thresholds from a labeled validation set.
Stopping all review of high-confidence extractions once they are automated
That removes the only way to see drift and novel error patterns. Keep a stratified random sample under review.
Sampling uniformly at random, so rare document types barely appear in the audit
Stratify by document type so every segment gets enough samples to measure.
Sending reviewers a random slice of all extractions instead of the low-confidence and contradictory ones
Reviewer capacity is limited. Prioritize low confidence and ambiguous or contradictory sources.
Using a single document-level confidence score when errors concentrate in specific fields
Field-level confidence lets you route only the uncertain fields to review.
Sources
- Claude Developer Platform Docs
- Anthropic, Claude Certified Architect – Foundations Exam Guide v1.0 (July 2026)