CCAR-F scenarios › Structured Data Extraction · Question 56 of 60
Scenario 6 · Structured Data Extraction
You run a system that uses Claude to turn unstructured documents into structured records, validates every result against JSON schemas, and passes the output to downstream systems. Accuracy has to stay high across messy and unusual documents, and edge cases must be handled cleanly rather than with plausible-looking guesses.
Context Management & Reliability · Select TWO
Select two. Your extraction system scores 97% field-level accuracy on last quarter's validation set, and leadership wants to stop human review of every extraction the model marks as high confidence. Which two steps should come before you reduce human review?
Show answer and explanation
Answer: A and C — Break accuracy down by document type and by field to confirm that performance holds in every segment, not only in the overall average. / Set up ongoing stratified random sampling of high-confidence extractions to measure their error rate and catch new error patterns.
An aggregate figure like 97% can hide poor performance on particular document types or fields, so accuracy has to be checked by segment before automating (a). Once high-confidence output skips review, stratified random sampling keeps measuring its error rate and catches new error patterns (c). (b) assumes confidence is calibrated and still doesn't show per-segment accuracy. (d) is the aggregate-metric trap itself.
More from this scenario
- Your pipeline sends an extraction to human review when the model's self-reported…
- Commercial leases often include amendments. On one lease, the main body states base rent…
- Your system extracts obligations from each section of a 60-section supplier agreement…
- During extraction, the agent calls a lookup_vendor tool to match each vendor named in a…
- Your invoice extraction tool's JSON schema marks purchase_order_number as required…