CCAR-F scenarios › Structured Data Extraction · Question 56 of 60

Scenario 6 · Structured Data Extraction

You run a system that uses Claude to turn unstructured documents into structured records, validates every result against JSON schemas, and passes the output to downstream systems. Accuracy has to stay high across messy and unusual documents, and edge cases must be handled cleanly rather than with plausible-looking guesses.

Context Management & Reliability · Select TWO

Select two. Your extraction system scores 97% field-level accuracy on last quarter's validation set, and leadership wants to stop human review of every extraction the model marks as high confidence. Which two steps should come before you reduce human review?

Show answer and explanation

Answer: A and C — Break accuracy down by document type and by field to confirm that performance holds in every segment, not only in the overall average. / Set up ongoing stratified random sampling of high-confidence extractions to measure their error rate and catch new error patterns.

An aggregate figure like 97% can hide poor performance on particular document types or fields, so accuracy has to be checked by segment before automating (a). Once high-confidence output skips review, stratified random sampling keeps measuring its error rate and catches new error patterns (c). (b) assumes confidence is calibrated and still doesn't show per-segment accuracy. (d) is the aggregate-metric trap itself.