CCAR-F scenarios › Structured Data Extraction · Question 57 of 60

Scenario 6 · Structured Data Extraction

You run a system that uses Claude to turn unstructured documents into structured records, validates every result against JSON schemas, and passes the output to downstream systems. Accuracy has to stay high across messy and unusual documents, and edge cases must be handled cleanly rather than with plausible-looking guesses.

Context Management & Reliability

Your pipeline sends an extraction to human review when the model's self-reported confidence for any field is below 0.8. An audit finds that fields reported at 0.9 or higher on handwritten intake forms are wrong 25% of the time, while typed forms at the same confidence are almost always right. Reviewer capacity is fixed. What should you change?

Show answer and explanation

Answer: C — Calibrate field-level confidence against a labeled validation set for each document type, and set routing thresholds from those calibrated results.

Raw self-reported confidence isn't reliable, and the audit shows it means different things for different document types. Section 6 calls for field-level confidence calibrated with labeled validation sets, with review thresholds set from that calibration, so that limited reviewer time goes where errors actually happen. (a) is a prompt tweak to a number that isn't calibrated in the first place. (b) averages away the weak field signals that matter. (d) ignores the fixed capacity constraint.