CCAR-F scenarios › Structured Data Extraction · Question 57 of 60
Scenario 6 · Structured Data Extraction
You run a system that uses Claude to turn unstructured documents into structured records, validates every result against JSON schemas, and passes the output to downstream systems. Accuracy has to stay high across messy and unusual documents, and edge cases must be handled cleanly rather than with plausible-looking guesses.
Context Management & Reliability
Your pipeline sends an extraction to human review when the model's self-reported confidence for any field is below 0.8. An audit finds that fields reported at 0.9 or higher on handwritten intake forms are wrong 25% of the time, while typed forms at the same confidence are almost always right. Reviewer capacity is fixed. What should you change?
Show answer and explanation
Answer: C — Calibrate field-level confidence against a labeled validation set for each document type, and set routing thresholds from those calibrated results.
Raw self-reported confidence isn't reliable, and the audit shows it means different things for different document types. Section 6 calls for field-level confidence calibrated with labeled validation sets, with review thresholds set from that calibration, so that limited reviewer time goes where errors actually happen. (a) is a prompt tweak to a number that isn't calibrated in the first place. (b) averages away the weak field signals that matter. (d) ignores the fixed capacity constraint.
More from this scenario
- Commercial leases often include amendments. On one lease, the main body states base rent…
- Your system extracts obligations from each section of a 60-section supplier agreement…
- During extraction, the agent calls a lookup_vendor tool to match each vendor named in a…
- Your invoice extraction tool's JSON schema marks purchase_order_number as required…
- A mixed document inbox contains invoices, receipts, and service contracts. You defined…