CCAR-F scenarios › Structured Data Extraction · Question 53 of 60
Scenario 6 · Structured Data Extraction
You run a system that uses Claude to turn unstructured documents into structured records, validates every result against JSON schemas, and passes the output to downstream systems. Accuracy has to stay high across messy and unusual documents, and edge cases must be handled cleanly rather than with plausible-looking guesses.
Prompt Engineering & Structured Output
Your extraction pipeline validates each result with Pydantic. When validation fails, the retry re-sends the same prompt, and only about a third of retries succeed. Failure analysis shows two groups: dates written as "03/04/25" when the schema expects ISO 8601, and contract_value missing on contracts that only say "fees as set out in Schedule B," which is a separate document not included in the request. What is the best redesign?
Show answer and explanation
Answer: D — Retry date failures with the document, failed output, and specific errors; for Schedule B values, return null and flag them rather than retrying.
Retrying with error feedback works when the follow-up request includes the original document, the failed extraction, and the specific validation errors, and it fixes format problems like the dates. Retries can't fix information that isn't in the document, such as values in a Schedule B that was never provided, so those should be returned as null and flagged. (a) repeats a blind retry with more noise. (b) leaves out the document the model needs to correct itself. (c) can't produce information that isn't in the input.
More from this scenario
- Your contract schema has a nullable effective_date. Spot checks show it comes back null…
- A nightly job submits 10,000 scanned-and-OCR'd claim documents through the Message…
- Select two. Your extraction system scores 97% field-level accuracy on last quarter's…
- Your pipeline sends an extraction to human review when the model's self-reported…
- Commercial leases often include amendments. On one lease, the main body states base rent…