CCAR-F scenarios › Structured Data Extraction · Question 53 of 60

Scenario 6 · Structured Data Extraction

You run a system that uses Claude to turn unstructured documents into structured records, validates every result against JSON schemas, and passes the output to downstream systems. Accuracy has to stay high across messy and unusual documents, and edge cases must be handled cleanly rather than with plausible-looking guesses.

Prompt Engineering & Structured Output

Your extraction pipeline validates each result with Pydantic. When validation fails, the retry re-sends the same prompt, and only about a third of retries succeed. Failure analysis shows two groups: dates written as "03/04/25" when the schema expects ISO 8601, and contract_value missing on contracts that only say "fees as set out in Schedule B," which is a separate document not included in the request. What is the best redesign?

Show answer and explanation

Answer: D — Retry date failures with the document, failed output, and specific errors; for Schedule B values, return null and flag them rather than retrying.

Retrying with error feedback works when the follow-up request includes the original document, the failed extraction, and the specific validation errors, and it fixes format problems like the dates. Retries can't fix information that isn't in the document, such as values in a Schedule B that was never provided, so those should be returned as null and flagged. (a) repeats a blind retry with more noise. (b) leaves out the document the model needs to correct itself. (c) can't produce information that isn't in the input.