CCAR-F scenarios › Structured Data Extraction · Question 59 of 60
Scenario 6 · Structured Data Extraction
You run a system that uses Claude to turn unstructured documents into structured records, validates every result against JSON schemas, and passes the output to downstream systems. Accuracy has to stay high across messy and unusual documents, and edge cases must be handled cleanly rather than with plausible-looking guesses.
Context Management & Reliability
Your system extracts obligations from each section of a 60-section supplier agreement, then concatenates all the per-section outputs into one input for a final consolidation pass that produces the downstream JSON. Obligations from the first and last sections always appear in the final output, but ones from sections 20–45 are often missing. What change best addresses this?
Show answer and explanation
Answer: A — Put a summary of key findings at the start of the consolidated input, and organize the detailed per-section results under explicit section headers.
This is the "lost in the middle" effect: models handle the beginning and end of long inputs reliably and may drop findings in the middle. Section 6's remedy is to put key findings summaries at the beginning and organize details under explicit section headers. (b) adds room in the window but doesn't change how attention is spread across the input. (c) doubles the cost to work around position effects instead of fixing the input's structure. (d) assumes the output is being cut off, but the middle items are being dropped, not truncated.
More from this scenario
- During extraction, the agent calls a lookup_vendor tool to match each vendor named in a…
- Your invoice extraction tool's JSON schema marks purchase_order_number as required…
- A mixed document inbox contains invoices, receipts, and service contracts. You defined…
- Your extraction pipeline validates each result with Pydantic. When validation fails, the…
- Your contract schema has a nullable effective_date. Spot checks show it comes back null…