CCAR-F scenarios › Structured Data Extraction · Question 58 of 60
Scenario 6 · Structured Data Extraction
You run a system that uses Claude to turn unstructured documents into structured records, validates every result against JSON schemas, and passes the output to downstream systems. Accuracy has to stay high across messy and unusual documents, and edge cases must be handled cleanly rather than with plausible-looking guesses.
Context Management & Reliability
Commercial leases often include amendments. On one lease, the main body states base rent as $4,200/month, and an amendment later in the same file states $4,650/month. Your extraction returns only one of the two values, apparently at random from run to run, and the billing system has charged the wrong rent several times. What is the best design change?
Show answer and explanation
Answer: B — Extract every stated rent value with its source section and date, set a conflict_detected flag, and let downstream rules or a reviewer reconcile them.
When the source is inconsistent, keep every value with its source and date and flag the conflict (a conflict_detected boolean) instead of picking one arbitrarily. Reconciliation then happens downstream with full information, and the dates make clear whether a later amendment supersedes the earlier figure. (a) and (c) are fixed positional rules that fail whenever document order doesn't match legal precedence. (d) produces a number no document actually states.
More from this scenario
- Your system extracts obligations from each section of a 60-section supplier agreement…
- During extraction, the agent calls a lookup_vendor tool to match each vendor named in a…
- Your invoice extraction tool's JSON schema marks purchase_order_number as required…
- A mixed document inbox contains invoices, receipts, and service contracts. You defined…
- Your extraction pipeline validates each result with Pydantic. When validation fails, the…