CCAR-F scenarios › Structured Data Extraction · Question 52 of 60
Scenario 6 · Structured Data Extraction
You run a system that uses Claude to turn unstructured documents into structured records, validates every result against JSON schemas, and passes the output to downstream systems. Accuracy has to stay high across messy and unusual documents, and edge cases must be handled cleanly rather than with plausible-looking guesses.
Prompt Engineering & Structured Output
A mixed document inbox contains invoices, receipts, and service contracts. You defined three extraction tools (extract_invoice, extract_receipt, extract_contract) and call the API with tool_choice set to "auto". About 4% of responses contain only prose, such as "This appears to be a service contract with…", and no tool call, so they break ingestion. What is the most effective fix?
Show answer and explanation
Answer: A — Set tool_choice to "any" so Claude must call one of the extraction tools while still choosing which tool fits the document type.
With "auto", the model is allowed to answer in text. "any" guarantees a tool call and still lets the model choose which tool fits, which is the case section 6 describes for multiple schemas and an unknown document type. (b) guarantees a tool call but forces the invoice schema onto receipts and contracts. (c) parses natural language to drive control flow and adds a second request. (d) is a prompt-only fix for something the API can enforce.
More from this scenario
- Your extraction pipeline validates each result with Pydantic. When validation fails, the…
- Your contract schema has a nullable effective_date. Spot checks show it comes back null…
- A nightly job submits 10,000 scanned-and-OCR'd claim documents through the Message…
- Select two. Your extraction system scores 97% field-level accuracy on last quarter's…
- Your pipeline sends an extraction to human review when the model's self-reported…