Validation, retry, and feedback loops

What you’ll be able to do

Build a retry request that carries the original document, the failed extraction, and the specific validation errors
Predict when a retry will succeed (format or structural errors) and when it can’t (information absent from the source)
Distinguish semantic validation errors from the syntax errors tool use already eliminates
Add calculated_total, conflict_detected, and detected_pattern fields so errors and false positives become measurable

What you need to know

Two kinds of errors, two different defenses

Tool use with a strict JSON schema (4.3) eliminates schema syntax errors: malformed JSON, wrong types, missing required keys. It doesn't touch semantic validation errors, where the output is well-formed but wrong:

Line items that don't sum to the stated total
A value in the wrong field (the vendor's address recorded as the billing address)
A date that parses but falls after the document's own signature date

Semantic errors need validation code after extraction, such as a Pydantic model with validators or explicit business checks. When that validation fails, you have a choice: retry, or route the document somewhere else.

Retry with error feedback

Re-sending the same prompt after a failure mostly reproduces the same failure. What makes a retry work is specific feedback: a follow-up request containing the original document, the extraction that failed, and the exact validation errors.

before — blind retry

for attempt in range(3): result = extract(document) # same prompt each time if validate(result).ok: break

after — retry carries the document, the failed output, and the errors

result = extract(document) errors = validate(result) if errors: result = extract_with_feedback( document=document, # the source, again failed_extraction=result, # what the model produced errors=[ "line_items sum to 1,240.00 but total is 1,420.00", "invoice_date '14/03/2026' does not match YYYY-MM-DD", ])

With all three, the model can see what it got wrong and where to look in the source, which makes the retry a self-correction step instead of a fresh draw.

Know when a retry can’t succeed

Retries fix problems the model can fix by reading the document again: format mismatches (a date in the wrong convention), structural output errors (a value in the wrong field, a list flattened into a string), or a misread number. They can't fix absent information. If the PO number exists only in a separate purchase-order document that wasn't provided, no amount of retrying will produce it. The model can only return null or invent a value. Repeated retries in that situation waste calls and, worse, increase pressure on the model to make something up.

So classify validation failures before retrying. For format and structural errors, retry with feedback. For "the information isn't in this document," don't retry: record the field as null, flag the gap, and route the case for human review or to a step that fetches the missing source.

Make the output check itself

Some semantic checks work best when the model produces the values to compare in the same extraction:

calculated_total alongside stated_total. Have the model sum the line items itself and also record the total printed on the document. A mismatch flags either an extraction error or a genuine discrepancy in the source, and either way a person should look.
conflict_detected boolean. When the source contradicts itself (two different due dates, a quantity that differs between table and text), the model sets the flag rather than quietly picking one value.

self-checking fields in the extraction schema

"stated_total": {"type": "number"}, "calculated_total": {"type": "number", "description": "Sum of line_items[].amount, computed by you"}, "conflict_detected": {"type": "boolean", "description": "True if the document gives inconsistent values for any field"}

Feedback loops for findings: detected_pattern

In a review pipeline (Scenario 5), developers dismiss some findings. A dismissal count on its own tells you the false positive rate but not what to change. Add a detected_pattern field to each structured finding, naming the code construct that triggered it ("broad except with re-raise," "string-built SQL," "mutable default argument"). Now dismissals can be grouped by pattern, and a pattern that is dismissed 90% of the time points at the exact criterion or example to fix (4.1, 4.2). Without the field, you only know that something is noisy.

Key concept

Retry with the original document, the failed extraction, and the specific validation errors, and only for errors the model can fix by rereading: format and structural problems. When information isn’t in the source, retries can’t help. Add fields like calculated_total, conflict_detected, and detected_pattern so semantic errors and false positives become visible and measurable.

When a scenario describes a retry loop that keeps failing on the same field, ask first whether the information is in the document at all.

Practice scenario

ScenarioAn expense-report extractor retries up to five times when validation fails. Logs show two dominant failures. One is receipt_date in DD/MM/YYYY instead of YYYY-MM-DD; these usually pass on the second try. The other is cost_center missing; these fail all five attempts, and occasionally a later attempt returns a cost center that isn't in any company list. The cost center is assigned in the HR system, not printed on the receipts.

Work it through, then open this

The date failure is a format mismatch, which is exactly what retry with error feedback fixes. Make sure each retry includes the document, the failed output, and the specific error. The cost center failure is absent information: it isn’t on the receipt, so retries can’t recover it, and the invented values on later attempts show the pressure to fabricate. Stop retrying that failure. Make cost_center nullable in the extraction, and fill it from the HR system or route the report for review.

Build exercise — Build a validation-retry loop that knows its limits

Intermediate · 35 min

What you’ll learn

Writing semantic validators for tool-use output
Sending retries with document, failed output, and errors
Separating retryable from non-retryable failures

Add semantic validation after an extraction step: line items sum to the total, dates within plausible ranges, values in the right fields. Add calculated_total and conflict_detected to the schema.
- Why: Tool use already removed syntax errors; the remaining failures are semantic and need explicit checks.
- You should see: Some schema-valid extractions failing validation, and a few documents where calculated_total and stated_total disagree.
On failure, send a follow-up request containing the original document, the failed extraction, and the list of specific errors. Compare success rates against a blind retry.
- Why: Specific feedback turns a retry into self-correction.
- You should see: Format and structural errors resolved on the first feedback retry far more often than with a blind retry.
Tag each failure as format/structural or information-absent, and stop retrying the second kind: return null, flag the gap, and route it.
- Why: Retries can’t produce information that isn’t in the source, and they encourage fabrication.
- You should see: Fewer wasted calls, no invented values for absent fields, and a clear list of documents that need another source.

Exam traps

Retrying with the identical prompt and hoping for a different result

The same input mostly reproduces the same error. Include the specific validation errors so the model knows what to correct.

Retrying with only the error message, without the original document and failed output

Self-correction needs all three: the source to reread, the output that failed, and what was wrong with it.

Retrying repeatedly when the required information isn’t in the provided document

Retries fix format and structural errors, not missing information. If the data is in an external document that wasn’t provided, retries fail or fabricate.

Assuming tool use makes output correct, so no semantic validation is needed

Tool use eliminates syntax errors only. Totals that don’t sum and values in the wrong field still get through.

Extracting only the stated total, with no independent calculation to compare it to

Extracting calculated_total alongside stated_total turns a silent discrepancy into a flag.

Logging dismissed findings without recording which code pattern triggered them

A detected_pattern field lets you group dismissals and find the specific pattern behind the false positives.

Sources

Claude Developer Platform Docs
Anthropic, Claude Certified Architect – Foundations Exam Guide v1.0 (July 2026)