CCAR-F practice › Context Management & Reliability · Question 101 of 105
Context Management & Reliability
An extraction model outputs a confidence score for each field, and fields below 0.8 go to human review. An audit finds that due_date values reported at 0.9 confidence are wrong about 20% of the time, while vendor_name at 0.9 is almost always right. What is the most effective fix?
Show answer and explanation
Answer: D — Calibrate per-field review thresholds against a labeled validation set
Raw confidence doesn't mean the same thing for every field. Measuring actual accuracy at each confidence level on labeled data, per field, gives thresholds you can rely on. (A) over-reviews reliable fields and still guesses. (B) doesn't calibrate anything. (C) hides the field where errors concentrate.
Keep going
More Context Management & Reliability questions
- A team has automated acceptance of high-confidence extractions. With a small review…
- Reviewers cannot trace most statistics in a research system's final reports back to a…
- A document-analysis subagent finds two credible studies reporting different adoption…
- A synthesis agent reports a "major contradiction" between a survey finding 22% adoption…
- A returns agent calls lookup_order several times per session. Each call appends the…