CCAR-P practice › Evaluation, Testing & Optimization · Question 56 of 110
Evaluation, Testing & Optimization
An engineering lead reports that a new configuration of a document-processing agent reduced cost per API call by 35%. Finance is ready to approve rollout. You notice the new configuration also raised the rate at which documents fail validation and get re-sent to the agent, and the rate escalated to human processors. Which metric should decide the rollout?
Show answer and explanation
Answer: C — Cost per successfully processed document, including retries and the cost of human escalations
(c) is right: the business pays per processed document, so cost must be measured at that unit, including retries and the human escalation cost. A cheaper call that fails more often can be more expensive per outcome. (a) optimizes a proxy. (b) mixes in a different dimension and averages hide the slow tail. (d) treats one guardrail as the whole decision; a small failure increase could be acceptable if cost per outcome and quality still clear their thresholds.
Keep going
More Evaluation, Testing & Optimization questions
- A healthcare provider's patient-intake assistant passed launch review with a zero rate…
- A team building a contract-review assistant has an evaluation set of 200 questions…
- A customer-support assistant must (1) return a JSON object matching a fixed schema, (2)…
- An LLM judge is used to compare two versions of a research summarizer pairwise. The new…
- An agent that files insurance claims passes a test case on its first run, and the team…