CCAR-P practice › Evaluation, Testing & Optimization · Question 56 of 110

Evaluation, Testing & Optimization

An engineering lead reports that a new configuration of a document-processing agent reduced cost per API call by 35%. Finance is ready to approve rollout. You notice the new configuration also raised the rate at which documents fail validation and get re-sent to the agent, and the rate escalated to human processors. Which metric should decide the rollout?

Show answer and explanation

Answer: C — Cost per successfully processed document, including retries and the cost of human escalations

(c) is right: the business pays per processed document, so cost must be measured at that unit, including retries and the human escalation cost. A cheaper call that fails more often can be more expensive per outcome. (a) optimizes a proxy. (b) mixes in a different dimension and averages hide the slow tail. (d) treats one guardrail as the whole decision; a small failure increase could be acceptable if cost per outcome and quality still clear their thresholds.