CCAR-P practice › Evaluation, Testing & Optimization · Question 61 of 110

Evaluation, Testing & Optimization

An agent that files insurance claims passes a test case on its first run, and the team marks the case green. In production, the same kind of claim fails roughly one time in five because the agent sometimes skips a required attachment step. Which change to the test framework would most directly have caught this before release?

Show answer and explanation

Answer: C — Run key cases several times, gate on a pass rate, and assert on the final filed state

(c) is right: model behaviour varies between runs, so a single pass doesn't establish reliability; repeated trials with a pass-rate threshold expose an intermittent failure, and checking the final filed state catches the missing attachment. (a) changes the system under test rather than measuring it as deployed. (b) adds coverage but still treats one pass per case as proof. (d) tests tools in isolation and would miss an agent that skips a step.