CCAR-P practice › Evaluation, Testing & Optimization · Question 61 of 110
Evaluation, Testing & Optimization
An agent that files insurance claims passes a test case on its first run, and the team marks the case green. In production, the same kind of claim fails roughly one time in five because the agent sometimes skips a required attachment step. Which change to the test framework would most directly have caught this before release?
Show answer and explanation
Answer: C — Run key cases several times, gate on a pass rate, and assert on the final filed state
(c) is right: model behaviour varies between runs, so a single pass doesn't establish reliability; repeated trials with a pass-rate threshold expose an intermittent failure, and checking the final filed state catches the missing attachment. (a) changes the system under test rather than measuring it as deployed. (b) adds coverage but still treats one pass per case as proof. (d) tests tools in isolation and would miss an agent that skips a step.
Keep going
More Evaluation, Testing & Optimization questions
- A travel company's booking assistant is a multi-turn chat. The team wants to A/B test a…
- After an online test, a new retrieval configuration for an internal IT helpdesk…
- A team is comparing two prompt variants for a claims-summary feature over a four-week…
- A pharmaceutical company's RAG assistant answers questions about drug-interaction…
- A procurement agent's responses intermittently end mid-sentence when it summarizes long…