CCAR-P practice › Evaluation, Testing & Optimization · Question 71 of 110
Evaluation, Testing & Optimization
A financial-services firm's research assistant shows healthy dashboards: 99.9% uptime, low HTTP error rates, stable average latency. Yet analysts say answer quality has slipped over the past month, and nobody can say when it started or why. Per-request logs contain only timestamps, status codes and latency. What is the most appropriate monitoring improvement?
Show answer and explanation
Answer: A — Log versions, usage, stop_reason and retrieved chunk IDs per request, and grade sampled traffic online
(a) is right: infrastructure metrics can be healthy while quality degrades. Version identifiers let onset be tied to a change, retrieved chunk IDs connect answers to their context, and online evaluation on sampled traffic produces a quality trend. (b) still watches only latency. (c) gathers raw material without a quality signal and ignores the governance of financial and personal data in logs. (d) is useful but only tests the offline dataset, not drift in real traffic.
Keep going
More Evaluation, Testing & Optimization questions
- A SaaS company's monitoring shows that, after a release, cost per conversation rose 30%…
- A bank is preparing to launch a Claude-based assistant that answers customer questions…
- An engineering lead reports that a new configuration of a document-processing agent…
- A healthcare provider's patient-intake assistant passed launch review with a zero rate…
- A team building a contract-review assistant has an evaluation set of 200 questions…