CCAR-P practice › Evaluation, Testing & Optimization · Question 71 of 110

Evaluation, Testing & Optimization

A financial-services firm's research assistant shows healthy dashboards: 99.9% uptime, low HTTP error rates, stable average latency. Yet analysts say answer quality has slipped over the past month, and nobody can say when it started or why. Per-request logs contain only timestamps, status codes and latency. What is the most appropriate monitoring improvement?

Show answer and explanation

Answer: A — Log versions, usage, stop_reason and retrieved chunk IDs per request, and grade sampled traffic online

(a) is right: infrastructure metrics can be healthy while quality degrades. Version identifiers let onset be tied to a change, retrieved chunk IDs connect answers to their context, and online evaluation on sampled traffic produces a quality trend. (b) still watches only latency. (c) gathers raw material without a quality signal and ignores the governance of financial and personal data in logs. (d) is useful but only tests the offline dataset, not drift in real traffic.