CCAR-P practice › Governance, Safety & Risk Management · Question 76 of 110

Governance, Safety & Risk Management

An insurer's policy Q&A assistant has run well for months. The week after the content team migrated the knowledge base to a new document management system, customer complaints rise about answers that cite deductible amounts confidently but wrongly. Model version, prompts and latency are unchanged. What is the most appropriate first step?

Show answer and explanation

Answer: D — Inspect the retrieved chunks for failing queries for parsing, chunking or index-freshness breaks from the migration

The symptom lines up with a change in the retrieval layer, and the model and prompt didn't change. Inspecting the context the model actually received shows whether tables or numbers were mangled in migration. (a), (b) and (c) treat it as a model problem. None of them would fix wrong or stale content being fed into the context, and (c) adds cost without addressing the cause.