CCAR-P practice › Evaluation, Testing & Optimization · Question 66 of 110

Evaluation, Testing & Optimization

A procurement agent's responses intermittently end mid-sentence when it summarizes long supplier contracts, and the downstream parser then rejects the output. Shorter contracts work fine. The team suspects the model can't handle long documents and plans to switch to a different model. What should they check first?

Show answer and explanation

Answer: B — The response's stop_reason, since "max_tokens" there means truncation at the output limit

(b) is right: output ending mid-sentence on long inputs is the signature of hitting the output token limit, and stop_reason reports it directly. The fix is a higher limit or a more compact required output, not a different model. (a) would cause wrong or incomplete content, not mid-sentence cut-offs. (c) hides the failure. (d) measures quality but doesn't diagnose why the output is cut off.