CCAR-P practice › Evaluation, Testing & Optimization · Question 72 of 110
Evaluation, Testing & Optimization
A SaaS company's monitoring shows that, after a release, cost per conversation rose 30% while traffic, model and output length were unchanged. The usage logs show input tokens flat but cache-read tokens near zero, where they had previously covered most of the input. What is the most likely cause to investigate first?
Show answer and explanation
Answer: C — The release put per-request content, such as a timestamp or request ID, into the previously stable prefix
(c) is right: cache reads dropping to near zero right after a release, with input volume flat, indicates something now changes the prefix on each request, so it can no longer be reused. Typical culprits are a timestamp, a request ID or non-deterministic tool ordering near the top of the prompt. (a) wouldn't show as a change in cache-read tokens. (b) and (d) would raise input tokens, which are flat.
Keep going
More Evaluation, Testing & Optimization questions
- A bank is preparing to launch a Claude-based assistant that answers customer questions…
- An engineering lead reports that a new configuration of a document-processing agent…
- A healthcare provider's patient-intake assistant passed launch review with a zero rate…
- A team building a contract-review assistant has an evaluation set of 200 questions…
- A customer-support assistant must (1) return a JSON object matching a fixed schema, (2)…