CCAR-P practice › Claude Models, Prompting & Context Engineering · Question 20 of 110
Claude Models, Prompting & Context Engineering
A personal-finance app categorizes every card transaction with Claude, millions of events a month, and the result must appear in the app within a tight latency budget. All traffic runs on the largest model tier. On a labeled set of 2,000 real transactions, both the mid and the smallest tier meet the agreed accuracy bar, with no failure cluster on any merchant type. What is the most appropriate change?
Show answer and explanation
Answer: B — Move the step to the smallest tier that passed, pin its version, and keep sampling production accuracy against the same bar
(b) is right: the step is high volume, latency sensitive and well specified, and the smallest tier passed a task-specific eval with no failure cluster, so it is the right tier; pinning and ongoing sampling guard against drift. (a) pays the largest-tier price and latency on a step that evidence shows doesn't need it. (c) overrides the eval with a guess; if unseen inputs worry you, extend the eval set. (d) adds latency and output tokens to a classification task that already passes without reasoning.
Keep going
More Claude Models, Prompting & Context Engineering questions
- A contract-analysis assistant on the mid tier keeps giving wrong termination dates…
- You are designing a cascade for an internal IT help assistant: the smallest tier answers…
- A document-extraction service calls Claude through a model alias that always resolves to…
- An employee self-service assistant answers pay questions. Its HR database tool runs…
- A SaaS vendor runs a Claude assistant for 30 enterprise tenants. Each tenant has a…