What you’ll be able to do
- Apply the four cost levers in the order the exam expects
- Measure token usage before optimising anything
- Place cache checkpoints inside long, partly-stable prompts
- Pair every cost change with an evaluation check
What you need to know
Measure first — the step people skip
Two named practices come before any optimisation: token usage tracking and cost modeling. Track what you're actually spending per request and where; project the bill at target volume before you scale into it.
The tool for this is the count_tokens endpoint: send the same model, system prompt, messages, and tools you intend to use, and it returns input_tokens without generating a response. It's free to call, which means there is no excuse for modelling costs from guesswork.
Optimising blind is how teams shave 8% off a component that was 3% of spend while the real cost sat untouched in an uncached system prompt. The measurement tells you which lever to pull.
The four levers, in order
- Cache the stable prefix. Quality-neutral, often the largest single win, and the most commonly missed. Cached reads bill at 0.10× the base input rate — a 90% discount — against a 1.25× write premium on the 5-minute TTL.
- Trim bloated context. Prune tool output and compact history. Still quality-neutral, sometimes quality-positive.
- Right-size the tier. Now you're trading, so this needs an eval check.
- Batch anything latency-tolerant. A flat 50% off both input and output tokens, with a 24-hour ceiling. Costs nothing when nobody is waiting.
Order is the tested part. Jumping to step three skips two free wins and buys a quality risk you didn't need to take.
Cache checkpointing inside long prompts
A prompt isn't always cleanly split into "all stable, then all volatile." A long system prompt may have several stable sections with something changeable in between. Cache checkpointing is placing reuse boundaries inside the prompt so the stable sections are still reused.
The placement rule is unchanged and unforgiving: everything before a checkpoint must be genuinely stable. One volatile value ahead of it invalidates from that point on, silently, with no error raised.
Verify with usage.cache_read_input_tokens on a repeat request rather than trusting that setting the field was enough.
Note what a wasted checkpoint actually costs. A cache that is written and never read doesn't cost you nothing — it costs you 1.25× the base input rate for the write, every call. Misplaced caching is worse than no caching.
Every cost change needs an eval check
The levers that trade quality for cost — tier changes especially — need a measured before-and-after, not a vibe. Without an evaluation set you have no way to know whether a cheaper configuration degraded output until users notice.
Two mistakes, compounding. The checkpoint never engaged, so the cost saving came almost entirely from the tier downgrade — the one lever that trades quality — and with no eval gate, nobody knew until users did.
Key concept
Measure, then pull levers in order: cache, trim, right-size, batch. The first two are quality-neutral; the third trades and needs an eval check. Verify caching engaged rather than assuming it did.
Practice scenario
Work it through, then open this
Wrong lever first. Cache the stable prefix and trim bloated context before touching tier — both are quality-neutral, and caching alone often closes the gap. Tier is the third step and the first that trades quality, so it doesn’t move without an eval comparison.
Build exercise — Re-price a pipeline in the right order
Intermediate · 35 min
What you’ll learn
- Measuring before optimising with count_tokens
- Pulling the four cost levers in sequence
-
Use the count_tokens endpoint on a representative request to get input_tokens without generating anything, then project the monthly bill at your target volume.
- Why: It’s free to call, which means guessing at cost is a choice. Measurement tells you which lever matters.
- You should see: A number that identifies which component actually dominates spend — often not the one you assumed.
-
Apply the levers in order and record the saving after each: cache the stable prefix, trim bloated context, right-size the tier, batch anything latency-tolerant.
- Why: Order is the tested part. The first two are quality-neutral; jumping to tier skips free wins and takes on risk you didn’t need.
- You should see: A meaningful drop before you’ve touched a single model tier.
-
If anything is going through Batch with caching on, check the cache TTL.
- Why: The default 5-minute TTL expires partway through anything but a tiny batch, so the prefix stops paying mid-run.
- You should see: A 1-hour TTL, or a cache quietly dying in the middle of your batch.
Exam traps
Downgrading the tier before caching and trimming
It skips two quality-neutral wins and takes on quality risk unnecessarily.
Assuming a cache checkpoint engaged because the field was set
Placement decides it. Check cache_read_input_tokens on a repeat request for evidence.
Optimising before measuring
Without token tracking you cannot tell which component is actually driving spend.
Changing tier without an evaluation gate
The cost drop is immediate and visible; the quality drop is delayed and arrives as user complaints.