What you’ll be able to do
- Separate standing instructions from per-turn and per-tool content
- Keep the system prompt stable enough for prompt caching to actually engage
- Recognize when tool-selection logic belongs in tool descriptions, not the system prompt
What you need to know
The system prompt is identity, not inventory
A system prompt answers one question: who is this agent and what are its non-negotiable constraints? It is not the place to enumerate every tool, describe every edge case, or carry information that changes between requests.
Three kinds of content get dumped into system prompts that don't belong there:
- Tool inventory — restating what each tool does. The tool's own
descriptionfield is where a model reads that, and duplicating it doubles the token cost and creates a place for the two to drift out of sync. - Per-turn state — today's date, the current user's ID, a session counter. Anything that changes between calls invalidates the cached prefix it sits in.
- One-off instructions — a fix for last week's specific complaint, bolted onto a prompt meant to describe stable identity and scope.
Growth by accretion is the default failure mode
An agent architecture rarely starts with a bloated system prompt. It gets there one tool at a time: a new capability ships, and a paragraph explaining when to use it gets appended. After a dozen tools, the system prompt is a decision tree written in prose, and the model has to hold all of it in mind on every single turn.
refund_tool description (in the tool definition, not here):
“Issue a refund for orders under 90 days old, excluding digital goods. Requires order_id. Checks the fraud flag automatically — do not call if fraud_flag is true.”
The "after" version is shorter, and — because it no longer changes every time a tool's eligibility rule is tweaked — it stays cacheable. The eligibility logic moved to where the model actually needs it: right next to the tool it governs.
Stability is a caching requirement, not a style preference
The system prompt is typically the first block in the request and the natural place to put a cache checkpoint. If any part of it varies per user, per session, or per request, the cache read never engages for that call — and every downstream block after it inherits the miss.
Architected correctly: the system prompt is identical across every call to a given agent, and anything that varies — user identity, session data, today's date — is injected in a later message, after the cache checkpoint.
Key concept
The system prompt describes who the agent is and what it must never do — everything else belongs in a tool description, a later message, or nowhere at all.
When a scenario describes a system prompt that "keeps growing" or a cache that "used to work," the fix is almost always to move volatile or tool-specific content out of it, not to trim it for length.
Practice scenario
Work it through, then open this
The fix is architectural, not a rewrite for brevity. Audit the system prompt for tool-selection logic — anything that says “use tool X when Y” — and move it into that tool’s own description, where the model reads it exactly when it’s evaluating that tool. What’s left in the system prompt should be identity and constraints that apply regardless of which tool gets called. This also restores cacheability if any of those 40 paragraphs had drifted into carrying per-session specifics.
Build exercise — Audit a system prompt for misplaced content
Intermediate · 25 min
What you’ll learn
- Spotting tool-selection logic that belongs in a tool description instead
- Finding volatile content that breaks caching
- What should actually remain in a system prompt
-
Take your longest system prompt and highlight every sentence that starts with or implies “use tool X when…”.
- Why: That’s tool-selection logic, and it belongs on the tool, not the agent’s identity statement.
- You should see: A system prompt with most of its length attributable to a handful of tools’ eligibility rules.
-
Search the same prompt for anything that could change between two calls to the same agent — a date, a name, a counter.
- Why: Volatile content anywhere in the system prompt sits before the natural cache checkpoint and can invalidate the whole block.
- You should see: Either a clean, static prompt, or a specific line that’s been silently breaking your cache hit rate.
-
Rewrite the prompt with only identity and non-negotiable constraints, and move everything else to tool descriptions or later messages.
- Why: This is the shape that stays stable as the tool set grows and stays cacheable as usage scales.
- You should see: A shorter, stable system prompt and tool descriptions that now carry their own selection logic.
Exam traps
Putting per-request or per-user data in the system prompt
Anything that varies between calls sits before the cache checkpoint and invalidates the read on every single request.
Letting the system prompt grow by one paragraph per new tool
This is the default failure mode. Selection logic belongs in the tool’s own description, not appended to a growing prose decision tree.
Duplicating a tool’s description inside the system prompt
Doubles token cost and creates two copies of the same rule that will eventually disagree with each other.
Treating the system prompt as the place for one-off instructions
A fix for one specific complaint doesn’t belong in a prompt meant to describe stable, general identity and scope.
Assuming a longer system prompt is a more thorough one
Length buried in the middle of a long prompt is exactly where instructions get lost. Thoroughness is about correct placement, not volume.
Never revisiting the system prompt as the agent’s capability set changes
A prompt that made sense with three tools rarely still makes sense with fifteen. It needs deliberate architecture review, not just append-only edits.