What you’ll be able to do
- Treat the context window as a budget to allocate, not a wall to hit
- Reserve headroom for tool output and generation before a session starts
- Design a pruning strategy that triggers before the window fills, not after
- Identify which content classes are safe to prune and which aren’t
What you need to know
A window is a budget, and budgets get allocated in advance
A context window — commonly 200K tokens, with a long-context beta stretching some models to 1M — looks like a single large number, so teams treat it like a cap they'll deal with when they hit it. An architect treats it like a budget, allocated before the first request goes out, the same way you'd size memory for a service before deploying it, not after it OOMs.
A rough allocation for a tool-using agent session might look like: system prompt and tool definitions — stable, small, cached; conversation history — grows every turn, the main pressure source; tool output — the least predictable and most dangerous, since one verbose tool call can eat thousands of tokens in a single turn; reserved headroom — space deliberately left empty for the model's own response, because a request that fills the window leaves no room to generate an answer.
Skipping that last category is the single most common budgeting mistake: a request sized right up to the window ceiling succeeds at being sent, then fails or truncates because there was never room for output.
Not all content is equally prunable
Reaching for "just summarize everything" treats every token the same, and it isn't. A useful split:
- Load-bearing state — decisions already made, constraints already agreed to, IDs and identifiers something downstream depends on. Pruning this silently changes the agent's behavior, not just its verbosity.
- Working detail — the reasoning and intermediate tool output that got you to a decision. Usually safe to compress once the decision is captured, because the decision itself is what matters going forward.
- Pure verbosity — restated context, redundant tool calls, retries that succeeded on a later attempt. Safe to drop outright.
A pruning strategy that can't tell these apart tends to either keep everything (and still blow the budget) or drop everything indiscriminately (and lose a constraint the agent agreed to three turns ago, which then resurfaces as a correctness bug, not a context error).
Trigger pruning on a threshold, not on failure
The "before" version works, technically — until the exception path itself becomes a production incident: a user-facing request that fails once, retries, and now costs twice. The "after" version treats the threshold as a design parameter, checked before the request goes out, not discovered by catching an error after it's already too late to avoid the extra round trip.
Key concept
A context window is a budget you allocate in advance, not a wall you discover by hitting it. Reserve headroom for the response, and prune on a threshold before the ceiling, not on a failure after it.
When a scenario describes a long-running agent that degrades unpredictably — sometimes truncated, sometimes just slow, sometimes seemingly "forgetting" a constraint — the fix is almost never a bigger window. It's a budget that was never actually allocated.
Practice scenario
Work it through, then open this
Two separate symptoms, one root cause. The truncation is a missing reserved-output allocation — as history grows, nothing was held back for the response, so the model runs out of room to finish. The “forgotten” constraint is a pruning strategy (if one exists at all) that isn’t distinguishing load-bearing state from working detail — the early constraint got compressed away along with everything else instead of being carried forward explicitly.
Build exercise — Size a context budget before you need one
Intermediate · 25 min
What you’ll learn
- Allocating a context budget across stable content, history, tool output, and reserved output
- Setting a pruning threshold instead of a pruning exception handler
- Telling load-bearing state apart from prunable detail
-
For a real or planned agent session, list what occupies context: system prompt, tool definitions, conversation history, typical tool output size, and the response itself. Assign each a rough token budget.
- Why: You can’t design a threshold for a budget you haven’t actually allocated.
- You should see: A number for reserved output that isn’t zero — most first attempts at this exercise leave nothing held back for the response.
-
Identify three pieces of state in a typical session that are load-bearing (a decision, a constraint, an ID) versus three that are pure working detail.
- Why: A pruning strategy that can’t make this distinction will either keep everything or drop something it shouldn’t.
- You should see: A short list you could hand to a summarization step as “always keep” versus “safe to compress.”
-
Set a token threshold below the actual window ceiling that triggers pruning, and check it fires before a request would fail, not after.
- Why: Reactive pruning (triggered by a caught exception) always costs an extra round trip; proactive pruning doesn’t.
- You should see: A pruning check that runs before the request is sent, using an estimate of current token usage.
Exam traps
Sizing a system prompt and tool set against the full context window with no headroom
Leaves no room for the response. The request itself succeeds at being sent; the model has nowhere to generate an answer.
Pruning only when a call already failed
Reactive pruning costs an extra round trip every time it fires and turns a design decision into an exception handler.
Treating all history as equally prunable
Compressing load-bearing state (a decision, a constraint) along with pure verbosity produces a correctness bug that looks like the agent “forgot” something.
Assuming a bigger context window removes the need for a budget
A larger window delays the problem; it doesn’t remove the need to allocate reserved output and a pruning threshold.
Summarizing tool output only after it has already blown the budget once
The same reactive-versus-proactive mistake applied specifically to the most unpredictable content class in the budget.
Forgetting that the response itself consumes budget the request has to leave room for
The most common single cause of “successfully sent, then truncated” failures.