What you’ll be able to do
- Distinguish context bloat from context drift by symptom
- Match pruning, compaction, and isolation to the right failure
- Reject remedies that address neither (bigger window, lower temperature)
- Recognise when a long session needs restructuring rather than tuning
What you need to know
Two different diseases with one vague symptom
"The agent gets worse over long conversations" describes two unrelated failures. Telling them apart is the tested skill, because the remedies don't overlap at all.
- Bloat — accumulated history and tool output pile up. The relevant content is still in there, but it's competing with noise, and you're paying for all of it. Symptoms: rising cost, slower responses, details getting missed that are demonstrably present in the context.
- Drift — the model loses its grip on instructions and facts established early. Symptoms: cost and speed are fine; it has simply stopped following the format you set forty turns ago.
Same complaint from the user, completely different fix. Diagnose before you reach.
What bloat looks like
Solid blocks are content the model still needs. Faded blocks are accumulated noise it doesn't.
The three remedies, and when each applies
- Tool output pruning — a search returned 40 results and the agent used 3. Drop the other 37 before they ever enter history. Cheapest of the three; try it first.
- Compaction — summarize older turns into a compact form that preserves decisions and key facts, then release the raw transcript. Use when the history itself is the bulk, not the tool output.
- Context isolation — run independent subtasks in separate subagent contexts so their intermediate noise never touches the main thread. Use when one messy subtask would otherwise pollute everything else.
Notice these are ordered by cost and disruption. Pruning changes nothing about your architecture. Isolation changes quite a lot. Reach in that order unless the scenario tells you otherwise.
The remedies that fix neither
Two answers appear constantly as distractors, and both sound reasonable:
A bigger context window gives the model more room to hold tokens. It says nothing about which of those tokens the model still actively weights. It doesn't fix drift, and for bloat it makes the bill worse while postponing the real fix.
Lowering temperature reduces sampling variation. Instruction memory isn't a sampling property. A model that has drifted from your format at turn 50 will drift just as reliably at temperature 0.
A worked case
That last sentence is the diagnostic. Unchanged cost rules out bloat — this is drift. The fix is reinforcing the instruction or compacting so it stays live in the model's active attention. Raising max_tokens, enlarging the window, and lowering temperature all address something that isn't happening here.
Key concept
Bloat is wasted tokens; drift is lost instructions. Cost tells you which one you have. Pruning and compaction fix bloat, reinforcement and isolation fix drift, and a bigger window fixes neither.
When a question describes degradation over a long session, look for whether cost moved. That single detail usually decides the answer.
Practice scenario
Work it through, then open this
That last sentence is the diagnostic. Unchanged cost rules out bloat, so this is drift: the instruction is still present but no longer actively weighted. Reinforce or compact so it stays live. A bigger context window and a lower temperature both address something that isn’t happening.
Build exercise — Find your own drift
20 min
-
Pull a real conversation that ran past 20 turns. Check whether an instruction from turn one is still being followed by turn twenty.
- Why: Drift is invisible until you deliberately go looking for it.
- You should see: At least one instruction that quietly stopped applying somewhere in the middle.
-
Rewrite one vague instruction from that transcript into something specific, positively framed, with an explicit format constraint.
- Why: This is the exact before/after pattern the exam rewards.
- You should see: A rewritten instruction the model could actually be graded against.
-
Add one schema-validation check around wherever you parse model output today.
- Why: Defensive parsing is a production habit, not a nice-to-have — models occasionally wrap JSON in prose.
- You should see: Code that handles a malformed response instead of crashing on it.
Exam traps
Fixing drift by increasing context window size
More room to hold tokens says nothing about which tokens the model still weights. Drift is an attention problem, not a capacity one.
Fixing drift by lowering temperature
Instruction memory is not a sampling property. A drifted model drifts identically at temperature 0.
Reaching for compaction when the tool output is the bulk
Pruning the tool output at source is cheaper and less lossy. Compaction is for when conversation history itself is the weight.
Treating any long-session degradation as a prompting problem
If cost rose alongside the degradation, it is bloat and belongs in context management, not in rewording the prompt.