Before you buy a bigger model: look at what's in the context
- On-Premise
- Production
When answer quality drops the first instinct is to scale up the model. If you are running locally that is the most expensive button you have: more VRAM, fewer concurrent requests, higher latency. And most of the time it is not the button that needs pressing.
Before asking how good the model is, look at what you are handing it.
The window is a budget, not a capacity
Thinking of the context window as capacity is misleading. Budget is closer: everything you put in takes attention away from everything else.
Looking at how the window actually breaks down on an agent request, the striking thing was not the proportions but the ordering. Tool definitions and retrieved documents take up the most room. Conversation history follows. System instructions are fixed and small.
The question you actually want answered sits at the bottom of that list, as the smallest piece.
The biggest waste is in tool definitions
Thirty tool definitions enter the context on every request and most requests use two of them. The rest take up space and hurt selection at the same time - the model reads twenty-eight options it will not use while deciding which one to pick.
Filtering tools by context was the single biggest gain we measured. When the agent is working on one incident type it only sees the tools for that type.
The same logic works on the retrieval side. Fetching three documents and verifying their relevance beats fetching eight and dumping them all in. Extra documents do not support the model; they split its attention.
History ages fastest
In agent flows, conversation history is worth less than it is in a chat app. The raw output of a tool call made five steps ago is usually dead weight by step six.
We split history in two. The state side holds which steps completed, which decisions were made, which identifiers came back - structured and small. The raw output side never sits in context at all; it gets re-fetched by identifier when needed.
Window saturation on long flows dropped noticeably after that split.
When you really do need a bigger model
If quality problems survive a fixed context, scaling up is a legitimate move. But three possibilities should be eliminated first.
Retrieval quality: is the right document arriving, or is the model fluently summarising the wrong one. Instruction ambiguity: does the model not know what to do, or can it not do it. And window saturation: is the important information getting in but landing late.
The third one is sneaky. In long contexts, material in the middle gets used less well than material at the start or end. Supplying the same information in a shorter context can beat a larger model - and costs nothing.
The quantisation decision hangs off this
The other way to reclaim VRAM locally is quantisation. On most tasks the quality loss goes unnoticed, and where it is noticed is predictable: long context, numerical reasoning, structured output.
Tightening the context reduces the first of those directly. So getting the window budget in order makes hardware decisions easier, not just quality ones.
Scaling the model locally charges you twice, once in hardware and once in concurrency. Fixing the context costs nothing and usually does more.
The order that worked for us: filter the tools, narrow retrieval, split history into state and raw output. Once those are done, the argument about model size turns into a much smaller argument.