Ollama or vLLM: we're asking the wrong question
- On-Premise
- Production
In a company whose data cannot leave the building, running inference locally is not a preference but a constraint. The real question is which inference layer.
That choice usually gets argued along the wrong axis - "which one is faster". The right axis is the concurrency profile.
They are not doing the same job
Ollama optimises for one user and one request. Pulling a model, running it and hitting an endpoint takes minutes. During development that is invaluable.
vLLM is a serving engine. PagedAttention splits the KV cache into pages to cut memory fragmentation; continuous batching keeps blending incoming requests onto the same GPU. On a single request you will not feel the difference. At twenty concurrent requests it becomes categorical.
| Ollama | vLLM | |
|---|---|---|
| Setup | Minutes | Hours |
| Single-request latency | Good | Good |
| Concurrent throughput | Limited | High |
| GPU memory efficiency | Standard | PagedAttention |
| Best fit | Development, low traffic | Production, many users |
The decision happens at concurrency
An agent system usually makes several model calls for a single user request - planning, tool selection, output validation, summarisation. So the system's concurrency grows as a multiple of the user count.
Capacity should be planned by model calls per second, not by number of users. A setup that looks perfectly fine on one request saturates far earlier than expected once an agent chain is running.
The split we landed on: Ollama for development and prototyping, where the speed that matters is the developer's rather than the model's. vLLM in production, because without batching the GPU sits largely idle.
Model size comes second
Reaching for a bigger model when quality drops is a common reflex. On a local deployment it is expensive: a larger model wants more VRAM and lowers concurrent capacity at the same time.
In most cases what determines quality is not model size but the quality of the context. If retrieval is poor, a bigger model simply misreads the wrong context more fluently.
The order that works: fix retrieval, narrow the prompt, and only then scale the model.
Where quantisation stops paying
Quantisation cuts VRAM substantially and on most tasks the quality loss goes unnoticed. Where it is noticed is predictable: long context, numerical reasoning, structured output generation.
All three are critical in agent systems, so before moving to a quantised model you need to measure the schema validation failure rate. If that rate climbs afterwards, the memory you gained does not cover the reliability you lost.
In local inference the choice comes before the "which model" question. An inference layer picked without establishing the concurrency profile turns, in production, into either an idle GPU or a queue of waiting requests.
Ollama in development, vLLM in production. And when quality suffers, look at the context before the model size.