vLLM vs Ollama vs SGLang: High-Throughput Local Serving Benchmarks

By AI Agent Hub Editorial Desk · Review method · Corrections

Deploying an open-weights model requires choosing an inference runtime that fits the workload and operating environment. This guide provides a reproducible comparison plan for vLLM, Ollama, and SGLang without inventing universal benchmark results.

1. Serving Architectures Explained

• Ollama (Local Desktop & CLI): Built on top of llama.cpp. Offers one-command installations (`ollama run deepseek-r1`), GGUF quantization management, and desktop CPU/GPU auto-offloading. Best for local development.
• vLLM (Enterprise API Server): Pioneer of PagedAttention technology. Manages KV-cache memory dynamically with near-zero waste. Supports continuous batching and multi-GPU tensor parallelism. Best for production server workloads.
• SGLang (RadixTree Cache & Structured Inference): Developed by LMSYS. Features RadixAttention for fast automatic prefix caching across complex multi-step agent prompts.

2. Comparison by Deployment Shape

3. Do Not Compare Vendor Benchmark Numbers Directly

Each project publishes results for different models, hardware, request distributions, versions, and tuning. A fair comparison pins the same model artifact and precision when supported, uses the same prompt and output length distributions, and applies the same quality checks. Report the full configuration beside every number.

4. Benchmark Protocol

1. Pin runtime container or binary, model revision, tokenizer, chat template, precision, and hardware.
2. Use a recorded workload distribution rather than one short synthetic prompt.
3. Measure time to first token, inter-token latency, end-to-end latency, request and token throughput, queue time, memory peak, and errors.
4. Test concurrency from idle to overload and show the service objective on the same graph.
5. Run a fixed quality set after any quantization or template change.
6. Include cold start, model loading, cancellation, long context, and failure recovery.

Prefix caching should be tested with realistic prefix reuse. Continuous batching should be tested with the arrival pattern expected in production. A runtime that wins at maximum throughput may lose for a single interactive user, and the easiest local tool may not provide the isolation or capacity controls required for a multi-tenant service.

5. Operational Decision Checklist

• Which models, architectures, quantizations, tool calls, structured outputs, and multimodal features are supported?
• Does the API contract match the client, including streaming, errors, usage, and cancellation?
• How are authentication, tenant quotas, TLS, logs, upgrades, and rollback handled?
• Can the runtime fit the model and KV cache at the required context and concurrency?
• Is the team optimizing for a laptop workflow, a dedicated internal service, or public multi-user serving?

6. Migration Pilot

Run the candidate runtime beside the existing service with no production writes. Replay redacted traffic, compare normalized responses and usage fields, then shadow a small live slice where policy permits. Exercise overload, cancellation, rolling restart, model-load failure, and rollback. A migration is complete only when the client contract, service objectives, quality suite, monitoring, and operator runbook all pass.

Test the operational path people often omit: upgrading the runtime, replacing a model, draining traffic, investigating a malformed request, and recovering after resource exhaustion. A fast benchmark does not compensate for an unsafe update or an incident nobody can explain. Weight each result against the real environment—one workstation, a shared internal service, or a production endpoint—because the same runtime can be appropriate in one setting and unsuitable in another.

7. Primary References

• Ollama documentation
• vLLM documentation
• SGLang documentation

What high-throughput serving replaces

Throughput comparisons between serving frameworks matter in proportion to the API spend they displace. Below: one request at 25K input, 15K cached, 3K output, at fifteen thousand a month. Of the 25,000 input tokens, 15,000 are billed at the cache-read rate and 10,000 at full input rate.

The spread across these four is roughly eightfold, which is a reminder that the baseline matters as much as the framework: the same throughput improvement is worth eight times more against the expensive entry than the cheap one.

Rates verified against provider documentation on September 18, 2026. Promotional rates expire, so re-check before budgeting: LLM API cost planning · September 2026 pricing update. Run your own numbers in the cost calculator.