SGLang vs vLLM: Install, Serve, and Benchmark

Two open-source engines dominate self-hosted LLM serving right now, and both promise the same thing: feed them a Hugging Face model, get an OpenAI-compatible API with high throughput. vLLM is the one most teams reach for first. SGLang is the one that keeps showing up in throughput charts and frontier-lab deployments. The obvious question, once you have both installed, is whether the newer engine actually beats the incumbent on the hardware you can afford.

So we put them on the same box. This SGLang vs vLLM guide installs SGLang from scratch, serves a model through its OpenAI endpoint, then runs the exact same load against vLLM on the same GPU and the same model. Every command and number below came from SGLang 0.5.15 and vLLM 0.25.1 serving Qwen2.5-7B-Instruct on a single NVIDIA RTX 4090 in July 2026, so the comparison is apples to apples rather than two vendors’ marketing decks.

What SGLang is, and where it differs from vLLM

SGLang is a serving framework for large language models built around a fast runtime and a structured generation front end. Its headline feature is RadixAttention, which caches and reuses the attention state of shared prompt prefixes across requests. For workloads with heavy prompt reuse (agents replaying a long system prompt, few-shot templates, multi-turn chat) that reuse is where SGLang earns its reputation.

vLLM built the same category with PagedAttention, which treats the KV cache like virtual memory pages so the GPU wastes almost nothing on fragmentation. Both engines do continuous batching, both do chunked prefill, both expose a drop-in /v1 OpenAI API, and both run the same GGUF-free Hugging Face safetensors weights. The install paths and the defaults are what differ, and so does the latency profile under load, which is the part you cannot read off a README.

Prerequisites

The sizing driver for a serving box is simple: the model weights have to fit in VRAM, and whatever is left becomes KV cache, which sets how many concurrent tokens you can hold in flight. A 7B model in bf16 is roughly 15 GB of weights. On a 24 GB card that leaves about 6 to 7 GB for KV cache at a 0.9 memory fraction, which is enough for real concurrency on 7B. Move up to a 13B/14B model in bf16 and you want 32 to 48 GB; a 70B model in bf16 needs multiple 40 GB-plus GPUs or an AWQ/INT4 quant on a single 48 GB card.

Everything here was tested on a single RTX 4090 (24 GB), Ubuntu 22.04, CUDA 12.4 toolchain, NVIDIA driver 595. That card is a sensible floor for following along with a 7B model; it is not a recommendation for a production fleet, where you size from your real working set and request volume. If you are still deciding on hardware, our notes on how much VRAM it takes to run an LLM and the best GPUs for local LLM work cover the trade-offs.

You need a working NVIDIA driver plus CUDA. Confirm the GPU is visible before installing anything:

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader

You should see the card, its total memory, and the driver version on one line:

NVIDIA GeForce RTX 4090, 24564 MiB, 595.71.05

With the driver and card confirmed, SGLang installs from pip in a couple of minutes.

Install SGLang

Keep SGLang in its own virtual environment. It pins specific versions of PyTorch and FlashInfer, and isolating it avoids fighting whatever else lives in your system Python. Create the environment first:

sudo apt update
sudo apt install -y python3-venv python3-pip git ninja-build build-essential
python3 -m venv /opt/sglang
source /opt/sglang/bin/activate

The ninja-build package matters. FlashInfer is SGLang’s default attention backend, and vLLM pulls it in too (on this GPU it uses FlashInfer for the sampling step), so both engines compile CUDA kernels on first launch. Without ninja on the path that build fails with FileNotFoundError: [Errno 2] No such file or directory: 'ninja' the moment the server tries to serve its first request. The official Docker images ship it prebuilt; a bare pip install on a clean box does not, so install it up front.

Now install SGLang with its full set of runtime extras:

pip install --upgrade pip
pip install "sglang[all]"

Confirm the version that landed:

python -c "import sglang; print(sglang.__version__)"

The build tested here reported:

0.5.15.post1

If you would rather not manage Python at all, the project publishes a container image (lmsysorg/sglang) that bundles the runtime, FlashInfer, and the build tools. That is the fastest route on a machine where you already run Docker with the NVIDIA container toolkit, and it sidesteps the ninja gotcha entirely.

Serve a model with SGLang

SGLang serves any Hugging Face model through sglang.launch_server. Point it at a model ID, bind it to a port, and set the fraction of VRAM it may reserve for weights plus KV cache:

python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 --port 30000 \
  --mem-fraction-static 0.9

On the first run it downloads the weights, loads them, allocates the KV cache, and captures CUDA graphs. When it is ready it prints the pool sizing, which tells you exactly how much room you have for concurrent requests:

KV Cache is allocated. #tokens: 113739, K size: 3.04 GB, V size: 3.04 GB
max_total_num_tokens=113739, chunked_prefill_size=2048, max_prefill_tokens=16384, max_running_requests=2048, context_len=32768
The server is fired up and ready to roll!

On the RTX 4090 that startup took 58 seconds cold and the server held about 23.3 GB of VRAM once the pool was allocated. The endpoint is OpenAI-compatible, so a health probe and any OpenAI client library work unchanged. Check it is up from another shell:

curl -s http://localhost:30000/health

A running server answers with HTTP 200. From here you point any tool that speaks the OpenAI API at http://your-host:30000/v1 and it just works, which is the whole appeal of these engines.

The vLLM side of the comparison

If you have not set vLLM up yet, our production vLLM install guide walks the full path, and there is a separate write-up for running vLLM on Kubernetes. Installed the same way, in its own virtual environment, serving the same model is a single command:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --gpu-memory-utilization 0.9

vLLM 0.25.1 took 79 seconds to become ready on the same card and reserved slightly more of the memory budget for KV cache. Both engines page the cache, so this is a difference in how much each left unallocated, not a win for either layout:

Available KV cache memory: 6.51 GiB
GPU KV cache size: 121,968 tokens
Maximum concurrency for 32,768 tokens per request: 3.72x

Two engines, same GPU, same model, KV pools within about 7% of each other (113,739 tokens for SGLang, 121,968 for vLLM). That parity in cache size is worth remembering when you read the throughput numbers, because it means neither engine won or lost on raw memory headroom.

How we benchmarked

Both engines were driven by SGLang’s own load generator, sglang.bench_serving (recent builds also expose it as sglang.benchmark.serving), which can target any OpenAI-compatible server, so the same client and the same request stream hit both. We used the random dataset with fixed lengths and a fixed seed so the two runs saw identical work, and ran two load profiles that stress different things:

• Saturation: 300 requests, 1024 input tokens and 512 output tokens each, capped at 64 concurrent requests. This measures peak throughput when the engine is never idle.
• Moderate load: 200 requests arriving at 8 per second, 1024 input and 256 output tokens. This measures the latency a real user would feel when the server is busy but not pinned.

The saturation run against SGLang looked like this:

python -m sglang.bench_serving \
  --backend sglang --host 127.0.0.1 --port 30000 \
  --model Qwen/Qwen2.5-7B-Instruct \
  --dataset-name random --random-input-len 1024 --random-output-len 512 \
  --random-range-ratio 1 --num-prompts 300 --max-concurrency 64 --seed 42

The identical command with --backend vllm --port 8000 drove vLLM. The GPU was rented from vast.ai, a cheap way to get an hour on a 4090 without owning one; if you go that route, our notes on automating on-demand GPU jobs and renting cloud GPUs save some money.

Throughput: a near tie

Under saturation, the two engines finished within about a second of each other and pushed almost identical token volume. This is the headline that marketing charts tend to obscure: on a single consumer GPU with a 7B model, SGLang and vLLM are within a few percent on raw throughput.

vLLM came out about 1.3% ahead on total throughput here (5,221 versus 5,155 tokens per second). That gap is inside the run-to-run noise you would see across repeated benchmarks, so the honest read is that they tie. The full result block SGLang printed:

============ Serving Benchmark Result ============
Backend:                                 sglang
Max request concurrency:                 64
Successful requests:                     300
Benchmark duration (s):                  89.39
Total input tokens:                      307200
Total generated tokens:                  153600
Request throughput (req/s):              3.36
Output token throughput (tok/s):         1718.25
Total token throughput (tok/s):          5154.74
Median TTFT (ms):                        3044.85
Median TPOT (ms):                        29.06
Median ITL (ms):                         23.60
==================================================

Those totals are the whole throughput story, and they are close enough to call even. The latency numbers underneath them are not.

Latency: where they actually diverge

Throughput is a tie, but latency is not, and the split is the most useful thing this benchmark surfaced. The two engines make opposite trade-offs, and which one you want depends on what your users notice.

vLLM reaches the first token far sooner. At 8 requests per second its median time to first token was 941 ms against SGLang’s 4,942 ms, a five-fold difference with default settings. vLLM’s scheduler paces prefill work so new requests start streaming quickly, which is exactly what a chat user feels as responsiveness.

SGLang flips the result on per-token decode speed. Its median time per output token was 29.1 ms under saturation versus vLLM’s 34.0 ms, and 55.8 ms versus 68.2 ms under moderate load. Once a response is streaming, SGLang emits tokens about 15 to 18% faster. For long generations, where most of the wall-clock is decode rather than that first token, SGLang’s steadier per-token rate is the number that adds up.

Put plainly: vLLM feels snappier at the start of a response, SGLang finishes a long response sooner. Neither of those is visible in a throughput chart, which is why running the test yourself beats trusting a single headline number.

SGLang vs vLLM: which to run

On this hardware and model, the decision is not about who is faster overall, because they are not meaningfully different on throughput. It comes down to workload shape:

• Interactive chat and agents: vLLM’s low time to first token makes the UI feel responsive, and its ecosystem (Kubernetes deployments, LoRA serving, structured output, broad model coverage) is the most mature. It is the safe default and the one with the most documentation, integrations, and community answers.
• Long-generation and prefix-heavy workloads: SGLang’s faster decode and RadixAttention prefix reuse pull ahead when generations are long or when many requests share a big system prompt. If you serve agents that replay the same long context repeatedly, benchmark SGLang with your real prompts, because the reuse advantage does not show up in a random-prompt test like this one.
• Startup churn: SGLang came up in 58 seconds against vLLM’s 79 seconds. On a fleet that scales pods up and down constantly, 20 seconds of cold start per replica is not nothing.

The larger point is that both are excellent, and the gap between them on a single 4090 is small enough that your prompt shape, your model size, and your tolerance for first-token latency should decide it, not a leaderboard. For the models themselves, our open-source LLM comparison covers what to actually serve on either engine.

Reproduce it on your own hardware

None of this is expensive to repeat, and you should, because the defaults and the version numbers move fast. Rent or borrow a 24 GB GPU, install both engines in separate virtual environments, and point sglang.bench_serving at each one with the request lengths that match your real traffic rather than the 1024/512 split used here. Swap in your production system prompt and the RadixAttention story changes; push output lengths to a few thousand tokens and the TPOT gap compounds. The engine that wins on your workload is the one worth deploying, and now you have the exact commands to find out which that is.