vLLM optimization & tuning — a production deep dive.

Every knob in vLLM's tuning surface exists to answer one question: given a fixed amount of GPU memory and compute, how many tokens of useful work can you push through per second, at what latency, before something breaks? This article works through the real mechanism behind each major lever — optimization levels, automatic prefix caching, chunked prefill, KV-cache preemption, the four parallelism strategies (TP/PP/DP/EP), CPU/NUMA provisioning, and multimodal caching — grounded directly in vLLM's own documentation and source, not paraphrase. It closes by comparing two different "tuning priority" orderings you'll see in the wild and explaining why they disagree.

1. Two prerequisites: continuous batching and PagedAttention

Everything below assumes two foundational vLLM mechanisms that this site has covered in full elsewhere and won't re-derive here:

• Continuous batching — rather than waiting for a fixed batch of requests to all finish before starting the next batch, the scheduler adds and removes individual sequences from the running batch every iteration, as they arrive and complete. This is what makes GPU utilization high under bursty, variable-length real traffic instead of the padding waste of static batching.
• PagedAttention — the KV-cache memory manager that treats cache storage like OS virtual memory: fixed-size blocks, non-contiguous physical allocation, a block table per sequence. This is the single biggest reason vLLM can run the high concurrency this article's tuning knobs assume is available. Covered in full, with the memory-fragmentation math, in this site's dedicated PagedAttention deep dive.

Everything that follows is about how you configure the scheduler and memory manager built on top of those two mechanisms — not the mechanisms themselves.

2. Optimization levels: -O0 through -O3

vLLM V1 integrates torch.compile by default. On first ("cold") start, it traces the model's forward pass, compiles it into fused kernels, and — for the parts of the graph with static shapes — captures CUDA graphs that replay the entire forward pass with a single kernel launch instead of hundreds of small ones. Both steps take real wall-clock time before the server can accept its first request, which is exactly the tradeoff the -O<n> flag exposes directly:

Level	What it does	Startup	Steady-state performance
-O0	No compilation, no CUDA graphs (equivalent to --enforce-eager)	fastest	lowest
-O1	Simple compilation + fusions, PIECEWISE CUDA graphs	fast	good
-O2 (default)	Wider compilation ranges, more fusions, FULL_AND_PIECEWISE CUDA graphs	slower	best available today
-O3	Currently equal to -O2; reserved for future aggressive/experimental passes	same as O2	same as O2 (for now)

Compiled artifacts (FX graphs, Triton kernels) are cached to disk under ~/.cache/vllm/torch_compile_cache by default, so the expensive compilation cost is a one-time-per-model-and-config tax, not a per-restart one — subsequent warm starts load the cached artifacts directly. Two flags matter specifically for iterating during development or shortening cold starts in autoscaled/serverless deployments: --enforce-eager skips compilation entirely (development), and passing --kv-cache-memory with a previously-logged value lets vLLM skip the memory-profiling pass it would otherwise run to determine how much KV-cache space is available.

3. Automatic prefix caching

The mechanism is simple to state precisely: vLLM hashes the KV-cache blocks (in PagedAttention's fixed-size block scheme) belonging to a request's prompt. If a new request's prompt shares a prefix with a block hash already resident in the cache — a system prompt, a repeated document, the earlier turns of a multi-turn conversation — the new request reuses those blocks instead of recomputing them, and only pays the prefill cost for the new, non-shared suffix.

Enable with enable_prefix_caching=True (the flag most 2026-era deployments leave on by default for chat and agentic workloads specifically because system prompts and tool schemas are large, static, and repeated on every single request).

4. Chunked prefill — the ITL/TTFT tradeoff, made explicit

Without chunking, a long prompt's entire prefill must run as one uninterrupted scheduling step — during which every other in-flight request's decode step waits, because the GPU is busy on that one prefill. Under bursty traffic with a mix of short and long prompts, this is exactly the "short requests stuck behind a huge prompt" problem the scheduler needs to avoid. Chunked prefill (default-enabled in current vLLM) splits a large prefill into smaller pieces and interleaves them with pending decode steps in the same scheduling iteration, governed by a single token budget:

Scheduling policy: at each step, the scheduler first batches all pending decode requests (decode gets priority), then fills whatever token budget remains — max_num_batched_tokens — with pending prefill work, chunking any prefill too large to fit in one pass.

\[ \text{tokens scheduled this step} \le \text{max\_num\_batched\_tokens} \]

max_num_batched_tokens	Effect on ITL (inter-token latency)	Effect on TTFT (time to first token)
Smaller (e.g. 2,048, the current default)	Better — fewer prefill chunks interrupt each decode step	Worse — long prompts take more scheduling rounds to finish prefilling
Larger (e.g. >8,192, recommended for throughput)	Worse — bigger prefill chunks steal more time from decode	Better — more of a long prompt fits in one scheduling pass

The default value optimizes for ITL (a chat-latency-sensitive default), which is why vLLM's own docs note the default can have lower throughput than a purely throughput-optimized configuration — if your workload is offline batch scoring rather than interactive chat, raising max_num_batched_tokens is one of the highest-leverage single changes available.

5. KV-cache preemption: what happens when memory runs out

PagedAttention's block-based allocation still has a hard ceiling — the fraction of GPU memory reserved for KV cache, set by gpu_memory_utilization. When too many concurrent sequences would exceed that budget, vLLM must preempt one or more running sequences to free blocks for the others. vLLM V1's default preemption mode is RECOMPUTE, not SWAP:

• RECOMPUTE (default in V1) — drop the preempted sequence's KV cache entirely; when it's rescheduled, redo its prefill from scratch. Simpler, lower overhead in V1's architecture, but the cost scales with how much context has to be redone — expensive for sequences deep into a long generation.
• SWAP — move the preempted sequence's KV cache to CPU memory instead of discarding it, then transfer it back on resume. Avoids recomputation but adds PCIe transfer overhead and CPU memory pressure of its own.

Frequent preemption is a visible warning in the logs, and it directly costs throughput — recomputed prefill is pure waste relative to a system with enough headroom to never evict. The standard remediation, in order of what to try first: increase gpu_memory_utilization (more memory reserved for KV cache, if headroom exists), decrease max_num_seqs or max_num_batched_tokens (admit fewer concurrent sequences so each gets guaranteed cache space), or — if neither is enough — add parallelism to spread the model and its cache across more accelerators (§6).

6. Parallelism strategies: TP, PP, DP, and EP

These four are not interchangeable — each solves a different constraint, and production MoE deployments typically combine three of them simultaneously.

Strategy	What it shards	When you need it	Main cost
Tensor Parallel (TP)	weight matrices within each layer, across GPUs	model too large for one GPU's memory	an all-reduce per layer — needs fast intra-node interconnect (NVLink)
Pipeline Parallel (PP)	whole layers, sequentially across GPUs/nodes	model too large even after TP, or scaling across nodes with slower interconnect	pipeline bubbles — GPUs idle waiting for earlier/later stages
Data Parallel (DP)	nothing — replicates the whole model, splits requests across replicas	model fits on one (TP-)group already; you just need more aggregate throughput	full memory cost per replica; no inter-replica communication needed
Expert Parallel (EP)	MoE expert weights, one subset of experts per GPU	MoE models specifically — dense-layer TP wastes the sparsity	an all-to-all token shuffle: dispatch each token to its expert's GPU, then gather results back

Expert Parallel deserves its own sentence because the mechanism is genuinely different from TP: with EP enabled, each GPU holds a disjoint subset of the model's experts (rather than a shard of every expert). Because MoE routing sends different tokens to different experts, this requires an all-to-all communication step twice per MoE layer — once to send each token's hidden state to the GPU that owns its selected expert, once to gather the results back — which is the dominant communication cost EP-based serving has to amortize.

7. CPU provisioning and NUMA binding

vLLM V1's multi-process architecture — separate API server, engine core, and per-GPU worker processes — means the CPU is not a passive bystander to GPU-bound inference. Tokenization, request scheduling (the engine core runs a tight busy loop), and output detokenization/streaming all run on CPU, and under-provisioning it produces exactly the kind of tail-latency jitter that's hard to diagnose from GPU metrics alone, because the GPU looks idle while the CPU is the actual bottleneck.

vLLM's documented minimum physical-core formula for a deployment with \(A\) API server processes, data-parallel size \(DP\), and \(N\) GPU workers:

\[ \text{minimum physical cores} = A + DP + N + \begin{cases}1 & DP > 1\\0 & DP = 1\end{cases} \]

Worked example straight from the docs: DP=4, TP=2 across 8 GPUs needs 4 API-server processes + 4 engine cores + 8 GPU workers + 1 DP coordinator = 17 physical cores minimum. The explicit warning worth internalizing: hyperthreads are not physical cores — 1 vCPU on a hyperthreaded system is roughly 0.5 physical core for this budget, so a cloud instance advertising 17 vCPUs may only deliver ~8.5 physical cores against a 17-core requirement.

On multi-socket (NUMA) hosts, cross-socket memory access for KV-cache and activation buffers is measurably slower than same-socket access. vLLM exposes explicit binding controls: --numa-bind auto-detects GPU-to-NUMA-node mapping, --numa-bind-nodes lets you specify the mapping explicitly (e.g. --numa-bind-nodes 0 0 1 1 pins the first two workers to NUMA node 0 and the next two to node 1), and --numa-bind-cpus pins specific CPU core ranges per worker. The Python API additionally requires VLLM_WORKER_MULTIPROC_METHOD=spawn for NUMA binding to take effect correctly.

8. Multimodal optimization: encoder parallelism and processor caching

Vision-language serving adds two cost centers dense text serving doesn't have: running the vision encoder itself, and repeatedly preprocessing images/video that may recur across requests (the same product photo queried by many users, a document re-referenced across turns).

8.1 mm_encoder_tp_mode

Two modes for sharding the vision encoder's own compute: "weights" mode shards each layer's weights across the TP ranks (standard tensor parallelism, applied to the encoder). "data" mode instead uses the TP group to shard the input data — effectively turning the TP size into an encoder-level data-parallel factor. vLLM's own reported result: mm_encoder_tp_mode="data" at tensor_parallel_size=8 improved multimodal throughput by roughly 10%, at the cost of replicating encoder weights across ranks (more memory, less communication) rather than sharding them (less memory, more communication).

8.2 Processor cache: mm_processor_cache_gb and mm_processor_cache_type

The multimodal processor cache avoids re-running the (CPU-side) preprocessing pipeline — resize, normalize, patchify — on an image the server has already processed once. Default size is 4 GiB; set it to 0 to disable caching entirely. The type of cache changes where memory actually gets spent — this is the detail most deployments get wrong by assuming the config value is the whole story:

Cache type	Where data lives	Total memory formula
LRU (default, no IPC)	Keys + values on P0 (API process) only	mm_processor_cache_gb × data_parallel_size
LRU with IPC	Keys on P0; keys + values duplicated into each engine core	mm_processor_cache_gb × api_server_count
Shared memory ("shm", for TP>1)	Keys on P0; values in shared memory reachable by all workers (no duplication)	mm_processor_cache_gb × api_server_count

The practical implication: with multiple API server processes or a large data-parallel size, the default LRU-with-IPC scheme duplicates the actual cached image/video data into every engine-core process — at scale, that duplication can dominate host memory. mm_processor_cache_type="shm" exists specifically to eliminate that duplication for TP>1 deployments by putting the actual cached values in one shared-memory region instead of copying them per-process.

9. Two "tuning priority" orderings — and why they disagree

The infographic that prompted this article lists a production tuning priority starting with prefix caching, then chunked prefill, then GPU memory utilization. vLLM's own official documentation lists a different order: verify CPU provisioning first, then pick an optimization level, then tune the batching token budget, then parallelism, then cache sizes. Both are defensible, and the disagreement is informative rather than a contradiction to resolve in favor of one:

Ordering	Implicit assumption	Right for…
Prefix caching → chunked prefill → memory utilization → batching → sequences → parallelism → NUMA	Infrastructure (GPUs, CPU core count, node topology) is already fixed and adequate; you're purely tuning request-handling behavior for a known chat/agentic workload shape.	An already-provisioned production service being tuned for a specific traffic pattern.
CPU provisioning → optimization level → batching → parallelism → caches → fastokens → API servers → NUMA	You might discover the deployment is fundamentally under-resourced (not enough physical cores for the process topology) before any request-level tuning could possibly help.	Standing up a new deployment, or diagnosing a service that's underperforming for reasons no amount of flag-tuning will fix.

The synthesis, not a pick-one: verify the infrastructure can support the topology you intend to run before optimizing request-serving behavior on top of it — the official docs' order is closer to correct as a global default, especially for a new deployment, but once CPU/GPU provisioning and parallelism are settled and stable, the infographic's ordering (prefix caching → chunked prefill → memory headroom) is exactly the right day-to-day tuning loop for a running service, because those are the levers that respond to changes in traffic pattern rather than to fixed infrastructure decisions made once at deployment time.

FAQ

Does prefix caching help a workload with long generations and short prompts?

No — prefix caching only accelerates the prefill phase. A decode-bound workload (short prompt, long generation) gets essentially no benefit from it; look at chunked-prefill tuning and parallelism instead.

Why does vLLM V1 default to RECOMPUTE instead of SWAP for preemption?

Lower overhead in V1's architecture specifically — swapping avoids redoing prefill compute but costs PCIe transfer bandwidth and CPU memory, and V1's design found recompute the better default tradeoff. It remains configurable per deployment.

Should I always use Expert Parallel for MoE models?

EP is the standard choice specifically because plain TP applied to MoE experts wastes the sparsity — TP shards every expert's weights onto every GPU (as if all experts were always active), while EP puts only a subset of experts per GPU, matching how sparse routing actually works. Dense (non-MoE) models have no equivalent choice to make; TP is simply TP for them.

Is 4 GiB always the right size for mm_processor_cache_gb?

It's the default, not a recommendation — the correct size depends on your working set of distinct images/video and your cache type's memory-duplication formula (§8.2). Compute the actual total memory footprint for your api_server_count / data_parallel_size before assuming the default is sized correctly for your deployment.

Takeaways

• Every optimization here trades one resource for another — startup time for steady-state speed (-O levels), memory for compute (prefix caching, KV preemption headroom), latency for throughput (max_num_batched_tokens), and communication for memory (TP/EP vs. DP replication). There is no universally-correct setting; there is only the correct setting for your traffic shape and hardware.
• Prefix caching and chunked prefill both act on prefill only — a decode-bound workload needs parallelism and memory tuning, not these two levers.
• MoE serving needs a mixed parallelism strategy, not a single choice — DP for the (small, replicable) attention weights, EP for the (large, sparse) expert weights is the documented production pattern, not an edge case.
• CPU provisioning is a real, formula-computable constraint in vLLM V1's multi-process architecture, and hyperthreads don't count as full cores against it — check this before assuming a GPU-side change will fix a latency problem.
• The multimodal processor cache's memory cost depends on cache type, not just the size you set — the default IPC scheme duplicates cached data per engine-core process, which mm_processor_cache_type="shm" exists to fix at TP>1.
• Two reasonable tuning-priority orderings can both be right for different situations — provisioning-first for new/underperforming deployments, request-behavior-first for an already-stable service responding to traffic changes.

References & further reading

• vLLM — Optimization and Tuning (official docs) — the primary source this article is grounded in throughout.
• vLLM — Automatic Prefix Caching — mechanism and use-case detail for §3.
• vLLM — Parallelism and Scaling — TP/PP/DP reference for §6.
• vLLM — Expert Parallel Deployment — the EP all-to-all mechanism and the DeepSeek-V3 deployment shape cited in §6.
• vLLM — CPU installation & NUMA binding — source for the core-provisioning formula and NUMA flags in §7.
• vLLM — Multimodal configuration reference — mm_encoder_tp_mode and processor-cache flags for §8.
• cvam.sight — PagedAttention deep dive — the KV-cache memory manager every optimization in this article builds on top of.