Inference Engineering, Monthly 2026.

Summary: September closed the year with consolidation and with correctness work. vLLM made Model Runner V2 the default for every model, TensorRT-LLM disclosed a silent adapter bug and then removed its AutoDeploy integration, and Ollama added decision models that return scores instead of text.source ↗

The default is also the test

Imagine you are a user of an engine that changes its default. You did not ask for the change. You did not set the flag. Your configuration simply starts running a new path. If that path works, nobody notices. If it has a bug in a corner you depend on, you find out when a customer complains. The default is a test that every user runs whether they want to or not.

This year the defaults flipped one after another. Asynchronous scheduling in January, the newer speculative decoding path in May, the cache manager in August, and now the runner for every model in September. The September flip is the broadest, because it covers all models rather than a subset. When a runner becomes the only path, every bug it hides becomes a bug every user can meet. That is why a month's known-issue lists matter as much as its feature lists.

The practical advice is simple. Treat each default as a moving target. The configuration you tested in March is not the configuration the engine runs by default in September. A quarterly review of defaults, with a short list of the settings that matter to your product, catches silent changes before an incident does.

One runner for every model

vLLM v0.29.0, released September 9 with 594 commits from 277 contributors, finished the rollout of Model Runner V2 to all models.source ↗ The runner had first become the default for dense models in July, and for pooling models earlier. This release added kernel.">CUDA graph memory profiling for automatic decode does not recompute attention over the whole history.">KV cache sizing, which means the engine measures how much memory its graphs need rather than guessing. It also added batch-sharded sampling.

Batch-sharded sampling is worth a moment, because it shows the memory budget from January reaching into a part of the pipeline most people never think about. At every step, the engine computes a score for every possible next token, for every request in the batch. For a large vocabulary and a large batch, that table is big. When the model runs across several GPUs, sharding the sampling across them divides the table per device. The memory saved goes back to the cache, which is where capacity always ends up.

vLLM v0.30.0, released September 22, was the largest release of the year, with 762 commits from 315 contributors. It added DeepSeek-V4.1-Flash, which the notes describe as storing its whole KV cache in the MXFP8 block format on SM100 hardware. A low-precision cache is a memory win with a quality question attached, so it should be tested the way the May quantization checks were. The same release added a CPU backend for DeepSeek V4 with vectorized sparse attention kernels, and GLM-5.3-Flash with expert load balancing.

Bugs that do not crash

Some bugs announce themselves. A server crashes, an error rate spikes, a dashboard turns red. Others are quiet. TensorRT-LLM's September candidates included two of the quiet kind, and they are worth reading slowly.

The first, in rc26 on September 9, disclosed that LoRA adapters targeting the routed experts of Qwen3 mixture-of-experts models were silently ignored. The output came back as the base model's answer. Adapters that targeted attention layers behaved correctly, which made the bug harder to spot. A customer with a tuned adapter gets the generic model, and nothing in the logs says so. The defense is an adapter-level test that compares the adapted output against a known-good reference, run on every deployment of the adapter, not only the first.

The second bug is the opposite kind of problem. The same notes said the V2 KV cache manager could refuse to start with an insufficient-memory error, even while enough GPU memory was free, because its size estimate was too conservative. The server does not run, and the message suggests a capacity problem. A team that responds by buying more GPUs has paid to fix a guess. The useful response is to measure free memory after a startup, compare it with what the estimate predicted, and set an explicit cache size if the guess is consistently off.

Removing the AutoDeploy path

TensorRT-LLM v1.3.0rc29, released September 29, removed the AutoDeploy integration and its public entry points.source ↗ This was a breaking change, and it came two months after July's deprecation notice. A project that gives that much warning is being careful, and a team that was watching the notices had time to plan. A team that was not watching finds its deployment failing at the next upgrade. The lesson from June applies again: list the backends and integrations you use, and map each one to its replacement before the removal lands.

The same candidate listed many model items, including a fix for Qwen3.5 checkpoints with global FP8 scales, a vision rotary change for Gemma 4 that avoids a matrix multiply, and extended piecewise CUDA graphs for MiniMax-M3. That breadth tells you how much of the runtime is now shared across model families.

A very large model with a small active set

SGLang v0.5.19, released September 5, brought 786 pull requests from 214 contributors.source ↗ Its headline model was Qwen3.8, listed at 2.4 trillion total parameters with 95 billion active per token. The naming teaches a lesson. The active count sets the per-token compute, and the total sets the memory needed to hold the weights. A model like this is a memory problem first and a compute problem second.

That raises a serving question: where do the weights live? A model too large for one node must be split across nodes, and the cost of moving expert outputs between nodes becomes the limit. Offloading cache to host memory helps, but weights that every token reads cannot sit in slow memory. The memory budget from January now applies to the weights as well as to the cache.

SGLang v0.5.20, released September 18, brought 713 more pull requests and added GLM-5.3-Flash and Tencent's Hy4-Preview. The release notes read more like a model catalog than a feature list. For an engine, the useful question is which models it supports well on your hardware, and how quickly new models arrive after release.

Scores instead of sentences

Ollama v0.35.0, released September 28, introduced decision models behind a new endpoint.source ↗ The notes say the endpoint is based on the Jev API from TypeSafe. A decision model does not write prose. It returns choices, probabilities, and scores. The examples are ticket triage, model routing, and content classification. Two models are listed, Nimble from Bespoke Labs and Tev1 from Together AI, and both are pulled like any other Ollama model.

The cost profile is different from a text generator. A generator writes a long sequence token by token, and its cost grows with the answer. A decision model returns a short structured result, closer in cost to a single forward pass. That makes it practical to run a cheap decision in front of an expensive generator, which is a common pattern for cost-sensitive products. Returning scores also lets the caller set thresholds and log them over time, which a text answer does not allow without parsing.

A score is useful only if it is calibrated. Calibration means that a score of 0.8 is right about 80 percent of the time on your data. Uncalibrated scores look precise and mislead. The endpoint gives you the number. It does not tell you whether the number means what you think on your traffic, so test that before you set a threshold.

Ollama's September patches continued. v0.33.3, on September 2, added image and audio input for Gemma 4 on the MLX engine, reported cached prompt tokens, and honored defaults stored in GGUF files. Reporting cached tokens is a small change with a large effect, because it makes the prefix cache's benefit visible to the application for the first time.

Boring, on purpose

llm-d v0.10.0, released September 29, set its themes in plain language: operational hardening, which means rollouts, high availability, and failure behavior operators can trust, and production readiness, which means moving the flagship paths from working to default.source ↗ In infrastructure, boring is a compliment. A boring deployment is one where a rollout does what it says and a failed node is replaced without a page.

The release also moved changes upstream into the engine projects, which reduces the cost of maintaining forks. That ties the fleet layer more closely to the engine, so the two should be upgraded as a pair, as the earlier months recommended for kernels and frameworks.

Connectors and a DeepEP upgrade

LMDeploy v0.17.0, released September 1, integrated DeepEP v2, a communication library for mixture-of-experts dispatch, and added Kimi K2.6 support on its PyTorch path.source ↗ It also added a connector for the Mooncake store, so LMDeploy can use the same storage layer as vLLM and LMCache. A shared storage standard makes it easier to move a cache between engines.

A faster MoE combine

llama.cpp published a build on September 1 with a CUDA change that fuses the weighted sum of mixture-of-experts outputs into one kernel.source ↗ The commit message explains the mechanism. The combine step used to write each weighted expert output to global memory before summing them, and that traffic was the main cost. The fused path runs one kernel where the baseline ran two. Decode is limited by memory traffic, so removing a round trip through global memory helps on every layer of every token. For local users on consumer GPUs, that can be the difference between usable and not.

FlashAttention 4 reaches beta 33

The FlashAttention repository published fa4 betas 29 through 33 in September, the last on September 30.source ↗ Beta 29 sped up mask generation on SM100 hardware. By the end of the month, the fa4 line had run for most of 2026 without a stable tag. A kernel that must serve many shapes, hardware generations, and precisions takes a long time to harden, and teams that adopted it early should keep their own test matrix of shapes until a stable tag is out.

Parallel diffusion fixes

xDiT released 0.6.0 on September 1 and 0.7.0 on September 25.source ↗ The 0.7.0 notes are mostly bug fixes, including a timing correction. Performance measured across several ranks is wrong if each rank starts its clock at a different moment, because the slowest rank sets the real finish time. Taking the start of all ranks into account corrects the number, and a benchmark that ignored it would overstate some speeds.

A quiet month for Ray

Ray had no release in September. The last changes were in August. A quiet month is a window to test the August release in staging, because the next release will bring new changes and the August behavior is the baseline to compare against.

The concepts September adds

September adds three ideas to the set, and each one is easy to get wrong.

The first is calibration. A decision model is only as useful as its scores are honest. Test calibration before you set a threshold, because a score can rank well and still be miscalibrated.

The second is the latency metric, such as an ignored adapter. Only an end-to-end check against a known-good reference catches it.">silent failure. A silently ignored adapter does not show up as an error rate or a latency change. The defense is an end-to-end check against a reference, kept current as the engine changes.

The third is the active parameter count. A model with 2.4 trillion parameters and 95 billion active is a memory problem and a compute problem measured in different units. Plan from both numbers.

What to check this month

1. Test LoRA adapters end to end. Compare adapted output to a known-good reference on every deployment.
2. Check cache sizing after startup. Measure free memory and set an explicit cache size if the estimate is off.
3. Plan the AutoDeploy migration. The removal is in rc29, so verify your replacement before the next stable release.
4. Calibrate decision scores before using thresholds. Check the scores against outcomes on your data.
5. Log cached prompt tokens. Track the prefix cache's benefit per request.
6. Upgrade the fleet layer with the engine. Plan them as a pair.

Frequently asked questions

Is a runner that handles every model the same as a faster runner?

Not necessarily. It is more uniform, which means fewer special cases and fewer silent fallbacks. Speed depends on the model and hardware, so test speed on your models and correctness on your adapters.

Does a decision model replace a classifier?

For some tasks, yes. It returns structured scores, as a classifier does. Whether its training matches your labels is something only your own data can answer.

Glossary

Reading links

• vLLM v0.29.0 release notes Model Runner V2 for all models.
• vLLM v0.30.0 release notes DeepSeek-V4.1-Flash with an MXFP8 cache.
• Ollama v0.35.0 release notes Decision models behind the systemone endpoint.

Summary: August was the month the cache became shared state. Ray Serve LLM completed routing that reads the cache, Ollama made cancelled prefills resume instead of restarting, and SGLang v0.5.17 served Kimi K3 from its first day.source ↗

The cache is no longer a private notebook

For most of this series, the KV cache was a private notebook. One engine process wrote it, read it, and threw it away. A router only needed to know which worker to pick. Over the spring and summer, the notebook was passed around: first to host memory, then to peers, then to shared stores. August shows the next step. The cache became part of the request protocol itself. The router asks what each replica holds. The ingress tokenizes once and hands the tokens along. The engine tells every interested party when a block is created or removed.

That changes who is responsible for correctness. When the cache lives in one process, only that process can get it wrong. When the cache is shared, the router's view, the engine's state, and the client's retries all have to agree. A cancelled request that leaves half a cache entry behind is now a problem for every replica that reads it. A stale router view sends every request routed on it to the wrong place. August's releases are mostly about making those agreements explicit.

A router that reads the cache

Ray 2.58.0, released August 23, completed a feature that Ray 2.57 had previewed: KV-aware routing for Ray Serve LLM. Tokenization now happens once, at the ingress replica, and the tokens travel out of band so the engine does not repeat the work. KV lifecycle events are broadcast to every ingress replica, so each one sees the same picture of who holds what.

The small saving of tokenizing once matters at high request rates, because it happens at the moment a request enters the system. The bigger idea is the broadcast. Several ingress replicas each route requests. If each one has a different view of the cache, they make different decisions, and some of those decisions are wrong. Broadcasting keeps the views in step. The cost is network traffic that grows with how fast blocks come and go. A workload with heavy churn pays more than one with stable prefixes.

Ray 2.57.0, released August 11, made DataSourceV2 the default for reading data, with row-group aware chunking, and replaced the old hash shuffle's aggregator pool with a version that does not need its size guessed up front. That guessing used to cause either wasted capacity or out-of-memory failures in batch jobs.

Remembering how far you got

Picture an agent that is writing a long report. It sends a huge prompt, the prefill starts, and then the user clicks cancel. The engine stops, but the work it did is still in the cache, partly. When the agent retries, a good system resumes from where it stopped. A bad system starts over and repeats all of the work.

Ollama v0.33.0, released August 21, fixed a hang where agent clients cancel long prefills, and it made prefill restore points trustworthy. A cancelled prefill now keeps every restore point it passed, so a retry resumes where it stopped. The notes give a sharp example. On models with recurrent layers, a resumed prefill recorded restore points that did not cover what they claimed. A request that matched 46,000 of 47,000 tokens had to be reprocessed from zero. That is a brutal penalty for a near-complete match, and it only appears when clients cancel and retry, which agent clients do often.

The lesson is about what a promise in a cache means. A restore point claims that the state is complete up to a token. If the claim is false, the system must recompute, and in the worst case it recomputes everything. Making the claim true by construction means a point is recorded only when the state behind it is complete. The release also added a Claude Desktop setting that lets the app use Ollama as a third-party gateway, which shows the runner becoming a gateway for other tools.

Kimi K3 on its first day

SGLang v0.5.17, released August 8, described Kimi K3 support on day zero.source ↗ The model has about 2.8 trillion parameters and a latent mixture-of-experts layout with 896 experts, using the top 16 for each token. It handles a one-million-token context. Its layer stack mixes 69 linear-attention layers with 24 multi-head latent attention layers, and it has a vision tower. The release says it ships as a native MXFP4 checkpoint.

The layer mix is the most interesting detail. A linear-attention layer keeps a running summary of the past. Its memory does not grow with the conversation. A latent attention layer keeps a cache that grows, but in a compressed form. Mix the two and most of the stack pays a flat cost per token, while only a minority of layers pay a growing one. That is the architectural answer to the memory budget from January. The cache is smaller because most layers do not keep a growing one.

The trade is recall. A fixed-size summary cannot hold every detail, so hybrid stacks keep some full-attention or latent layers to recover specifics from far back. The number of each kind is a design choice that trades memory against recall. Kimi K3's split is one point on that curve, and it is why a recall test at several depths matters more for it than the raw context size suggests.

vLLM v0.27.0, on August 10, brought the full Kimi K3 stack in one release, including Python and Rust frontends. vLLM v0.28.0, on August 26, added Decode Context Parallel for Kimi K3, fused kernels for its linear-attention decode and prefill, and an adaptive speculative token budget. That budget sizes the number of speculative tokens under a shared limit, and the notes credit it with about 60 percent better time to first token under DSpark.

KV cache manager V2 becomes the default

TensorRT-LLM v1.3.0rc25, on August 31, announced that its KV cache manager V2 is now the default for a long list of model families, including DeepSeek V3, V3.2, and V4, GLM-5, GPT-OSS, the Kimi K2 line, MiniMax, Nemotron H, and the Qwen3 line.source ↗ The candidate before it had listed a teardown failure among its known issues, which is a reminder that a default flip usually follows a stretch of known problems being fixed.

The cache manager decides what stays, what gets evicted, and how blocks are reused. A bug there affects output correctness as well as speed. The safe way to adopt the new default is to run the replay and the long-output tests, then compare hit rates and output drift against the old manager before promoting. Keep the old manager available for rollback.

The storage layer keeps moving

Mooncake v0.3.13, on August 26, and v0.3.13.post1, on August 31, fixed several things in its transfer engine.source ↗ One fix corrects a stale status in an io_uring path, which affects how transfers report completion. Another fixes a leak in a request map. Leaks are slow killers. They do not fail a test, but they exhaust memory over days of uptime. The post-release also restored CUDA-free wheels for machines without GPUs.

LMCache v0.5.5 release candidates arrived on August 28 and 31, with builds for CUDA 12.9, ROCm, and Intel XPU alongside the standard wheel. A cache library that ships on several accelerator stacks at once is being treated as portable infrastructure. Test each accelerator path separately, because a wheel can install cleanly and then fail at import time on the wrong stack.

Local builds, every few days

The llama.cpp project sits underneath much of local inference, and its build cadence matters more each month.source ↗ In August it published 38 tagged builds, several of them in a single batch on August 31. The pace reflects an active project shipping small, targeted changes. Local products that use it should pin a build and update on a schedule. Running whatever is current means the output of a model can change between two runs with no change on your side.

Ollama's engine changes track that upstream work, so a local product that uses Ollama inherits a chain of upstream choices it does not control. That chain is worth watching.

Parallel diffusion, briefly

xDiT released version 0.5.0 on August 31.source ↗ The project is a parallel inference framework for diffusion transformers, which spreads one image or video generation across several GPUs. Most of this series covers language models, but the same habits apply: pin the parallel layout, measure the tail, and test on the hardware you will run.

The concepts August adds

August adds three ideas that will be on the exam of every serving team.

The first is the restore point, a promise that a stored state is complete up to a given token. A false promise is worse than no promise, because it forces silent recomputation or silent errors.

The second is the mixed-state cache. Hybrid models hold two kinds of state: a fixed running summary and a growing cache. Each kind needs its own correctness rule, and a prefix hit is only valid when both were captured at the same point. This is the same boundary problem that block-aligned Mamba caching solved in March.

The third is the broadcast churn of shared state. When many routers depend on the same cache, the cost of keeping them in step grows with how fast the cache turns over. Stable prefixes are cheap to share, and fast-changing ones are expensive.

What to check this month

1. Measure KV-aware routing on your churn. Record hit rate and broadcast traffic at peak churn before and after enabling it.
2. Test cancellation and resume. Cancel long prefills on purpose and check that the retry resumes from its restore point.
3. Compare the cache manager defaults. Replay traffic and compare hit rates and output drift before promoting.
4. Test long-context recall for hybrid models. Speed over a million tokens means nothing if the facts are missed.
5. Pin local builds. Track llama.cpp and Ollama versions in any local product.
6. Watch for transfer leaks. Track memory over multi-day runs after upgrading Mooncake.

Frequently asked questions

Is KV-aware routing worth it for a small fleet?

Usually not until you have several replicas and repeated prefixes. With one or two replicas the router has little to choose between.

Does a million-token context mean the model remembers everything?

No. It means the model can accept that many tokens. Recall depends on training and layout, and it must be tested at several depths.

Glossary

Reading links

• Ray 2.58.0 release notes KV-aware and token-aware routing in Ray Serve LLM.
• Ollama v0.33.0 release notes Restore points for cancelled prefills.
• SGLang v0.5.17 release notes Kimi K3 day-zero support.

Summary: July was the month the engines started deleting their old paths. vLLM v0.25.0 made Model Runner V2 the default for all dense models and removed the legacy PagedAttention implementation, while SGLang v0.5.16 added DSpark, a speculative method that sizes its verification window from the draft's own confidence.source ↗

The month the old road was closed

Every long-running project has a road that everybody still uses, even though a newer road was built beside it. Keeping both open is expensive. Each change to the shared code must be checked against the old road as well as the new one. In July, vLLM closed the old road. Its v0.25.0 release, on July 11, made Model Runner V2 the default for all dense models and removed the legacy decode does not recompute attention over the whole history.">KV cache in fixed-size blocks, like virtual memory pages, so requests share memory without waste.">PagedAttention implementation, because the newer backends are now the standard path.

That sentence can alarm a reader who remembers PagedAttention as the heart of vLLM. It is worth being precise. The idea of managing the cache in fixed blocks lives on in the newer backends. What was deleted is one particular implementation of that idea. A project can only delete code after the replacement has earned its place through months of testing on common setups. The removal is therefore a sign of confidence, and it is also a deadline for anyone whose configuration depended on the old code.

The same release made the Transformers modeling backend as fast as native vLLM, and added FP8 mixture-of-experts support to it. A team can now run a new architecture through the Transformers code and get close to native speed. The native path may still carry model-specific tuning, so compare on the models you actually run.

Sizing the window by confidence

Think of speculation as sending a scout ahead to guess the next few steps on a trail. In the fixed-length method, the scout always goes the same distance, whether the trail is clear or not. If the scout is unsure, the distance wastes effort. If the scout is confident and the distance is short, the group walks slower than it could.

SGLang v0.5.16, released July 25, introduced DSpark, which it describes as confidence-driven speculative decoding. The draft is produced in blocks, and each verification window is sized from the draft's own confidence. A confident scout gets a longer leash, and an uncertain one comes back sooner. The reported result is 383.7 tokens per second at an average accept length of about five, on DeepSeek V4 Pro with tensor parallel 8 on B300 hardware at batch size one.

Batch size one is the best case for speculation, because there is no crowd to share the work with. Treat the number as proof that the mechanism works, and then test it at the batch sizes you actually run. A miscalibrated confidence signal can be worse than a fixed length, because it can send the scout too far when the trail is bad. The hard engineering problem is making that signal cheap and well calibrated.

SGLang v0.5.15, on July 10, had made Spec V2 the default, with zero-overhead scheduling through graph-friendly draft extension and fewer synchronizations between device and host. The same release described GLM-5.2 NVFP4 serving on Kernels often need a specific build for it.">Blackwell at over 500 tokens per second per user on eight B300 GPUs at batch size one. A post-release on July 14 fixed NaN outputs from a FlashInfer FP4 kernel on long inputs, which is the kind of correctness fix that matters more than any headline latency, or it means little.">throughput.

Reusing text that moved

Normal prefix caching has a strict rule: a cached block only helps if the request starts with exactly the same prefix. But a retrieval system often inserts the same document at a different position in each prompt. The document is the same, the offset is different, and the cache misses.

LMCache v0.5.2, released July 22, brought CacheBlend under vLLM's hybrid memory manager. CacheBlend reuses cached text computed in another context, corrects its position, and recomputes a small part of the cache to keep the answer right. The subtle part is position. A block cached at one offset carries the wrong position if it is reused at another. The release adds support for the two positional schemes that sliding-window models use. Test reused-context answers against a fresh computation before you trust them, because a positional error does not throw an exception. It just gives a slightly wrong answer.

The same release made NIXL an optional install, and it needs vLLM 0.26 or later. Pin the cache library and the engine together, as one unit.

A deprecation with a reason

TensorRT-LLM v1.3.0rc21, on July 15, announced that its AutoDeploy backend is being deprecated.source ↗ The notice gives its reasoning in plain language. The team is working on agent-driven porting to the PyTorch backend, and it cites MiniMax M3 functional support delivered within the first week of that model's release as an early sign. The bet is that porting can become fast enough that a single primary backend covers the need.

That reasoning describes a real tension. Teams want new models quickly, and a backend that loads models without hand-written code is valuable for exactly that. Keeping several backends alive has a cost. The project is betting that the cost of porting is falling, and the next few months of notes will show whether that bet holds.

The command that became an agent

Ollama v0.32.0, released July 11, changed the meaning of its simplest command.source ↗ Running plain ollama now starts an interactive agent that can help with code and delegated tasks, and the example uses a cloud model. The release also renamed the Codex app integration to ChatGPT and simplified how a launch chooses its integration.

The change matters because it changes what a local runner is for. An agent sends a stream of requests with a growing context, tool calls, and sometimes web search. That is the traffic profile of an agent workload, which is full of repeated prefixes. A runner's speed now depends on how well it handles that pattern, not just on single prompts. Ollama's v0.32.5 patch on July 27 fixed an MLX bug that reduced output quality for NVFP4 models, a reminder that quality checks belong in local workflows too.

Prefix hits even when speculating

LMDeploy v0.15.0, on July 31, supported long-context and multi-token-prediction prefix-cache hits.source ↗ Speculation and prefix caching used to interact badly in some engines, because the draft state muddied what counted as a reusable block. Making the hit work for both means a long shared prefix pays off even when speculation is on.

The same release added guided decoding for speculative decoding. Guided decoding forces output into a grammar, such as valid JSON. Combining it with speculation is harder than it sounds, because the grammar has to reject drafted tokens that would be invalid. For structured-output products, the schema validity rate is the number to test, because a broken combination can pass every latency test while producing invalid output.

Moving big blocks across busy links

Mooncake v0.3.12, on July 23, added transport-level work.source ↗ Its notes mention path diversity for RDMA queue pairs through UDP source ports, link aggregation balance, and quality of service with slice spraying for its transfer engine. Spraying splits one transfer across several paths, so a single hot link cannot become the limit.

These changes are invisible to most users, and critical to a cache-heavy cluster. When many machines move large blocks at once, the busiest link sets the speed, and the whole system waits on it. Spreading the traffic keeps tail latency under control, and the benefit is easiest to see under concurrent load, so benchmark it there.

Early warnings from a memory monitor

Ray 2.56.1, on July 17, fixed regressions that the 2.56 release had introduced in Ray Data's conversion to pandas, including an overflow crash and a crash on empty tensor columns.source ↗ It also added early detection of memory pressure on system slices, where the memory monitor takes snapshots of usage. Early detection is valuable because an out-of-memory kill destroys work in progress. A monitor that notices pressure building lets the system shed load or spill data while the work can still be recovered.

The concepts July adds

July adds three ideas that make the stack easier to reason about.

The first is removal as maintenance. A deleted path is a sign that its replacement has earned the default. Treat a removal as a forced migration, and read its notice as carefully as a new feature.

The second is confidence as a control signal. A system that knows how sure it is can allocate work to match. The pattern will appear in batching and cache policy too. The risk is miscalibration.

The third is position in cache reuse. A block is only useful at the position it was computed for, unless the system corrects it. The test is a comparison against fresh computation, not a latency number.

What to check this month

1. Find configs that depended on the removed path. Audit your settings before moving past v0.25.0.
2. Check optional dependencies at build time. Make sure every model you serve starts in the container.
3. Pair LMCache with vLLM by version. Pin them together.
4. Test DSpark against fixed-length speculation. Use your batch sizes and content, not the batch-one headline.
5. Test CacheBlend answers against fresh computation. Check the documents you actually serve.
6. Check schema validity under speculation. Constrained output must stay valid with speculation on.

Frequently asked questions

Did vLLM remove PagedAttention entirely?

The legacy implementation was removed. The block-based idea remains inside the newer backends, which are now the standard path.

Is DSpark faster than a fixed draft length?

The strong result is at batch size one on one model and hardware. Whether it wins for you depends on batch size, content, and calibration. Measure both methods on your traffic.

Glossary

Reading links

• vLLM v0.25.0 release notes Model Runner V2 default and the legacy path removed.
• SGLang v0.5.16 release notes DSpark confidence-driven speculative decoding.
• LMCache v0.5.2 release notes CacheBlend and the vLLM 0.26 requirement.

Summary: June was about cache that travels and paths that retire. LMCache v0.5.0 added peer-to-peer cache transfer, and TensorRT-LLM announced that its TensorRT backend would be removed after one more candidate.source ↗

A cache that asks its neighbours

Until this year, most of the cache in this series stayed inside one machine. Offload moved blocks to host memory or disk on the same box. Shared stores moved blocks across machines, but every worker talked to one central library. June brought a different idea. Picture a neighbourhood where each house keeps its own pantry. When you need an ingredient, you first ask the house next door. If they have it, they hand it over. Nobody has to walk to a central warehouse.

LMCache v0.5.0, released June 23, added full peer-to-peer support for its multiprocess mode. One cache instance can look up a block on another instance and fetch it directly. The release adds a P2P adapter, a transfer channel abstraction, and a new remote call interface for peer lookup, along with an end-to-end example. Those are the parts you need for a neighbourhood where the houses talk to each other.

The benefit is clearest for replicas that serve the same prompts. Say a fleet of replicas all answer questions about the same long manual. Without peer lookup, each replica computes the same prefix again. With it, a replica that lacks the prefix fetches it from a neighbour that has it. The network cost is real, but it is often far lower than recomputing a long prefill.

The break-even question

Every fetch has a price, and the question is whether it beats the alternative. Recomputing a prefix costs GPU time that grows with the prefix length. Fetching it costs network time that grows with the size of the cached block, plus the lookup overhead. For long prefixes on a fast network, fetching wins easily. For short prefixes, or on a busy network, recomputing can be cheaper.

That point where the two costs cross is the break-even point. Measure it on your own hardware and network, and set a minimum prefix length below which the system recomputes. A team that skips this step may find that its fast network is busy with other traffic, and the fetches slow down the very requests they were meant to help.

There is also a correctness issue. A block fetched from a neighbour must match the model, the tokenizer, and the cache layout of the requester. A mismatch can produce wrong output with no error at all. Version cache entries with the model identity and the engine version, and refuse to load blocks whose identity does not match. The check costs almost nothing, and it blocks the most dangerous latency metric, such as an ignored adapter. Only an end-to-end check against a known-good reference catches it.">silent failure in this design.

A backend that is leaving

TensorRT-LLM's v1.3.0rc20 candidate, dated June 30, opened with a bold notice.source ↗ This candidate is the last to support the TensorRT backend, and the backend will be removed in the next version. The candidate also listed known issues, including a warmup crash in DeepSeek V3 models and an autotuning failure in Qwen3-family models.

Engines collect backends over time, and each one costs maintenance. A healthy project removes the paths that a newer design has replaced. For an operator, a retirement is a migration with a deadline. The practical move is to inventory which backend each deployment uses, then run the replacement on a replay of production traffic while the old one still works. A team that discovers the removal after it happens must migrate under pressure.

The same candidate also made the request's chat template opt-in, which is a breaking API change. Two breaking changes in one candidate are a good reason to run the full integration suite before any production move.

Spreading the work of many experts

Imagine a mixture-of-experts model where each token is sent to a few expert specialists, and the specialists live on different GPUs. If the router keeps sending tokens to the same popular specialists, their GPUs are overloaded while the others wait. The slowest GPU sets the pace for the whole step, so the imbalance is a direct tax on latency.

SGLang v0.5.14, released June 26, highlighted two MoE load-balancing methods, Waterfill and LPLB. They attack the same problem from different angles, and the notes give only a brief description, so read the linked documentation before choosing one. The measurement to watch is the spread of work per GPU at each step. Log the busiest device against the average. If the ratio is close to one, the router already spreads the work and balancing will change little.

vLLM's v0.23.0 release, on June 15, added expert-parallel load balancing for its Mega-MoE path. Several engines are now tackling this problem, so a team comparing them should use the same spread metric to judge each one.

vLLM in June

vLLM shipped three releases this month. v0.22.1 on June 5 added a mixture-of-experts code model from JetBrains, plus accelerated quantized layers for AMD Zen CPUs and fixes for multi-node Ray serving. v0.23.0 on June 15 brought 408 commits from 200 contributors, and v0.24.0 on June 29 brought 571 commits from 256 contributors and added MiniMax-M3.

The v0.23.0 notes include a change that matters for long-context teams. Sliding-window layers forget old tokens, so a prefix that scrolls out of the window stops being useful to those layers, even if it still helps the full-attention layers. Selective retention keeps the parts of a prefix that remain useful, rather than treating it as one lump. It is the hybrid thinking from May, applied to prefix reuse.

The DeepSeek V4 entries across v0.22.0, v0.23.0, and v0.24.0 show a steady hardening pattern. First came support, then patch releases, then backend-specific kernels and load balancing. A team that adopted the model in April had a clear path to a stable setup by midsummer, if it followed the releases.

A throughput claim with its setup attached

SGLang v0.5.14 highlighted DeepSeek V4 on Nvidia GB300 with five times the throughput at the same interactivity, in a joint post with PyTorch.source ↗ The phrase at the same interactivity is the one to read carefully. Throughput is often measured alongside a latency target, and a system can raise throughput by accepting slower responses. Holding interactivity fixed makes the comparison fair. Even so, it is one configuration on one hardware generation, so treat it as a direction until you reproduce it.

The same release added several models, including GLM-5.2, LiquidAI's LFM2.5, and Poolside's Laguna-M.1. The breadth shows how quickly serving engines absorb new architectures. The operator's question is not which models the engine supports, but how quickly it supports the models you need, and whether those models are tuned for your hardware.

FP8 cache, the familiar format

LMDeploy v0.14.0, released June 24, added FP8 KV cache quantization.source ↗ In May, the same project brought in TurboQuant. FP8 is a more standard format, supported on more hardware, so it is easier to test and reason about. A team choosing a quantized cache can now compare an 8-bit floating format against the earlier option on its own tasks.

The comparison has to include long outputs, because drift accumulates. Keep a 16-bit baseline for the same prompts, so the difference is measured against a fixed reference rather than against memory of how the model used to answer. The release also added an OpenAI-style Responses endpoint, which means applications written for that interface can run against LMDeploy without a shim.

Drafting faster on a laptop

Ollama v0.31.1, released June 30, said Gemma 4 is now significantly faster on Apple silicon.source ↗ On a coding-agent benchmark it generated tokens nearly 90 percent faster on average, by using multi-token prediction. Ollama picks how many tokens to draft as it runs, and the feature is on by default. This is speculative decoding delivered inside a local runner, with the draft length chosen by the software rather than by the user.

An adaptive draft length is a good pattern. The right number depends on the acceptance rate, which depends on the content. Code and structured text accept more drafts than creative prose. A fixed setting is a compromise across content types, and an adaptive one can follow the content. The 90 percent figure comes from a coding benchmark, so expect a smaller gain on chat. Benchmark the mix your users actually send.

Ray and the memory of a pipeline

Ray 2.56.0, released June 29, focused on data stability.source ↗ It added the ability to run several datasets in one cluster, automatic batch size choice for CPU map-batches, and a default logical memory setting that helps prevent out-of-memory failures. Iteration also buffers less and shuts its executor down when the consumer leaves early. The notes say this reduces object-store spilling for common training workloads.

Spilling is a sign of memory pressure. When the shared object store fills, data goes to disk, which is slow. A pipeline that spills heavily is usually buffering too much or using batches that are too large. The fixes here attack those causes, which is good news for batch inference jobs that run for hours.

The beta train keeps going

The FlashAttention repository published fa4 betas 16 through 19 in June.source ↗ Beta 19, on June 24, added backward kernels for the sparse multi-head latent attention used by DeepSeek V4. Earlier betas fixed FP8 forward passes, an int32 overflow in the tile scheduler for long contexts, and a shared-memory overflow for small head sizes. The int32 fix deserves a note. A position counter in 32 bits runs out past two billion, which long contexts times large batches can reach. Long-context tests need a case at the largest context the product supports, because short tests never touch that range.

The concepts June adds

June adds three ideas that will shape the second half of the year.

The first is the break-even point for transfer. Fetching beats recomputing only past a certain prefix length on a given network. Measure that point, do not assume it.

The second is load spread. Expert layouts make some devices busier than others, and the busiest device sets the step time. Track the ratio, and enable balancing only if it costs latency.

The third is the deprecation window. A notice is a deadline. The teams that inventory their backends early are the ones that migrate calmly.

What to check this month

1. Measure the transfer break-even. Compare recomputing a long prefix with fetching it from a peer on your own network.
2. Version cache entries. Refuse blocks whose model or engine identity does not match.
3. Inventory your TensorRT backend use. Test the replacement on a replay before the removal lands.
4. Log MoE load spread. Track the busiest and average device per step.
5. Judge speculation on your content mix. Benchmark the content your users actually send.
6. Check data pipeline defaults after the Ray upgrade. Confirm the memory and batch settings fit your jobs.

Frequently asked questions

Is peer transfer always faster than recomputing?

No. It depends on prefix length, network speed, and lookup overhead. Short prefixes and slow networks can make recomputing the better choice.

Why does a backend removal matter if the old path still works today?

Because the next release will not include it, and every upgrade after that fails to find it. Migrating early is far easier than migrating under a forced upgrade.

Glossary

Reading links

• LMCache v0.5.0 release notes Peer-to-peer KV transfer in multiprocess mode.
• TensorRT-LLM releases The v1.3.0rc20 backend sunset notice.
• SGLang v0.5.14 release notes GB300 result and MoE load balancing.

Summary: May was the month memory stopped being a background detail. vLLM v0.21.0 connected its KV offload to a hybrid memory allocator, SGLang v0.5.11 made speculative decoding version 2 the default, and LMDeploy v0.13.0 added TurboQuant for cache quantization.source ↗

Two kinds of memory in one house

Think of a hybrid model as a house with two kinds of rooms. Some rooms, the full-attention layers, remember everything said so far. Other rooms, the sliding-window layers, only keep the most recent few hundred tokens and forget the rest. If you give every room the same storage plan, the sliding rooms waste most of their space holding things they will never read again.

vLLM v0.21.0, released May 15, connected its KV offload system with a hybrid memory allocator. The scheduler now understands sliding-window groups, and the allocator can treat the two kinds of layer differently. The sliding rooms can drop old blocks on a known schedule, and the full rooms keep what they need. The result is that the same GPU holds more useful state, and the offload system knows which blocks are worth moving to host memory.

A practical way to size this is to ask what fraction of your cache belongs to sliding-window layers. If most of it does, the savings can be large. If most of it belongs to full-attention layers, the gain comes mainly from a better offload policy.

Speculation that respects a budget

Many reasoning products give a model a budget, a cap on how many thinking tokens it may spend before answering. The budget is a product decision, and it interacts with speculative decoding in a subtle way. A draft can propose tokens that the verifier later rejects. If the engine counts drafted tokens against the budget, the model may run out of thinking room early, or the accounting may be off in ways the user never asked for.

vLLM v0.21.0 says speculative decoding now respects reasoning and thinking budgets. The lesson generalizes. Any feature that counts tokens has to agree with every feature that changes the token stream. Budgets, stop sequences, structured output, and logprob requests all depend on the exact tokens the user sees. A bug in that accounting does not crash. It produces an answer that stops too early or runs too long.

SGLang v0.5.11, released May 5, made speculative decoding version 2 the default. The notes say it uses overlap scheduling to hide CPU overhead for EAGLE, multi-token prediction, and DFLASH paths. Making a path the default is a strong statement of confidence, and it moves every deployment that did not set the flag. Teams running older settings should compare outputs and latency before and after the change.

The vLLM month in brief

vLLM had four releases in May. v0.20.1 on May 4 stabilized DeepSeek V4. v0.20.2 on May 10 was a six-commit patch that fixed a DeepSeek V4 hang, a KV block allocation failure, and an MXFP4 path under compilation. v0.21.0 on May 15 brought 367 commits from 202 contributors, and v0.22.0 on May 29 brought 459 commits from 230 contributors.

v0.21.0 carries two changes that will bite some builds. It formally deprecates Transformers version 4 in favor of version 5, and it requires a C++20-compatible compiler. Source builds in an old container will fail in a confusing way if nobody checks. v0.22.0 moved DeepSeek V4 into its own package, added full and piecewise kernel.">CUDA graphs, and made Model Runner V2 the default for Qwen3 dense models. A fallback to the older runner can hide a slower path, so check which runner your configuration actually uses.

DeepSeek V4 across three engines

DeepSeek V4 moved through several engines in May, and the pattern is a useful one to study. vLLM went from stabilization to a package reorganization and graph support. SGLang's v0.5.12, released May 16, described a complete inference path with tensor, expert, and context parallelism, prefill-decode disaggregation, and a list of hardware targets that included Nvidia's Blackwell and Hopper parts and AMD's MI35X. LMCache v0.4.5 added DeepSeek V4 support on its multiprocess path.

When three engines support one model within two months, the model's features have become standard. Its sparse attention, its multi-token prediction, and its expert layout are well enough understood that each engine can implement them with its own kernels. The choice of engine for this model is now an operations decision. Each path has different kernels, different offload options, and different failure behavior.

SGLang's release also described HiSparse, which offloads inactive cache to CPU memory. The word inactive does real work. Sparse attention means many cached tokens are not read at each step, so their cache can sit in host memory until a query needs them. That pairing is one reason long-context serving of sparse models is becoming practical on fewer GPUs.

Quantizing the cache

LMDeploy v0.13.0, released May 12, added TurboQuant for KV cache quantization.source ↗ Quantizing weights and quantizing the cache are different jobs. Weights are fixed after loading, so their error is set once. The cache is written at every decode step, and its error piles up over a long answer. The design problem is to keep that pile small enough that a long answer still holds together.

The payoff is capacity. Store the cache in 8 bits instead of 16 and the same memory holds twice the tokens. Go lower and the gain grows, but so does the risk. A summary can tolerate a lot of cache error. An extraction task that copies identifiers, such as account numbers or order codes, cannot. The test that matters is long-output fidelity: generate long answers with and without the quantized cache, and compare facts and identifiers. A short benchmark will miss drift that shows up only after thousands of tokens.

Caching on the decode side

Disaggregation solved one problem and created another. Separating prefill from decode stops the long prefills from stalling decode. But a prefix computed on one worker had to travel, and decode workers could not reuse prefixes they had never seen. SGLang v0.5.11 restored KV cache of a shared prompt prefix across requests, so the prefix is computed once.">prefix caching on the decode side, so shared prefixes can be found and reused where the decoding happens. The release says this recovers hit rates and time-to-first-token savings for long shared prefixes.

The story repeats across the year. A separation brings a benefit and a cost, and a later release wins back part of the cost inside the new structure. A team that adopts disaggregation should measure the cost it created, not only the benefit it was promised.

Two engines inside one runner

Ollama v0.30.0, released May 13, said it uses llama.cpp to improve compatibility and performance, and that this augments the MLX engine on Apple silicon.source ↗ Two engines in one local runner is a real design choice. MLX fits Apple's unified memory. llama.cpp runs on a wide range of hardware and reads a broad set of model formats. Offering both lets the runner pick the best path per platform, but it also means the same model name can behave differently on different machines.

The release also changed how embedding inputs are handled. It lowercases input for one embedding model to match its model card. Preprocessing changes like that alter retrieval quality across a whole index, and the fix may mean re-embedding everything. Treat a preprocessing change as a data migration, not a release note.

Storage that other people depend on

LMCache v0.4.5, on May 15, made the CUDA 13 wheel the default and added DeepSeek V4 support and a TensorRT-LLM integration.source ↗ Mooncake v0.3.11, on May 21, added an eviction policy to its bucket storage backend, with batch notification to the master service. Eviction is the storage counterpart of the pluggable policy in vLLM's offload. Once a backend evicts a block, the index must learn about it, or lookups will point at ghosts.

llm-d v0.7.0, on May 12, carried the most urgent operational note of the month. Its CUDA images moved to CUDA 13.0.2, which needs NVIDIA driver 580 or later on the host. Nodes with older drivers have to be upgraded before the image is deployed. The same release made standalone mode the default, using a simpler proxy, while still recommending the full gateway for production. The default got simpler and the driver requirement got stricter. Read both changes together.

The concepts May adds

May adds three ideas that show up in almost every serious deployment.

The first is the KV cache as a budget that is shared across layer types. A hybrid model keeps two kinds of memory, and the allocator has to treat them as two budgets. Count the layers of each kind before you plan the memory.

The second is accepted-token accounting. Speculation produces tokens in bursts and discards some of them. Anything that counts tokens, including a reasoning budget, must count only what was accepted.

The third is cache quantization as a quality decision. It is an easy gain on paper and a real risk on identifiers and long answers, so it needs its own test.

What to check this month

1. Pin the allocator and offload settings. Record the active allocator, eviction policy, and offload tier.
2. Test thinking budgets with speculation on and off. Use a fixed seed and compare stop behavior and accepted-token counts.
3. Check the C++20 and Transformers 5 requirements. Test the build pipeline before upgrading.
4. Verify the driver for llm-d and CUDA 13 images. Driver 580 or later is required for the CUDA 13.0.2 images.
5. Run long-output tests for cache quantization. Compare identifiers and facts across long generations.
6. Re-check decode-side hit rates. After moving to a disaggregated layout, confirm shared prefixes are reused on decode workers.

Frequently asked questions

Should I turn on KV cache quantization everywhere?

No. Start with the tasks where long-output fidelity matters least, measure drift on each, and expand only after the tests pass.

Does a hybrid allocator help a pure transformer model?

Much less. Its benefit comes from mixed layer types. A pure full-attention model still gains from offload policy, but the sliding-window logic has little to act on.

Glossary

Reading links

• vLLM v0.21.0 release notes KV offload with the hybrid allocator and reasoning budgets.
• SGLang v0.5.11 release notes Speculative decoding v2 as default and decode-side radix cache.
• LMDeploy v0.13.0 release notes TurboQuant cache quantization.

Summary: April was the month the stack learned to survive. vLLM added Gemma 4 and DeepSeek V4 work and a general CPU offload for the KV cache, while SGLang v0.5.10 added elastic expert parallelism so a serving deployment can keep going when one GPU fails.source ↗

The night a GPU fell over

Imagine a large mixture-of-experts model served across eight GPUs. At two in the morning one of them throws a memory error and drops off the bus. In most setups, the whole replica goes down with it. The load balancer marks it unhealthy, the other replicas absorb the traffic, and an engineer wakes up to a restart that takes many minutes, because the weights have to load again and the cache has to warm up. Users see slow answers or none at all.

That is the failure story April set out to change. In a mixture-of-experts model, the experts live on different GPUs. Losing one GPU means losing some experts, not every expert. SGLang's v0.5.10 release, dated April 6, integrated an elastic version of its expert-parallel system, which the notes describe as partial failure tolerance. When a GPU fails, the system redistributes the expert weights and keeps serving without a full restart. The blog post behind the feature explains that the router and the expert placement change while traffic flows.

The trade is capacity for continuity. Serving at reduced capacity is not the same as serving at full capacity, but it is far better than serving nothing while a restart runs. A team should measure how fast the system detects the failure, how fast it moves the experts, and how much capacity remains. Those three numbers belong in any plan for a large deployment.

Overlap without bubbles

January's story was about overlapping the CPU's bookkeeping with the GPU's work. Speculative decoding was the hard case, because the result of one step decides what the next step proposes. vLLM v0.19.0, released April 3, says asynchronous scheduling now works with speculative decoding and overlaps the two. The release calls this zero-bubble overlap, meaning the GPU has no idle gap between steps in the common case.

The notes do not give one end-to-end number for this change, only that it significantly improves latency, or it means little.">throughput. That is the honest level of detail. The gain depends on how much of the step was idle before, which depends on batch size, the cost of the proposal, and the acceptance rate. A cheap proposer with high acceptance has the most to gain. The test is the same one from January: profile the gap, then compare throughput with the overlap on and off on your traffic.

Overlap also creates a correctness question. If a later step begins before the earlier step's result is final, the engine has to roll back or discard speculative work when a token is rejected. A mistake there does not crash anything. It shifts the output distribution in ways that show up only under certain sampling settings. Any change in this area deserves a fixed-seed comparison against a reference run.

Cache that moves by policy

Picture a warehouse with a small front room and a large back room. The front room is the GPU: fast, expensive, and small. The back room is host memory: slower and much bigger. Every request brings boxes, and the warehouse has to decide which boxes stay in the front room. Until now, the decision was mostly first-come, first-served. vLLM v0.19.0 added a general CPU offload for its V1 engine, with a pluggable policy and block-level preemption.

The pluggable part is the interesting bit. A policy decides which boxes are cold. A simple rule, least recently used, works well when reuse is recent. A smarter rule keeps boxes that many requests share, even if nobody touched them lately. Agent workloads with shared prefixes benefit from the smarter rule, while one-off chat prompts gain little from any rule. The right choice is a measurement, not a principle.

Block-level preemption is the other half of the story. When memory runs short, the engine can pause a request, move only its blocks out, and resume later. Older approaches moved whole requests and wasted the work done so far. Moving blocks one at a time keeps the request's place in line and reduces the cost of memory pressure. Long-context deployments hit that pressure sooner or later, so this is a feature most of them will eventually need.

Day zero, and the bugs that follow

Gemma 4 and DeepSeek V4 both arrived in April, and both were a stress test for the engines. Gemma 4 combined mixture-of-experts layers, multimodal input, reasoning, and tool use in one family. vLLM v0.19.0 required a recent Transformers version and recommended a prebuilt image. Then came v0.19.1 on April 18, a patch that fixed streaming tool calls. The bugs were in the layer that streams tokens and parses tool calls as they arrive. A partial token can look like the end of a delimiter, and the parser can produce invalid JSON or duplicate text.

That story has a useful moral. The model was correct. The bug lived in the protocol layer that wraps the model. A team that ships an agent should test streaming tool calls with the exact model and client it will use, not only with a regular request. The tests that matter are the ones with split numbers and split booleans, because those are where the parser can lose its place.

DeepSeek V4 arrived in vLLM v0.20.0 at the end of April as initial support. The notes listed fixes for a token leakage problem, an illegal memory access in the combination of sparse attention and multi-token prediction, and a limit on a shared expert's activation. Two of the three are correctness and safety fixes. A new combination of features almost always produces a new bug, and a day-zero release is the start of hardening, not the end.

Ollama shipped Gemma 4 on April 2 in four sizes. The small ones are effective 2 and 4 billion parameter models. The 26 billion parameter model is a mixture-of-experts design with about 4 billion active per token, and the 31 billion model is dense. The mix lets you trade quality, speed, and memory according to your machine.

The base moves under your feet

vLLM v0.20.0 made CUDA 13.0 the default wheel, moved to PyTorch 2.11, added Python 3.14, and runs on Transformers version 5.source ↗ The notes say the CUDA version policy follows PyTorch's, and they recommend installing with uv and a cu129 backend for teams still on CUDA 12.9. That is a lot of base change in one release.

An upgrade like that is an environment migration. The GPU driver, the CUDA runtime, PyTorch, Transformers, and the engine all have to agree. A mismatch rarely fails loudly. It shows up as a slow fallback path or a memory error under load. The safe pattern is to freeze the whole environment as one container image, test it end to end, and promote the image as a unit. Rollback is then one command.

Kernels, vision, and a correctness alarm

FlashAttention 4 kept moving through its betas in April, from beta 7 on April 1 to beta 11 on April 29.source ↗ Five betas in one month show a team hardening the kernel across many shapes. Vision encoders also gained full CUDA graph capture in v0.19.0. Vision transformers have fixed input shapes once images are resized, which suits graphs well.

TensorRT-LLM v1.2.1, dated April 20, was short and urgent. Its headline was a fix for an issue that caused decode does not recompute attention over the whole history.">KV cache corruption. Corruption is the worst kind of inference bug, because the output still looks plausible. Any team on that line should move this patch to the top of the list, and should run a long multi-turn comparison against a saved reference run after upgrading.

Autoscaling the expensive stages

Ray released 2.55.0 on April 15 and a patch on April 22.source ↗ Ray Data gained a new data source API, GPU shuffle, and two-phase checkpointing. The change that matters most for inference is autoscaling for GPU stages. GPUs are expensive and slow to start. A policy that scales too eagerly wastes money on idle devices, and one that scales too slowly leaves upstream data waiting. A queue-based signal reacts to the backlog itself, which is closer to the true bottleneck than CPU utilization.

Storage gets sturdier

LMCache v0.4.3, on April 6, made its sessions thread-safe for concurrent tensor-parallel workers and started a command-line interface.source ↗ v0.4.4 on April 22 supported different cache shapes and data types across layers, which real hybrid models need. It also added a multi-path disk backend, so one disk does not become the bottleneck for many workers. Mooncake's post-releases on April 1 and April 22 continued the storage work.

The concepts April adds

April adds three ideas that a team will feel in its on-call rotation.

The first is expert parallelism under failure. When experts are spread across devices, a dead device is a partial outage. Degraded mode keeps the service alive by moving the experts that were on the dead device to healthy ones, at reduced capacity until a replacement arrives.

The second is the eviction policy. Once a team can pick the rule that decides which blocks leave the GPU, it has to measure which rule its traffic prefers. There is no universal winner.

The third is correctness under overlap. Zero-bubble scheduling, graphs, and offload all run work earlier or in parallel. Every overlap needs a test that checks outputs against a reference, not only a test that checks speed.

What to check this month

1. Upgrade the environment as one image. Driver, CUDA, PyTorch, Transformers, and the engine move together.
2. Test streaming tool calls with the shipped model. Include split numbers and split booleans.
3. Measure the zero-bubble gain on your traffic. Compare throughput with overlap on and off.
4. Pick an offload policy by measurement. Compare hit rate and reload latency across two policies on a replay.
5. Plan for degraded capacity. Test the degraded state under peak load, not only the healthy state.
6. Patch TensorRT-LLM to v1.2.1 if you use that line. It is a correctness fix.

Frequently asked questions

Is degraded mode a replacement for spare capacity?

No. It keeps serving at reduced capacity when a part fails. Spare capacity keeps full speed. Critical services still need both.

Should I enable CPU offload by default?

Not until you measure reload latency and hit rate on your traffic. Offload trades GPU memory for transfer time, and it pays only when reuse is high.

Glossary

Reading links

• vLLM v0.19.0 release notes Gemma 4, zero-bubble speculation, general CPU offload.
• SGLang v0.5.10 release notes Elastic expert parallelism and partial failure tolerance.
• TensorRT-LLM v1.2.1 release notes The KV cache corruption fix.

Summary: March was the month the pieces started to fit. vLLM v0.17.0 made its Model Runner V2 a real path with pipeline and context parallelism, FlashAttention 4 went into beta, and TensorRT-LLM v1.2.0 became stable with disaggregated serving that can scale up and down.source ↗

The runner, the quiet middle

Most of the attention in inference goes to two places: the scheduler that decides what runs, and the kernels that do the math. Between them sits a part that gets less credit, the model runner. It takes a batch the scheduler has built and turns it into work on the GPU. It decides how memory is laid out for that batch, how the graphs are captured, and how the pieces of a parallel model are passed along. When the runner is full of special cases, every new feature has to be wired in one exception at a time.

vLLM's Model Runner V2 spent a long time as a second path. In v0.17.0, released March 7 with 699 commits from 272 contributors, it reached a set of milestones that had been missing. It gained pipeline parallelism, decode context parallelism, Eagle3 speculative decoding with kernel.">CUDA graphs, pooling model support, and piecewise and mixed CUDA graph capture. Each item removes a reason someone would stay on the older path. When the runner handles these natively, features that used to be exceptions become ordinary combinations, and the test matrix gets smaller.

The release also upgraded the stack to PyTorch 2.10, which the notes flag as a breaking change for environments. It also listed a known issue with a CUDA library mismatch, and the fix was to remove a system CUDA path from the library search path. That small detail is a good lesson. A runner depends on the whole stack beneath it, and the cheapest bugs often live in the environment rather than in the code.

FlashAttention 4 and the beta that never ends

Attention is the most expensive step for long contexts, and FlashAttention is the kernel most engines rely on. The original paper, by Tri Dao and colleagues, showed how to tile the computation so that the working set stays in fast on-chip memory and the slow memory is touched less often. Each generation since has chased the hardware it runs on. FlashAttention 3 targeted the Hopper generation with asynchrony and low precision.

FlashAttention 4 arrived as a series of betas.source ↗ The repository published fa4-v4.0.0.beta0 on March 5, and beta releases followed on March 5, 23, and 25, labeled up to beta 6 by the end of the month. vLLM listed FlashAttention 4 as a supported backend in v0.17.0. A beta is not a failure. It means the kernel is correct on the cases the team tested, and the users collect the failures on shapes and hardware the team did not. The honest question for you is whether your head sizes, your grouping of heads, and your cache layout are among the tested shapes.

Kernel gains are also hardware specific. A speedup measured on one GPU generation may not carry over to another. The right habit is to write down the exact shape you care about, then run the kernel on that shape before you assume the headline applies.

Splitting the work before the GPU

Picture a multimodal request. A user uploads a photo, and before the model sees a single token, a lot of work happens on the CPU. The image is decoded, resized, tiled into patches, and the prompt is tokenized with placeholders for the picture. None of this needs a GPU, but in a shared process it competes with the GPU workers for CPU time and memory. A burst of large images can stall decoding for every request sharing that process.

vLLM v0.18.0, released March 20 with 445 commits from 213 contributors, introduced vllm launch render, a way to run that preprocessing without a GPU. The render stage becomes its own service. You can scale the CPU part separately from the GPU part, and the GPU workers spend their time on the work that needs them. The price is one extra hop and a serialized handoff between the two stages. A team should measure how much of each request is spent before the GPU sees it, and split the stages only when that share is large.

The same release also added gRPC serving through a new flag, giving teams an alternative to the REST route with a binary protocol, and it moved n-gram speculative decoding onto the GPU so it can overlap with the rest of the step. The n-gram method guesses the next tokens by matching recent text earlier in the same context. It costs almost nothing to propose, which makes it attractive for code and quoted answers, and useless for open-ended creative text.

The cost of guessing

Think of speculative decoding as a student who writes an answer quickly and then has a teacher check it in one pass. The student's guess is cheap. The check is what costs time. If the teacher accepts most of the guesses, the class finishes faster. If the teacher rejects most, the student wasted effort.

For an n-gram proposer, the student is a pattern-matcher. It looks at the last few words, finds the same words earlier in the text, and copies what came next. That works on code diffs and on quoted passages, where repetition is common. It fails on creative writing. Moving that matching onto the GPU removes a wait that sat on the critical path of every step. The number to watch is not raw speed but the acceptance rate: how many guessed tokens survive the check on your traffic.

TensorRT-LLM goes stable

TensorRT-LLM v1.2.0 arrived on March 12 as a stable release, the end of a long run of candidates.source ↗ Its notes read like a catalog of progress. It added beta support for several new model families, improved validation for others, and widened the hardware list to Blackwell and Ampere variants. It also enabled multi-token prediction for DeepSeek V3.2, which lets a model guess several future tokens from its own heads.

The entry that matters most for operators is prefill and decode stages of a model on separate pools of workers, so each can be sized for its own job.">disaggregated serving. In a disaggregated deployment, prefill and decode live on separate pools of workers. The notes add a service discovery mechanism, which means the pools can grow and shrink while traffic flows. That is the difference between a demo and a service. A disaggregated setup that cannot add a worker without a restart is a fixed topology. One that can is a system you can run through a busy afternoon.

Cache that moves before you ask

LMCache v0.4.0, released March 10, marked its maturity and its move toward the multiprocess mode.source ↗ The release added an L2 prefetch controller that works with the storage manager. Prefetch is the cache version of reading ahead. If the system can guess that a block will be needed soon, it moves that block from a cold tier to a warm one before the request asks for it.

Prefetch works only when the guess is good. Consider an agent that resends its whole conversation on every turn. The next prefix is almost certain, because it is the last turn plus a few new tokens. Warming those blocks in the background pays off. A one-off question with a unique prompt gives the prefetcher nothing to predict, and the bandwidth is wasted. Before you turn prefetch on, measure how often the block you prefetched is the block that was actually needed.

Mooncake's March release, v0.3.10, added work on high availability for its master service, using etcd, and copy and move support for stored objects. A cache store that loses its master loses track of where its blocks are, which turns a speedup into a stall. Availability is part of performance in this kind of system.

Apple silicon gets its own runtime

Ollama v0.19.0, released March 27, put an MLX-based preview on Apple silicon.source ↗ MLX is Apple's machine learning framework, designed for unified memory, where the CPU and GPU share one pool. In that setting, the cost of moving data between them drops, and large weights do not need to be copied across a bus. For a laptop, that changes which models fit and how fast they feel.

The lesson for the series is about where benchmarks come from. Numbers measured on a server GPU do not transfer to a laptop. The right optimizations differ too. When you compare local and server results, compare on the same task and the same quality bar, not on tokens per second alone.

Concepts the month makes practical

March added three ideas that will return again and again in this series.

The first is the CUDA graph. Every decode step launches many small kernels, and each launch costs a little CPU time. A CUDA graph records a sequence of those launches once and replays it. The catch is that graphs need fixed shapes. Piecewise capture is the compromise: capture the stable parts of a step as graphs, and run the rest eagerly. That balance is why it became the common default.

The second is pipeline parallelism inside the runner. Once the runner knows how to pass activations between stages natively, the stages can be scheduled with the same overlap that works on a single GPU. That is the real meaning of the v0.17.0 milestone.

The third is the prefetch idea, which is the cache's version of reading ahead. It is cheap when the next access is predictable, and wasteful when it is not.

What to check this month

1. Test FlashAttention 4 on your shapes. Check head sizes and cache layout against the kernel's supported list before enabling it.
2. Upgrade the framework and the kernel as a pair. Plan PyTorch 2.10 together with any kernel change, and keep the previous environment ready.
3. Measure preprocessing share. For multimodal products, record how much time goes to rendering before the GPU sees the request.
4. Measure acceptance for n-gram speculation. Turn it off for workloads where acceptance stays low.
5. Test disaggregated scale events. Scale pools up and down under load and watch for failures in service discovery.
6. Check prefetch predictability. Measure how often prefetched blocks were actually used before tuning the policy.

Frequently asked questions

Is a beta FlashAttention kernel safe for production?

Only behind a test gate you control. Run your own accuracy and long-context tests, and keep the previous kernel ready for rollback until a stable tag is out.

Does n-gram speculation help chat assistants?

Usually less than it helps code or quoted retrieval, because open-ended text repeats less. Measure acceptance on your traffic and turn it off when the gain is not there.

Glossary

Reading links

• FlashAttention, Dao et al. (arXiv 2205.14135) The tiling idea behind modern attention kernels.
• vLLM v0.17.0 release notes Model Runner V2 milestones and the FlashAttention 4 backend.
• FlashAttention releases The fa4 beta series.

Summary: February was the month the ideas from January stopped being experiments. vLLM v0.16.0 declared asynchronous scheduling with pipeline parallelism fully supported, while SGLang v0.5.9 brought sparse attention kernels and overlapped LoRA loading to real users.source ↗

The month the exclusion list shrank

Every serious feature in this field begins with a list of things it does not support yet. In January, the most useful line in the vLLM notes was a list of configurations that asynchronous scheduling did not cover. Pipeline parallelism was on that list. Picture a team that runs its model across several GPUs in a pipeline. Each stage hands its output to the next, so the step of one stage depends on the result of the stage before. That dependency is exactly what makes overlapping CPU and GPU work hard.

On February 25, vLLM v0.16.0 removed that line. Its notes say asynchronous scheduling combined with pipeline parallelism is now fully supported, and they report a 30.8 percent end-to-end latency, or it means little.">throughput gain and a 31.8 percent improvement in decode cost.">time per output token for that setup. The release was branch-cut on February 8, so anything merged after that date is not in it. That detail is a small reminder that a release is a snapshot of a moving target.

Those percentages deserve a careful reading. They are real numbers from the notes, and they describe a particular model, a particular parallel layout, and a particular load. If your pipeline is shallower, or your prompts are shorter, you may see a smaller gain. The right move is to reproduce the setup that matches your traffic before you put the number in a planning document.

What vLLM shipped this month

vLLM had two releases in February. v0.15.1 on February 4 was a patch. v0.16.0 was the feature release, with 440 commits from 203 contributors, seven of them new. Its highlights tell a story with three threads. The first is the scheduler thread already described. The second is a WebSocket-based Realtime API for streaming audio, which builds on the streaming work that appeared in v0.15.0. A voice agent wants to keep one connection open across many turns, and a socket fits that better than a new request each time.

The third thread is quieter and matters a lot for teams that train as well as serve. The notes mention native NCCL-based weight syncing, layerwise weight reloading, and pause and resume for the engine. Reinforcement learning loops repeatedly generate answers from the current policy, then update the weights and send them back. If the engine can take new weights quickly, without a restart, a stall of several minutes becomes a short pause. In that loop, the inference engine is part of training.

SGLang and the LoRA stall

SGLang v0.5.9 arrived on February 24 with a short list of highlights, and the first one solves a problem most people never see.source ↗ A LoRA adapter is a small add-on that changes how a base model behaves. Serving many adapters on one base model is attractive, because the big weights are shared. But when an adapter is not already in memory, loading it stops the batch. Every user sharing that batch waits.

SGLang's fix is to load the adapter while the GPU is busy with other work. The notes report that this cut prefill cost plus queueing.">time to first token by roughly 78 percent and time per output token by about 35 percent on large adapters. Those are large numbers, and they describe the case where loading was a visible stall. If your adapters already live in memory, the gain is smaller. The lesson is to measure the cold load before you assume the fix applies.

The second highlight is the one that shaped the rest of the year. SGLang integrated TensorRT-LLM kernels for sparse attention in DeepSeek V3.2, and the notes claim three to five times faster performance on Blackwell hardware, with a minor accuracy drop. Sparse attention changes the story of the attention step. Instead of reading every past token, each query reads a chosen few. The cost per step falls as the context grows. The cache still has to hold everything, so the memory budget from January does not go away. What changes is how much of that cache each step must touch.

A closer look at sparse attention

Think of a long document as a library. Dense attention means that every time the model writes a word, it reads every book on every shelf. Sparse attention means it first picks a few shelves that look relevant, then reads only those. The job is faster, but the choice of shelves matters. Pick the wrong ones and the answer misses a fact that was in a book nobody opened.

That is why the accuracy note matters. A minor drop can be fine for a summary and unacceptable for extracting a contract clause. The honest test is a long-context recall check, where a fact is placed at different depths in a long input and the model is asked to find it. Speed numbers tell you the library is faster to search. Recall tells you the search still finds the right book.

The engineering challenge is memory access. Dense attention reads a contiguous block of cache. Sparse attention reads scattered entries, and GPUs prefer contiguous reads. That is why sparse support arrives as a separate kernel integration with its own tuning, instead of a single flag that works everywhere.

The fleet layer grows up

llm-d is a Kubernetes-native project for serving models across a cluster, and its v0.5.0 release, dated February 3, set four goals: reproducible benchmarking, scaling the KV cache, improving autoscaling, and a better metrics and tracing story.source ↗ It also made breaking changes to the gateway and mesh components. Read the goals together and you get a picture of what a platform team struggles with.

A single engine is limited by its own GPU memory and scheduler. A cluster is limited by its routing, its scaling, and its cache. A new replica starts with an empty cache, so its first requests are slow. Autoscaling a model server is harder than autoscaling a stateless web service, because a new replica is not useful right away. Reproducible benchmarks matter for the same reason. If the gateway version, the engine version, and the autoscaler timing all change between runs, you cannot tell which change moved the number.

Caches that become shared services

In a single engine, the KV cache belongs to one process. Across a cluster, it starts to look like a shared service. Mooncake's February release, v0.3.9, shows the direction. Its notes mention fixes to an offset allocator, support for Ascend fabric memory mode, batch publishing with tensor-parallel awareness, and a dummy client that lets SGLang's hierarchical cache use the store. Those are storage details, but they add up to one idea: cache blocks now move between processes and machines.

LMCache v0.3.14, released on February 17, added a new L1 manager for its multiprocess storage path. A multiprocess design puts the cache manager in a separate process from the engine. Several engine workers can then share one cache, and the cache can survive a worker restart. The cost is that every lookup now crosses a process boundary, and the manager has to handle locks and eviction for everyone at once.

It helps to think of the cache as a set of shelves with different distances from the cook. The hottest blocks live on the GPU, next to the stove. Warm blocks sit in CPU memory, a few steps away. Cold blocks go to disk or remote storage, a walk to the back room. A lookup searches the shelves from the nearest one outward, and a hit at a far shelf costs more time. The right layout depends on reuse, meaning how long a block waits before someone needs it again.

Ollama becomes a launcher

Ollama published thirteen releases in February, from v0.15.3 on February 1 to v0.17.4 on February 27.source ↗ The story is consistent. Version 0.16.0 on February 12 brought GLM-5 and MiniMax-M2.5 to the model library, and introduced a new ollama command for launching apps that use Ollama models. Version 0.17.0 on February 21 added automatic setup of OpenClaw through ollama launch openclaw, with web search for cloud models. It also made the tokenizer faster.

Look at the model list and you see how far local runners have come. GLM-5 is listed at 744 billion total parameters with 40 billion active. Running a model that size locally depends on offload, quantization, or cloud routing. The workload also changes. An agent tool sends repeated calls with a growing conversation, which is exactly the traffic that KV cache of a shared prompt prefix across requests, so the prefix is computed once.">prefix caching loves. A runner that serves agents has to think like a cache.

TensorRT-LLM keeps its candidate train

TensorRT-LLM kept its release-candidate cadence through February, moving through rc3 on February 12, rc4 on February 17, and rc5 on February 24.source ↗ The rc3 notes show a range of work: LoRA on BF16 checkpoints for a large FP8 model, Eagle3 speculative decoding for a hybrid model, and an integration of a new KV cache manager into the runtime. That last item is the one to watch. The cache manager is where memory policy lives, and a change there can move latency in ways no single kernel benchmark shows.

The release-candidate label is a signal too. It says the design is still being shaken out. Teams that depend on this engine should test each candidate against a replay of real traffic before they move a deployment forward. Cache behavior is often the first thing to change and the last thing a microbenchmark reveals.

The concepts this month adds

February adds three concepts to the January set, and each one shows up in a real product decision.

The first is the pipeline handoff. When stages depend on one another, a stall in one stage shows up in the next. Keeping the pipeline full is what turns a multi-GPU model from a slow chain into a steady flow. The v0.16.0 release made that flow fully supported with asynchronous scheduling, which is why its numbers matter.

The second is the adapter stall. LoRA adapters are cheap individually, but loading one on demand can freeze a batch. The fix is to overlap the load with compute, so the GPU never waits for the disk. The question to ask is how long a cold load takes on your storage, not on a developer's laptop.

The third is speculative decoding spreading to new architectures. Eagle3 support for a hybrid model shows the technique moving beyond plain transformers. The principle is unchanged: a cheap guess, checked in parallel. The practical question is the acceptance rate on your prompts, because a draft trained on general text may guess poorly on code or on domain language.

What to check this month

1. Re-test pipeline parallelism. Measure throughput and time per output token with your parallel layout, before and after the upgrade.
2. Measure adapter load time. Record how long a cold adapter takes to load from your real storage.
3. Test sparse attention for recall. Place facts at several depths in a long input and confirm the model finds them.
4. Plot scale events against GPU use. A replica that scales up with an empty cache adds cost before it adds throughput.
5. Count model calls per user action. Agent products often make several calls for one click. Add their latencies together.

Frequently asked questions

Does full support for async scheduling mean I should turn it on everywhere?

Supported means the team has tested that path and considers it safe. It does not mean it wins for every workload. Keep the default unless your own test shows a regression.

Are sparse attention kernels safe for accuracy-sensitive work?

Only after you test them on your own tasks. The note reports a minor accuracy drop, and its size depends on the task. Long-context recall is the right gate.

Glossary

Reading links

• vLLM v0.16.0 release notes Async scheduling with pipeline parallelism.
• SGLang v0.5.9 release notes LoRA loading overlap and sparse attention kernels.
• llm-d v0.5.0 release notes Cluster goals for KV cache scaling and autoscaling.

Summary: January was the month the engines started caring about idle time. vLLM made asynchronous scheduling the default, and the SGLang Model Gateway rebuilt its cache-aware router so that a request lands where its prefix already lives.source ↗

The story of a GPU that kept waiting

Picture a busy inference server on a Tuesday morning. Hundreds of users are chatting with a model, and the GPU is the most expensive thing in the room. It should be working all the time. But if you watch it closely, it is not. It runs a few kernels, then stops. It waits while the CPU decides what to do next. It runs a few more, then waits again. Each wait is short, a few milliseconds, but there are thousands of them every minute, and together they eat a real share of the machine.

That waiting is the story of January 2026. The engineers who build vLLM and SGLang had been staring at profiles like this for years. The GPU was fast. The bottleneck was the loop around it. Every decode step, the scheduler decides which requests run, builds the batch, launches the work, waits for the tokens, and processes the results. Done one after another, the CPU steps and the GPU steps take turns, and the GPU spends a surprising amount of time doing nothing.

The fix is simple to describe and hard to build. You let the CPU prepare the next step while the GPU is still busy with the current one. The engineers call this asynchronous scheduling. On January 20, vLLM v0.14.0 made it the default. Users who saw problems could turn it off, and the release notes listed the setups that were not yet covered: pipeline parallelism, the CPU backend, and some speculative decoding methods. That list is the honest part of the story. It tells you where the feature was still growing up.

Why this matters more than it sounds

Most months in this field bring a parade of new model names. January brought fewer names and more plumbing. That is the kind of change that lasts. A new model is useful for a few weeks. A change to the scheduler changes the speed of every model that runs on that engine afterward, and it keeps doing so as the models come and go.

It helps to see the cost in numbers. Suppose one decode step takes 20 milliseconds of GPU time, and the CPU work around it takes 5 milliseconds in series. The GPU sits idle for a quarter of the step. Overlap the CPU work with the GPU work, and the idle time mostly disappears. The step gets closer to 20 milliseconds, and the same hardware serves more tokens per second. Those are invented round numbers, but the shape of the gain is what the vLLM team was chasing.

The router that remembers

Now picture a fleet of ten replicas behind a load balancer. A user sends a long system prompt, and the first replica computes its KV cache for it. A minute later, a second user sends the same prompt. A simple round-robin load balancer sends that request to a different replica, which computes the whole cache again. The first replica had the answer already, and nobody asked it.

This is where the SGLang Model Gateway, called SMG in its notes, comes in. Version 0.3.1, released on January 9, rebuilt the router's cache-aware engine. It keeps a radix tree of the prefixes each replica holds, and it sends each request to the replica with the longest match. The release reports that cache insertions rose from about 18,900 to 216,000 per second, and that memory per tree node fell by roughly 99 percent. Those are router numbers. They say the router got faster at deciding. They do not say the model got faster.

The router story has a catch that every reader should keep in mind. The router learns what each replica holds from the requests it has sent. A replica can evict an entry under memory pressure and not tell anyone right away. Then the router believes a prefix is cached when it is gone, and the request pays for a full recompute. Fast routing with a stale view is still a slow answer.

When one GPU is not enough

January also brought a quieter story about very long contexts. A context of a million tokens does not fit on one GPU, so the model has to be split. Pipeline parallelism cuts the model into stages, one stage per group of layers. Chunked pipeline parallelism cuts the input into pieces that flow through the stages in order, so later stages start before earlier ones finish. SGLang's v0.5.8 release, on January 23, claimed close to linear scaling for that case. The honest question is whether the scaling holds for your context length and your hardware. The notes do not answer that for you.

The same release made its diffusion models up to 1.5 times faster, and it added a production change for a mixture-of-experts model that its notes credit with cutting prefill cost plus queueing.">time to first token by 65 percent. That last number is a good example of a claim with a setup attached. It describes one model, one serving path, and one measurement. Keep the setup next to the number when you repeat it.

What else moved in January

vLLM v0.15.0 arrived on January 29 with 335 commits from 158 contributors.source ↗ Its most useful change for hybrid models was block-aligned prefix caching for Mamba-style states, which the notes report gives about a 2 times speedup on the workloads they measured. The same release made asynchronous scheduling work with pipeline parallelism, closing one of the gaps from v0.14.0.

Ollama had a busy month of its own. Its v0.14.0 release on January 10 added an experimental command-line agent loop with a bash tool, plus compatibility with the Anthropic messages API. v0.15.0 on January 21 added ollama launch, which points coding tools such as Claude Code at local models. A local runner that starts acting like an agent platform sends a different kind of traffic: many short, repeated calls with a growing conversation. That traffic loves prefix caching.

TensorRT-LLM spent January in release candidates on its v1.2 and v1.3 lines, so the month was about direction more than stable features. Mooncake and LMCache shipped patch releases. Mooncake, described in its 2024 paper, is built around moving cache between machines. LMCache offloads cache beyond GPU memory. Both were quietly becoming the storage layer that the engines lean on.

The concepts behind the story

Every month in this series leans on the same small set of ideas, and January touches nearly all of them. Here they are as a story you can follow.

The first idea is the split between prefill and decode. A request arrives with a prompt. Prefill reads the whole prompt once and builds the cache. Decode then writes the answer one token at a time, and each token needs the whole cache read again. Prefill is a sprint through many tokens at once. Decode is a slow walk with a heavy bag. Almost every serving trick is a way to stop the two from fighting over the same resources.

The second idea is the cache itself. The KV cache is the memory of the conversation, and it is usually the largest consumer during serving. A useful way to picture it is a notebook where the model writes a note for every token it has seen. Long conversations fill the notebook fast. The PagedAttention paper, from the team behind vLLM, changed the notebook from one long page per request into small pages that can be shuffled around, much like virtual memory in an operating system. That single change let many requests share one GPU without wasting space.

The third idea is continuous batching. In the old way, a batch waited for its slowest member. A short question sat idle behind a long essay. Continuous batching lets a new request join the batch at the next step, and lets a finished one leave. Asynchronous scheduling is the next refinement: the bookkeeping for that loop now overlaps with the GPU work.

The fourth idea is prefix caching. Imagine a company support bot whose system prompt is two thousand tokens long. Every customer conversation starts with that same prompt. If the cache for that prompt is kept, each new customer skips the most expensive part of the first step. The gain depends on how often a prefix repeats and how long the cache survives before it is evicted.

The fifth idea is speculative decoding. Decode is sequential, so each token normally costs a full pass through the big model. A small draft model proposes a few tokens, the big model verifies them together, and the ones that match are kept. The output is the same as the big model would have produced, so the speedup is real, not an approximation. The gain depends on how often the draft is accepted, which is why its acceptance rate matters more than its raw speed.

A memory budget you can do on a napkin

Let's make the cache concrete. Take a model with 32 layers, 8 key-value heads, a head size of 128, and 16-bit values. Each token needs a key and a value per layer. That comes to roughly 128 kilobytes of cache per token. A single request with a 32,000-token context holds about 4 gigabytes. Ten such requests at once need about 40 gigabytes before the weights, the activations, and the fragmentation are counted.

That arithmetic explains a lot. It explains why a server admits requests by free cache blocks instead of by how many requests it has. It explains why a shared 20,000-token system prompt is worth caching: reused across many concurrent requests, it saves gigabytes. It explains why an 8-bit cache holds twice the tokens of a 16-bit one. When a release note says it reduced memory per request, convert that into tokens per GPU. That is the number that sets your cost.

What to check this month

1. Measure your idle gap. Profile a decode step and see how much time the GPU waits between kernel batches.
2. Check your exact combination. Speculative decoding with pipeline parallelism was on vLLM's exclusion list in v0.14.0. Confirm your setup is covered before you upgrade.
3. Watch the strict sampling change. vLLM v0.14.0 makes unsupported sampling parameters fail under speculative decoding instead of ignoring them.
4. Replay routing on real traffic. Compare round-robin with a prefix-aware policy using realized cache hit rates and time to first token.
5. Pin your engine versions. Track the latest release in staging, and promote to production on a schedule.

Frequently asked questions

Is asynchronous scheduling always faster?

No. The gain is largest when CPU scheduling time is a real share of each step, which happens at moderate batch sizes and fast models. With very large batches the GPU is already saturated, and there is little idle time to recover.

Do I need the SGLang gateway for cache-aware routing?

You need some router that knows where prefixes are cached. The gateway is one implementation. A simpler approach hashes a stable prompt prefix to a fixed replica, which needs no tracking but cannot adapt when one prefix becomes very popular.

Glossary

Reading links

• Efficient Memory Management for LLM Serving with PagedAttention (arXiv 2309.06180) The paper behind vLLM's memory design. Read first.
• vLLM v0.14.0 release notes Async scheduling default and its exclusion list.
• SGLang releases Gateway (SMG v0.3.1) and runtime notes.