Kimi K3 — the architecture and infrastructure, explained.

Moonshot AI's Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with native vision and a 1-million-token context window — and, per Moonshot, the largest open-weights release ever (weights out July 27, 2026). The headline numbers are big, but the interesting part is how it stays affordable at that size: only 16 of 896 experts fire per token, the attention layer is mostly Kimi Delta Attention (a linear-attention variant that keeps a fixed-size memory instead of an ever-growing KV cache), weights ship in MXFP4 from quantization-aware training, and serving runs on 64+-accelerator supernodes with a prefill cache hitting >90% in coding workloads — which is how cache-hit input lands at $0.30 per million tokens. This post walks through the architecture and the infrastructure in plain language, building each concept from the basics first.

The three basics you need first

Three ideas carry this whole article. If you know them, skip ahead; if not, two minutes here makes everything else readable.

1. Mixture of Experts — a big model that only wakes up a little

A dense transformer runs every parameter for every token — a 2.8T dense model would be absurdly expensive to run. A Mixture-of-Experts (MoE) model replaces each feed-forward block with a set of parallel "experts" (small feed-forward networks) plus a router that picks a handful of them per token. Total parameters measure what the model knows (capacity); active parameters measure what you pay per token (compute). K3's bet: 2.8T of capacity, a tiny active slice per token — 16 experts chosen out of 896.

2. The KV cache — why long context is usually expensive

Standard attention lets each new token look back at every previous token. To avoid recomputing the past, inference keeps every previous token's key/value vectors in GPU memory — the KV cache. It grows linearly with context length: at 1M tokens it becomes the dominant memory cost, and every generated token must scan it, so decoding slows as context grows. This is why "1M context" claims are cheap to print and expensive to serve — and why K3's attention layer is the most consequential design choice in the model.

3. Linear attention — a fixed-size memory instead of a growing one

Linear attention variants replace the look-at-everything mechanism with a running, fixed-size state that is updated as each token arrives — like keeping a continuously-revised summary instead of a full transcript. Cost per token stays constant regardless of context length. The catch: a fixed-size state must overwrite something to learn something new, so naive linear attention forgets — the exact trade the HOLA paper in Digest #3 attacks from another angle. The delta rule family fixes much of this by updating the state only by the error between what the state predicts and what actually arrived — writing the new, not rewriting everything.

K3 by the numbers, next to K2

Dimension	Kimi K2	Kimi K3
Total parameters	~1T	2.8T
Experts	384 routed + 1 shared	896, 16 active per token (Stable LatentMoE)
Attention	MLA (K2) / KDA hybrid (Kimi Linear line)	KDA + Attention Residuals + Gated MLA
Context window	256K class	1M tokens
Modality	Text	Native vision + text
Scaling efficiency	baseline	~2.5× better (per Moonshot)
Weights format	bf16/int lines	MXFP4 weights, MXFP8 activations (QAT)

The MoE design: 16 of 896, and why sparser won

K2 already showed Moonshot's preference for wide sparsity — 384 experts with very few active. K3 pushes the same axis harder: 896 experts, 16 active, under a framework they call Stable LatentMoE. The intuition for why sparser-and-wider keeps winning: every expert added grows what the model can store, while the per-token bill only depends on how many fire. As long as the router gets better at picking the right 16, capacity is nearly free at inference time.

That "as long as" is where MoE designs live or die, and two named K3 components target it directly:

• Quantile Balancing — expert load-balancing derived from the quantiles of router scores rather than blunt auxiliary losses. The classic MoE failure is collapse: a few popular experts absorb all traffic while the rest atrophy. Balancing on score quantiles pushes traffic toward even utilization without distorting what the router actually wants to choose.
• Sigmoid Tanh Unit (SiTU) — a gated activation used for finer control over what flows through, part of the stability story that lets a 2.8T model with this much sparsity train without the router oscillating.

Kimi Delta Attention: the load-bearing choice

KDA is not new with K3 — it debuted in Moonshot's Kimi Linear research line (the published 48B-A3B models and the Kimi Linear paper), which is exactly why it's credible at this scale: K3 is the production bet on an architecture they de-risked in the open first.

Building on the linear-attention basics above, KDA's specific contributions:

• Channel-wise gated delta rule. Plain delta-rule attention decides how much to forget with one gate per attention head. KDA gates per channel — each dimension of the state can independently hold or decay its memory. Fine-grained forgetting means the state keeps what matters longer, which is where linear attention usually loses to full attention on recall.
• Hardware-shaped computation. KDA's update is organized blockwise and context-parallel so it maps onto GPU matrix units efficiently — an architecture designed around the accelerator, not just around the math.
• Constant-size state ⇒ long-context economics. No million-token KV cache to store and scan. Decoding throughput stays roughly flat as context grows — the property that makes a 1M-token window servable rather than just advertisable.

How the delta rule actually works (the one piece of math worth it)

Skip this subsection if formulas aren't your thing — the rest of the article doesn't depend on it. But it's the mechanism everything else rests on, so here it is in the gentlest form. A linear-attention layer carries a state matrix S — think of it as an associative memory: give it a key, it returns the value it associated with a similar key. As each token arrives with its own key k and value v, the state must be updated. Naive linear attention just adds the new association: \( S \leftarrow S + v\,k^{\top} \). The problem: additions never subtract, so old and new associations pile on top of each other and blur.

The delta rule is smarter. Before writing, it asks what the state already predicts for this key — call it \( \hat{v} = S\,k \) — and writes only the correction:

\[ S \leftarrow S - \beta\,(S k - v)\,k^{\top} \]

If the state already knew the right value, \( S k - v \approx 0 \) and almost nothing is written — no wasted capacity. If it was wrong, the error is exactly what gets stored. This is the same "learn from the surprise" idea as gradient descent, running inside the forward pass, once per token. KDA's contribution is the \( \beta \): instead of one forget-strength per head, KDA makes it a vector with one entry per channel — replace the scalar with a diagonal gate \( \operatorname{diag}(\beta) \), so each feature dimension decays independently. Some dimensions can hold a fact for a million tokens while others refresh every few tokens — fine-grained memory management that a single scalar gate can't express. That channel-wise gating is the single biggest reason KDA recalls long-range facts that ordinary linear attention forgets.

The new architecture, component by component

K3's launch post names eight building blocks that are either new or freshly combined. Here is what each one actually does, in the order a token meets them. The one-line version lives in the table; the prose after it is the "not just a heads-up" detail.

Component	Layer it lives in	What it changes
Kimi Delta Attention	most attention layers	Linear attention with a channel-wise gated delta rule — constant-size memory, long recall
Gated MLA	periodic attention layers	Full multi-head latent attention, gated — kept for exact recall the linear state can't do
Attention Residuals	across depth	Layers selectively retrieve earlier-depth representations instead of blindly summing them
Stable LatentMoE	every feed-forward block	The framework that keeps 896-expert / 16-active routing stable while training
Quantile Balancing	the router	Load-balances experts by router-score quantiles, preventing expert collapse
Sigmoid Tanh Unit (SiTU)	activations	A gated activation for finer control of what passes through
Per-Head Muon	the optimizer	Moonshot's Muon optimizer applied independently per attention head
MXFP4 QAT	whole model, from SFT	Trains the model to live in 4-bit weights so the served model isn't a degraded copy

Gated MLA — the exact-recall escape hatch

A fixed-size state is a lossy summary. For most of what a language model does — tracking topic, style, general facts — lossy is fine. But some tasks need exact lookup: "what was the variable named on line 40,000," "quote that clause verbatim." A compressed state genuinely cannot guarantee that; only real attention over the actual tokens can. So K3 does not go all-linear. It interleaves: most layers are KDA, and every few layers there is a Gated MLA layer — full Multi-head Latent Attention (the KV-compressed full attention K2 already used) with an added gate that lets the layer modulate how much of that exact-recall signal it admits. In the published Kimi Linear line this ran at roughly a 3-to-1 ratio (three linear layers per full-attention layer); K3 builds on that hybrid rather than replacing it. The result is a model that pays linear-attention prices on the vast majority of layers while keeping a thin backbone of true attention for the moments exactness is non-negotiable.

Attention Residuals — depth without drowning

A transformer's residual stream is the running sum every layer reads from and writes back to. It works beautifully for tens of layers. At the depth a 2.8T model runs, the naive "every layer adds its output to the same stream" starts to hurt: early, sharp signals get buried under hundreds of later additions, and deep layers see a muddy average of everything. Attention Residuals change the read side — a layer can selectively retrieve representations from specific earlier depths rather than only seeing the uniform accumulated sum. Concretely, it gives deep layers a way to reach back to a clean earlier representation when that is what they need, instead of reconstructing it from the pile. It is the depth-axis cousin of what KDA does on the sequence axis: replace "sum everything equally" with "fetch what's relevant." Both are the same design instinct — selective retrieval beats uniform accumulation once scale makes the pile too big.

Stable LatentMoE, Quantile Balancing & SiTU — keeping 896 experts honest

Pushing sparsity this far (16 of 896) is unstable by default: the router and experts co-adapt, a few experts win early, traffic concentrates, and the losers never get enough gradient to become useful — expert collapse. The classic fix is an auxiliary load-balancing loss that punishes imbalance, but a heavy one distorts the router into spreading tokens it didn't actually want to spread, costing quality. K3's three-part answer:

• Stable LatentMoE is the overall framework that makes very-wide, very-sparse routing trainable at 2.8T scale — the scaffolding the other two hang on.
• Quantile Balancing derives the balancing signal from the quantiles of the router's own scores rather than a blunt penalty. It nudges the distribution toward even utilization while respecting the router's ranking — balance without lobotomizing the router's preferences.
• SiTU (Sigmoid Tanh Unit) is a gated activation — a sigmoid gate times a tanh transform — giving each unit finer control over how much signal passes. In a model this sparse, activation stability is part of what stops the routing from oscillating during training.

Per-Head Muon — the optimizer detail

Training is where big MoE models actually break, and the optimizer matters. Moonshot has used the Muon optimizer since K2 (Muon updates weight matrices using their spectral structure rather than treating every parameter independently, which tends to train large models more efficiently than Adam). Per-Head Muon applies it independently to each attention head, so a head that needs a different effective step size gets one instead of being averaged in with its neighbors. It is a small-sounding change with an outsized role in the "~2.5× scaling efficiency vs K2" claim — that number is a product of architecture and training-method gains, and the optimizer is squarely in the training half.

Native vision and the 1M window

K3 is natively multimodal — vision is trained in, not bolted on through a separate encoder-adapter released later. Combined with the 1M-token window, the practical shape of the model is: entire codebases, hours of transcripts, or document piles plus screenshots and figures in one context. The launch post leans on this for the agentic demos — video editing with frame-accurate cuts, dashboards generated from raw data — tasks where text-only models need lossy detours.

Training infrastructure: quantization-aware from SFT onward

The most operationally interesting training decision: K3 runs quantization-aware training (QAT) from the supervised fine-tuning stage onward, targeting MXFP4 weights with MXFP8 activations.

Basics, briefly: models train in 16-bit precision, but serving in 4-bit cuts memory ~4× and speeds inference. The usual route — quantize after training — always costs some quality, because the model never learned to live inside 4-bit precision. QAT flips this: the model trains with quantization in the loop, learning weights that are already robust to the low-precision grid. MX (microscaling) formats make this practical by attaching a shared scale to each small block of values, so 4-bit numbers cover a usable dynamic range — and they're the formats current accelerator generations execute natively.

• Why it matters at 2.8T: in MXFP4, the weights are ~4× smaller than bf16 — the difference between "needs a football field of accelerators" and "fits a supernode." The 4-bit release is the model, not a degraded copy.
• "Broad hardware compatibility" is Moonshot's stated goal — MXFP4/MXFP8 are open microscaling standards adopted across vendors, which matters for a lab whose accelerator supply has geopolitical constraints.
• ~2.5× scaling efficiency vs K2 is the compound claim: the architecture (sparser MoE + KDA), the data recipe, and the training-method changes together yield ~2.5× more capability per unit of training compute. That multiplier, not any single benchmark, is the strategic number in the launch post.

Serving infrastructure: supernodes and the prefill cache

Moonshot recommends serving K3 on supernode configurations of 64 or more accelerators — a tightly-coupled, high-bandwidth-interconnect domain, in the spirit of NVL72-class rack systems or comparable scale-up fabrics. Why the model wants that shape:

• Expert parallelism: 896 experts spread across the node; every token's hidden state must reach its 16 chosen experts and return. That all-to-all exchange is the MoE serving tax, and it's only cheap inside one fast interconnect domain — cross the node boundary and latency eats the sparsity win.
• Even in MXFP4, 2.8T ≈ 1.4TB of weights — far beyond one accelerator's memory, so the weights physically need the node.

The second pillar is the prefill cache riding on KDA. Agentic workloads resend nearly-identical context every turn — system prompt, repo state, conversation so far. Caching the computed prefill state means a returning request only pays for the new suffix. Because KDA's state is compact (no giant per-token KV tensors), cached prefill states are cheap to keep and restore at 1M-token scale. Moonshot reports the official API's cache hit rate above 90% in coding workloads — and prices accordingly:

Meter	Price / MTok	Read
Input, cache hit	$0.30	the price agentic loops mostly pay (>90% hits in coding)
Input, cache miss	$3.00	10× the hit price — cache design is now product design
Output	$15.00	frontier-tier output pricing

What it does in practice — and the honest limits

The launch post's demos target the agentic frontier: K3 built a MiniTriton compiler from scratch with performance on par with or better than Triton, optimized GPU kernels competitively with Claude Fable 5, and produced a complete 45nm chip design in a single 48-hour autonomous run (with an 8,700 tokens/s decode figure attached to that workload). Benchmarks put it at 88.3 on Terminal-Bench 2.1 (GPT-5.6 Sol: 88.8), 67.5 on DeepSWE (Sol: 73.0), 42.0 on SWE Marathon (ahead of Fable 5's 35.0), 93.5 GPQA-Diamond, and strong multimodal scores (81.6 MMMU-Pro, 94.3 MathVision).

Unusually for a launch post, Moonshot lists limits plainly: sensitivity to thinking history (the harness must manage reasoning traces carefully — mishandled history degrades it), excessive proactiveness (it may take unrequested decisions mid-task), and a "noticeable gap in user experience" versus Claude Fable 5 and GPT-5.6 Sol despite the benchmark parity. That candor tracks exactly with Digest #3's Harness Effect finding: the model is the engine, and Moonshot is telling you the engine needs a good car.

FAQ

Is Kimi K3 actually open?

Full model weights released July 27, 2026 — at 2.8T parameters, the largest open-weights release to date. "Open weights" means you can download and run it (on a 64+-accelerator supernode…), not that the training data or full recipe is open.

What's genuinely new here vs K2?

Scale (2.8T vs ~1T), much wider sparsity (896 vs 385 experts), the KDA + AttnRes + Gated MLA attention stack replacing K2's MLA, native vision, the 1M window, and MXFP4 quantization-aware training. The Muon-optimizer lineage and the wide-sparse-MoE philosophy carry over.

Why does linear attention matter more than the parameter count?

Because the parameter count sets a one-time hardware bar, but attention sets the per-request economics. A 1M-token window with a growing KV cache is mostly a demo; with KDA's constant-size state plus prefill caching it's a $0.30/MTok product surface.

Can I run it locally?

Realistically, no — ~1.4TB of MXFP4 weights and an all-to-all expert exchange want a supernode. The open weights matter for labs, clouds, and researchers (distillation, fine-tunes, serving providers), not laptops.

Should I trust the benchmark table?

Treat it like every launch table: vendor-run, snapshot-in-time, own harness. The credible core is the architecture lineage (KDA is published research with open smaller models) and the pricing, which is a falsifiable public commitment.

Takeaways

• K3 is an economics design, not just a scale flex: every headline component — 16-of-896 sparsity, KDA's fixed-size state, MXFP4 QAT, the prefill cache — exists to make 2.8T parameters and 1M context servable at a price.
• KDA is the load-bearing choice: a channel-wise gated delta-rule linear attention, de-risked publicly in the Kimi Linear line, now carrying a frontier flagship — the strongest production endorsement linear attention has ever had.
• Quantization moved into training: QAT from SFT onward means the 4-bit model is the model — the open-weights release and the served product are the same artifact.
• The pricing teaches the architecture: $0.30 hit / $3.00 miss / $15 out is the whole infrastructure story compressed into three numbers — build agents that keep prefixes stable.
• Moonshot's own caveats are the roadmap: thinking-history sensitivity, over-proactiveness, and a UX gap vs Fable 5/GPT-5.6 Sol — capability arrived; harness polish is the open front.

References & further reading

• Moonshot AI — Kimi K3 launch post — the primary source for every spec, benchmark, and pricing figure here.
• Kimi Linear: An Expressive, Efficient Attention Architecture — the published research behind KDA.
• Kimi-Linear-48B-A3B-Instruct — the open smaller model where the KDA hybrid was de-risked.
• Kimi API — K3 quickstart — model id kimi-k3, thinking-effort modes, API details.
• cvam.sight — Frontier Digest #3 — the Harness Effect context for K3's own harness caveats.
• cvam.sight — PagedAttention deep dive — the KV-cache problem KDA sidesteps, explained at length.