GLM-5.3-Flash ("Ox Alpha") — the architecture, explained for graduate students.

For a week in August 2026, a model called "Ox Alpha" ran anonymously on OpenRouter and OpenCode, and the LLM community spent that week reverse-engineering it from tokenizer fingerprints and output style alone. On August 26, Z.ai (Zhipu) confirmed it: Ox Alpha is GLM-5.3-Flash, a 320B-parameter (18B active), natively multimodal Mixture-of-Experts model, released under the MIT license, served throughout its stealth week entirely on domestically-produced Chinese AI chips. Architecturally it is the most interesting open release since Kimi Linear: it replaces most of its predecessor's MLA/DSA attention layers with Kimi Delta Attention (KDA), keeps a thin backbone of sparse full attention, and wraps every sublayer in Manifold-Constrained Hyper-Connections (mHC) — four parallel residual streams held numerically stable by a matrix-scaling algorithm borrowed from optimal transport. This article works through every one of those components from first principles, grounded in Z.ai's own GLM-5 technical report and the primary papers behind KDA, DSA, and mHC — not just the marketing numbers.

1. The Ox Alpha reveal, and why it matters as a data point

Stealth model launches on OpenRouter have become a recognizable release pattern in 2026: a lab ships a model anonymously under a codename, watches how it performs against real traffic and community scrutiny with no brand attached to the number, then reveals it once confident. Ox Alpha's week fits the pattern precisely — strong agentic coding results, a distinctive tokenizer, and infrastructure fingerprints (serving latency profile, error message formats) that the community used to narrow the candidate list before Z.ai's own confirmation closed the question.

Two claims from the reveal are worth treating as separate facts, because they carry different evidentiary weight: the architecture and benchmark numbers are Z.ai's own technical disclosure (verifiable against the released weights and code), while "served entirely on domestically-produced Chinese AI chips throughout the stealth week" is a claim about infrastructure that is far harder for an outside party to verify independently — plausible given Z.ai's documented multi-chip-vendor strategy (§8), but stated here as a reported claim, not confirmed telemetry.

2. GLM-5.3-Flash by the numbers

Property	Value
Total / active parameters	320B / 18B (5.6% active)
Decoder layers	45 (text decoder) + 24-layer vision encoder
Layer mix	34 KDA + 11 MLA/DSA + 1 MTP layer (first 3 blocks use dense SwiGLU FFNs, not MoE)
Attention head count	64
Embedding dimension	4,096
Vocabulary	154,880 (~155k)
Context window	1,048,576 tokens
MoE experts	288 total — 1 shared + 8 routed active per token
FFN intermediate dim	2,048 per expert
mHC residual streams	4 parallel streams
KV cache per token (bf16)	≈11 KiB — very low for this scale
Weights	native FP8, 306 GiB checkpoint
Pre-training corpus	30T multimodal tokens
License	MIT
Released	2026-08-26 (as GLM-5.3-Flash; stealth-launched as "Ox Alpha" the week prior)

3. Where this sits relative to GLM-5 and GLM-5.3

GLM-5.3-Flash is a distinct branch from the dense-attention GLM-5.3 text model this site covered separately (GLM-5.3: Frontier Coding with Emergent Cyber Capabilities) — that article's subject uses the MLA+DSA attention stack described in Z.ai's GLM-5 technical report with no linear-attention layers and no mHC. GLM-5.3-Flash is the natively multimodal, KDA-hybrid, efficiency-first sibling, built for high-throughput serving rather than peak single-query capability. Z.ai's own comparison states the Flash architecture reduces per-token attention computation by ≈3.0× and average KV-cache size by ≈4.4× relative to GLM-5.3 — the central engineering trade this article explains mechanism-by-mechanism.

The base for both lines is the GLM-5 technical report ("GLM-5: from Vibe Coding to Agentic Engineering," Zhipu AI & Tsinghua University), which establishes the MLA design, the Muon-Split optimizer fix, DSA's continued-pretraining recipe, and multi-token prediction with parameter sharing — all of which GLM-5.3-Flash inherits and then partially replaces with newer components. Sections 4–7 below work through each layer type using that report's own reported numbers wherever GLM-5.3-Flash reuses the mechanism unchanged, and the relevant external papers where GLM-5.3-Flash's own additions (KDA, mHC) come from further afield.

4. Multi-head Latent Attention (MLA), NoPE, and why it needed a Muon fix

MLA is the attention mechanism GLM-5.3-Flash's remaining 11 non-KDA layers use. The idea, introduced by DeepSeek-V2 and adopted broadly since: instead of caching full per-head key/value vectors, project keys and values down into a single shared low-rank latent vector per token, cache that (much smaller), and up-project back to per-head K/V only at attention-computation time. This is what makes MLA's KV cache dramatically smaller than standard multi-head attention (MHA) or even grouped-query attention (GQA) at matched quality — the entire point of using it in a serving-optimized model.

Two details from the GLM-5 report are worth carrying into a graduate reading, because they are genuine engineering findings, not just architecture description:

• MLA alone underperforms GQA-8 under the Muon optimizer. Z.ai reports that with a 576-dimension latent KV cache, plain MLA could not match an 8-query-group GQA baseline on their evaluation suite when trained with Muon. Muon orthogonalizes weight-update matrices; applied naively to MLA's larger, shared up-projection matrices \((W^{UQ}, W^{UK}, W^{UV})\), it apparently could not condition the different attention heads' effective learning rates correctly.
• The fix — "Muon Split" — is architectural, not just a hyperparameter change. Splitting each shared up-projection matrix into smaller per-head matrices before applying Muon's orthogonalization lets different heads update at different effective scales. Z.ai's own ablation (GLM-5 report, Table 1) shows this closes the gap: MLA + Muon Split matches or beats GQA-8 across HellaSwag, MMLU, C-Eval, RACE, and GSM8K, while plain MLA trails GQA-8 by roughly 5–6 points on GSM8K and HumanEval.

NoPE in MLA's Q/K projections (visible in the diagram as a separate annotated component) means these specific layers apply no rotary or learned positional encoding to the query/key vectors before the attention dot product — the model relies entirely on the KDA layers' inherent sequential, recurrent structure (§5) and on RoPE-bearing layers elsewhere in the stack (where present) to carry positional information. NoPE is a documented, if less common, design choice in the wider literature: some architectures find that at long context lengths, RoPE's periodicity can itself become a length-generalization liability, and removing position information from selected layers — especially ones that mostly attend to very recent tokens or rely on other layers for long-range signal — is a stability trade some hybrid architectures accept deliberately rather than by omission.

5. DeepSeek Sparse Attention (DSA): the lightning indexer

DSA is the mechanism that makes GLM-5.3-Flash's MLA layers sparse rather than fully dense, and it is worth explaining as a two-stage pipeline rather than a single trick, because that is exactly how the original DeepSeek-V3.2-Exp paper structures it:

1. The lightning indexer — a small, cheap neural network trained to predict, for each query token, which key-value blocks the full dense-attention computation would actually attend to most strongly. It is explicitly a lightweight approximation model, trained (in DeepSeek's original recipe) via distillation from a dense-attention teacher over roughly 2.1B tokens.
2. Top-\(k\) block selection and sparse attention — using the indexer's predictions, only the top-ranked KV blocks are materialized and attended to densely; the rest are skipped entirely. This is a retrieve-then-attend pattern, converting the attention computation's complexity from \(O(L^2)\) to roughly \(O(L \cdot k)\) for a fixed per-query budget \(k\), where \(L\) is sequence length.

Z.ai's own ablation is unusually candid about the accuracy cost: on their long-context benchmark suite (MQ-NIAH, MV-NIAH, SQuAD, HotpotQA, all at 128K context), DSA scores 100.0 / 97.0 / 86.0 / 63.0 against dense MLA's 100.0 / 95.5 / 79.7 / 66.3 — DSA actually wins on three of four, and only trails on HotpotQA-128k (63.0 vs. 66.3), a multi-hop reasoning benchmark where losing some genuinely-relevant-but-low-scored context plausibly hurts more. The overall picture the report argues for: roughly 90% of attention entries in long contexts are redundant, and DSA is close to lossless at recovering the 10% that matter, at 1.5–2× less attention compute for long sequences.

5.1 IndexPool — compressing the indexer's own cache at 1M-token scale

A subtlety that matters specifically at GLM-5.3-Flash's 1,048,576-token context: the lightning indexer itself needs a cache of index key vectors to score against, and at a million tokens, that cache becomes a new bottleneck — the exact "who watches the watchmen" problem sparse-retrieval systems run into once the retrieval index itself gets large. IndexPool is GLM-5.3-Flash's answer: it compresses groups of cached indexer key vectors via weighted pooling — reported at a 4:1 ratio (four cached index vectors pooled into one) at the 1M-context operating point — trading a small amount of indexer resolution for materially lower indexer memory and lookup latency at extreme context lengths.

6. Kimi Delta Attention (KDA) — the layer that does most of the work

34 of GLM-5.3-Flash's 45 layers are KDA, not MLA/DSA — this is the layer type that actually dominates the model's compute and memory profile, and it comes from a different lineage than DSA/MLA entirely: Moonshot AI's Kimi Linear research line, most recently productized at frontier scale in Kimi K3 (covered in depth in this site's Kimi K3 architecture article).

6.1 The delta rule, briefly

KDA belongs to the family of linear attention mechanisms: instead of attending over the full history of past tokens (the mechanism behind MLA's KV cache), it maintains a fixed-size state matrix \(S\) — an associative memory updated once per token — and pays constant compute and memory per generated token regardless of sequence length. The naive update rule, \(S \leftarrow S + v k^{\top}\), just accumulates outer products, and accumulation without subtraction means old and new associations blur together as the sequence grows. The delta rule fixes this by writing only the prediction error:

\[ S \leftarrow S - \beta\,(Sk - v)\,k^{\top} \]

If the state already predicts the right value for key \(k\), \(Sk - v \approx 0\) and almost nothing is written; if it was wrong, the error is exactly what gets stored — the same "learn from the surprise" principle as gradient descent, applied once per token inside the forward pass. KDA's specific contribution is making \(\beta\) a per-channel vector rather than one scalar per head — a channel-wise gated delta rule — so individual feature dimensions can independently hold information for very long spans while others refresh rapidly, which is precisely the fine-grained forgetting control that lets a fixed-size state approach full-attention recall quality on many tasks.

6.2 Why 34:11 (≈3:1), not all-KDA

A fixed-size linear-attention state is a lossy summary — adequate for most of what a language model tracks (topic, style, diffuse context) but structurally incapable of guaranteeing exact retrieval of one specific fact from deep in a million-token context, the way real attention over the actual tokens can. GLM-5.3-Flash's design keeps a thin backbone of the sparse MLA/DSA layers specifically for that exact-recall capability, interleaved through the stack at roughly one MLA/DSA layer for every three KDA layers — the same hybrid ratio the Kimi Linear research line established as its working point, now validated a second time by an independent lab (Z.ai) making the same architectural bet.

6.3 ShortConv — local mixing before the recurrent state

Linear-attention and state-space hybrids commonly pair the recurrent state update with a small, cheap short causal convolution over a handful of adjacent tokens (typically kernel width 3–4) applied to the query/key/value projections before they enter the recurrence — the same pattern used in Based, Griffin, and Mamba-family hybrids. The intuition: a fixed-size recurrent state is well-suited to carrying long-range, slowly-changing information but is a comparatively expensive way to represent "the previous three tokens" — a cheap local convolution handles that immediate n-gram-like structure directly, leaving the KDA state free to specialize in genuinely long-range dependencies rather than spending capacity re-deriving local patterns at every step.

7. Manifold-Constrained Hyper-Connections (mHC)

7.1 The problem: one residual stream is a bottleneck

The standard transformer residual connection, \(x_{l+1} = x_l + F_l(x_l)\), routes every layer's output through a single shared pathway. ByteDance's original Hyper-Connections paper (Zhu et al., ICLR 2025) proposed widening this into \(n\) parallel residual streams with a learned mixing matrix controlling how each layer reads from and writes to each stream — strictly more expressive than a single stream, in principle letting the network route different kinds of information along different pathways and dynamically reweight how much each layer's output propagates forward.

7.2 Why unconstrained Hyper-Connections breaks at scale

The catch, reported directly by DeepSeek's follow-up work: an unconstrained learned mixing matrix has no guarantee of preserving signal magnitude across depth. At 27B parameters, DeepSeek reports unconstrained Hyper-Connections produced signal amplification exceeding 3,000× through the network, causing complete training divergence — the multi-stream generalization reintroduces, in a more dangerous form, exactly the exploding/vanishing-signal problem identity residual connections were originally invented to solve.

7.3 The fix: constrain the mixing matrix to the Birkhoff polytope

mHC's answer is to stop treating the per-layer mixing matrix as a free parameter and instead force it to be a doubly stochastic matrix — every row and every column sums to 1, with all entries non-negative. The set of all such matrices is the Birkhoff polytope, and a doubly-stochastic mixing matrix is guaranteed, by construction, to preserve the total signal magnitude flowing through the residual streams: it can redistribute signal among streams but cannot amplify or attenuate the total. The matrix is produced via the Sinkhorn–Knopp algorithm — an iterative row/column-normalization procedure, originally developed for optimal-transport problems, that projects an arbitrary non-negative matrix onto the Birkhoff polytope by alternately rescaling rows and columns until both converge to sum-to-1.

\[ \text{Sinkhorn–Knopp: } M^{(t+1)} = D_r^{(t)} M^{(t)} D_c^{(t)}, \quad \text{iterated until row- and column-sums} \to 1 \]

GLM-5.3-Flash uses 4 parallel residual streams. Each attention or MoE sublayer is preceded and followed by an mhC mixer block (visible in the architecture diagram as the constrained-mixing layers wrapping each attention/MoE sublayer) that reads from and writes back to all 4 streams through a Sinkhorn-Knopp-constrained matrix, rather than the plain single-stream \(x + F(x)\) update. DeepSeek's reported overhead for 4 parallel lanes is modest — roughly 6.7% additional training time — for the reported benefit of stable scaling that a naive multi-stream design cannot achieve.

8. The MoE layer: 288 experts, 1+8 active

Each KDA and MLA/DSA block (except the first 3, which use dense SwiGLU FFNs) ends in an MoE feed-forward layer: a router scores all 288 experts for each token, and the token's hidden state is processed by exactly 1 shared expert (always active, capturing common cross-token patterns) plus the 8 highest-scoring routed experts, each expert an independent SwiGLU FFN with a 2,048-dimension intermediate layer. Total active feed-forward compute per token comes from 9 of 288 experts — the arithmetic behind the model's headline 5.6% active-parameter ratio (18B / 320B).

9. Multi-token prediction with parameter sharing

The final layer is a single MTP (multi-token prediction) layer — a draft head trained to predict more than one future token per forward pass, used at inference time for speculative decoding. The GLM-5 report identifies a real design tension here: naively, predicting \(n\) tokens ahead needs \(n\) separate MTP layers (as in DeepSeek-V3's original recipe), and both the memory footprint and the KV-cache cost of the draft model scale linearly with \(n\). GLM's fix is parameter sharing — train a single MTP layer and reuse its parameters across multiple speculative steps rather than instantiating a separate layer per step, which keeps the draft model's memory cost constant instead of growing with speculation depth.

The reported trade normally expected from this simplification is a lower speculative acceptance rate (draft/target mismatch grows across shared-parameter steps) — but Z.ai's own measurement (GLM-5 report, Table 2) shows GLM-5's shared-parameter MTP achieves a 2.76 average accept length at 4 speculative steps, actually higher than DeepSeek-V3.2's 2.55 with per-step dedicated layers — evidence that the training-time/inference-time mismatch from single-layer training (predicting only the next-2-tokens during training, per DeepSeek's original recipe) is a more significant source of acceptance-rate loss than the parameter-sharing simplification itself.

10. RMSNorm, briefly

Every sublayer in the diagram is bracketed by RMSNorm (root-mean-square layer normalization) rather than LayerNorm — a now-standard simplification that normalizes activations by their root-mean-square magnitude alone, \(\hat{x} = x / \sqrt{\tfrac{1}{d}\sum_i x_i^2 + \epsilon}\), without separately centering the mean the way LayerNorm does. It is cheaper (no mean computation or re-centering step) and empirically matches LayerNorm's training stability for transformer-scale models, which is why essentially every modern open LLM — GLM, Kimi, DeepSeek, Llama since v2 — uses it as the default normalization layer. Nothing GLM-5.3-Flash-specific happens here; it is included in the architecture diagram, and in this glossary, because "RMSNorm" is exactly the kind of load-bearing-but-invisible component a careful reading should not skip past without naming.

11. The (omitted) vision encoder — natively multimodal, briefly

The architecture diagram this article works from explicitly notes it shows the text decoder only; GLM-5.3-Flash also ships a 24-layer vision encoder, omitted from the diagram and from the parameter/layer counts in §2's text-decoder-focused table. "Natively multimodal" here means the vision encoder was trained jointly within the 30T-token multimodal pre-training corpus rather than bolted on via a later adapter stage — the same design philosophy Kimi K3 and other 2026-generation frontier releases have converged on, discussed in that article's native-vision section.

12. KV-cache and serving economics

The ≈11 KiB/token bf16 KV-cache figure is the single number that most directly explains why this architecture exists. Compare it to what a comparably-sized dense-attention model would need:

Mechanism	KV cache / token (illustrative, bf16)	Scaling with context
Standard MHA (64 heads, no compression)	hundreds of KiB, model-size-dependent	linear in context length, full head count
GQA-8 (grouped-query, 8 groups)	a fraction of MHA, still linear	linear in context, reduced constant
MLA (dense, all layers)	substantially below GQA-8	linear in context, low-rank constant
GLM-5.3-Flash (34 KDA + 11 MLA/DSA hybrid)	≈11 KiB	KDA layers: constant in context; only the 11 MLA/DSA layers scale with length

Because 34 of 45 layers carry a fixed-size recurrent state rather than a growing KV cache, and the remaining 11 are sparsified by DSA, the effective per-token memory cost stays low even as the context grows toward 1M tokens — the architectural reason a 320B model can serve a million-token context at 48.7 tokens/sec output and 1.52s time-to-first-token (Z.ai's own reported serving numbers) rather than the multi-second-per-token collapse a dense-attention model of comparable scale would suffer at that context length.

13. Benchmarks, as reported

Benchmark	GLM-5.3-Flash	Reference point
Terminal-Bench 2.1	84.3	Claude Opus 4.8: 85.0
DeepSWE v1.1	63.4	GLM-5.2: 46.2
AutomationBench	48.8	GLM-5.2: 26.2
Z.ai Code Bench v1.0	29.0	Opus 4.8: 29.5
Artificial Analysis Intelligence Index	57	—

FAQ

Is GLM-5.3-Flash the same model as the GLM-5.3 covered elsewhere on this site?

No — that article covers the dense-attention (MLA+DSA only, no KDA, no mHC) GLM-5.3. GLM-5.3-Flash is a separate, natively multimodal MoE release built for efficient long-context serving, using a different attention stack entirely.

Why replace MLA/DSA layers with KDA instead of using KDA everywhere?

A fixed-size linear-attention state is a lossy summary — it cannot guarantee exact retrieval of one fact from deep in a long context the way real (even sparse) attention can. The ≈3:1 hybrid keeps a thin backbone of exact-recall-capable MLA/DSA layers for that reason, discussed in §6.2.

What specifically does mHC add over a standard residual connection?

Four parallel residual streams instead of one, mixed by a matrix constrained to be doubly-stochastic (the Birkhoff polytope, enforced via Sinkhorn-Knopp) — which provably preserves signal magnitude across depth, unlike an unconstrained multi-stream mixing matrix, which DeepSeek reports can amplify signal over 3,000× and diverge at scale. See §7.

Is the "served entirely on Chinese chips" claim verified?

It is Z.ai's own claim about their Ox Alpha stealth-week infrastructure, consistent with their documented multi-vendor chip strategy, but not independently verifiable telemetry from outside the company — treated in this article as reported, not confirmed (§1).

Takeaways

• GLM-5.3-Flash is a hybrid-attention efficiency play, not a straightforward scale-up: 34 KDA layers (constant-size state, cheap) plus a thin 11-layer MLA/DSA backbone (sparse, exact-recall) at roughly a 3:1 ratio — the same hybrid philosophy Kimi Linear established, now independently validated by a second lab.
• Every sparsity/compression mechanism targets the same bottleneck from a different angle: DSA sparsifies attention itself, IndexPool sparsifies the DSA indexer's own cache at extreme context, and KDA's constant-size state sidesteps the KV-cache growth problem entirely for 34 of 45 layers — together landing at ≈11 KiB/token.
• mHC is not a free efficiency win — it's the fix for a specific, documented failure mode (>3,000× signal amplification in unconstrained multi-stream residuals at 27B scale) rather than an unconditional architectural upgrade, and it costs a real ≈6.7% training-time overhead for that stability.
• Multi-token prediction with parameter sharing beat the naive multi-layer approach on Z.ai's own numbers (2.76 vs. 2.55 accept length) — evidence the acceptance-rate cost people expect from parameter sharing is dominated by a different factor (training-inference mismatch) in practice.
• The benchmark table is a snapshot; the architecture is the durable content. Every mechanism above is checkable against the open MIT-licensed weights and code — that is the appropriate object of scrutiny, not the vendor's own leaderboard position at launch.

References & further reading

• Z.ai — GLM-5.3-Flash on Hugging Face — model card, weights, and the primary specs table this article works from.
• Sebastian Raschka — LLM Architecture Gallery — source of the annotated GLM-5.3-Flash architecture diagram this article is a written companion to.
• MarkTechPost — Z.ai releases GLM-5.3-Flash — benchmark, pricing, and deployment details cited in §12–13.
• Hyper-Connections (Zhu et al., ICLR 2025) — the original multi-residual-stream architecture mHC constrains.
• mHC: Manifold-Constrained Hyper-Connections — the Birkhoff-polytope/Sinkhorn-Knopp fix, and the >3,000× signal-amplification failure mode it addresses.
• DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention — the lightning indexer and sparse retrieve-then-attend mechanism behind DSA.
• Kimi Linear: An Expressive, Efficient Attention Architecture — the KDA lineage GLM-5.3-Flash's 34 linear-attention layers build on.
• cvam.sight — Kimi K3: Architecture and Infrastructure — the fuller derivation of the delta rule and channel-wise gating this article reuses in §6.1.
• cvam.sight — GLM-5.3: Frontier Coding with Emergent Cyber Capabilities — this site's coverage of the sibling dense-attention GLM-5.3 model.