Every one of today's frontier open-weight models (DeepSeek, Qwen, GLM, Kimi, Gemma, Mistral, Llama, gpt-oss) is really a remix of a fairly small set of building blocks. This is a plain-language field guide to all 26 of them, grouped the way they group in practice — normalization, attention mechanisms, mixture-of-experts, training tricks, sequence-mixing hybrids, and a few odds and ends. Every concept gets a short, no-jargon explanation of what it does and why it exists, plus two links (a primary paper and a good deep-dive) if you want to go further than this article goes.
Index
| # | Concept | Category | One-line summary |
|---|---|---|---|
| 1 | RMSNorm | Normalization | Rescale activations by their root-mean-square; drop the mean-centering LayerNorm does. |
| 2 | QK-Norm | Normalization | Normalize queries and keys before the dot product so attention logits can't blow up. |
| 3 | PolyNorm | Normalization | A learned polynomial of the input, used as a normalization/activation hybrid. |
| 4 | MHA | Attention | The original transformer attention: every head gets its own Q, K, and V. |
| 5 | GQA | Attention | Groups of query heads share one K/V head — shrinks the KV cache for cheap. |
| 6 | MLA | Attention | Compress K and V into one small latent vector per token; decompress on the fly. |
| 7 | SWA | Attention | Each token only attends to a fixed nearby window, not the whole history. |
| 8 | NoPE | Attention | Skip explicit position embeddings entirely — causal masking alone gives position info. |
| 9 | RoPE | Attention | Encode position by rotating query/key vectors — the dominant positional scheme today. |
| 10 | Gated Attention | Attention | Multiply attention's output by a learned sigmoid gate before it leaves the block. |
| 11 | Hybrid Attention | Attention | Mix cheap linear/local layers with a few expensive full-attention layers in one stack. |
| 12 | DeepSeek Sparse Attention | Attention | A lightweight indexer picks the top-k relevant tokens; attention only runs over those. |
| 13 | Attention Budgeting | Attention | Deliberately vary how much attention compute/cache each layer gets, not a flat rate. |
| 14 | Attention Residuals | Attention | Replace the fixed residual-stream add with a learned weighted sum over all past layers. |
| 15 | CCA | Attention | Do the whole attention operation inside a compressed latent space, not full dimension. |
| 16 | CSA / HCA | Attention | Two flavors of KV-cache token-group compression for extreme long-context efficiency. |
| 17 | KV Sharing | Attention | Later layers reuse an earlier layer's K/V instead of computing their own. |
| 18 | MoE | MoE / Sparsity | Route each token to a handful of specialist sub-networks instead of one giant one. |
| 19 | LatentMoE | MoE / Sparsity | Run MoE experts in a compressed latent space for better accuracy per FLOP. |
| 20 | IndexShare | MoE / Sparsity | Share DeepSeek Sparse Attention's indexer across neighboring layers, not every layer. |
| 21 | MTP | Prediction / Training | Train the model to predict several future tokens at once, not just the next one. |
| 22 | ShortConv | Sequence-Mixing | A tiny causal convolution that gives cheap local context before attention runs. |
| 23 | Looped Depth Sharing | Sequence-Mixing | Run the same block of layers multiple times instead of stacking unique layers. |
| 24 | mHC | Sequence-Mixing | Route the residual stream through multiple parallel, constrained streams instead of one. |
| 25 | PLE | Other | A second, per-layer embedding table injected fresh into every decoder layer. |
| 26 | SiLU | Other | The smooth, self-gated activation function (x · sigmoid(x)) used in nearly every modern MLP. |
Normalization
Normalization layers exist to keep activation values in a numerically well-behaved range as they pass through dozens of transformer layers — without it, training either explodes (values grow without bound) or collapses (gradients vanish). These three are the current state of the art.
1. RMSNorm normalization
Classic LayerNorm does two things to every vector: it subtracts the mean (re-centers it around zero) and divides by the standard deviation (rescales it to unit variance). RMSNorm keeps only the second step. It divides each vector by its root-mean-square value — literally \( x / \sqrt{\text{mean}(x^2) + \epsilon} \) — then multiplies by a learned per-dimension scale. Dropping the mean-centering step turns out not to hurt quality, and it removes one full reduction pass per normalization call, which is why essentially every LLM built since 2023 (Llama, DeepSeek, Qwen, GLM, Mistral, Gemma) uses RMSNorm instead of LayerNorm.
2. QK-Norm normalization
Attention scores are computed as a dot product between a query vector and a key vector. If either vector's magnitude grows large during training — which happens naturally as models scale — the dot product can grow with it, pushing the softmax into a saturated region where gradients vanish and training destabilizes. QK-Norm fixes this at the source: it applies its own normalization (L2-norm or RMSNorm) to the query and key vectors before the dot product, then multiplies by a small learned scale. The result is attention logits that stay in a predictable range no matter how large the model gets, which is why QK-Norm has become close to standard in 2025-2026 training recipes (Qwen3, gpt-oss, and others use it). One real limitation: QK-Norm needs the full, uncompressed query and key vectors to normalize — that's directly incompatible with MLA's whole trick of never materializing full-size keys, which is why MLA-based models use other stabilization tricks instead.
3. PolyNorm normalization
Where RMSNorm and LayerNorm are single fixed formulas, PolyNorm learns a small polynomial function of the input instead — something in the shape of \( a_2 x^2 + a_1 x + a_0 \) with learned coefficients, applied alongside the usual normalizing division. The intuition: a straight rescale (what RMSNorm does) is a very restrictive transformation; letting the network learn a bit of curvature in how it reshapes activations gives it more expressive power for a nearly-free parameter cost. Reported results show measurably lower training and validation loss at matched compute versus plain RMSNorm, though it's newer and far less battle-tested at frontier scale than RMSNorm itself.
Attention mechanisms
This is the biggest category because attention — specifically, its memory cost — is the single largest engineering constraint in serving long-context LLMs. Every mechanism below exists to answer one of two questions: how do we make the KV cache smaller, or how do we make attention compute cheaper, without losing quality. Read the first three (MHA → GQA → MLA) in order; they're a direct evolutionary chain.
Fig. 1 — The three big-picture strategies for shrinking what has to be cached per token: don't shrink (MHA), share across heads (GQA), or compress into a small latent (MLA).
4. MHA — Multi-Head Attention attention
The original transformer mechanism, unchanged in its core idea since 2017. Split the model's hidden dimension into several "heads," and let each head learn its own independent query (Q), key (K), and value (V) projection. Each head can then specialize — one might track syntax, another might track long-range coreference — and their outputs are concatenated back together. The catch, at inference time: every head needs its own K and V vectors cached for every past token, so MHA's KV cache grows linearly with both sequence length and number of heads. For a modern long-context model this cache becomes the dominant memory cost, which is exactly the problem every other mechanism in this section exists to solve.
5. GQA — Grouped-Query Attention attention
The simplest possible fix to MHA's cache problem: instead of every query head getting its own K/V head, group several query heads together and have them share one K/V head. 8 query heads split into 2 groups of 4 means only 2 K/V heads need caching instead of 8 — a 4× cache reduction, tunable by choosing the group size. At the extreme (all query heads share one K/V head) this becomes Multi-Query Attention (MQA); GQA is the middle ground that keeps most of MHA's quality while getting most of MQA's memory savings. This is genuinely the most widely deployed attention variant in production today — Llama 3/4, Mistral, and Qwen all use it as their default.
6. MLA — Multi-Head Latent Attention attention
DeepSeek's answer to the same problem, and a genuinely different idea from GQA rather than a variant of it. Instead of caching K and V per head at all, MLA compresses a token's entire K and V information down into one small "latent" vector (a few hundred dimensions, versus the thousands MHA would need across all heads), and caches only that. At attention time, up-projection matrices decompress the latent back out into per-head keys and values on the fly. Because the decompression is a cheap fixed matrix multiply, MLA gets close to MHA's full quality while caching roughly what a single small head's worth of data would take — DeepSeek-V2 reported roughly a 93% KV-cache reduction versus MHA at matched benchmark performance. The tradeoff is architectural complexity (the up/down-projection matrices need to be learned and add compute) and, as noted under QK-Norm above, incompatibility with some other stabilization tricks that need fully materialized keys.
7. SWA — Sliding Window Attention attention
A completely different lever from GQA/MLA: instead of shrinking what gets cached per token, shrink how many tokens get attended to at all. Under SWA, each token only attends to the most recent W tokens (say, 4096), not the entire history — a fixed-size local window that slides forward with the sequence. This bounds both compute and cache to a constant size regardless of total sequence length. The catch is that information more than W tokens back is invisible to any single layer — but stacking layers still lets information propagate: after k layers of window W, a token's effective "reach" back into the past is roughly k × W tokens, which is how models like Mistral and Gemma get away with SWA-only or SWA-mostly stacks. Mixing a handful of full-attention layers with mostly-SWA layers (see Hybrid Attention, below) is the more common pattern in current frontier models.
8. NoPE — No Positional Embedding attention
Transformers have no inherent sense of token order — attention is a set operation, so without some explicit signal, "the cat sat" and "sat the cat" would look identical to it. The obvious fix is to inject position information (see RoPE, next). NoPE's finding is that in a causal (decoder-only) model, this turns out to be less necessary than assumed: causal masking already means token \(i\) can only ever see tokens \(1..i\), and that alone leaks enough ordering information for the model to learn positional structure implicitly through training, in layers that use NoPE. Several 2025-2026 hybrid architectures now use NoPE for a subset of layers (interleaved with RoPE layers elsewhere in the same stack) specifically because it has been shown empirically to generalize to longer sequences than pure RoPE stacks do — removing an explicit position signal removes a specific length the model could "memorize" as its training limit.
9. RoPE — Rotary Position Embedding attention
The dominant way modern LLMs actually do inject position information. Instead of adding a position vector to the token embedding (the original transformer's approach), RoPE rotates each query and key vector by an angle proportional to its position in the sequence, treating pairs of dimensions as 2D coordinates. The elegant property that makes this work: when you take the dot product of a rotated query and a rotated key, the result depends only on their relative position (the angle difference), not their absolute positions. That relative-position property is exactly what attention needs, and it's cheap — no extra parameters, just a fixed rotation applied at attention time. Nearly every model built since 2022 (Llama, DeepSeek, Qwen, Mistral, GLM) uses RoPE as its default positional scheme, often varying the rotation's base frequency to extend context length after initial training.
\[ q'_i = R_{\theta i}\, q_i, \quad \langle q'_i, k'_j\rangle = f(q_i, k_j, i-j) \]10. Gated Attention attention
A small, cheap addition to standard attention: after the usual scaled-dot-product-attention output is computed, multiply it element-wise by a learned, head-specific sigmoid gate — a value between 0 and 1 computed from the input itself, deciding how much of that head's output actually passes through. This adds a touch of non-linearity into what is otherwise a purely linear combination of value vectors, and — more importantly in practice — the gate can learn to fully suppress a head for a given input. That turns out to fix a known pathology called "attention sink," where models waste a disproportionate amount of attention weight on the first token of a sequence regardless of relevance, purely as an artifact of how softmax normalizes. Gated Attention is now in production in Qwen3-Next and was a NeurIPS 2025 Oral.
11. Hybrid Attention attention
Not one specific mechanism but a design pattern that now describes most frontier architectures: interleave cheap sequence-mixing layers (linear attention, SWA, state-space layers) with a small minority of expensive full-attention layers, rather than using one attention type for every layer. The logic is that most layers don't need full quadratic global attention — a cheap local or linear mechanism captures most of what they're doing — but the model as a whole still needs a few layers with genuine unrestricted long-range access to do tasks like retrieval or multi-hop reasoning correctly. GLM-5.3-Flash's 34 Kimi-Delta-Attention layers + 11 MLA/DSA layers (roughly a 3:1 ratio) is a concrete example of exactly this pattern — covered in depth in this site's GLM-5.3-Flash architecture teardown. Jamba (Mamba + attention) was one of the earliest production examples of the pattern.
12. DeepSeek Sparse Attention (DSA) attention
A learned alternative to SWA's fixed local window. Rather than hard-coding "attend to the nearest W tokens," DSA trains a small, cheap "lightning indexer" network that scores every earlier token for relevance to the current query, then selects only the top-k highest-scoring tokens (DeepSeek-V3.2 uses k=2,048) for the actual expensive attention computation. This turns attention's cost from scaling with the full sequence length \(L\) down to scaling with the fixed budget \(k\) — \(O(L^2) \to O(L \cdot k)\) — while, unlike SWA, letting the model choose which distant tokens matter rather than assuming only nearby ones do. The indexer itself is cheap enough to run over the full sequence because it's a much smaller computation than full attention would be.
13. Attention Budgeting attention
A higher-level design decision rather than a single mechanism: deliberately give different layers in the same model different amounts of attention "budget" — different KV-cache retention, different window sizes, different sparsity levels — instead of applying one uniform attention config to every layer. The motivation is that a flat, uniform budget is provably wasteful: not every layer needs the same amount of long-range access, and a uniform per-query KV-cache budget performs badly across requests of very different lengths and attention patterns. As reasoning models and long-agentic workflows keep more and more tokens in context, this kind of deliberate per-layer budget allocation — rather than a single global knob — is becoming a standard architectural lever alongside layer type itself (see Hybrid Attention above).
14. Attention Residuals attention
The standard transformer residual connection adds every layer's output to the running stream with an implicit, fixed weight of 1 — layer 40's contribution is added exactly as strongly as layer 2's, forever, regardless of whether that's actually useful. Attention Residuals (introduced by the Kimi team) replaces that fixed-weight accumulation with a learned, input-dependent weighted sum over all previous layers' outputs — the model can learn to weight a distant early layer more heavily than an adjacent recent one, if that's what the data calls for. The mechanism is cheap to add (one extra RMSNorm and one small "pseudo-query" vector per layer) and reported results show consistent, if modest, improvements in validation loss for roughly 4% extra training cost and 2% extra inference cost.
15. CCA — Compressed Convolutional Attention attention
MLA compresses K and V into a latent space; CCA takes the same instinct further and compresses Q, K, and V all into one shared, low-dimensional latent space, then runs the entire attention operation — scores and all — inside that compressed space, rather than decompressing back to full size before attending. To keep this from being too limiting, CCA adds a short causal convolution over the compressed Q and K sequences (mixing nearby compressed tokens together) before the attention step, recovering expressiveness that pure compression would otherwise lose. Combined with head-sharing (GQA-style grouping) into "CCGQA," reported results show it beating both plain GQA and MLA at matched KV-cache size, with up to 8× cache compression versus GQA at no accuracy loss on tested MoE models.
16. CSA / HCA — Compressed / Heavily-Compressed Sparse Attention attention
DeepSeek-V4's answer to million-token context windows, and a natural extension of DSA's "select the top-k tokens" idea combined with KV compression. Instead of keeping one cache entry per past token, CSA first compresses every m consecutive tokens' KV entries down into a single summarized entry, then applies DSA-style sparse selection on top of those already-compressed entries — combining sequence-dimension compression with learned sparsity in a single mechanism. HCA is the same idea taken to an extreme: it consolidates a much larger group of m' ≫ m tokens into one entry, for even more aggressive compression at long range. DeepSeek-V4 interleaves both — gentler CSA compression in some layers, aggressive HCA compression in others — which is reported as the specific combination that makes a genuine one-million-token context window computationally practical.
Mixture-of-Experts & sparsity
MoE is the other major axis of efficiency gains, orthogonal to the attention tricks above — instead of shrinking attention's cost, it shrinks the cost of the feed-forward (MLP) blocks that make up the bulk of a transformer's parameters, by activating only a fraction of them per token.
Fig. 2 — A trained router selects a small subset of expert sub-networks per token; a shared expert (present in most modern MoE designs) always fires to hold common knowledge.
18. MoE — Mixture of Experts moe / sparsity
Replace a transformer's single dense feed-forward block with many smaller "expert" feed-forward blocks, plus a small learned router that looks at each token and decides which handful of experts should process it — typically 2 to 9 out of anywhere from 64 to nearly 300 total experts in current models. Only the selected experts' weights are actually used (and need to be loaded/computed) for a given token, so a model can have a huge total parameter count (which determines how much it can know) while keeping its active parameter count — and therefore inference compute and latency — much smaller. DeepSeek-V3, at 671B total / 37B active parameters, is the canonical modern example; nearly every frontier open-weight model released in the last two years uses some form of MoE.
19. LatentMoE moe / sparsity
Applies MLA's core insight — do the expensive operation in a compressed latent space instead of full dimension — to MoE experts rather than to attention. Instead of each expert operating on the model's full hidden dimension, LatentMoE routes tokens through experts that operate in a smaller shared latent space, re-architected from a hardware-software co-design perspective to specifically maximize accuracy achieved per unit of FLOP and per parameter, across both offline high-throughput and online latency-critical serving regimes. In published design-space experiments up to 95B parameters and a 1T-token training horizon, LatentMoE consistently outperformed standard MoE at matched compute — and it's reportedly already been adopted in Nvidia's Nemotron-3 Super/Ultra model line.
Prediction & training tricks
21. MTP — Multi-Token Prediction prediction / training
Standard LLM training asks the model to predict only the single next token at every position. MTP adds extra lightweight prediction heads that simultaneously try to predict the token 2, 3, or more steps ahead, using shared underlying representations — turning next-token prediction into next-few-tokens prediction during training. The main payoff isn't really about training loss; it's about inference: a model trained this way can be paired with speculative decoding, where the MTP heads' extra-step predictions are proposed as draft tokens and verified in parallel by the main model, meaningfully increasing tokens generated per forward pass. DeepSeek-V3 reports this pattern giving a meaningfully higher accept-length in speculative decoding versus a model without any MTP training at all — turning a training-time addition into a real inference-speed win.
Sequence-mixing & state-space hybrids
These three concepts are about what happens to the residual stream and token sequence between attention layers — cheap local mixing, depth reuse, and how the residual stream itself is structured.
Fig. 3 — ShortConv sits before the main sequence-mixing layer, giving cheap, fixed-cost local context that plain attention or a state-space layer alone doesn't get for free.
22. ShortConv sequence-mixing
A small causal 1D convolution (kernel width typically 3-4 tokens) applied to a layer's input before the main sequence-mixing operation (attention, linear attention, or a state-space model) runs. It's borrowed directly from architectures like Mamba and Based, where research found that pure subquadratic sequence mixers (state-space models without any local convolution) systematically fail at associative-recall-style tasks — remembering "the value that followed this specific key earlier in the sequence" — and that a short convolution fixes this specific failure mode cheaply, without needing full attention. Liquid's LFM2 models build an entire hybrid architecture primarily out of gated ShortConv layers plus a smaller number of GQA attention layers, and dynamic (input-dependent) variants have been shown to give a 1.33× compute advantage over compute-matched pure transformers.
23. Looped Depth Sharing sequence-mixing
A different way to get more "depth" of computation without more parameters: instead of stacking N unique transformer layers, take a smaller stack of shared layers and run the input through that same stack multiple times (looped), with each pass's output feeding back in as the next pass's input. This decouples computational depth (how many times data gets transformed) from parameter depth (how many distinct sets of weights exist) — a model can "think longer" on a hard input by looping more times, using the same weights each time, similar in spirit to how a recurrent network reuses its weights across time steps. The idea traces back to the 2018 Universal Transformer and has seen a resurgence in 2026 research specifically as a mechanism for implicit iterative reasoning without the token overhead of writing out chain-of-thought explicitly.
24. mHC — Manifold-Constrained Hyper-Connections sequence-mixing
A structural change to the residual stream itself. A standard transformer has exactly one residual stream running through the whole network; Hyper-Connections generalized this to multiple parallel streams that layers can read from and write to in flexible, learned combinations, rather than one fixed path. The unconstrained version of this idea, though, is numerically unstable — signal amplification through unconstrained combination weights has been documented to blow up by over 3,000× in some configurations. mHC's fix is to constrain the combination weights to live on a specific mathematical manifold — a doubly-stochastic matrix living inside the Birkhoff polytope, kept there during training via Sinkhorn-Knopp normalization — which mathematically guarantees the signal can be redistributed across streams but never amplified without bound. GLM-5.3-Flash's four-stream mHC design (covered in depth on this site) is a concrete production use of exactly this fix.
Other building blocks
25. PLE — Per-Layer Embeddings other
A parameter-efficiency trick, not an attention or normalization change. Alongside the normal token embedding table, PLE adds a second embedding table that produces a small dedicated vector for every decoder layer, for every token — computed once in a single lookup, then sliced and injected fresh into each layer as it runs. The point is to let a small model carry more token-specific information than its "effective" transformer-stack parameter count would otherwise suggest, without making the repeated transformer blocks themselves more expensive to run. Google's Gemma 4 E2B/E4B models are the production example: E2B is described as 2.3B "effective" parameters used in the repeated stack, but 5.1B when the PLE table is counted — the extra capacity comes essentially for free at inference since PLE lookups are cheap compared to running them through the transformer blocks themselves. There's no standalone peer-reviewed paper for PLE yet — it currently exists only in Google's model cards and the open-source implementation.
26. SiLU other
The activation function used inside nearly every modern LLM's feed-forward block (usually as part of a "SwiGLU" gated variant). SiLU (Sigmoid Linear Unit, also called Swish) is defined as \( \text{SiLU}(x) = x \cdot \sigma(x) \) — multiply the input by its own sigmoid. Unlike ReLU, which hard-clips every negative input to exactly zero, SiLU is smooth and non-monotonic: it dips slightly negative for small negative inputs before flattening out, and that smoothness gives better gradient flow during training. It's called "self-gated" because the sigmoid term acts like a soft, learned gate applied to the value itself, rather than the input being gated by some separate signal — a cheap way to add adaptive, input-dependent non-linearity with no additional parameters over the plain linear term.
\[ \text{SiLU}(x) = x \cdot \sigma(x) = \frac{x}{1+e^{-x}} \]FAQ
Do I need to understand all 26 to understand one modern model?
No single model uses all 26 — most use somewhere between 6 and 12. A given model's technical report will typically name its specific combination directly (e.g. "RMSNorm + GQA + RoPE + SwiGLU" for a simple dense model, or a longer combination like GLM-5.3-Flash's RMSNorm + Hybrid Attention + MLA + DSA + NoPE + mHC + MoE + MTP for a frontier hybrid). This glossary is meant to be a reference you return to per-concept, not something to memorize end to end.
Which of these are actually mutually exclusive?
Very few. MHA, GQA, and MLA are alternatives to each other (a layer picks one). NoPE and RoPE are alternatives at the layer level, but a single model can mix both across different layers. Nearly everything else — normalization choice, MoE, MTP, ShortConv, mHC, PLE, gating — is independent and layered on top of whichever base attention/positional choice a model makes.
Which of these concepts are the "safest bets" — well-established versus experimental?
RMSNorm, RoPE, GQA, MoE, and SiLU/SwiGLU are essentially industry-standard at this point — you'll find them in nearly every serious open-weight model. MLA, DSA, QK-Norm, and MTP are proven at frontier scale (DeepSeek, Qwen) but not yet universal. PolyNorm, LatentMoE, IndexShare, Attention Residuals, CSA/HCA, and mHC are newer, published within roughly the last year, and worth watching rather than assuming as defaults yet.
Why do so many of these trace back to DeepSeek and the Raschka architecture gallery?
DeepSeek's V2/V3/V3.2/V4 line has been unusually willing to publish detailed technical reports explaining novel mechanisms (MLA, DSA, CSA/HCA, MTP) rather than just releasing weights — which is exactly the kind of primary source this kind of glossary needs. Sebastian Raschka's LLM Architecture Gallery independently tracks and names nearly this same set of concepts as they appear across different labs' models, which is why several of the newer or less-formalized concepts here (PolyNorm, Attention Budgeting, PLE, KV Sharing, CSA/HCA, Attention Residuals) point to his gallery as the best available secondary source — there often isn't a single dedicated paper yet.
Takeaways
- Nearly every attention-mechanism entry in this list is solving one of two problems: shrink the KV cache (GQA, MLA, CCA, CSA/HCA, KV Sharing) or shrink attention's compute (SWA, DSA, Hybrid Attention, Attention Budgeting). Knowing which problem a mechanism solves tells you when it matters for your use case.
- MHA → GQA → MLA is a genuine evolutionary chain, each step trading a bit more architectural complexity for a bit more cache savings at roughly matched quality — understanding this progression makes the rest of the attention section much easier to place.
- MoE and attention efficiency are orthogonal axes — a model's total-vs-active parameter ratio (MoE) and its per-token attention cost (everything in the attention section) are independent design choices that combine multiplicatively in a real model's efficiency profile.
- "Hybrid" is the dominant 2025-2026 design pattern, not an exception — mixing cheap and expensive layers (Hybrid Attention), cheap and expensive normalization tricks, and even multiple residual streams (mHC) rather than picking one mechanism uniformly across the whole stack.
- Several of the newest concepts here don't have a single canonical paper yet (PLE, Attention Budgeting as a named pattern, some of the gallery-tracked variants) — for those, a technical blog or a model's own card is currently the best available source, which is exactly what the further-reading links above route to when that's the case.
References & further reading
- Sebastian Raschka — The LLM Architecture Gallery — the closest thing to a canonical index of nearly every concept in this article; start here for anything not yet covered by a dedicated paper.
- Raschka — Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention — a good single-article overview tying several of these concepts together in context.
- cvam.sight — GLM-5.3-Flash ("Ox Alpha") architecture teardown — a full graduate-level walkthrough of one real model combining Hybrid Attention, MLA, DSA, NoPE, mHC, MoE, and MTP together.
- cvam.sight — PagedAttention deep dive — the KV-cache memory-management layer that every attention mechanism in this article's cache numbers assumes is running underneath.
- cvam.sight — vLLM Optimization & Tuning — how these architectural choices translate into actual serving-time configuration and tradeoffs.