Introduction to DeepSeek Architecture
Before you understand why DeepSeek is different, you need to know what every LLM shares. MLA, MoE, MTP — the three innovations that matter.
From Transformer internals to DeepSeek-V4. A complete technical deep-dive into every architectural innovation — MLA, MoE, MTP, FP8 training, GRPO, and more. Grounded in original papers and code.
Build the mental model of how a Large Language Model works — from raw text to generated tokens. No assumed ML background.
Before you understand why DeepSeek is different, you need to know what every LLM shares. MLA, MoE, MTP — the three innovations that matter.
LLMs don't read words. They read numbers. The full journey from your sentence to token IDs to embeddings to predictions and back.
The attention mechanism is the one idea that changed AI forever. Query, Key, Value — explained without assuming you know linear algebra.
Stop using attention as a black box. Every matrix multiplication explained — Q, K, V projections, scaled dot-product, and a worked numeric example.
Why can't an LLM see into the future? The causal mask is why — and it's elegantly simple. Plus: the full autoregressive loop explained.
One attention head learns one pattern. GPT-4 has 96 of them. Here's why multiple heads matter and what each one actually learns.
Theory is nice. Running code is better. Full PyTorch MHA from scratch — no nn.MultiheadAttention, just raw matrix ops.
Explain the #1 memory bottleneck in LLM inference (the KV cache) and trace the evolution of attention variants: MQA → GQA → MLA.
The #1 memory bottleneck in LLM inference. Exactly what gets cached, why it grows linearly, and the concrete numbers that make it a hard constraint.
Shazeer's 2019 paper collapsed H KV heads to 1. H× memory reduction — and what you pay in quality. Used by PaLM, Falcon, StarCoder.
G groups of shared KV heads — the middle ground. How G=8 became the production standard in LLaMA-2/3 and Mistral. Quality-memory tradeoff mapped precisely.
The O(n²) compute wall, the memory hierarchy problem, and Flash Attention's IO-aware tiling solution. How the score matrix moves from HBM to SRAM.
The centrepiece technical innovation of DeepSeek-V2/V3 — Multi-Head Latent Attention. 93% KV cache reduction without quality loss.
The core insight: compress K,V to a 512-dim latent, cache it, expand to full attention on demand. 64× cache compression with MHA-level quality.
Full PyTorch implementation — down/up projections, latent cache management, absorbed projections for inference speed. Every tensor, every line.
Side-by-side: head-count compression vs representation compression. Memory tables, quality benchmarks, and when to use each approach.
Exact memory budgets for LLaMA-3-8B/70B and DeepSeek-V2-236B. How MLA enables 40× more concurrent requests per GPU at 128K context.
Why RoPE breaks inside latent compression, and how decoupled RoPE solves it. The d_h^R hyperparameter, real cache overhead, and context extension implications.
How Transformers understand word order — from integer position to RoPE. The modern standard used by DeepSeek, LLaMA, and Mistral.
Self-attention is order-blind. The obvious fix — tag each token with its index — fails three ways and defines the requirements list every later encoding must satisfy.
One number → a vector of bits. Bounds magnitude and length-dependence, and introduces the load-bearing idea: position is a multi-frequency signal.
The original Transformer scheme. The angle-addition identity reveals the deep payoff — shifting position is a rotation — the insight RoPE later applies directly to Q/K.
Rotate, don't add. Turn position into angle so the dot product depends only on relative offset — exactly, every layer, parameter-free. Full derivation and the rotate_half trick.
RoPE vs learned PE, sinusoidal, T5 bias, and ALiBi on five axes — plus the catch: extrapolation needs NTK/YaRN scaling, the path that gives DeepSeek 128K context.
DeepSeek's second pillar — fine-grained expert segmentation, shared experts, and auxiliary-loss-free load balancing.
The FFN is where most params and compute live. MoE scales that half to hundreds of billions of params while paying for only a small slice per token.
The router decides which experts run for every token at every layer. The gating math, the choices that matter, and the non-differentiability hiding inside top-k.
"Expert 7 handles Python" is almost entirely wrong. What experts actually specialise in, why the clean story fails, and what it implies for MoE design.
A router left alone starves most experts. The load-balancing aux loss stops it — the formula, why it quietly damages the model, and the bias trick V3 used to delete it.
Hardware needs fixed-size buffers known before the batch runs. Where capacity factor and token dropping come from, and how DeepSeek drops almost nothing.
Fine-grained expert segmentation and shared expert isolation — the two changes that make DeepSeek's MoE work at 671B params with only 37B active.
A faithful, readable PyTorch MoE block: SwiGLU experts, sigmoid routing, top-k dispatch, shared experts, balancing bias, tests, and cost accounting.
Predict beyond the next token, train sequential MTP modules safely, implement them from scratch, and reuse their drafts for verified speculative decoding.
The standard language-model objective is powerful but locally supervised; this article separates what it learns brilliantly from what it does not explicitly reward.
Predict several future offsets from every position, understand the loss, and see why this is a training objective rather than permission to emit tokens independently.
DeepSeek-V3’s sequential MTP modules preserve a causal chain, share expensive endpoints, and can disappear cleanly at inference.
Tensor alignment, gradient flow, loss scheduling, parameter sharing, distributed placement, and diagnostics for a stable MTP training run.
Add a sequential DeepSeek-style MTP branch to a tiny causal Transformer with correct shifts, shared parameters, losses, and tests.
Draft several tokens cheaply, verify them in parallel, preserve the target distribution, and understand when MTP helps—or does not.
Build the full precision ladder from quantization basics to DeepSeek-V3's fine-grained FP8 training pipeline, including the numerical traps that decide convergence.
From FP32 to FP8 and INT4: scales, zero points, granularity, calibration, outliers, PTQ, QAT, memory, bandwidth, and accuracy.
Why BF16, FP16, and FP32 coexist in one training step, how autocast and loss scaling work, and which operations must stay precise.
DeepSeek-V3’s tile and block scaling: why activation outliers demand local scales and what custom kernels must do.
Low-precision products, wide accumulators, partial sums, error growth, and the hardware behavior DeepSeek had to work around.
Dynamic activation scales, static weight scales, distribution drift, calibration, kernel fusion, and the cost of quantizing every step.
The complete low-precision training system: operator policy, fine-grained scales, accurate accumulation, communication precision, stability evidence, and cost claims.
Connect V2/V3 architecture to the 2,048-GPU training system: sparse economics, expert parallelism, DualPipe, communication, and memory.
How MLA and DeepSeekMoE combine in the 236B/21B model, with configuration, training, post-training, economics, and the design lineage to V3.
A source-checked tour of 671B/37B, 14.8T tokens, MLA, 256 routed experts, MTP depth one, FP8, DualPipe, and post-training.
The bias controller behind V3’s balanced experts: selection versus weighting, update dynamics, residual sequence loss, and operational telemetry.
Why 671B total and 37B active change FLOPs, memory, communication, utilization, serving, and cost in different ways.
Data, pipeline, expert, tensor, and optimizer-state parallelism; how V3 combines them and why DualPipe is necessary.
Token dispatch, expert placement, node-limited routing, grouped GEMM, load balance, and the two all-to-all operations inside every MoE layer.
A practical network model for all-to-all traffic, topology, overlap, congestion, precision, and why communication can dominate accelerator arithmetic.
Weights, gradients, optimizer state, activations, communication buffers, KV cache, checkpointing, ZeRO, precision, and a defensible memory budget.
Separate R1's base architecture from its four-stage post-training pipeline, then derive RLVR, GRPO, emergent reasoning, verification, and test-time scaling.
R1-Zero, cold-start data, reasoning RL, rejection-sampling SFT, final RL, distilled models, rewards, and what 'architecture' really means here.
Why math and code permit objective rewards, how outcome and format signals work, where reward hacking appears, and how RLVR differs from preference RL.
Policy ratios, clipping, KL control, group-normalized advantages, the removed critic, memory savings, bias, and implementation pitfalls.
What the paper actually shows about longer reasoning, reflection, self-correction, benchmark growth, readability problems, and the limits of the word 'emergent.'
Separate internal checking, sampled reflection, external verifiers, process rewards, tool use, and redundant loops in reasoning models.
Longer reasoning, best-of-N, majority voting, search, verifiers, adaptive budgets, scaling curves, latency, and the economics of spending inference compute.
Bring the series current through DeepSeek-V4, million-token serving, agent systems, post-CUDA infrastructure, a dated frontier comparison, and an open-weight outlook.
Perceive-plan-act loops, tool schemas, state, memory, verification, context management, security, and how V3.2/V4 reasoning changes the agent stack.
The official 2026 preview: V4-Pro and Flash, CSA+HCA hybrid attention, mHC, Muon, 32T+ tokens, reasoning modes, and million-token efficiency.
Attention, KV cache, positional generalization, retrieval quality, prefill, decode, memory, distributed serving, and evaluations that expose fake long context.
What DeepSeek’s workloads demand from accelerators and interconnects: FP8 tiles, sparse experts, low-latency collectives, memory, compiler stacks, and portability.
A dated, methodology-aware comparison of openness, modality, context, deployment, agent interfaces, economics, and benchmark evidence as of July 30, 2026.
A grounded series conclusion: capacity, efficient attention, reasoning RL, agents, hardware, evaluation, governance, and concrete ways readers can contribute.