Original research papers that form the foundation of the articles on this site. Sorted by topic.
Vaswani et al. — the Transformer architecture that started it all.
2018Bidirectional Transformer pre-training with masked language modelling, built to fine-tune across language tasks.
2021Encodes position by rotating queries and keys, bringing explicit relative-position structure into self-attention.
2024Multi-Head Latent Attention — 93% KV cache reduction. The core innovation.
2024671B params, $5.58M training cost. MLA + MoE + MTP at scale.
2025Emergent reasoning from pure RL. GRPO, chain-of-thought, self-verification.
The modern MoE starting point: a learned gate activates only a sparse subset of thousands of expert networks.
2021Simplifies MoE to top-1 routing and scales sparse language models to a trillion parameters with stable training.
2024Fine-grained expert segmentation and shared-expert isolation—the specialization recipe later used by DeepSeek-V2.
IO-aware exact attention via tiling and online softmax. 2–4× speedup, linear memory.
20232× faster via better work partitioning. 230 TFLOPs/s on A100. Single author: Tri Dao.
2024H100-optimized with async execution, warp specialization, and FP8. 740 TFLOPs/s.
Visual language model for few-shot learning. Perceiver Resampler + gated cross-attention. Beats fine-tuned models with 32 examples. NeurIPS 2022.
2023Self-supervised vision by predicting abstract representations, not pixels. No augmentations, 10× cheaper than MAE. CVPR 2023.
2026Parallel Box Decoding for vision-language grounding. Emits whole boxes at once — up to 10× faster than Qwen3-VL and more accurate. NVIDIA.
Parallelizes the draft↔verify loop of speculative decoding. ~30% over speculative baselines, up to 5× over plain decoding. Kumar, Dao, May.
2024Multi-step action proposal + dynamics model, both diffusion, for online MPC. Matches SOTA offline RL on D4RL. TMLR.
2026Stable end-to-end JEPA from pixels. Two loss terms, one hyperparameter, 15M params, 48× faster planning. LeCun et al.
2025Benign overfitting, double descent, overparametrization — none unique to neural nets. Soft inductive biases + PAC-Bayes. Andrew Gordon Wilson.
2025Data fixed, compute free. Weight decay 30× standard, asymptote-fitting, ensemble scaling, distillation. Kim, Kotha, Liang, Hashimoto.
2025Eight primitives and a unified template for overlapped multi-GPU kernels. Up to 4.08× on sequence-parallel workloads with fewer than 50 lines of device code.
2026Accuracy per watt across 20+ local models, eight accelerators and one million real queries. Local IPW improved 5.3× from 2023–2025.
2023A GPU-native ECS for batched learning environments. Hide-and-Seek with physics and ray tracing exceeds 1.9M environment steps per second.
Supervised instruction tuning followed by reinforcement learning from human feedback—the foundational RLHF recipe.
2024Open mathematical reasoning at scale and the introduction of Group Relative Policy Optimization without a critic model.
2026When OPD works, when it silently fails. Two conditions, 97–99% shared top-k mass, and why long-horizon reasoning may break the recipe.