One landmark 2024 paper per week — 52 picks, curated down from the year's flood, ordered as a year-long reading plan. Not every model release, not every benchmark: the papers that mattered, the ones the 2025 reasoning explosion was built on (GRPO, test-time scaling, MoE, latent reasoning) plus the architecture bets that paid off later (Mamba-2, BitNet, KAN, byte-level). Read one a week and you walk into 2025 knowing exactly how the field got there.
2024 was the year the modern LLM stack solidified. Open mixture-of-experts went mainstream, RL post-training got a real algorithm (GRPO), test-time compute became a scaling axis, and a dozen teams quietly shipped the architecture ideas that 2025 and 2026 would run with.
There were thousands of papers. A flat list helps nobody. So this is 52 — one per week — picked for impact and for how well they explain the year. I leaned on the famous public 2024 roundups for coverage, then cut hard and added the papers I think actually moved the needle. If you read this list a week at a time, by week 52 you understand 2024 well enough that the 2025–2026 list reads like a sequel you already know the plot of.
2402.03300) encodes year-month — 24=2024, 02=February. Every row keeps the id so you can verify and cite. Links go to the /abs/ page, so you land on versions, code, and BibTeX.How to use this list
As a calendar. One paper a week, in order. The weeks are roughly chronological, so you experience 2024 as it unfolded — the MoE wave, the architecture experiments, the model-report summer, the reasoning-and-efficiency finale in December.
As a reference. Skim the "why read" column, jump to whatever theme you need. The emoji tags let you triage in seconds.
As prep for 2025. Anything tagged 🏛️ is load-bearing — those are the papers the next year cites constantly. If you are short on time, read only the 🏛️ rows (there are about a dozen) and you have the spine of the year.
Fig 1 — The four moods of 2024, one quarter at a time.
Q1 · January – March (weeks 1–16)
The year opens with the mixture-of-experts wave and, quietly, the single most consequential method of the decade: GRPO, buried inside a math paper. Also the 1-bit experiments and the first 2M-token context tricks.
| Wk | Paper | Why read |
|---|---|---|
| W1 · Jan 📦 | Mixtral of Experts | The open MoE that made sparse models mainstream — 8 experts, 2 active. The template every 2024 MoE follows. |
| W2 · Jan 🛠️ | DeepSeekMoE | Fine-grained experts + shared experts for common knowledge. The MoE recipe DeepSeek rode all the way to V3. |
| W3 · Jan 🛠️ | Self-Play Fine-Tuning (SPIN) | A model improves by playing against its own past outputs — no new human data. Early seed of self-improvement. |
| W4 · Jan 🛠️ | Self-Rewarding Language Models | The model is its own judge, generating its own rewards during iterative DPO. The idea behind a whole 2025 sub-field. |
| W5 · Jan 🔬 | RAG vs Fine-Tuning | The clean comparison everyone cites when deciding retrieval vs weights. Settles a lot of arguments. |
| W6 · Feb 🏛️ | DeepSeekMath (introduces GRPO) | The most important 2024 paper for what came next — GRPO, the RL algorithm behind DeepSeek-R1 and the entire 2025 reasoning boom. Read this one closely. |
| W7 · Feb 🏛️ | The Era of 1-bit LLMs (BitNet b1.58) | Ternary {-1,0,1} weights match fp16 at the same size. A genuinely radical efficiency claim that still echoes in 2026. |
| W8 · Feb 🛠️ | Self-Discover | LLMs compose their own reasoning structures from atomic modules. Pre-R1 reasoning, prompt-only. |
| W9 · Feb 🛠️ | DoRA: Weight-Decomposed LoRA | Splits weight magnitude and direction for better PEFT — a near-free upgrade over plain LoRA. |
| W10 · Feb 🛠️ | LongRoPE | Extends context past 2M tokens by reworking positional interpolation. The long-context groundwork. |
| W11 · Feb 🛠️ | OneBit: Extremely Low-bit LLMs | Pushes weight quantization to ~1 bit. Read alongside BitNet to see the low-bit frontier. |
| W12 · Feb 🏛️ | Genie: Generative Interactive Environments | An 11B world model that turns videos into playable worlds — no action labels. The foundation-world-model opener. |
| W13 · Mar 📦 | Gemini 1.5 | 1M-token multimodal context. The release that made "long context" a headline capability. |
| W14 · Mar 🛠️ | GaLore | Full-parameter training in a fraction of the memory via low-rank gradients. Train bigger on less. |
| W15 · Mar 🛠️ | Vision-RWKV | An RNN-style linear-attention backbone for vision — early signal of the post-transformer architecture interest. |
| W16 · Mar 🏛️ | Stable Diffusion 3 (Rectified Flow Transformers) | The MM-DiT architecture + rectified flow that set the text-to-image standard. The image-gen landmark of the year. |
Q2 · April – June (weeks 17–27)
The architecture-bet quarter. Mamba-2 unifies SSMs with attention, KAN questions the MLP itself, DeepSeek-V2 ships Multi-head Latent Attention, and the alternatives (xLSTM, Mixture-of-Depths) pile up.
| Wk | Paper | Why read |
|---|---|---|
| W17 · Apr 🏛️ | KAN: Kolmogorov-Arnold Networks | Learnable activations on edges instead of fixed ones on nodes — a from-scratch rethink of the MLP. The year's most-discussed "what if everything is different" paper. |
| W18 · Apr 🛠️ | Infini-attention | Unbounded context with bounded memory via compressive memory. The "infinite context" idea, distilled. |
| W19 · Apr 🛠️ | Mixture-of-Depths | Tokens dynamically skip layers under a compute budget. MoE's lesser-known cousin — compute where it's needed. |
| W20 · Apr 📦 | Phi-3 Technical Report | A 3.8B model rivaling Mixtral/GPT-3.5, runnable on a phone. The data-quality-over-scale thesis, demonstrated. |
| W21 · May 🏛️📦 | DeepSeek-V2 (MLA) | Multi-head Latent Attention — ~93% KV-cache reduction. The attention innovation the whole DeepSeek line is built on. |
| W22 · May 🏛️ | Mamba-2 (Transformers are SSMs) | State-space duality unifies attention and SSMs; the core layer runs 2–8× faster. The theoretical bridge that 2026's Mamba revival stands on. |
| W23 · May 🛠️ | xLSTM: Extended LSTM | Scales the LSTM to billions of params with exponential gating. The "what if RNNs were never the problem" bet. |
| W24 · May 🛠️ | SimPO | Reference-free preference optimization — simpler and often stronger than DPO. The practical alignment upgrade of the year. |
| W25 · May 📦 | Chameleon: Mixed-Modal Early-Fusion | One model, text and images interleaved from the start (not bolted on). The early-fusion multimodal direction. |
| W26 · Jun 📦 | Nemotron-4 340B | NVIDIA's open 340B family — and a synthetic-data masterclass (98%+ of alignment data is generated). Read for the data pipeline. |
| W27 · Jun 🛠️ | RouteLLM | Learn to route each query to a strong or weak model — 2× cost cuts with no quality loss. The pragmatic cost play. |
Q3 · July – September (weeks 28–37)
The model-report summer — Llama 3, Qwen2, Gemma 2 — plus the paper that reframed the whole roadmap: scaling test-time compute can beat scaling parameters.
| Wk | Paper | Why read |
|---|---|---|
| W28 · Jul 🏛️📦 | The Llama 3 Herd of Models | Meta's 90-page report on building a frontier open model end to end. The most complete public model report of the year. |
| W29 · Jul 📦 | Qwen2 | The release that began Qwen's rise to the dominant open family. Baseline for everything Qwen ships after. |
| W30 · Aug 📦 | Gemma 2 | Google's open report — distillation + local/global attention. The small-open reference before Gemma 3. |
| W31 · Aug 🏛️ | Scaling LLM Test-Time Compute Optimally | Spending compute at inference can beat scaling parameters by 4×+. The paper that made test-time scaling a first-class axis — and set up o1. |
| W32 · Aug 📦 | Jamba-1.5 (Hybrid Transformer-Mamba) | A production hybrid SSM-transformer at scale. The blueprint 2026's hybrid models (Nemotron 3) follow. |
| W33 · Aug 🏛️ | The AI Scientist | Sakana's end-to-end automated research loop — idea → code → experiment → paper → review. The "AI does science" milestone (precursor to AlphaEvolve). |
| W34 · Sep 📦 | OLMoE: Open MoE | Fully open MoE — weights, data, code, logs. The most transparent MoE to learn the architecture from. |
| W35 · Sep 📦 | Qwen2-VL | Dynamic resolution + M-RoPE for images and video. The open VLM reference of late 2024. |
| W36 · Sep 🛠️ | SCoRe: Self-Correct via RL | DeepMind teaches genuine self-correction with multi-turn RL on self-generated data. The serious version of "think again." |
| W37 · Sep 🔬 | In Defense of RAG in the Long-Context Era | Even with million-token windows, retrieval still wins on cost and accuracy. Settles the "is RAG dead?" debate. |
Q4 · October – December (weeks 38–52)
The finale, and the densest stretch. o1 lands and the reasoning race begins; Apple punctures the hype with GSM-Symbolic; and December alone ships DeepSeek-V3, latent-space reasoning, byte-level models, and a modern BERT. This quarter is the launchpad for everything in the 2025 list.
| Wk | Paper | Why read |
|---|---|---|
| W38 · Oct 🏛️ | Differential Transformer | Attention as the difference of two softmaxes — cancels noise, sharpens focus, cuts hallucination. A clean, adoptable architecture tweak. |
| W39 · Oct 🔬 | GSM-Symbolic (Apple) | Shows models wobble when you just rename variables — questioning whether GSM8K gains are real reasoning. The skeptic anchor for the o1 era. |
| W40 · Oct 🛠️ | O1 Replication Journey | An open, honest attempt to reverse-engineer o1-style reasoning. The bridge from o1's announcement to R1's open recipe. |
| W41 · Nov 🏛️🛠️ | Tülu 3 | A fully open post-training recipe — SFT, DPO, and the first big use of RL with Verifiable Rewards (RLVR). The open playbook the 2025 RLVR wave standardized. |
| W42 · Nov 🛠️ | BitNet a4.8 | 4-bit activations for 1-bit models — pushing the low-bit frontier toward real deployment. Pairs with W7. |
| W43 · Nov 🛠️ | Marco-o1 | An open o1-style reasoner with MCTS + reflection, aimed at open-ended (not just verifiable) tasks. Early open-reasoning attempt. |
| W44 · Dec 🏛️📦 | DeepSeek-V3 | 671B MoE (37B active), MLA + auxiliary-loss-free load balancing + multi-token prediction, trained shockingly cheap. The base model R1 was built on. Essential. |
| W45 · Dec 🏛️ | Coconut: Reasoning in Continuous Latent Space | Reason by feeding hidden states back in, not tokens — thinking without words. A genuinely new reasoning paradigm. |
| W46 · Dec 🏛️ | Byte Latent Transformer (BLT) | Tokenizer-free, entropy-based byte patches that match BPE at scale. The strongest case yet for killing the tokenizer. |
| W47 · Dec 🏛️ | ModernBERT | The encoder finally modernized — RoPE, local/global attention, 8K context, 2T tokens. The new default for retrieval/classification. |
| W48 · Dec 📦 | Qwen2.5 Technical Report | 18T-token pretraining, 1M+ SFT samples, multistage RL. The report that set up Qwen3. The most-downloaded open family's blueprint. |
| W49 · Dec 📦 | Phi-4 | A 14B model that surpasses its GPT-4 teacher on STEM via synthetic-data curriculum. The data-centric thesis at its peak. |
| W50 · Dec 🛠️ | Large Concept Models (Meta) | Predict sentence-level concepts in an embedding space instead of tokens. An ambitious bet on a higher unit of language. |
| W51 · Dec 🛠️ | Deliberative Alignment (OpenAI) | Teach the model to reason over the safety spec before answering — reasoning as a safety tool. How the o-series is aligned. |
| W52 · Dec 📦 | DeepSeek-VL2 | DeepSeek's MoE vision-language model — closes out the year and the multimodal thread. A fitting week 52. |
The load-bearing dozen
If 52 is too many, read only these 🏛️ papers. They are the ones 2025 and 2026 cite on every page — the actual spine of the year.
| # | Paper | What it set up |
|---|---|---|
| 1 | DeepSeekMath / GRPO | The RL algorithm behind R1 and the entire 2025 reasoning era. |
| 2 | DeepSeek-V2 / MLA | The cheap-attention innovation the DeepSeek line rides. |
| 3 | DeepSeek-V3 | The base model R1 was fine-tuned from. |
| 4 | Scaling Test-Time Compute | Made inference compute a scaling axis — the o1 thesis. |
| 5 | Mamba-2 | The SSM/attention bridge behind 2026's Mamba revival. |
| 6 | BitNet b1.58 | The 1-bit efficiency frontier still cited in 2026. |
| 7 | Tülu 3 | Open post-training + the first big RLVR demonstration. |
| 8 | Coconut | Latent-space reasoning — a new paradigm 2025 builds on. |
| 9 | Llama 3 Herd | The reference for how to build a frontier open model. |
| 10 | GSM-Symbolic | The reasoning-skepticism anchor for the o1 era. |
| 11 | Byte Latent Transformer | The strongest case for tokenizer-free models. |
| 12 | KAN | The year's boldest "rethink the building block" paper. |
Takeaways
2024 built the launchpad; 2025 lit the fuse. Nearly every big 2025 story traces to a 2024 paper here — GRPO (W6) became R1, test-time compute (W31) became o1 and the reasoning race, DeepSeek-V3 (W44) became R1's base, Tülu 3's RLVR (W41) became the standard signal, and the 2024 architecture bets (Mamba-2, BitNet, MLA) became the 2026 efficiency era.
Read it as a calendar, not a pile. One paper a week. By December you will have watched MoE go mainstream, the tokenizer come under threat, reasoning get an algorithm, and efficiency become the headline — which is exactly the run-up the 2025–2026 list continues.
References & extra reads
- The 2025–2026 reading list · the direct sequel to this one — pick up where week 52 leaves off
- DeepSeekMath / GRPO · the single most consequential 2024 paper for what followed
- DeepSeek-V3 · the base model the 2025 reasoning era was built on
- arXiv cs.CL recent · the firehose, for finding the next year's 52
A curated 52 of 2024, one per week. Dates are first arXiv submission month; links go to abstract pages so you get versions, code, and BibTeX. Emoji tags (🏛️🛠️📦🔬) are my own triage marks. Coverage informed by the public 2024 roundups, then cut and re-picked for impact.