Brennan Shacklett asks a literal systems question: what if an entire game engine—capable of simulating many kinds of games—lived on the GPU? Games are low-cost learning environments for robotics, self-driving research, game development, and reinforcement learning, but conventional engines are inefficient when training needs enormous throughput.
Instead of rendering one beautiful world faster, Madrona runs thousands of small, independent worlds together and asks the GPU to do the same kind of work across all of them.
Why normal game engines are the wrong shape
Running a thousand ordinary engine copies makes them fight for CPU and GPU resources and prevents costs from being amortized. Madrona instead treats a thousand learning environments as one throughput-oriented GPU batch. Brennan says the Hide-and-Seek example generates millions of experience frames per second.
The hard part is gameplay logic: branchy code, dynamic memory allocation, different object counts, and runtime changes are a poor fit for conventional fixed-tensor GPU frameworks.
The many-world idea
Madrona batches entire environments. One GPU holds the state of thousands of worlds and runs the same systems across them. Some worlds contain hiding agents, others seekers, but their components—position, velocity, shape, team, reward state—share layouts that the GPU can process coherently.
The architectural key is an Entity Component System (ECS). An entity is an ID. Components are columns of data such as positions. Systems operate over entities possessing the relevant components. This data-oriented layout avoids chasing object pointers and exposes broad, regular loops that map well to GPU threads.
From game logic to GPU work
Systems declare the components they need—action processing asks for position and action; collision asks for position and bounding box—and Madrona maps each matching row to a GPU thread. Systems join into a task graph for actions, physics, observations, and rewards.
For dynamic allocation, Madrona appends rows, marks deletions, then uses a fast GPU sort to compact the tables. A persistent mega-kernel works through the task graph, using GPU atomics for coordination.
The results
In Brennan’s profiler view, synchronization is under 1% of the work and an RTX 4090 stays nearly full while simulating about 4,000 Hide-and-Seek worlds. For the baseline environments shown, he reports the full GPU version as over 100× faster in many cases than the original CPU reference implementations.
He says later projects extended the result to end-to-end training, and ML researchers without low-level GPU experience successfully built new environments over roughly two years of use.
| Design | Traditional engine | Madrona |
|---|---|---|
| Primary unit | one rich world | batch of many worlds |
| Game state | CPU objects / scene graph | GPU-native ECS columns |
| Learning handoff | frequent copies | tensor-compatible GPU state |
| Goal | human-visible frame quality | experience throughput |
Why faster simulation changes research
The talk’s broader lesson is about GPU programming abstractions. Python syntax layered over CUDA improves readability but does not solve dynamic allocation or irregular parallelism. Brennan argues for higher-level, scripting-like GPU systems with good default performance for workloads that do not look like tensor algebra.
The caveats
The previous article mixed in later project benchmarks, exact step rates, and claims about simulation fidelity that Brennan did not make in this presentation. They have been removed from the talk summary; the paper and project page remain linked for readers who want results beyond the video.
The takeaway
Madrona flips the game-engine question. Instead of “how realistic can one world be?” it asks “how many useful worlds can one GPU advance together?” For RL, where experience is the raw material of learning, that turns the simulator from a bottleneck into a factory.
Why simulation throughput matters to learning
Reinforcement learning improves a policy through interaction. An agent observes a state, takes an action, receives a new state and reward, and repeats. Data-hungry algorithms require enormous numbers of these transitions. If environment stepping is slow, faster neural-network training cannot help because the learner waits for experience.
Games are attractive environments because they provide controllable rules, cheap resets, procedurally generated situations, and measurable outcomes. They support research in robotics, driving, multi-agent coordination, and game AI without running every experiment in the physical world.
Why launching a thousand engines is inefficient
The obvious parallel strategy is many processes. Each engine owns objects, physics state, scripts, and rendering resources. Thousands of copies duplicate memory and scheduling overhead. They contend for CPU caches and operating-system resources. Small GPU calls from many processes fail to form one coherent workload.
Costs cannot be amortized because each engine believes it is the only world. The hardware sees many irregular tasks rather than one large batch. Madrona changes the unit of execution from one engine process to a batch of worlds managed by one engine.
Latency-oriented versus throughput-oriented design
A commercial game wants one world to react immediately to a player and render the next frame on time. An RL simulator can tolerate that any individual world waits briefly if thousands of worlds advance efficiently together.
This freedom permits batching allocation, sorting work, and using persistent kernels. Operations that would add unacceptable latency to one interactive frame can improve total environment steps per second across the batch.
Why gameplay logic is difficult on a GPU
Dense tensor operations apply the same instruction pattern to regular arrays. Gameplay contains branches: an agent may hold an object, collide with a wall, lock a box, or trigger a rule that other agents do not. Worlds contain different numbers of entities. Objects appear and disappear.
Traditional GPU frameworks prefer fixed shapes and predictable control flow. Dynamic allocation and virtual dispatch can create divergence and unpredictable memory access. Brennan’s challenge was to preserve expressive environment construction while presenting enough regularity to the GPU.
Entity: identity without a heavy object
In an entity-component system, an entity is primarily an identifier. It does not carry a large object hierarchy. A hiding agent, movable box, and wall can all be entities.
This separation avoids pointer-heavy object graphs. Identity remains stable while data is stored in structures optimized for iteration.
Component: data in columns
Components hold facts such as position, rotation, velocity, action, team, or reward state. Instead of storing every property together inside one object, the engine stores components in columns. A system reading positions can traverse contiguous position data.
Madrona combines component rows from many worlds in GPU memory and records which world each row belongs to. This produces a large data-parallel table even when individual worlds are small and different.
System: behavior as a query
A system declares the components it needs. An action-processing system asks for entities with position and action. Collision asks for position and bounds. Reward computation asks for the state relevant to the task.
The engine finds matching rows and maps invocations to GPU threads. Because behavior depends on component presence rather than a virtual class hierarchy, the engine gets a form of runtime polymorphism without expensive per-thread virtual dispatch.
Procedural generation and variable world sizes
Hide-and-Seek worlds can contain different obstacles and layouts. Fixed tensors would require padding every world to the largest possible count or complex masking. ECS tables simply contain the entities that exist.
The developer still uses a direct interface such as creating an obstacle. Madrona translates those operations into batched table management. This programming model is a central contribution: performance would be less useful if every environment required hand-written allocation kernels.
Dynamic allocation through append and compact
Immediate fine-grained allocation is difficult on a GPU. Madrona exploits throughput tolerance. New rows are appended. Deleted rows are marked rather than immediately removed. Later, a high-performance GPU sort or compaction pass reorganizes the tables.
This resembles garbage collection: defer cleanup, gather enough work, then process it efficiently in bulk. One world may temporarily carry dead rows, but the entire batch gains throughput.
The task graph for a frame
A complete environment step contains dependencies. Actions must be available before movement. Physics updates positions before observations. Rewards depend on resulting state. Madrona expresses these systems as a task graph.
The graph tells the runtime which work can execute and where synchronization is required. Developers reason about environment stages while the engine schedules large batches of system invocations.
Persistent mega-kernel scheduling
Launching a separate GPU kernel for every tiny system would add overhead and repeatedly return control to the CPU. Madrona uses a persistent mega-kernel that remains active and pulls ready tasks from the graph.
Fast GPU atomics coordinate workers. SMs can take different systems as dependencies clear. Brennan’s visualization shows colored activity across SMs and narrow synchronization gaps, indicating that the runtime keeps the device busy.
Reading the utilization result
The demonstration uses roughly 4,000 Hide-and-Seek worlds on an RTX 4090. Brennan says synchronization consumes under one percent of the shown work. This supports the scheduling claim: irregular gameplay does not automatically force the GPU to sit idle.
Utilization alone is not success. A kernel can keep hardware busy doing unnecessary work. The environment-throughput comparison is needed to show useful progress.
The CPU baselines
The team chose environments with existing CPU implementations, including Hide-and-Seek and Overcooked. The original machine-learning environments provide one baseline. A CPU implementation using the same ECS organization provides another, separating the benefit of data-oriented design from GPU execution.
Brennan reports that the complete GPU version is over 100× faster in many shown cases. “Many cases” matters: environment complexity and baseline quality change the factor. The claim should not be generalized to every game engine.
From simulator speed to end-to-end training
A simulator benchmark can produce frames faster than a policy consumes them. Later projects tested the complete loop, including policy inference and learning. Brennan says the architecture extended successfully beyond isolated simulation throughput.
Keeping observations, actions, rewards, and model computation on the GPU can avoid CPU transfers between every step. The exact training architecture determines whether simulation, inference, or optimization becomes the next bottleneck.
Why accessibility to ML researchers matters
A framework succeeds only if researchers can create environments without becoming GPU-runtime specialists. Brennan reports that users with little low-level GPU knowledge built environments and obtained strong performance.
The ECS interface, task graph, and runtime absorb complexity. This is analogous to a tensor framework: most users describe computation, while experts optimize execution beneath the abstraction.
What Python-like syntax does not solve
Many GPU languages make CUDA concepts easier to write by adopting Python syntax. Brennan appreciates that improvement but argues it leaves the underlying model unchanged. Dynamic allocation, irregular entities, and runtime polymorphism still require architectural support.
A true high-level GPU environment should provide data structures and scheduling suited to irregular workloads, not merely translate a thread kernel into prettier syntax.
GPU compute beyond tensor cores
Modern GPUs contain massive bandwidth, general arithmetic units, atomics, and scheduling capacity in addition to tensor cores. Workloads such as simulation, graph processing, and dynamic systems may exploit these resources without looking like matrix multiplication.
Madrona demonstrates that the obstacle is often the programming model. When data and scheduling are reorganized, branchy game logic can become a productive throughput workload.
What Madrona does not automatically guarantee
High simulation speed does not prove that a simulated world matches reality. A policy can learn artifacts of the environment. Reward design can be wrong. Physics fidelity may be insufficient for a target task. Those questions belong to the research using the engine.
Madrona is infrastructure for generating experience efficiently. Researchers still need validation, randomization, and evaluation appropriate to their domain.
A practical environment-building workflow
- Define entities and the minimum component data each requires.
- Write systems as queries over components.
- Express frame dependencies in the task graph.
- Batch many worlds and vary their layouts procedurally.
- Keep observations, actions, and rewards in GPU-accessible structures.
- Profile synchronization, allocation, and per-system utilization.
- Compare against the same environment on a strong CPU baseline.
- Validate learning quality, not only steps per second.
When a GPU-native engine is the wrong tool
If an experiment needs only a few worlds, strict per-world latency, or an existing complex commercial engine, rewriting everything around ECS tables may not repay the effort. CPU engines have mature tooling and debugging ecosystems.
The approach is most compelling when environment throughput is a measured bottleneck, many independent worlds exist, and the research program will run enough experiments to amortize the port.
The broader systems lesson
Madrona specializes the software architecture around the workload rather than merely adding a faster device. It changes storage, allocation, dispatch, synchronization, and the definition of a frame. That is why it belongs in this Edition 03 sequence.
The final insight is transferable: a GPU is not limited to conventional tensor operations, but irregular software must be reorganized around bulk data and throughput. Good abstractions make that reorganization available to people who should be thinking about environments and learning—not atomic instructions.