MEM — A Robot Needs More Than One Kind of Memory.

A robot making grilled cheese needs to remember two very different facts. It needs recent images to recover a chopstick hidden by its wrist or notice that the last grasp failed. It also needs the semantic fact that the sandwich has already been cooking for several minutes. Feeding every past frame into one ever-growing context is both computationally impossible and conceptually wasteful.

Short-term memory needs visual detail. Long-term memory needs semantic compression. MEM gives each timescale the representation it deserves.

MEM: Multi-Scale Embodied Memory for Vision Language Action Models, presented by Marcel Torne, integrates that split into Physical Intelligence's $\pi_{0.6}$ VLA.

Why memoryless VLAs loop

Most robot policies condition on the current observation. That works for short demonstrations, but the policy cannot distinguish “I have washed this plate for two seconds” from “I have washed it for a minute.” It may repeat the same failed grasp because the current image does not say which strategy it already tried. The result is familiar: washing forever, burning food or pulling the same side of a door again and again.

The obvious fix—append more frames—creates an efficiency problem. Minutes of multi-camera video become enormous token sequences, while real control allows only a few hundred milliseconds of latency. Earlier memory additions can also reduce effectiveness through spurious correlations and train–test distribution shift.

The architecture: two policies, two memories

The high-level policy reads the task goal and its compressed memory, then issues the next subtask in language. The low-level policy takes that instruction, recent observations and proprioception, and emits an action chunk.

Compress video without throwing time away

MEM starts from a pretrained single-frame vision transformer and inserts temporal-attention layers. Frames can exchange information through time, but the encoder keeps tokens only for the current image. Those tokens now carry selected history without paying the full downstream cost of every frame.

This design has three practical advantages: most weights initialize from an existing vision model; temporal interaction happens before compression; and the VLA sees roughly a single image's token count. The short horizon helps with self-occlusion, counting and adaptation. In the paper's examples, it remembers items inside a grocery bag, which parts of a window were wiped and how long food has cooked.

Compress minutes into a memory scratchpad

For distant history, the high-level policy recurrently predicts a compact text memory $m_t$. The next high-level decision receives that memory, the goal and current context. Completed facts can remain while irrelevant attempts disappear.

That last property matters. A human demonstration might contain “pick up bowl” once. At deployment, a failing policy may produce it five times. Naively concatenating every instruction creates a sequence unlike training data. MEM can leave its summary unchanged until the bowl is actually picked up, reducing this distribution shift.

What the experiments establish

The combined system performs tasks requiring memory for up to fifteen minutes: preparing ingredients across a kitchen, cleaning a kitchen and making grilled cheese. Ablations show that the two memories are complementary. Remove video memory and the robot struggles with timing and local recovery. Remove language memory and it loses distant task progress. Naively retaining language history performs worse than learned compression.

Memory also produces a quieter capability: in-context motor adaptation. A memoryless policy keeps grasping at the wrong height or pulling a fridge door from the wrong side. With recent visual history, the policy can recognize the failed attempt and change strategy. On challenging dexterous tasks that do not require memory, MEM matches the underlying $\pi_{0.6}$ rather than paying a general performance penalty.

What is learned—and what is hand-specified

The talk's Q&A supplies an important boundary. The long-term memory is trained with supervised fine-tuning on annotated data. Humans decided which facts annotators should preserve. Text is practical because it is compact, inspectable and easy to supervise, not because human memory is believed to be literal prose.

That leaves the next research question: can reinforcement learning or a latent objective discover what is worth storing? A latent memory could be richer than text, but is harder to inspect, debug and supervise. MEM chooses the engineering point that works now.

The caveats

• The long-horizon experiments use ten rollouts per policy and task or recipe; they demonstrate capability, not fleet-scale reliability.
• Language memory can omit visually precise information and inherits annotation choices.
• The model remembers within an episode. Memory spanning weeks, months and embodiment drift remains future work.
• Compression is selective forgetting. A bad summary can remove the detail needed later.

The takeaway

MEM's contribution is not merely “give the robot more context.” It is a systems claim about information: retain dense observations only where detail matters, and turn distant experience into a small semantic state. The result lets a VLA operate for minutes, remember progress and stop repeating the same mistake. In long-horizon robotics, forgetting intelligently is part of remembering.

Sources: paper, talk, and transcript.