Dexterous teleoperation has an awkward ceiling: the hardest motions to teach are precisely the motions a human operator struggles to express through a mismatched robot hand with little force feedback. SimToolReal replaces that bottleneck with a universal objective that simulation can generate at enormous scale.
Do not train “hammer,” “sweep” and “turn screwdriver” as separate skills. Train one policy to move arbitrary objects through requested 6D poses.
SimToolReal, presented by Tyler Ga Wei Lum, controls a 22-degree-of-freedom hand and seven-degree-of-freedom arm at 60 Hz. The same frozen policy grasps, reorients and uses unseen real tools without task-specific policy training.
The abstraction that unifies tool use
Most sim-to-real reinforcement learning systems engineer a new object model, reward and specialist policy for every behavior. SimToolReal asks what those behaviors share. A brush sweeps when its object frame follows one trajectory; a hammer strikes along another; a screwdriver rotates through another. The action semantics differ, but each can be specified as a sequence of desired object poses.
During training, the system procedurally generates tool-like primitive shapes, samples random target poses and trains a goal-conditioned policy to move each object to its goal. Massive GPU-parallel simulation produces decades of contact experience in days. Domain randomization and system identification vary the physics so the learned controller is not tied to one synthetic world.
What the policy sees and emits
The recurrent policy receives robot proprioception, the estimated current object pose, a grasp-region bounding box and the next goal pose. It outputs joint-position targets for the entire hand and arm. An LSTM gives the controller history across the motion.
At deployment, a human video can specify the task. SAM segments the tool and FoundationPose estimates its pose trajectory. Crucially, the video is not a robot-action demonstration and does not fine-tune the policy. It supplies only the desired object path—the prompt for the frozen goal-reaching controller.
One policy, full manipulation sequence
This removes hand-authored switching between grasp, reorientation and use policies. The controller can pick up a brush, rotate it inside the hand and sweep in one closed loop. A new task is a new object trajectory, not a new reward-design and training run.
The results support the abstraction. SimToolReal outperforms retargeting and fixed-grasp baselines by 37% while matching specialist RL policies trained on target objects and tasks. The real-world evaluation spans 120 rollouts, 24 tasks, 12 object instances and six tool categories: hammer, marker, eraser, brush, spatula and screwdriver.
Why the baselines fail
| Method | Failure |
|---|---|
| Fixed grasp | the arm must compensate for a bad hand pose, often causing collision |
| Kinematic retargeting | copied fingertip motion does not reason about robot contact forces |
| Per-task RL | works, but repeats reward engineering and training for every skill |
| SimToolReal | learns contact-rich goal reaching once, then takes trajectories as prompts |
The policy also shows recovery: after dropping a hammer, it can chase, re-grasp and finish. That behavior is difficult to obtain by simply imitating clean demonstrations because recovery states are underrepresented.
The real bottleneck moves to perception
Generalizing control does not generalize the entire stack. The paper's dominant failure mode is pose tracking, especially under hand occlusion. Small objects and heavy tools are harder; heavy tools drop more easily, while small objects give the tracker fewer reliable pixels. The controller assumes a usable object pose and grasp region.
“Zero-shot” therefore has a precise boundary. The policy is not retrained on the target tool or task, but deployment still requires an object model/perception pipeline and a target trajectory. It is zero-shot policy transfer, not zero-setup autonomy.
Why this matters beyond tools
SimToolReal is an example of representation doing the work that scale alone cannot. Object pose is independent enough of the robot's joints to transfer across tasks, yet concrete enough for fast feedback control. It sits between low-level torque and high-level language.
The follow-up idea mentioned in the talk, Play2Perfect, extends the pattern: pretrain a shared dexterous prior through free-space play, then fine-tune it on sparse-reward, contact-rich precision assembly. First learn broad control; only then spend expensive learning capacity on millimeter-level contact.
The takeaway
SimToolReal's biggest contribution is the interface between task and control. A task becomes a desired object trajectory; a universal simulated policy supplies the contact mechanics. That separation turns a new tool behavior from a training project into an inference-time prompt—while honestly revealing that perception, trajectory generation and difficult physics still remain.
Sources: paper, code and benchmark, talk.