Simulator-as-Proxy World Models for Robot Manipulation

Anonymous Authors

Anonymous Institution

Paper coming soon Code coming soon

Insights

Simulator as action reasoner, video model as physics corrector

Video World Model

Video model is weak at action-to-motion reasoning and object contact
input image
input image
Image + actions
Video model
contact is hallucinated
arm motion is inaccurate
Error vs GT

Agentic Real-to-Sim

Simulator is a better action-to-motion reasoner, but the agent-built scene is misaligned with reality, and simulator is not good at non-rigid contact and physics
input image
input image
Image + actions
Claude agent
+ robotic simulator
rope not carried
towel thrown off by contact

Ours: Real2Sim (simulator) + Sim2Real (video model)

Small real2sim gap: simulator is an accurate action reasoner, video model completes realistic physics and contact
first frame
Image
metric depth
Depth
Actions
Input
simulator
Proxy video
Video model
RGB video

Simulation Visualization

Image + actions our simulator 4D visualization
Loading 3D scene...
Drag to rotate · scroll to zoom

0.0 s

Method

inputmodeloutput

Real-to-Sim

Left image of the stereo pair
left image
Right image of the stereo pair
right image
stereo matchingobject segmentation
Metric depth
metric depth
Semantic map
semantic mapleft view

Simulation

point simulator

+

action sequence

openclosed
depth video
semantic video

Sim-to-Real

First frame
first frame
+
proxy video
video model
RGB video

Results

GTProxy videoGenerated video
Non-rigid objects

Put the yellow cloth in the laundry bag

Pick up the white ropes from the table and put them in the box

Hang the green towel on the dish rack

Put the black hoodie inside the storage box

Rigid objects

Put the cans on the tray

Move the can from the table to the sink

Stack the white cups together, then put the marker in the stacked cups

Put the blocks in the grey cup

Put the green cube on the wooden pallet

Comparison with Baselines

Same first frame and actions for every model. Ctrl-World smears the cloth, DreamDojo and Cosmos-Predict 2.5 misplace the arm; with our proxy, the arm and the held object follow the real robot.

GTCtrl-WorldDreamDojoCosmos-Predict 2.5SimPWM-Cosmos-TransferSimPWM-LTX-2.3
GT
Ctrl-World
DreamDojo
Cosmos-Predict 2.5
SimPWM-Cosmos-Transfer
SimPWM-LTX-2.3

Put the yellow cloth in the laundry bag

GT
Ctrl-World
DreamDojo
Cosmos-Predict 2.5
SimPWM-Cosmos-Transfer
SimPWM-LTX-2.3

Stack the white cups together, then put the marker in the stacked cups