SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting

M. Nomaan Qureshi, Sparsh Garg, Francisco Yandun, David Held, George Kantor, Abhisesh Silwal

🎉 Accepted at ICRA, 2025 🎉

🎉 Accepted for Spotlight Presentation (Top 7/53) at CoRL MRM-D Workshop 🎉

arXiv PDF Code Twitter Thread YouTube Video

Abstract

Sim2Real transfer, particularly for manipulation policies relying on RGB images, remains a critical challenge in robotics due to the significant domain shift between synthetic and real-world visual data. In this paper, we propose SplatSim, a novel framework that leverages Gaussian Splatting as the primary rendering primitive to reduce the Sim2Real gap for RGB-based manipulation policies. By replacing traditional mesh representations with Gaussian Splats in simulators, SplatSim produces highly photorealistic synthetic data while maintaining the scalability and cost-efficiency of simulation. We demonstrate the effectiveness of our framework by training manipulation policies within SplatSim and deploying them in the real world in a zero-shot manner, achieving an average success rate of 86.25%, compared to 97.5% for policies trained on real-world data.

[1/n] Key Idea: Use Gaussian Splatting as a Renderer over Existing Simulators

[2/n] Robot Interacting with Flying Objects

Here’s a fun collection of clips where we reversed gravity in the simulator, highlighting that these renderings are purely simulated.

[3/n] You can also control the splat models using Gello!

[4/n] Simulated Demonstrations for T_Push Task

[5/n] Simulated Demonstrations for Apple Picking Task

[6/n] Making Gaussian Splat of the Scene with Robot

[7/n] Then Align the 3D Gaussians of Robot with the Simulator Pointcloud

[8/n] Overall Pipeline of SplatSim

[9/n] Results : Zero-Shot Sim2Real Policy Deployment for Push-T

[10/n] Results : Zero-Shot Sim2Real Policy Deployment for Pick-Up-Apple

[11/n] Results : Zero-Shot Sim2Real Policy Deployment for Orange-On-Plate

[12/n] Results : Zero-Shot Sim2Real Policy Deployment for Assembly

[13/n] Results : Zero-Shot Sim2Real Policy Deployment for Pick-Up-Apple

BibTeX

@misc{qureshi2024splatsimzeroshotsim2realtransfer,
            title={SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting}, 
            author={Mohammad Nomaan Qureshi and Sparsh Garg and Francisco Yandun and David Held and George Kantor and Abhishesh Silwal},
            year={2024},
            eprint={2409.10161},
            archivePrefix={arXiv},
            primaryClass={cs.RO},
            url={https://arxiv.org/abs/2409.10161}, 
      }