SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting

M. Nomaan Qureshi, Sparsh Garg, Francisco Yandun, David Held, George Kantor, Abhisesh Silwal

🎉 Accepted for Spotlight Presentation (Top 7/53) at CoRL MRM-D Workshop 🎉

arXiv PDF Code Twitter Thread YouTube Video

Nomaan is applying for PhD programs for the Fall 2025 cycle

Sparsh is looking for full time Industry Positions starting May 2025

Abstract

Sim2Real transfer, particularly for manipulation policies relying on RGB images, remains a critical challenge in robotics due to the significant domain shift between synthetic and real-world visual data. In this paper, we propose SplatSim, a novel framework that leverages Gaussian Splatting as the primary rendering primitive to reduce the Sim2Real gap for RGB-based manipulation policies. By replacing traditional mesh representations with Gaussian Splats in simulators, SplatSim produces highly photorealistic synthetic data while maintaining the scalability and cost-efficiency of simulation. We demonstrate the effectiveness of our framework by training manipulation policies within SplatSim and deploying them in the real world in a zero-shot manner, achieving an average success rate of 86.25%, compared to 97.5% for policies trained on real-world data.

[1/n] Key Idea: Use Gaussian Splatting as a Renderer over Existing Simulators

[2/n] Robot Interacting with Flying Objects

Here’s a fun collection of clips where we reversed gravity in the simulator, highlighting that these renderings are purely simulated.

[3/n] You can also control the splat models using Gello!

[4/n] Simulated Demonstrations for T_Push Task

[5/n] Simulated Demonstrations for Apple Picking Task

[6/n] Making Gaussian Splat of the Scene with Robot

[7/n] Then Align the 3D Gaussians of Robot with the Simulator Pointcloud

[8/n] Overall Pipeline of SplatSim

[9/n] Results : Zero-Shot Sim2Real Policy Deployment for Push-T

[10/n] Results : Zero-Shot Sim2Real Policy Deployment for Pick-Up-Apple

[11/n] Results : Zero-Shot Sim2Real Policy Deployment for Orange-On-Plate

[12/n] Results : Zero-Shot Sim2Real Policy Deployment for Assembly

[13/n] Results : Zero-Shot Sim2Real Policy Deployment for Pick-Up-Apple

BibTeX

@misc{qureshi2024splatsimzeroshotsim2realtransfer,
            title={SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting}, 
            author={Mohammad Nomaan Qureshi and Sparsh Garg and Francisco Yandun and David Held and George Kantor and Abhishesh Silwal},
            year={2024},
            eprint={2409.10161},
            archivePrefix={arXiv},
            primaryClass={cs.RO},
            url={https://arxiv.org/abs/2409.10161}, 
      }