High-fidelity physics simulation is essential for scalable robotic learning, but the sim-to-real gap persists, especially for tasks involving complex, dynamic, and discontinuous interactions like physical contacts. Explicit system identification, which tunes explicit simulator parameters, is often insufficient to align the intricate, high-dimensional, and state-dependent dynamics of the real world.
To overcome this, we propose an implicit sim-to-real alignment framework that learns to directly align the simulator's dynamics with contact information. Our method treats the off-the-shelf simulator as a base prior and learns a contact-aware neural dynamics model to refine simulated states using real-world observations.
We show that using tactile contact information from robotic hands can effectively model the non-smooth discontinuities inherent in contact-rich tasks, resulting in a neural dynamics model grounded by real-world data. We demonstrate that this learned forward dynamics model improves state prediction accuracy and can be effectively used to predict policy performance and refine policies trained purely in standard simulators, offering a scalable, data-driven approach to sim-to-real alignment.
Model Architecture: The model utilizes a multi-modal encoder to fuse object states and robot actions with a Contact Predictor that infers future contact events, which then condition a Diffusion-UNet to generate physically-consistent future pose increments.
Sim-to-Real Alignment: The framework follows a two-stage data pipeline: it first learns contact-induced physical priors from large-scale simulation, then fine-tunes with real Tactile Sensor data to implicitly align simulated states with real-world contact patterns.
Real World
Neural Dynamics
Physics Simulator
Marginal Grasping (Mustard Bottle): Precise alignment for unstable grasping interactions.
Marginal Grasping (Bleach Cleanser): Precise alignment for unstable grasping interactions.
Robust Grasping (Mustard Bottle): Stable alignment for successful grasping.
Robust Grasping (Bleach Cleanser): Consistent alignment for reliable manipulation.
@misc{jing2026contactawareneuraldynamics,
title={Contact-Aware Neural Dynamics},
author={Changwei Jing and Jai Krishna Bandi and Jianglong Ye and Yan Duan and Pieter Abbeel and Xiaolong Wang and Sha Yi},
year={2026},
eprint={2601.12796},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2601.12796},
}