Gotta Scoop 'Em All: Sim-and-Real Co-Training of Graph-based Neural Dynamics for Long-Horizon Scooping
Abstract
We present a system to address a new problem of removing all granular objects from a container using a scoop, a task that requires long-horizon reasoning over the complex interactions between granular objects, the scoop, and the container. To tackle this challenge, we adopt a graph-based neural dynamics (GBND) model trained through sim-and-real co-training that leverages both rich synthetic data and a small amount of real-world data. Using the learned model, we plan long-horizon scooping sequences of behavior primitives with Monte-carlo Tree Search (MCTS). Extensive experimental evaluations demonstrate that our system with the co-training strategy is capable of removing all materials in a container with fewer than 20 scoops, significantly outperforming strategies that use only simulation or real data. Our approach can also generalize in a zero-shot manner to new materials, and quickly adapt to new containers with few additional data points.