Abstract

Kiran Lekkala, Chen Liu, Laurent Itti, Bird's Eye View Based Pretrained World model for Visual Navigation, In: International Symposium on Robotics Research (ISRR), Dec 2024. (Cited by 1)

Abstract: This paper presents a novel system that fuses components in a traditional World Model into a robust system, that is pretrained entirely on a dataset of unlabeled videos and random trajectories from a simulator. The pretrained model, when frozen and deployed, zero-shot transfers to unseen environments for fast Reinforcement Learning of a 1-layer policy or planning. To facilitate transfer, we use a representation that is based on Bird's Eye View (BEV) images. Thus, our agent learns representations of the observation by first learning to translate from complex First-Person View (FPV) based RGB images to BEV representations, then learning to navigate using those representations. Later, when deployed, the agent uses the perception model that translates FPV-based RGB images to embeddings that were learned by the FPV to BEV translator and that can be used by the downstream policy. The incorporation of state-checking modules using Anchor images and Mixture Density LSTM not only interpolates uncertain and missing observations but also enhances the robustness of the model in the real-world. We trained the model using data from a Differential drive robot in the CARLA simulator. Our methodology's effectiveness is shown through the deployment of trained models onto a real-world Differential drive robot, where using the BEV representation leads to better transfer and faster learning. Lastly we release a comprehensive codebase, dataset and models for training and deployment (https://sites.google.com/usc.edu/world-model-sim2real/).

Themes: Computer Vision, Machine Learning