Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation



  • Chuang Gan
  • Peihao Chen
  • Dongyu Ji
  • Kunyang Lin
  • Runhao Zeng
  • Thomas H. Li
  • Mingkui Tan

Published on




We address a practical yet challenging problem of training robot agents to navigate in an environment following a path described by some language instructions. The instructions often contain descriptions of objects in the environment. To achieve accurate and efficient navigation, it is critical to build a map that accurately represents both spatial location and the semantic information of the environment objects. However, enabling a robot to build a map that well represents the environment is extremely challenging as the environment often involves diverse objects with various attributes. In this paper, we propose a multi-granularity map, which contains both object fine-grained details (e.g., color, texture) and semantic classes, to represent objects more comprehensively. Moreover, we propose a weakly-supervised auxiliary task, which requires the agent to localize instruction-relevant objects on the map. Through this task, the agent not only learns to localize the instruction-relevant objects for navigation but also is encouraged to learn a better map representation that reveals object information. We then feed the learned map and instruction to a waypoint predictor to determine the next navigation goal. Experimental results show our method outperforms the state-of-the-art by 4.0% and 4.6% w.r.t. success rate both in seen and unseen environments, respectively on VLN-CE dataset. Code is available at

Please cite our work using the BibTeX below.

title={Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation},
author={Peihao Chen and Dongyu Ji and Kunyang Lin and Runhao Zeng and Thomas H. Li and Mingkui Tan and Chuang Gan},
booktitle={Advances in Neural Information Processing Systems},
editor={Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho},
Close Modal