SimpleMapping: Real-Time Visual-Inertial Dense Mapping with Deep Multi-View Stereo

Yingye Xin^1*    Xingxing Zuo^1,2*,†    Dongyue Lu¹    Stefan Leutenegger^1,2

^*equal contribution    ^†corresponding author

¹Smart Robotics Lab, Technical University of Munich    ²Munich Center for Machine Learning (MCML)

IEEE International Symposium on Mixed and Augmented Reality (ISMAR) 2023, Sydney, Australia

arXiv | Video | Project Page

TODOs

• 📣 C++ code is released with pretrained model in May 2024! Please check SimpleMapping/.
• 📣 SPA-MVSNet will be released soon under the folder SPA-MVSNet/.

Abstract

We present SimpleMapping, a real-time visual-inertial dense mapping method capable of performing incremental 3D mesh reconstruction with high quality using only sequential monocular images and inertial measurement unit (IMU) readings. 6-DoF camera poses are estimated by a robust feature-based visual-inertial odometry (VIO), which also generates noisy sparse 3D map points as a by-product. We propose a sparse point aided multi-view stereo neural network (SPA-MVSNet) that can effectively leverage the informative but noisy sparse points from the VIO system. The sparse depth from VIO is firstly completed by a single-view depth completion network. This dense depth map, although naturally limited in accuracy, is then used as a prior to guide our MVS network in the cost volume generation and regularization for accurate dense depth prediction. Predicted depth maps of keyframe images by the MVS network are incrementally fused into a global map using TSDF-Fusion. We extensively evaluate both the proposed SPA-MVSNet and the entire dense mapping system on several public datasets as well as our own dataset, demonstrating the system’s impressive generalization capabilities and its ability to deliver high-quality 3D reconstruction online. Our proposed dense mapping system achieves a 39.7% improvement in F-score over existing systems when evaluated on the challenging scenarios of the EuRoC dataset.

BibTeX

If you find this repository helpful, please cite our paper and give us a star!

@article{Xin2023ISMAR,
    author={Xin, Yingye and Zuo, Xingxing and Lu, Dongyue and Leutenegger, Stefan},
    title={{SimpleMapping: Real-Time Visual-Inertial Dense Mapping with Deep Multi-View Stereo}},
    booktitle={IEEE International Symposium on Mixed and Augmented Reality (ISMAR)},
    month={Oct},
    year={2023}
}