MVSGaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo

Abstract

We present MVSGaussian, a new generalizable 3D Gaussian representation approach derived from Multi-View Stereo (MVS) that can efficiently reconstruct unseen scenes. Specifically, 1) we leverage MVS to encode geometry-aware Gaussian representations and decode them into Gaussian parameters. 2) To further enhance performance, we propose a hybrid Gaussian rendering that integrates an efficient volume rendering design for novel view synthesis. 3) To support fast fine-tuning for specific scenes, we introduce a multi-view geometric consistent aggregation strategy to effectively aggregate the point clouds generated by the generalizable model, serving as the initialization for per-scene optimization. Compared with previous generalizable NeRF-based methods, which typically require minutes of fine-tuning and seconds of rendering per image, MVSGaussian achieves real-time rendering with better synthesis quality for each scene. Compared with the vanilla 3D-GS, MVSGaussian achieves better view synthesis with less training computational cost. Extensive experiments on DTU, Real Forward-facing, NeRF Synthetic, and Tanks and Temples datasets validate that MVSGaussian attains state-of-the-art performance with convincing generalizability, real-time rendering speed, and fast per-scene optimization.

Author abstract · arXiv abstract · v3

Publication

European Conference on Computer Vision (ECCV), 2024

Paper and resources

• Paper · arXiv v3 (PDF)
• arXiv:2405.12218
• Google Scholar
• Code · TQTQliu/MVSGaussian
• Project website

Research topics

MVSGaussian · 3D Gaussian Splatting · Multi-view stereo · Generalizable reconstruction · Novel-view synthesis · Real-time rendering · Geometry-aware representations · Per-scene optimization

Research problem and approach

Generalizable neural rendering can still require slow rendering or lengthy scene adaptation. MVSGaussian predicts Gaussian parameters from multi-view geometry, uses hybrid rendering and aggregates consistent point clouds to initialize further optimization. arXiv abstract · v3

Main contributions

• Combines multi-view stereo features with a generalizable 3D Gaussian representation and hybrid rendering. arXiv abstract · v3
• Introduces geometrically consistent aggregation for efficient per-scene initialization. arXiv abstract · v3

Method comparison

arXiv abstract · v3

Selected results

• MVSGaussian achieves PSNR 28.21 dB, SSIM 0.963 and LPIPS 0.076 on DTU, compared with ENeRF’s 27.61 dB, 0.957 and 0.089. The reported three-view rendering speed is 21.5 fps and memory use 0.876 GB. Table 1 · DTU, three input views · arXiv v3
• With cascaded depth estimation, combining Gaussian splatting and volume rendering raises DTU PSNR from the splatting-only setting’s 27.48 to 28.21 dB and Tanks-and-Temples PSNR from 21.70 to 23.28 dB. Table 4 · reconstruction ablation · arXiv v3

Cite this paper

Tianqi Liu, Guangcong Wang, Shoukang Hu, Liao Shen, Xinyi Ye, Yuhang Zang, Zhiguo Cao, Wei Li, Ziwei Liu. MVSGaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo. Computer Vision – ECCV 2024, 2025, pp. 37–53.

The conference took place in 2024; the proceedings volume was published in 2025.

@inproceedings{arxiv240512218,
  title     = {{MVSGaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo}},
  author    = {Tianqi Liu and Guangcong Wang and Shoukang Hu and Liao Shen and Xinyi Ye and Yuhang Zang and Zhiguo Cao and Wei Li and Ziwei Liu},
  booktitle = {Computer Vision – ECCV 2024},
  year      = {2025},
  publisher = {Springer Nature Switzerland},
  pages     = {37--53},
  doi       = {10.1007/978-3-031-72649-1\_3},
  url       = {https://doi.org/10.1007/978-3-031-72649-1_3}
}