Mip-Splatting: Alias-free 3D Gaussian Splatting

Zehao Yu · Anpei Chen · Binbin Huang · Torsten Sattler · Andreas Geiger

CVPR 2024 Best Student Paper

Paper | arXiv | Project Page | Online Viewer

We introduce a 3D smoothing filter and a 2D Mip filter for 3D Gaussian Splatting (3DGS), eliminating multiple artifacts and achieving alias-free renderings.

Update

We integrated an improved densification metric proposed in Gaussian Opacity Fields, which significantly improves the novel view synthesis results, please check the paper for details. Please download the lastest code and reinstall diff-gaussian-rasterization to try it out.

Installation

Clone the repository and create an anaconda environment using

git clone git@github.com:autonomousvision/mip-splatting.git
cd mip-splatting

conda create -y -n mip-splatting python=3.8
conda activate mip-splatting

pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html
conda install cudatoolkit-dev=11.3 -c conda-forge

pip install -r requirements.txt

pip install submodules/diff-gaussian-rasterization
pip install submodules/simple-knn/

Dataset

Blender Dataset

Please download and unzip nerf_synthetic.zip from the NeRF's official Google Drive. Then generate multi-scale blender dataset with

python convert_blender_data.py --blender_dir nerf_synthetic/ --out_dir multi-scale

Mip-NeRF 360 Dataset

Please download the data from the Mip-NeRF 360 and request the authors for the treehill and flowers scenes.

Training and Evaluation

# single-scale training and multi-scale testing on NeRF-synthetic dataset
python scripts/run_nerf_synthetic_stmt.py 

# multi-scale training and multi-scale testing on NeRF-synthetic dataset
python scripts/run_nerf_synthetic_mtmt.py 

# single-scale training and single-scale testing on the mip-nerf 360 dataset
python scripts/run_mipnerf360.py 

# single-scale training and multi-scale testing on the mip-nerf 360 dataset
python scripts/run_mipnerf360_stmt.py 

Online viewer

After training, you can fuse the 3D smoothing filter to the Gaussian parameters with

python create_fused_ply.py -m {model_dir}/{scene} --output_ply fused/{scene}_fused.ply"

Then use our online viewer to visualize the trained model.

Acknowledgements

This project is built upon 3DGS. Please follow the license of 3DGS. We thank all the authors for their great work and repos.

Citation

If you find our code or paper useful, please cite

@InProceedings{Yu2024MipSplatting,
    author    = {Yu, Zehao and Chen, Anpei and Huang, Binbin and Sattler, Torsten and Geiger, Andreas},
    title     = {Mip-Splatting: Alias-free 3D Gaussian Splatting},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {19447-19456}
}

If you find our improved densification metric useful, please kindly cite

@article{Yu2024GOF,
  author    = {Yu, Zehao and Sattler, Torsten and Geiger, Andreas},
  title     = {Gaussian Opacity Fields: Efficient High-quality Compact Surface Reconstruction in Unbounded Scenes},
  journal   = {arXiv:2404.10772},
  year      = {2024},
}