Reproducibility of "Campus3D:A Photogrammetry Point Cloud Benchmark for Outdoor Scene Hierarchical Understanding"
The repository contains the re-implementation of this ACM MM 2020 Paper based on the repository. It also presents the reproduced results of the supported paper with trained models in MODEL ZOO. The reduced version of Campus3D dataset can be donwloaded from the official website. Compared to the original repo, the main updates of this reproducibilty repo are:
- Transfer the model implementation from TenforFlow with precompling requiremnets to pure PyTorch
- Add DGCNN and PointCNN to the benchmark models. Report the detailed benchmarks with mutiple networks and their pre-trained models.
- Upgrade the sampling method with KDTree, more details can be found in
dataset/sampling/sampler_collections.py
The whole package can be downloaded by the following command.
git clone https://github.com/Yuqing-Liao/reproduce-campus3d.git
Dependencies can be installed using the provided script.
cd reproduce-campus3d
pip install -r requirements.txt
Compressed Campus3D dataset file campus3d-reduce.zip can be downloaded from official website. Put it into data/ and unzip with below script.
cd reproduce-campus3d/data
unzip campus3d-reduce.zip
To apply training of the model, please first check the configuration files in config/. Particularly you need to change the value of IS_PRETRAINED to False and then run experiments, eg:
cd reproduce-campus3d
python run.py --model 'pointnet2' --mc_level -1 --exp_name 'EXP_NAME'
The 'EXP_NAME' is the user-defined name. In this way, the models will be saved in checkpoints/EXP_NAME/models, and other output files will be saved in checkpoints/EXP_NAME.
Pretrained models are available on Google Drive, and they can be downloaded through the link presented in the following table. You can train either from the downloaded models or from your own pretrained models. To apply training of the model, please first check the configuration files in config/. Particularly you need to change the value of IS_PRETRAINED to True, PRETRAINED_MODEL_PATH to the path of the model to train and then run experiments, eg:
cd reproduce-campus3d
python run.py --model 'pointnet2' --mc_level -1 --exp_name 'EXP_NAME'
In this way, the models will be saved in checkpoints/EXP_NAME/models, and other output files will be saved in checkpoints/EXP_NAME.
To apply evaluation of the model on the test set, please first check the configuration files in config/. Particularly you need to change the value of PRETRIANED_MODEL_PATH to the path of the model to evaluate and then run experiments, eg:
cd reproduce-campus3d
python run.py --eval true --model 'pointnet2' --mc_level -1 --exp_name 'EXP_NAME'
In this way, the output files will be saved in check/EXP_NAME.
The hierarchical learning experiments were proposed to present the effectiveness of the Multi-task and Hierarchical Esemble(MT+HE) method. Multi-classifiers(MC) in each level were also proposed for comparison. To run the training, the argument --mc_level can be set as 0-4 and -1 for MC experiments in 0-4 levels and MT+HE experiments in all levels respectively. In addition, the MT training contains two stage Multi-task Learning without consistency loss(MTnc) and Multi-task Learning with consistency loss(MT), of which the MT is trained based on the pretrained MTnc model.
To run the evaluation, the argument --mc_level in run script can be set as any one from 0-4 to test 5 MC models altogether for MC and MC+HE results, or set as -1 to test the target model for MTnc or MT(+HE) results. Particularly for MC evaluation, the PRETRAINED_MODEL_PATH is supposed to be set as the list of 5 model paths in the config file.
The semantic segmentation bechmark were built with three models PointNet++, PointCNN and DGCNN. They are all conducted via the MT+HE method for hierarchical learning on the Campus3D dataset. To run different models, one can change the argument --model as the indicated model. Following are the reference repository for PyTorch implementation of 3D deep models.
PointNet++ GitHub Link
PointCNN GitHub Link
DGCNN GitHub Link
| No. | Model | Name | Method | MC Level | Training Process | Scheduler | Download Link |
|---|---|---|---|---|---|---|---|
| 0 | PointNet++ | 'pointnet2' | MC | 0 | 50 epochs(lr=0.01) | cos | MC0.t7 |
| 1 | PointNe++ | 'pointnet2' | MC | 1 | 50 epochs(lr=0.01) | cos | MC1.t7 |
| 2 | PointNet++ | 'pointnet2' | MC | 2 | 50 epochs(lr=0.01) | cos | MC2.t7 |
| 3 | PointNet++ | 'pointnet2' | MC | 3 | 50 epochs(lr=0.01) | cos | MC3.t7 |
| 4 | PointNet++ | 'pointnet2' | MC | 4 | 50 epochs(lr=0.01) | cos | MC4.t7 |
| 5 | PointNet++ | 'pointnet2' | MTnc | -1 | 50 epochs (lr=0.01) | cos | pointnet2_MTnc.t7 |
| 6 | PointCNN | 'pointcnn' | MTnc | -1 | 50 epochs (lr=0.01) | cos | pointcnn_MTnc.t7 |
| 7 | DGCNN | 'dgcnn' | MTnc | -1 | 50 epochs (lr=0.01) | cos | dgcnn_MTnc.t7 |
| 8 | Pointnet++ | 'pointnet2' | MT | -1 | 50 epochs (lr=0.01) + 20 epochs with consistency loss (lr=0.01) |
cos | pointnet2_MT.t7 |
| 9 | PointCNN | 'pointcnn' | MT | -1 | 50 epochs (lr=0.01) + 30 epochs with consistency loss (lr=0.01) |
cos | pointcnn_MT.t7 |
| 10 | DGCNN | 'dgcnn' | MT | -1 | 50 epochs (lr=0.01) + 20 epochs with consistency loss (lr=0.01) |
cos | dgcnn_MT.t7 |
| Benchmark | Model | C1 | C2 | C3 | C4 | C5 |
|---|---|---|---|---|---|---|
| OA% | PointNet++ | 91.4 | 87.5 | 86.7 | 85.0 | 75.1 |
| OA% | PointCNN | 88.9 | 79.3 | 78.7 | 76.8 | 63.8 |
| OA% | DGCNN | 94.7 | 90.6 | 89.1 | 87.2 | 81.5 |
| mIoU% | PointNet++ | 83.8 | 74.3 | 58.0 | 37.1 | 22.3 |
| mIoU% | PointCNN | 79.7 | 61.5 | 42.8 | 26.3 | 15.0 |
| mIoU% | DGCNN | 89.6 | 80.1 | 63.3 | 43.1 | 28.4 |
These results are produced by model No.8, No.9 and No.10.
| Method | C1 | C2 | C3 | C4 | C5 |
|---|---|---|---|---|---|
| MC | 90.8 | 86.2 | 84.4 | 83.6 | 73.6 |
| MC+HE | 91.4 | 87.4 | 86.5 | 84.8 | 74.9 |
| MTnc | 90.6 | 86.0 | 85.0 | 83.1 | 73.3 |
| MT | 91.4 | 87.4 | 86.7 | 84.9 | 75.2 |
| MT+HE | 91.4 | 87.5 | 86.7 | 85.0 | 75.1 |
These results are produced by model No.0-4, No.5 and No.8. They demonstrate the effectiveness of the MT+HE method for HL problem. Results with detailed per-class IoU are displayed below.
| Granularity Level | Class | MC | MC+HE | MTnc | MT | MT+HE |
|---|---|---|---|---|---|---|
| C1 | ground | 85.4 | 86.4 | 85.3 | 86.1 | 86.1 |
| C1 | construction | 79.9 | 80.8 | 79.4 | 81.4 | 81.5 |
| C2 | natural | 81.1 | 82.4 | 80.8 | 82.9 | 82.9 |
| C2 | man_made | 58.5 | 60.9 | 58.7 | 58.1 | 58.5 |
| C2 | construction | 78.8 | 80.8 | 78.5 | 81.3 | 81.5 |
| C3 | natural | 79.2 | 82.4 | 80.8 | 82.9 | 82.9 |
| C3 | play_field | 62.9 | 65.9 | 56.1 | 66.5 | 67.3 |
| C3 | path&stair | 8.7 | 8.2 | 8.7 | 0.0 | 0.0 |
| C3 | driving_road | 58.4 | 60.6 | 57.7 | 58.4 | 58.5 |
| C3 | construction | 76.6 | 80.8 | 78.2 | 81.4 | 81.5 |
| C4 | natural | 81.0 | 82.4 | 80.4 | 82.9 | 82.9 |
| C4 | play_field | 57.3 | 65.9 | 54.0 | 68.2 | 67.3 |
| C4 | path&stair | 9.3 | 8.2 | 8.7 | 0.0 | 0.0 |
| C4 | vehicle | 16.6 | 19.4 | 16.7 | 9.4 | 9.9 |
| C4 | not vehicle | 57.9 | 59.9 | 57.2 | 57.8 | 57.9 |
| C4 | building | 76.5 | 78.2 | 75.4 | 78.8 | 78.8 |
| C4 | link | 0.0 | 0.1 | 0.2 | 0.0 | 0.0 |
| C4 | facility | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| C5 | natural | 80.1 | 82.4 | 80.5 | 83.0 | 82.9 |
| C5 | play_field | 52.7 | 65.9 | 53.5 | 67.0 | 67.3 |
| C5 | sheltered | 10.6 | 7.9 | 9.0 | 0.0 | 0.0 |
| C5 | unsheltered | 7.6 | 7.9 | 8.3 | 0.0 | 0.0 |
| C5 | bus_stop | 0.0 | 0.0 | 0.1 | 0.0 | 0.0 |
| C5 | car | 18.1 | 19.5 | 16.7 | 10.6 | 9.9 |
| C5 | bus | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| C5 | not vehicle | 58.1 | 59.9 | 57.4 | 57.7 | 57.9 |
| C5 | wall | 46.5 | 47.3 | 45.8 | 47.3 | 47.1 |
| C5 | roof | 43.1 | 44.2 | 41.7 | 47.4 | 47.4 |
| C5 | link | 0.2 | 0.1 | 0.4 | 0.0 | 0.0 |
| C5 | artificial_landscape | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| C5 | lamp | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| C5 | others | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |