Skip to content

Latest commit

 

History

History
154 lines (114 loc) · 6.74 KB

README.md

File metadata and controls

154 lines (114 loc) · 6.74 KB
Raw

SimIPU

SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations

Zhenyu Li, Zehui Chen, Ang Li, Liangji Fang, Qinhong Jiang, Xianming Liu, Junjun Jiang, Bolei Zhou, Hang Zhao

AAAI 2021 (arXiv pdf)

Notice

  • Redundancy version of SimIPU. Main codes are in SimIPU/project_cl.
  • You can find codes of MonoDepth here. We provide detailed configs and results, even in an indoor environment depth dataset, which demonstrates the generalization of SimIPU. Since we enhance the depth framework, model performances are stronger than the ones presented in our paper.

Usage

Installation

This repo is tested on python=3.7, cuda=10.1, pytorch=1.6.0, mmcv-full=1.3.4, mmdetection=2.11.0, mmsegmentation=0.13.0 and mmdetection3D=0.13.0.

Note: since mmdetection and mmdetection3D have made huge compatibility change in their latest versions, their latest version is not compatible with this repo. Make sure you install the correct version.

Follow instructions below to install:

  • Create a conda environment
conda create -n simipu python=3.7
conda activate monocon
git clone https://github.com/zhyever/SimIPU.git
cd SimIPU
  • Install Pytorch 1.6.0
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch
  • Install mmcv-full=1.3.4
pip install mmcv-full==1.3.4 -f https://download.openmmlab.com/mmcv/dist/cu101/torch1.6.0/index.html
  • Install mmdetection=2.11.0
git clone https://github.com/open-mmlab/mmdetection.git
cd ./mmdetection
git checkout v2.11.0
pip install -r requirements/build.txt
pip install -v -e .
cd ..
  • Install mmsegmentation=0.13.0
pip install mmsegmentation==0.13.0
  • Build SimIPU
# remember you have "cd SimIPU"
pip install -v -e .
  • Others Maybe there will be notice that there is no required future package after build SimIPU. Install it via conda.
conda install future

Data Preparation

Download KITTI dataset and organize data following the official instructions in mmdetection3D. Then generate data by running:

python tools/create_data.py kitti --root-path ./data/kitti --out-dir ./data/kitti --extra-tag kitti

If you would like to run experiments on Mono3D Nus, you should follow the official instructions to prepare the NuScenes dataset.

For Waymo pre-training, we have no plan to release corresponding data-preparing scripts for a short time. Some of the scripts are presented in project_cl/tools/. I just have no effort or resources to reproduce the Waymo pre-training process. Since we provide how to prepare the Waymo dataset in our paper, if you have a problem to achieve it, feel free to contact me and I would like to help you.

Pre-training on KITTI

bash tools/dist_train.sh project_cl/configs/simipu/simipu_kitti.py 8 --work-dir work_dir/your/work/dir

Downstream Evaluation

1. Camera-lidar fusion based 3D object detection on kitti dataset.

Remember to change the pre-trained model via changing the value of key load_from in the config.

bash tools/dist_train.sh project_cl/configs/kitti_det3d/moca_r50_kitti.py 8 --work-dir work_dir/your/work/dir

2. Monocular 3D object detection on Nuscenes dataset.

Remember to change the pre-trained model via changing the value of key load_from in the config. Before training, you also need align the key name in checkpoint['state_dict']. See project_cl/tools/convert_pretrain_imgbackbone.py for details.

bash tools/dist_train.sh project_cl/configs/fcos3d_mono3d/fcos3d_r50_nus.py 8 --work-dir work_dir/your/work/dir

2. Monocular Depth Estimation on KITTI/NYU dataset.

See Depth-Estimation-Toolbox.

Pre-trained Model and Results

We provide pre-trained models. As default, the "Full Waymo or Waymo" presents Waymo dataset with load_interval=5. We use discrete frames to ensure training variety. Previous experiments indicate model improvement with load_interval=1 is slight. So actually, 1/10 Waymo means 1/5 (load_interval=5) times 1/10 (use first 1/10 scene data) = 1/50 Waymo data.

Dataset Model
SimIPU KITTI link
SimIPU Waymo link
SimIPU ImageNet Sup + Waymo SimIPU link

Fusion-based 3D object detection results.

AP40@Easy AP40@Mod. AP40@Hard Link
Moca 81.32 70.88 66.19 Log

Monocular 3D object detection results.

Pre-train mAP Link
Fcos3D Scratch 17.9 Log
Fcos3D 1/10 Waymo SimIPU 20.3 Log
Fcos3D 1/5 Waymo SimIPU 22.5 Log
Fcos3D 1/2 Waymo SimIPU 24.7 Log
Fcos3D Full Waymo SimIPU 26.2 Log
Fcos3D ImageNet Sup 27.7 Log
Fcos3D ImageNet Sup + Full Waymo SimIPU 28.4 Log

Citation

If you find our work useful for your research, please consider citing the paper

@article{li2021simipu,
  title={SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations},
  author={Li, Zhenyu and Chen, Zehui and Li, Ang and Fang, Liangji and Jiang, Qinhong and Liu, Xianming and Jiang, Junjun and Zhou, Bolei and Zhao, Hang},
  journal={arXiv preprint arXiv:2112.04680},
  year={2021}
}