This work is accepted for publication in IEEE Sensors Journal.
In addition to the original deep neural network architecture, we added different losses to constrain the convergence of the model. Also, the preprocessing of dataset is required for fast training and convergence.
We are releasing our approach's source code for auto-calibrating camera and lidar sensors to share with the scientific community and industry with the aim of the collaboration with people around the world to push the boundaries in the field of autonomous driving and intelligent transportation systems.
The figure shows the comparison between decalibration (x-axis, artificial errors) and errors after auto-calibration (y-axis) for rotational and translational axes.
Method | second per frame |
---|---|
Ours | 0.0047 |
Go-ICP | 1.8221 |
RGGNet | 0.0244 |
The method is tested based on the following setup, it may run on other setup as well. Version of packages may not necessarily fully match the numbers below.
Env | Version |
---|---|
System | 20.04 |
Python | 3.8 |
Numpy | 1.18.4 |
PyTorch | 1.5.0 |
torchvision | 0.6.0 |
tqdm | 4.46.0 |
Pillow | 7.1.2 |
For a full list of all dependencies, please visit here.
- Clone the repo to your local drive.
- Setup your own Python environment and install necessary dependencies (using a virtual environment is recommended).
- The installation of
pytorch
andtorchvision
may vary in the future because of variable CUDA versions, you can install manually via the official website.
In general, the inputs for the model are a pair of a LiDAR projected depth map and a camera depth map of the same dimension.
KITTI raw dataset from the date 26_09
is used and preprocessed before training.
Drive 0005
and 0070
are split out for testing.
Drive 0013
, 0020
, and 0079
are split out for validation.
The rest drives are used for training.
For all drives, depth map (located in image_c
folder) is calculated from stereo color images,
and LiDAR depth map (located in image_l
folder) is also calculated with an artificial error.
The artificial errors are recorded (in gt.txt
file) for each drive in the following format,
- img_filename, roll err, pitch err, yaw err, x err, y err, z err
Example for gt.txt
:
0000000000.png,0.14150073,0.059256576,0.011725272,0.16007939,-0.18555182,0.04104398
...
0000000107.png,-0.06508537,0.14815551,-0.15871142,0.07438427,0.09373859,0.12373618
The structure of the dataset should be organized like this:
├── KITTI ROOT
└── train
└── 2011_09_26
├── 2011_09_26_drive_0001_sync
├── image_c
├── 0000000000.png
├── ...
└── 0000000107.png
├── image_l
├── 0000000000.png
├── ...
└── 0000000107.png
└── gt.txt
├── 2011_09_26_drive_0002_sync
└── ...
├── ...
├── calib_cam_to_cam.txt
├── calib_imu_to_velo.txt
└── calib_velo_to_cam.txt
├── test
└── 2011_09_26
├── 2011_09_26_drive_0005_sync
└── ...
├── 2011_09_26_drive_0070_sync
└── ...
├── calib_cam_to_cam.txt
├── calib_imu_to_velo.txt
└── calib_velo_to_cam.txt
└── val
└── 2011_09_26
├── 2011_09_26_drive_0013_sync
└── ...
├── 2011_09_26_drive_0020_sync
└── ...
├── 2011_09_26_drive_0079_sync
└── ...
├── calib_cam_to_cam.txt
├── calib_imu_to_velo.txt
└── calib_velo_to_cam.txt
In dataset_preprocessing/
folder, there's a utility script (download.sh, run at the folder you want to download) for downloading KITTI dataset
(Note: the script may not work as intended now).
After downloading and unarchiving, the dataset should be reorganized as shown above.
There is another script (preprocess.sh) for generating both LiDAR depth maps and camera depth maps, but need a little modification. You need to add your selected algorithm for camera depth in preprocess.py in order to run the script.
Note that the preprocessed camera depth images and projected LiDAR depth images should be cropped into the shape of (352, 1216) for height and width (dim 0 & 1) respectively.
Modify the configurations in src/train.sh
and read the available parameters from src/train.py
.
Run the training process is as simple as run the following command after configured the parameters.
bash train.sh
In order to continue training, you should modify the configuration in train.sh
to use the checkpoint file you have saved.
You can test the trained model by the command below,
python eval.py --ckpt path/to/ckpt.tar --visualization --rotation_offsest 10 --translation_offsest 0.2
- Use
--visualization
flag to show the video feed window (OpenCV required). --rotation_offsest
and--translation_offsest
used during the training process are required to correctly recognize the output from the model.
NetCalib is released under a MIT License license.
For a closed-source version of NetCalib for commercial purposes, please contact the authors: Wu and Hadachi
Shan Wu; Amnir Hadachi; Damien Vivet; Yadu Prabhakar.
If you use NetCalib in an academic work, please cite:
@article{wu2021this,
title={This is The Way: Sensors Auto-calibration Approach Based on Deep Learning for Self-driving Cars},
author={Shan, Wu; Hadachi, Amnir; Vivet, Damien; Prabhakar, Yadu},
journal={IEEE Sensors Journal},
year={2021},
publisher={IEEE}
}
Published paper is here.