Skip to content
/ ALSO Public

ALSO: Automotive Lidar Self-supervision by Occupancy estimation

License

Notifications You must be signed in to change notification settings

valeoai/ALSO

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ALSO: Automotive Lidar Self-supervision by Occupancy estimation

Alexandre Boulch1    Corentin Sautier1,2    Björn Michele1,3    Gilles Puy1    Renaud Marlet1,2

1 Valeo.ai, Paris, France 2 LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Vallée, France

3 CNRS, IRISA, Univ. Bretagne Sud, Vannes, France


Arxiv


ALSO has been accepted at CVPR 2023


Overview


This is the reference PyTorch implementation for training and testing self-supervision for automotive lidar point cloud using the method described in our paper ALSO: Automotive Lidar Self-supervision by Occupancy estimation

Overview


Citation

Please acknowledge our work in your publications:

@InProceedings{ALSO,
author = {Alexandre Boulch and Corentin Sautier and Björn Michele and Gilles Puy and Renaud Marlet},
title = {{ALSO}: Automotive Lidar Self-supervision by Occupancy estimation},
booktitle = {International Conference on Computer Vision and Pattern Recognition (CVPR)},
year = 2023,
}

Dependencies

We give the library version(s), we used for experiments.

  • Hydra (1.2.0)
  • Pytorch-geometric (2.1.0)
  • Pytorch-Lightning (1.7.2)

Semantic segmentation specific libraries

  • torchsparse (1.4.0)

Detection specific libraries

  • spconv (2.1.21)
  • openpcdet (0.6.0)

Installation

At root of the folder run:

pip install -ve .

Then all scripts can be run from also_selfsup folder:

cd also_selfsup

Datasets

ONCE

We follow intruction from ONCE dataset to generate the information files of the raw_small dataset.

Semantic segmentation

Pre-training

python train_selfsupervised.py cfg=nuscenes

for a pre-training on nuScenes.

for other datasets, replace cfg=nuscenes by the desired dataset

Parameters

We use Hydra for easy configuration of trainings. The pretraining parameters are set for pre-training with a NVidia-V100 16Gb. In order to pre-train on other configurations, one can modify parameters directly in the command line, e.g., setting the batch size to 8:

python train_selfsupervised.py ... cfg.training.batch_size=8

or the backbone to SPVCNN:

python train_selfsupervised.py ... cfg.network.backbone=SPVCNN

Downstream

From scratch

# 100%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=30 cfg.val_interval=5 cfg.downstream.skip_ratio=1

# 50%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=50 cfg.val_interval=5 cfg.downstream.skip_ratio=2

# 10%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=100 cfg.val_interval=10 cfg.downstream.skip_ratio=10

# 1%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=500 cfg.val_interval=50 cfg.downstream.skip_ratio=100

# 0.1%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=1000 cfg.val_interval=100 cfg.downstream.skip_ratio=1000

With pretrained models

First, convert the pre-trained model to be loaded

python convert_models.py --ckpt path_to_pretraining_checkpoint

Second, run downstream training

python train_downstream_semseg.py cfg=nuscenes cfg.downstream.checkpoint_dir='path_to_checkpoint_directory' cfg.downstream.checkpoint_name='pretrained_backbone_XXX.ckpt'

Evaluation of downstream models

First, convert the trained model to be loaded

python convert_models.py --downstream --ckpt path_to_downstream_checkpoint

Second, run the evaluation script

python eval.py --split val --config path_to_downstream_model/config.yaml --ckpt path_to_downstream_checkpoint/trained_model_XXX.ckpt

Detection

Pre-training

Similar to semantic segmentation, pre-training for detection can simply be run using:

python train_selfsupervised.py cfg=kitti3d_second
python train_selfsupervised.py cfg=nuscenes_second_kitti

Downstream

First, convert the trained model to be loaded

python convert_models.py --ckpt path_to_pretraining_checkpoint

Second, use the pre-trained model with the official OpenPCDet code:

cd path_to_OpenPCDet/tools/
python train.py --cfg_file cfgs/kitti_models/second.yaml --extra_tag --pretrained_model path_to_pretrained_model/pretrained_backbone_XXX.ckpt

Pre-trained models

Task Pre-training dataset Model Link Notes
Semantic segmentation nuScenes MinkUNet34 download
    SPVCNN download
  SemanticKITTI MinkUNet18SC download
    SPVCNN download
Detection KITTI3D SECOND/PVRCNN download trained for KITTI3D
  nuScenes SECOND/PVRCNN download trained for KITTI3D
  KITTI360 SECOND/PVRCNN download trained for KITTI3D
  ONCE Small SECOND/PVRCNN download trained for ONCE

Here are links to pre-trained models

Acknowledgments

This project would not have been possible without many community resources and repositories. Among them:

Please, consider acknowleding these projects.