Skip to content

Latest commit

 

History

History
53 lines (37 loc) · 9.22 KB

README.md

File metadata and controls

53 lines (37 loc) · 9.22 KB
Raw

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

Abstract

In many robotics and VR/AR applications, 3D-videos are readily-available sources of input (a continuous sequence of depth images, or LIDAR scans). However, those 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose the generalized sparse convolution that encompasses all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in the 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and the trilateral-stationary conditional random field that enforces spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that convolutional neural networks with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise, outperform 3D convolutional neural networks and are faster than the 3D counterpart in some cases.

Introduction

We implement MinkUNet with TorchSparse / Minkowski Engine / Spconv backend and provide the result and checkpoints on SemanticKITTI datasets.

Results and models

SemanticKITTI

Method Backend Lr schd Amp Laser-Polar Mix Mem (GB) Training Time (hours) FPS mIoU Download
MinkUNet18-W16 torchsparse 15e ✔ ✗ 3.4 - - 60.3 model | log
MinkUNet18-W20 torchsparse 15e ✔ ✗ 3.7 - - 61.6 model | log
MinkUNet18-W32 torchsparse 15e ✔ ✗ 4.9 - - 63.1 model | log
MinkUNet34-W32 minkowski engine 3x ✗ ✔ 11.5 6.5 12.2 69.2 model | log
MinkUNet34-W32 spconv 3x ✔ ✔ 6.7 2 14.6* 68.3 model | log
MinkUNet34-W32 spconv 3x ✗ ✔ 10.5 6 14.5 69.3 model | log
MinkUNet34-W32 torchsparse 3x ✔ ✔ 6.6 3 12.8 69.3 model | log
MinkUNet34-W32 torchsparse 3x ✗ ✔ 11.8 5.5 15.9 68.7 model | log
MinkUNet34v2-W32 torchsparse 3x ✔ ✔ 8.9 - - 70.3 model | log

Note: We follow the implementation in SPVNAS original repo and W16\W20\W32 indicates different number of channels.

Note: Due to TorchSparse backend, the model performance is unstable with TorchSparse backend and may fluctuate by about 1.5 mIoU for different random seeds.

Note: Referring to PCSeg, MinkUNet34v2 is modified based on MinkUNet34.

Note*: Training Time and FPS are measured on NVIDIA A100. The versions of Torchsparse, Minkowski Engine and Spconv are 0.5.4, 1.4.0 and 2.3.6 respectively. Since spconv 2.3.6 has a bug with fp16 on in the inference stage, the actual FPS measurement using fp32.

Citation

@inproceedings{choy20194d,
  title={4d spatio-temporal convnets: Minkowski convolutional neural networks},
  author={Choy, Christopher and Gwak, JunYoung and Savarese, Silvio},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
  pages={3075--3084},
  year={2019}
}