DAMO-StreamNet: Optimizing Streaming Perception for Autonomous Driving

DAMO-StreamNet is a novel streaming perception framework for real-time video object detection in autonomous driving scenarios. It builds upon state-of-the-art models like YOLO and LongShortNet to achieve optimized accuracy under strict latency constraints.

Key Features

• Robust Neck Design: Incorporates deformable convolution to enhance receptive fields and feature alignment.

• Dual-Branch Structure: Fuses semantic and temporal features for accurate motion prediction.

• Asymmetric Distillation: Distills future knowledge from teacher to student network during training for performance gains.

• Real-time Forecasting: Continuously updates support frames for seamless streaming.

For more details, please see our full IJCAI 2023 paper.

Usage

DAMO-StreamNet supports real-time detection of 8 classes relevant to autonomous driving:

• Person, Bicycle, Car, Motorcycle, Bus, Truck, Traffic Light, Stop Sign

See ModelScope Documentation for code examples to run inference using our pretrained models.

Model Zoo

Model	Input Size	Velocity	sAP 0.5:0.95	sAP50	sAP75	COCO Weights	Checkpoint
DAMO-StreamNet-S	600x960	1x	31.8	52.3	31.0	link	link
DAMO-StreamNet-M	600x960	1x	35.5	57.0	36.2	link	link
DAMO-StreamNet-L	600x960	1x	37.8	59.1	38.6	link	link
DAMO-StreamNet-L	1200x1920	1x	43.3	66.1	44.6	link	link

Teacher models available here.

Installation

Follow install guidelines from StreamYOLO and LongShortNet.

Quick Start

Dataset Preparation

Follow Argoverse-HD setup instructions.

Model Preparation

Organize downloaded models:

./models
├── checkpoints
│   ├── streamnet_l_1200x1920.pth
│   ├── streamnet_l.pth
│   ├── streamnet_m.pth
│   └── streamnet_s.pth
├── coco_pretrained_models
│   ├── yolox_l_drfpn.pth
│   ├── yolox_m_drfpn.pth
│   └── yolox_s_drfpn.pth  
└── teacher_models
    └── l_s50_still_dfp_flip_ep8_4_gpus_bs_8
        └── best_ckpt.pth

Training

bash run_train.sh

Evaluation

bash run_eval.sh 

Training Details

• 8 Epochs on Argoverse-HD
• SGD Optimizer with Linear LR Schedule
• Random Flip Augmentation
• Multi-Scale Training

References

Please cite our paper:

@article{DAMO_StreamNet,
  title={DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Driving},
  author={Jun-Yan He, Zhi-Qi Cheng, Chenyang Li, Wangmeng Xiang, Binghui Chen, Bin Luo, Yifeng Geng, Xuansong Xie},
  journal={IJCAI},  
  year={2023}
}

DAMO-StreamNet builds on YOLO, LongShortNet and StreamYOLO.

License

For academic research only. Please contact authors for commercial licensing.