Vehicle Detection using Pre-trained Large Vision-Language Foundation Models
VFM-Det: Towards High-Performance Vehicle Detection via Large Foundation Models, Wentao Wu†, Fanghua Hong†, Xiao Wang*, Chenglong Li, Jin Tang [Paper] [Code] [DemoVideo]
- [2024.08.23] The source code is released.
Existing vehicle detectors are usually obtained by training a typical detector (e.g., YOLO, RCNN, DETR series) on vehicle images based on a pre-trained backbone (e.g., ResNet, ViT). Some researchers also exploit and enhance the detection performance using pre-trained large foundation models. However, we think these detectors may only get sub-optimal results because the large models they use are not specifically designed for vehicles. In addition, their results heavily rely on visual features, and seldom of they consider the alignment between the vehicle's semantic information and visual representations. In this work, we propose a new vehicle detection paradigm based on a pre-trained foundation vehicle model (VehicleMAE) and a large language model (T5), termed VFM-Det. It follows the region proposal-based detection framework and the features of each proposal can be enhanced using VehicleMAE. More importantly, we propose a new VAtt2Vec module that predicts the vehicle semantic attributes of these proposals and transforms them into feature vectors to enhance the vision features via contrastive learning. Extensive experiments on three vehicle detection benchmark datasets thoroughly proved the effectiveness of our vehicle detector. Specifically, our model improves the baseline approach by
Configure the environment according to the content of the requirements.txt file.
#If you training VFM-Det using a single GPU, please run.
CUDA_VISIBLE_DEVICES=0 python train.py
#If you testing VFM-Det, please run.
CUDA_VISIBLE_DEVICES=0 python validation.py
Datasets
Cityscapes dataset download address：https://www.cityscapes-dataset.com/
COCO2017 dataset download address： http://images.cocodataset.org/zips/train2017.zip http://images.cocodataset.org/annotations/annotations_trainval2017.zip http://images.cocodataset.org/zips/val2017.zip http://images.cocodataset.org/annotations/stuff_annotations_trainval2017.zip http://images.cocodataset.org/zips/test2017.zip http://images.cocodataset.org/annotations/image_info_test2017.zip
UA-DETRAC dataset download address：https://www.albany.edu/cnse/research/computer-vision-machine-learning-lab
- Thanks for the WZMIAOMIAO library for a quickly implement.
If you find this work helps your research, please cite the following work and give us a star. Any questions you have, please leave an issue.
@misc{wu2024VFMDet,
title={VFM-Det: Towards High-Performance Vehicle Detection via Large Foundation Models},
author={Wentao Wu and Fanghua Hong and Xiao Wang and Chenglong Li and Jin Tang},
year={2024},
eprint={2408.13031},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2408.13031},
}