HCAT

Official implementation of the HCAT , including training code and trained models.

Tracker

HCAT download model

In this work, we present an efficient tracking method via a hierarchical cross-attention transformer named HCAT. Our model runs about 195 f ps on GPU, 45 fps on CPU, and 55 fps on the edge AI platform of NVidia Jetson AGX Xavier. Experiments show that our HCAT achieves promising results on LaSOT, GOT-10k,TrackingNet, NFS, OTB100, UAV123, and VOT2020.

Results

Model	**LaSOT AUC(%) **	**TrackingNet **AUC(%)	GOT-10k AO (%)	Speed-GPU (fps)	Speed-CPU (fps)	Speed-AGX (fps)
Res18_N1_q16	57.9	74.2	61.3	240	46	69
Res18_N2_q16	59.1	76.6	65.3	195	45	55
Res50_N2_q16	59.1	77.9	67.8	115	22	42
Lighttrack_N2_q16	59.8	76.6	66.3	100	45	34
Convnext_tiny_N2_q16	63.1	80.5	70.1	136	21	34

• The reported speed is the speed of the model itself, and does not include pre- and post-processing of the image(e.g., cropping the search region)
• Download The RawResults

Installation

This document contains detailed instructions for installing the necessary dependencied for HCAT. The instructions have been tested on Ubuntu 18.04 system.

Install dependencies

• Create and activate a conda environment

conda create -n hcat python=3.7
conda activate hcat

• Install PyTorch

conda install -c pytorch pytorch=1.5 torchvision=0.6.1 cudatoolkit=10.2

• Install other packages

conda install matplotlib pandas tqdm
pip install opencv-python tb-nightly visdom scikit-image tikzplotlib gdown timm
conda install cython scipy
sudo apt-get install libturbojpeg
pip install pycocotools jpeg4py
pip install wget yacs
pip install shapely==1.6.4.post2

• Install onnx and onnxruntime
• Here the version of onnxruntime-gpu needs to be compatible to the CUDA version and CUDNN version on the machine. For more details, please refer to https://www.onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html . For example, on my computer, CUDA version is 10.2, CUDNN version is 8.0.3, so I choose onnxruntime-gpu==1.6.0

pip install onnx onnxruntime-gpu==1.6.0

• Setup the environment
  Create the default environment setting files.

# Change directory to <PATH_of_HCAT>
cd HCAT

# Environment settings for pytracking. Saved at pytracking/evaluation/local.py
python -c "from pytracking.evaluation.environment import create_default_local_file; create_default_local_file()"

# Environment settings for ltr. Saved at ltr/admin/local.py
python -c "from ltr.admin.environment import create_default_local_file; create_default_local_file()"

You can modify these files to set the paths to datasets, results paths etc.

• Add the project path to environment variables
  Open ~/.bashrc, and add the following line to the end. Note to change <path_of_HCAT> to your real path.

export PYTHONPATH=<path_of_HCAT>:$PYTHONPATH

• Download the pre-trained networks
  Download the network for HCAT and put it in the directory set by "network_path" in "pytracking/evaluation/local.py". By default, it is set to pytracking/networks.

Quick Start

TRAINING

• Modify local.py to set the paths to datasets, results paths etc.
• Runing the following commands to train the HCAT. You can customize some parameters by modifying hcat.py

conda activate hcat
cd HCAT/ltr
python run_training.py hcat hcat 
# for ddp
# python run_training_ddp.py hcat hcat --local_rank 4

Convert Model

• Convert model to onnx

conda activate hcat
cd HCAT/pysot_toolkit
python pytorch2onnx.py

Evaluation

• We integrated PySOT for evaluation. You can download json files in PySOT or here.

  For pytorch model, You need to specify the path of the model and dataset in the test.py.

  net_path = '/path_to_model' #Absolute path of the model
  dataset_root= '/path_to_datasets' #Absolute path of the datasets

  Then run the following commands

  conda activate hcat
  cd HCAT
  python -u pysot_toolkit/test.py --dataset <name of dataset> --name 'HCAT' #test tracker #test tracker
  python pysot_toolkit/eval.py --tracker_path results/ --dataset <name of dataset> --num 1 --tracker_prefix 'hcat' #eval tracker

  For onnx model, You need to specify the path of the model and dataset in the test_onnx.py.

  backbone_path = '/path_to_backbone' #Absolute path of the backbone
  model_path = '/path_to_model' #Absolute path of the model
  dataset_root= '/path_to_datasets' #Absolute path of the datasets

  Then run the following commands

  conda activate hcat
  cd HCAT
  python -u pysot_toolkit/test_onnx.py --dataset <name of dataset> --name 'HCAT' #test tracker #test tracker
  python pysot_toolkit/eval.py --tracker_path results/ --dataset <name of dataset> --num 1 --tracker_prefix 'hcat' #eval tracker

  The testing results will in the current directory(results/dataset/hcat/)

• You can also use pytracking to test and evaluate tracker. The results might be slightly different with PySOT due to the slight difference in implementation (pytracking saves results as integers, pysot toolkit saves the results as decimals).

Getting Help

If you meet problem, please try searching our Github issues, if you can't find solutions, feel free to open a new issue.

• ImportError: cannot import name region

Solution: You can just delete from pysot_toolkit.toolkit.utils.region import vot_overlap, vot_float2str in test.py if you don't test VOT2019/18/16. You can also build region by python setup.py build_ext --inplace in pysot_toolkit.

Acknowledgement

This is a modified version of the python framework PyTracking and Transt based on Pytorch , also borrowing from PySOT and GOT-10k Python Toolkit. We would like to thank their authors for providing great frameworks and toolkits.

Contact

• Xin Chen (email:chenxin3131@mail.dlut.edu.cn)
• Ben Kang(email:kangben@mail.dlut.edu.cn)