This repository provides the X-MARS training and test splits based on the MARS dataset. Details on the X-MARS dataset can be found in my masters thesis.
If you use this work in your research, please kindly cite my thesis as
@article{DBLP:journals/corr/abs-1803-08709,
author = {Andreas Eberle},
title = {Pose-Driven Deep Models for Person Re-Identification},
journal = {CoRR},
volume = {abs/1803.08709},
year = {2018},
url = {http://arxiv.org/abs/1803.08709},
archivePrefix = {arXiv},
eprint = {1803.08709},
timestamp = {Wed, 11 Apr 2018 11:12:46 +0200},
biburl = {https://dblp.org/rec/bib/journals/corr/abs-1803-08709},
bibsource = {dblp computer science bibliography, https://dblp.org}
}
and the work of Zheng et al. which created the MARS dataset as
@proceedings{zheng2016mars,
title={MARS: A Video Benchmark for Large-Scale Person Re-identification},
author={Zheng, Liang and Bie, Zhi and Sun, Yifan and Wang, Jingdong and Su, Chi and Wang, Shengjin and Tian, Qi},
booktitle={European Conference on Computer Vision},
year={2016},
organization={Springer}
}
With real-world applications of person re-id becoming more and more suitable, the amount of required data for training a re-id system becomes an important concern since labeling large amounts of images is a time-consuming and hence expensive task. This holds especially true when working with video datasets to do re-id on tracklets instead of single images. For example the MARS dataset providing tracklets is about 36 times larger than the Market-1501 dataset, while they are both based on the same data source and MARS even contains less IDs than Market-1501. In contrast, being able to train on a standard single-image person re-id dataset (like Market-1501) and using that network for tracklet detection (as provided by MARS) would drastically reduce the number of images to be labeled. Therefore, cross evaluating a person detection system trained on the Market-1501 dataset with the MARS dataset could give insights into these aspects.
Unfortunately, while the MARS and Market-1501 datasets are based on the same data source and labels are assigned consistently, they cannot be used for cross-evaluation since their test and training sets overlap largely. In fact, 48.3% of the MARS test set IDs are contained in the Market-1501 training set diminishing the significance of a cross evaluation.
To enable a meaningful cross-evaluation, a reordering of the MARS dataset's test and training splits called X-MARS is proposed. Since the IDs in Market-1501 and MARS are consistent and the IDs used by MARS are a subset of the IDs used by Market-1501, it is possible to reorder the tracklets of MARS based on the train/test split of Market-1501. This is done by assigning all IDs of MARS (i.e. the union of the test and training IDs) which are part of the Market-1501 training set to the X-MARS training set. The same procedure is applied for the test set of X-MARS.
In order to ease comparability between X-MARS and MARS and allow reusing the evaluation scripts for MARS with X-MARS, the query/gallery split and the used file format are created in the same way as it was done for MARS and provided in the info subfolder of this repository.
Furthermore we provide the code for generating the training, query and gallery splits with the x_mars_creator.py script. Note that the script generates csv files containing the data. The provided matlab files have been created manually from the generated csv files. Also note that the gallery split is chosen by random in the script. Therefore the gallery split will probably differ from the one provided in the files in the selected tracklets but not in the way they have been chosen.
Since X-MARS is just a reordering of the MARS dataset's IDs, the same images/tracklets are used. Therefore please refer to the MARS website for the download of the MARS dataset required to use X-MARS. Furthermore, Zheng et al. also provide their evaluation code for the MARS dataset.
To use the X-MARS dataset simply follow the same protocol as for the MARS dataset but use the train/test splits provided in the info subfolder of this repository. For the evaluation simply replace the info folder from the MARS evaluation code with our info folder and run the normal evaluation scripts.
The following table states the current state of the art. If your results are missing, please open an issue so I can add them.
The table gives the evaluation accuracies for the Market-1501 and X-MARS datasets. Market-1501 results are provided because the models have been trained on this dataset and thus it is of interest to see the drop in accuracies when applying this model on the X-MARS dataset.
| Reference | Market-1501 | X-MARS | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| mAP | R-1 | R-5 | R-10 | R-50 | mAP | R-1 | R-5 | R-10 | R-50 | |
| "Pose-Driven Deep Models for Person Re-Identification", Andreas Eberle, 2018: PSE (Inception-v4) | 64.9 | 84.4 | 93.1 | 95.2 | 98.4 | 58.7 | 75.3 | 85.6 | 88.6 | 93.9 |
| "Pose-Driven Deep Models for Person Re-Identification", Andreas Eberle, 2018: PSE (ResNet-50) | 69.0 | 87.7 | 94.5 | 96.8 | 99.0 | 59.2 | 74.9 | 85.2 | 88.3 | 92.9 |