Siamese-Image-Modeling

By Chenxin Tao, Xizhou Zhu, Weijie Su, Gao Huang, Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, Jifeng Dai

This is the official implementation of the CVPR 2023 paper Siamese Image Modeling for Self-Supervised Vision Representation Learning.

🏠 Introduction

SiameseIM is a new form of self-supervised learning that can learn semantic alignment and spatial sensitivity with a single dense loss. We note the following key observations from SiameseIM:

• Compared with MIM methods, SiameseIM shows that reconstructing another view helps to obtain good semantic alignment.

• Compared with ID methods, SiameseIM shows that dense supervision can be applied by matching the dense correspondence between two views strictly through their relative positions.

• SiameseIM is able to surpass both MIM and ID methods over a wide range of tasks. SiameseIM obtains more improvements in few-shot, long-tail and robustness-concerned scenarios.

📈 Main Results

	ImageNet			COCO		ADE20k	LVIS				Robustness
	FT	LIN	1% FT	AP box	AP mask	mIoU	AP box	AP box rare	AP mask	AP mask rare	IN-A top-1	IN-R top-1	IN-Sketch top-1	IN-C 1-mCE
MoCo-v3 (ID method)	83.0	76.7	63.4	47.9	42.7	47.3	37.3	25.5	35.3	25.8	32.4	49.8	35.9	55.4
MAE (MIM method)	83.6	68.0	51.1	51.6	45.9	48.1	40.1	29.3	38.1	29.1	35.9	48.3	34.5	48.3
SiameseIM	84.1	78.0	65.1	52.1	46.2	51.1	40.5	30.9	38.1	30.1	43.8	52.5	38.3	57.1
Improve w.r.t. MoCo-v3	+1.1	+1.3	+1.7	+4.2	+3.5	+3.8	+3.2	+5.4	+2.8	+4.3	+11.4	+2.7	+2.4	+1.7
Improve w.r.t. MAE	+0.5	+10.0	+14.0	+0.5	+0.3	+3.0	+0.4	+1.6	+0.0	+1.0	+7.9	+4.2	+3.8	+8.8

Note:

(1) Compared with MoCo-v3, SiameseIM improves dense prediction tasks (COCO detection, ADE20k segmentation, LVIS detection) significantly;

(2) Compared with MAE, SiameseIM improves long-tail, few-shot, robustness tasks (ImageNet linear evaluation & few-shot classification, ADE20k segmentation, LVIS detection) significantly;

(3) Notably, ADE20k segmentation and LVIS detection both contain long-tail classes, which put forward high requirement for semantic alignment, and detection tasks, which demand good spatial alignment. Thus, SiameseIM can surpass both MoCo-v3 and MAE by a large margin on these tasks.

🛠️ Usage

Preparation

See prepare.md

Model Checkpoint

See checkpoints.md

Pretrain

See pretrain.md

Finetune

See finetune.md

Linear Evaluation

See linear_eval.md

Few-shot Evaluation

See few_shot.md

COCO & LVIS Detection

We use ViTDet for detection tasks, please refer to detectron2.

ADE20k Segmentation

We follow MAE to use UPerNet for segmentation task, please refer to mmsegmentation.

Robustness Evaluation

We evaluate the ImageNet finetuned model on ImageNet-A, ImageNet-R, ImageNet-Sketch and ImageNet-C datasets.

📃 License

This project is released under the CC-BY-NC 4.0 license.

🖊️ Citing SiameseIM

If you find SiameseIM useful in your research, please consider citing:

@inproceedings{tao2023siamese,
  title={Siamese image modeling for self-supervised vision representation learning},
  author={Tao, Chenxin and Zhu, Xizhou and Su, Weijie and Huang, Gao and Li, Bin and Zhou, Jie and Qiao, Yu and Wang, Xiaogang and Dai, Jifeng},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={2132--2141},
  year={2023}