Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, Hongsheng Li

This repository contains code for the paper Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning and it was built based on LLaVA

We collect and introduce the Visual CoT dataset comprising 438k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions. The work also proposes a multi-turn processing pipeline for the multi-modal language model that dynamically focuses on visual inputs and provides interpretable thoughts. Importantly, the released benchmark is capable of evaluating MLLMs in scenarios requiring specific local region identification.

Project Page, Dataset, arXiv

News

• [12/22] We have updated the related images in the VisCoT dataset, now available in the VisCoT HF repo.
• [11/26] We have updated the VisCoT data with detailed reasoning steps here and fixed some bugs.
• [10/4] We have established the webpage for this project.
• [9/27] VisCoT is accepted by Neurps 2024, $${\color{red}Spotlight}$$ 🎉🎉🎉

Contents

• Install
• Model Zoo
• Train
• Evaluation
• Visualization

Install

1. Clone this repository and navigate to Visual-CoT folder

git clone https://github.com/deepcs233/Visual-CoT.git
cd Visual-CoT

2. Install Package

conda create -n viscot python=3.10 -y
conda activate viscot
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

3. Install additional packages for training cases

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

Demo

Please refer to https://github.com/haotian-liu/LLaVA/tree/main?tab=readme-ov-file#demo

Model Zoo

We propose a novel multi-turn processing pipeline for MLLMs that can dynamically focus on visual inputs and provide intermediate interpretable thoughts.

The model weights below are merged weights. You do not need to apply delta. The usage of VisCoT checkpoints should comply with the base LLM's model license.

Version	Size	Resolution	Checkpoint
VisCoT	7B	224	deepcs233/VisCoT-7b-224
VisCoT	7B	336	deepcs233/VisCoT-7b-336
VisCoT	13B	224	deepcs233/VisCoT-13b-224
VisCoT	13B	336	deepcs233/VisCoT-13b-336

Train

Our training steps are largely consistent with LLaVA; for further details, please refer to the LLaVA documentation/issues.

VisCoT training consists of two stages: (1) feature alignment stage: use our 558K subset of the LAION-CC-SBU dataset to connect a frozen pretrained vision encoder to a frozen LLM; (2) visual instruction tuning stage: use 665K dataset including multimodal instruction-following data and academic VQA tasks from LLaVA-1.5, 1.4M dataset with positional annotations from Shikra, and 373K visual CoT dataset from ours, to teach the model to follow multimodal instructions and obtain the visual CoT ability.

VisCoT is trained on 8 A100 GPUs with 80GB memory. To train on fewer GPUs, you can reduce the per_device_train_batch_size and increase the gradient_accumulation_steps accordingly. Always keep the global batch size the same: per_device_train_batch_size x gradient_accumulation_steps x num_gpus.

Hyperparameters

We use a similar set of hyperparameters as Vicuna in finetuning. Both hyperparameters used in pretraining and finetuning are provided below.

1. Pretraining

Hyperparameter	Global Batch Size	Learning rate	Epochs	Max length	Weight decay
LLaVA-v1.5-13B	256	1e-3	1	2048	0

2. Finetuning

Hyperparameter	Global Batch Size	Learning rate	Epochs	Max length	Weight decay
LLaVA-v1.5-13B	128	2e-5	1	2048	0

Download Vicuna checkpoints (automatically)

Our base model Vicuna v1.5, which is an instruction-tuned chatbot, will be downloaded automatically when you run our provided training scripts. No action is needed.

Pretrain (feature alignment)

In our work, we directly use the project's weight from LLaVA-1.5. If you do not need to train it by yourself, projector weights can be downloaded here: https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md#projector-weights

Please download the 558K subset of the LAION-CC-SBU dataset with BLIP captions used in LLaVA here.

Pretrain takes around 5.5 hours for VisCoT-13B on 8x A100 (80G), due to the increased resolution to 336px. It takes around 3.5 hours for VisCoT-7B.

Training script with DeepSpeed ZeRO-2: pretrain.sh.

• --mm_projector_type mlp2x_gelu: the two-layer MLP vision-language connector.
• --vision_tower openai/clip-vit-large-patch14-336: CLIP ViT-L/14 336px.

Visual Instruction Tuning

1. Prepare data

Please download the annotation of our mixed instruction tuning data viscot_mixed_2m.json to ./playground/data. We provide our 363k visual CoT dataset viscot_363k.json for building your own dataset. Additionally, the ./viscot_dataset directory contains metadata for the visual CoT dataset, which includes detailed information requiring further postprocessing before it can be used for training. The ./cot_with_detailed_reasoning_steps directory contains 98k data pairs, each accompanied by detailed reasoning steps. The train/validation split is maintained consistent with the original dataset. Necessary scripts are available in the ./tools directory. For instance, the script ./tools/convert_data_to_llava_format.py can convert the meta JSONL file into the required format for training. Please download the images for constituting datasets, and some of them may need to register/complete the form first.

We also prepare the image file in this link, you need to merge these split archive files and then extract them.

• COCO: train2017
• GQA: images
• OCR-VQA: download script, we save all files as .jpg
• TextVQA: train_val_images
• VisualGenome: part1, part2
• Visual7W: repo
• Flickr30k: homepage
• DocVQA: homepage
• InfographicsVQA: homepage
• Open images: download script, we only use 0-5 splits
• VSR: images
• DUDE: images
• SROIE: homepage
• CUB: images

After downloading all of them, organize the data as follows in ./playground/data,

├── coco
│   └── train2017
│   └── train2014
├── gqa
│   └── images
├── ocr_vqa
│   └── images
├── textvqa
│   └── train_images
└── vg
│   ├── VG_100K
│   └── VG_100K_2
└── v7w
│   └── images
└── flickr30k
│   └── images
└── cot
│   └── flickr30k
│   └── docvqa
│   └── gqa
│   └── infographicsvqa
│   └── openimages
│   └── textvqa
│   └── vsr
│   └── dude
│   └── sroie
│   └── cub

2. Start training!

We have prepared LLaVA's pretrained projectors in our repo (checkpoints/llava_7b_mm_projector.bin and checkpoints/llava_13b_mm_projector.bin). It is not recommended to use legacy projectors, as they may be trained with a different version of the codebase, and if any option is off, the model will not function/train as we expected.

Visual instruction tuning takes around 60 hours for VisCoT-7b-224 on 8x A100 (80G).

Training script with DeepSpeed ZeRO-3: finetune.sh.

If you are interested in finetuning LLaVA model to your own task/data, please check out Finetune_Custom_Data.md。

Some options to note:

• --mm_projector_type mlp2x_gelu: the two-layer MLP vision-language connector.
• --vision_tower openai/clip-vit-large-patch14-336: CLIP ViT-L/14 336px.
• --ft_vision_tower True: finetune the vision encoder with the same learning rate as the backbone.
• --vision_tower_lr 2e-6: use a specific vision encder learning rate.

Evaluation

Visual CoT Benchmark

1. Single-GPU inference, VisCoT-7b-336 can be changed to other model names saved in the ./checkpoints/

bash scripts/v1_5/eval/cot_benchmark.sh VisCoT-7b-336

2. Obtain the score using ChatGPT-3.5, the API KEY need to be set in llava/eval/eval_cot_score.py

bash scripts/v1_5/eval/cot_score.sh VisCoT-7b-336

3. Stat the overall score

python tools/cot_get_result.py VisCoT-7b-336

4. Stat the detection accuracy of visual CoT bounding boxes (optional)

python tools/cot_detection_get_result.py VisCoT-7b-336

RefCoCo

1. Single-GPU inference, VisCoT-7b-336 can be changed to other model names saved in the ./checkpoints/

bash scripts/v1_5/eval/refcoco.sh VisCoT-7b-336

2. Stat the overall accuracy

python tools/refcoco_get_result.py VisCoT-7b-336

General Benchmarks

Please refer to LLaVA's scripts.

Visualization

Acknowledgements

This implementation is based on code from several repositories.

• LLaVA
• Shikra
• Vicuna

Citation

If you find our repo, dataset or paper useful, please cite us as

@misc{shao2024visual,
      title={Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models}, 
      author={Hao Shao and Shengju Qian and Han Xiao and Guanglu Song and Zhuofan Zong and Letian Wang and Yu Liu and Hongsheng Li},
      year={2024},
      eprint={2403.16999},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Disclaimer

This dataset was collected and released solely for research purposes, with the goal of making the MLLMs dynamically focus on visual inputs and provide intermediate interpretable thoughts. The authors are strongly against any potential harmful use of the data or technology to any party.

Intended Use

The data, code, and model checkpoints are intended to be used solely for (I) future research on visual-language processing and (II) reproducibility of the experimental results reported in the reference paper. The data, code, and model checkpoints are not intended to be used in clinical care or for any clinical decision making purposes.

Primary Intended Use

The primary intended use is to support AI researchers reproducing and building on top of this work. \shortname{} and its associated models should be helpful for exploring various vision question answering (VQA) research questions.

Out-of-Scope Use

Any deployed use case of the model --- commercial or otherwise --- is out of scope. Although we evaluated the models using a broad set of publicly-available research benchmarks, the models and evaluations are intended for research use only and not intended for deployed use cases.

License

All code within this repository is under Apache License 2.0.