This codebase is adapted from Video-ChatGPT.
| Resource | Link |
|---|---|
| Paper |  |
| LLAVIDAL Weights |  |
| (ADL-X) Multi-modal Features |  |
| (ADL-X) Instruction Dataset |  |
| Data Curation |  |
| Training |  |
| Offline Demo |  |
| Quantitative Evaluation |  |
| Start with installation (click here) | |||
|---|---|---|---|
| I want to try the demo | I want to train LLAVIDAL | I want to reproduce LLAVIDAL results | I want to generate the ADL-X dataset |
Overview of LLAVIDAL, which utilizes an LLM to integrate multiple modalities, including video, pose, and object features. Videos are represented by embeddings obtained from a VLM, poses are processed through (PoseLM), and object embeddings are obtained through (ObjectLM). These embeddings are projected into the LLM space, where they are concatenated with tokenized text queries for instruction tuning.
Our python environement is identical to Video-ChatGPT, we recommend following their installation instructions:
conda create --name=llavidal python=3.10
conda activate llavidal
git clone https://github.com/ADL-X/LLAVIDAL.git
cd LLAVIDAL
pip install -r requirements.txt
export PYTHONPATH="./:$PYTHONPATH"Additionally, if you are using A100/H100 GPUs you can install FlashAttention,
pip install ninja
git clone https://github.com/HazyResearch/flash-attention.git
cd flash-attention
git checkout v1.0.7
python setup.py installWe provide a Gradio demo to run LLAVIDAL on your local machine. For the best performance, use a CUDA-enabled machine with at least 18GB of VRAM.
- Activate the llavidal Conda environment
conda activate llavidal - Download the following LLaVA weights: LLaVA-7B-Lightening-v1-1
- Download the LLAVIDAL weights from Available Resources
Finally, run the demo by executing the following command:
python llavidal/demo/video_demo.py \
--model-name <path to the LLaVA-7B-Lightening-v1-1 weights downloaded in step 2> \
--projection_path <path to the downloaded llavidal weights (llavidal_weights.bin
) downloaded in step 3>After running the command a URL will be provided. Click this URL and follow the on-screen instructions to use the demo.
LLAVIDAL is trained on ADL-X, an ADL dataset of over 100K video-instruction pairs. The weights of the model are initialized from LLaVA and it is trained for 3 epochs on 8 48GB NVIDIA RTX A6000 GPUs. To begin, download the LLaVA weights from this link: LLaVA-7B-Lightening-v1-1.
We provide two methods to prepare the ADL-X dataset:
- Download the pre-extracted features (recommended):
- Download the Multi-modal Features (
video_features.zip,object_features.zip,pose_features.zip) and Instruction Dataset (NTU_QA-for-training.json) from Available Resources - This should result in separate directories for each modality, and a json for training
- Download the Multi-modal Features (
- Curate the dataset and extract the features:
- Follow the steps in Data Curation Pipeline.
The command below will train the LLAVIDAL architecture for 3 epochs on all three modalities. This command is modular and will only train LLAVIDAL with the modalities whose folders are passed. For example, if only --object_folder and --pose_folder is passed, LLAVIDAL will drop the video modality and will only train with the object and pose modalities.
torchrun --nproc_per_node=8 --master_port 29001 llavidal/train/train_mem.py \
--version v1 \
--tune_mm_mlp_adapter True \
--mm_use_vid_start_end \
--bf16 True \
--num_train_epochs 3 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 3000 \
--save_total_limit 3 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 100 \
--tf32 True \
--model_max_length 2048 \
--gradient_checkpointing True \
--lazy_preprocess True \
--output_dir ./work_dirs/LLAVIDAL_video-object-pose-text_3epochs \
--model_name_or_path /path/to/LLaVA-7B-Lightening-v-1-1/ \
--data_path /path/to/NTU_QA-for-training.json \
--video_folder /path/to/video_features/ \
--object_folder /path/to/object_features/ /
--pose_folder /path/to/pose_features/This is the suggested method to train LLAVIDAL, in which we use a curriculum learning approach to sequentially introduce modalities into LLAVIDAL. In the implementation this consists of training many models independently in stage 1, merging their weights, and propogating those weights to the next stage. The following command can be use to train LLAVIDAL with MMPro training (UPDATE THE PATHS in mmpro_training.sh BEFORE RUNNING):
bash scripts/mmpro_training.shThe final model will be available in the directory you ran the above command at ./work_dirs/mmpose_training/stage3_video-pose-object-text/.
NOTE: You can skip this process entirely and download the ADL-X dataset in the Available Resources section above
Follow the steps below to recreate ADL-X using the data curation pipeline proposed in the paper.
Step 1: Download the NTU-RGB+D dataset
Step 2: Download the action combination here.
Step 3: Arrange the directory structure of the NTU-RGB+D videos in the following way:
NTU Videos
├── P001
│ ├── S001C001P001R001A001_rgb.avi
│ ├── S001C001P001R001A002_rgb.avi
│ ├── S001C001P001R001A003_rgb.avi
│ ...
├── P015
│ ├── S003C001P015R001A001_rgb.avi
│ ├── S003C001P015R001A002_rgb.avi
│ ├── S003C001P015R001A003_rgb.avi
│ ├── S003C001P015R001A004_rgb.avi
│ ├── S003C001P015R001A005_rgb.avi
│ ...
...
Step 4: Generate the temporally stitched videos with the following command, passing the action combination list and video folder path.
python /data_annotation/process_video_sequences.pyStep 5: Download and install CogVLM. Run the following command (which uses cli_demo_hf.py) to get frame-level annotations of the temporally stitched videos at 0.5fps:
python cli_demo_hf.py --from_pretrained THUDM/cogvlm-chat-hf --quant 4Step 6: Obtain dense video descriptions using GPT 3.5 Turbo with the following command, passing the frame-level annotations from Step 5 and your OpenAI API key:
python data_annotation/generate_dense_descriptions.pyStep 7: Generate QA pairs by running the following command, passing the dense video descriptions from Step 6 and your OpenAI API key:
python /data_annotation/generate_QA_pairs.pyStep 8: Convert the generated QA pairs from Step 7 (also availble at NTU_QA.json) into the required format for training:
python scripts/convert_instruction_json_to_training_format.py \
--input_json_file <path to json file downloaded in step 2> \
--output_json_file NTU_QA-for-training.jsonStep 9: Prepare Spatio-Temporal features using CLIP
For training efficiency, we pre-compute the spatio-temporal video features used during training. The following command will save one pickle file per video in directory specified by the --clip_feat_path argument. Run the following command to generate spatio-temporal features with CLIP:
python scripts/save_spatio_temporal_clip_features.py \
--video_dir_path <path to the directory containing videos generated in Step 4> \
--clip_feat_path <The output dir to save the features in>Step 10: Download the pre-computed pose (pose_features.zip) and object features (object_features.zip) from Available Resources and extract them
Our data curation pipeline can be adapted to trimmed (start at Step 2) or untrimmed (start at Step 5) video datasets.
We introduce two new evaluation for ADL centric tasks -- ADLMCQ-AR & ADLMCQ-AF which are MCQs conttaining Action Recognition and Action Forecasting Tasks. We also release SmartHome Untrimmed Descriptions for the first time.
Step 1: Download all the datasets-- Charades , LEMMA(We use the exo-view) ,SMARTHOME UNTRIMMED and TRIMMED.
Step 2: For Action Forecasting access the json files and slice the videos from the start frame and end frame.For action recognition nothing is needed.
Step 3: Arrange the data like that in the json file provided and run the command ,
cd llavidal/eval/python run_inference_action_recognition_charades.py
--video_dir /path/to/videos \
--qa_file /path/to/qa_file.json \
--output_dir /path/to/output \
--output_name results \
--model-name <LLAVA model path> \
--conv-mode llavidal_v1 \
--projection_path <path to LLAVIDAL WEIGHTS> Step 3: Evaulate using GPT3.5 Turbo api
cd quantitative_evaluation/evaluate_action_recognition_charades.pyand pass the above results in STEP 2.
For other methods the above steps are same
For video descriptions for Charades run command
cd llavidal/evalpython run_inference_benchmark_general.pyPass the appropiate paths to get the results josn
For video descriptions for Smarthome Untrimmed ,slice the videos in 1 minutes each and make a dense description like that of data curation process.
To get individual descriptions
cd llavidal/evalpython run_inference_descriptions_smarthome.pyWe closely follow the MEMENTOS EVALUATION to get the object and action F1 scores
We provide a notebook to achieve the execute the above approach. Follow this notebook to get the evaluation
cd quantitative_evaluation/mementos_evaluation.ipynbWe are introducing ADLX the first ADL centric video instruction dataset, due to licensing restrictions we cannot share the original videos but we are providing the video features of our dataset,we are also providing the object features and the pose features.
The dataset is in LINK. The folders are Video_features , Pose Features and Object Features
The video features are like
Video Features
├── 001_video_001.pkl
├── 001_video_002.pkl
├── 001_video_003.pkl
├── 001_video_004.pkl
├── 001_video_005.pkl
├── 001_video_006.pkl
...
each video feature is of dimension 356 x 1024.
The pose features are like
Pose Features
├── 001_001_video_001_pose.pickle
├── 001_001_video_002_pose.pickle
├── 001_001_video_003_pose.pickle
├── 001_001_video_004_pose.pickle
├── 001_001_video_005_pose.pickle
├── 001_001_video_006_pose.pickle
├── 001_001_video_007_pose.pickle
├── 001_001_video_008_pose.pickle
...
each pose feature is of the dimension 216 x 256
The object features are like
Object Features
├── 001_001_video_001_object.pkl
├── 001_001_video_002_object.pkl
├── 001_001_video_003_object.pkl
├── 001_001_video_004_object.pkl
├── 001_001_video_005_object.pkl
├── 001_001_video_006_object.pkl
├── 001_001_video_007_object.pkl
├── 001_001_video_008_object.pkl
...
each object feature is of the dimension n x 8x 512, where n is the number of objects present in the video.
- LLaMA: Great step towards bridging vision and language!
- VideoChatgpt: We thank for the foundational work.
- LLAVA : For inspiring the overall architecture
- CogVLM: For creating a strong captioning model.
If you're using LLAVIDAL in your research or applications, please cite using this BibTeX:
@misc{chakraborty2024llavidal,
title={LLAVIDAL: Benchmarking Large Language Vision Models for Daily Activities of Living},
author={Rajatsubhra Chakraborty and Arkaprava Sinha and Dominick Reilly and Manish Kumar Govind and Pu Wang and Francois Bremond and Srijan Das},
year={2024},
eprint={2406.09390},
archivePrefix={arXiv},
primaryClass={id='cs.CV' full_name='Computer Vision and Pattern Recognition' is_active=True alt_name=None in_archive='cs' is_general=False description='Covers image processing, computer vision, pattern recognition, and scene understanding. Roughly includes material in ACM Subject Classes I.2.10, I.4, and I.5.'}
}The dataset is protected under the CC-BY license of Creative Commons, which allows users to distribute, remix, adapt, and build upon the material in any medium or format, as long as the creator is attributed. The license allows ADL-X for commercial use. As the authors of this manuscript and collectors of this dataset, we reserve the right to distribute the data.