Skip to content

Latest commit

 

History

History
98 lines (83 loc) · 4.34 KB

README.md

File metadata and controls

98 lines (83 loc) · 4.34 KB
Raw

Click to Move

Click to Move: Controlling Video Generation with Sparse Motion

Pytorch implementation of our paper Click to Move: Controlling Video Generation with Sparse Motion In ICCV 2021. Please cite with the following Bibtex code:

@inproceedings{ardino2021click,
  title={Click to Move: Controlling Video Generation with Sparse Motion},
  author={Ardino, Pierfrancesco and De Nadai, Marco and Lepri, Bruno and Ricci, Elisa and Lathuili{\`e}re, St{\'e}phane},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={14749--14758},
  year={2021}
}

Please follow the instructions to run the code.

Scripts

1. Installation

  • See the c2m.yml configuration file. We provide an user-friendly configuring method via Conda system, and you can create a new Conda environment using the command:
conda env create -f c2m.yml
conda activate c2m
  • Install cityscapesscripts with pip
cd cityscapesScripts
pip install -e .

WIP

2. Data Preprocessing

2.1 Generate instance segmentation

We apply a modified version of Panoptic-deeplab to get the corresponding semantic and instance maps. You can find it into panoptic_deeplab folder. For this work we have used the HRNet backbone. You can download it from here.

Cityscapes

  • Please download the Cityscapes dataset from the official website (registration required). After downloading, please put these files under the ~/dataset_cityscape_video/ folder and run the following command in order to generate the correct segmentation maps

    cd panoptic_deeplab
python tools/generate_segmentation.py --cfg configs/cityscapes_{trainset/valset}.yaml TEST.MODEL_FILE YOUR_DOWNLOAD_MODEL_FILE

    Remember to set up the config file with the correct input folder, output folder and dataset split

    You should end up with the following structure:

    dataset_cityscape_video
├── leftImg8bit_sequence
│   ├── train
│   │   ├── aachen
│   │   │   ├── aachen_000003_000019_leftImg8bit.png
│   │   │   ├── ...
│   ├── val
│   │   ├── frankfurt
│   │   │   ├── frankfurt_000000_000294_leftImg8bit.png
│   │   │   ├── ...
│   ├── train_semantic_segmask
│   │   ├── aachen
│   │   │   ├── aachen_000003_000019_ssmask.png
│   │   │   ├── ...
│   ├── val_semantic_segmask
│   │   ├── frankfurt
│   │   │   ├── frankfurt_000000_000294_ssmask.png
│   │   │   ├── ...
│   ├── train_instance
│   │   ├── aachen
│   │   │   ├── aachen_000003_000019_gtFine_instanceIds.png
│   │   │   ├── ...
│   ├── val_instance
│   │   ├── frankfurt
│   │   │   ├── frankfurt_000000_000294_gtFine_instanceIds.png
│   │   │   ├── ...

2.2 Generate object trajectories

3 Train the model

We store the configuration of the model as a YAML configuration file. You can have a look at a base configuration in src/config/c2m_journal_cityscapes.yaml. The training file takes as input the following parameters:

  • config: path to configuration file
  • device_ids: names of the devices comma separated
  • seed: seed of the training
  • profile: debug using PyTorch profiler

Our code support multi-gpu training using DistributedDataParallel. Here's an example of how you can run the code with one or more gpus.

Single gpu

python train.py --device_ids 0 --config config/c2m_journal_cityscapes.yaml

Multi gpu

python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 train.py --device_ids 0,1 --config config/c2m_journal_cityscapes.yaml The example considers a scenario with a single node and two gpus per node. Please change according to your needs. For more information check the DDP example

4 Test the model

python test.py --device_ids 0 --config config/c2m_journal_cityscapes.yaml