ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation

Duolikun Danier, Fan Zhang, David Bull

Accepted in CVPR 2022.

Project | Paper | arXiv | Video | Poster

Dependencies and Installation

The following packages were used to evaluate the model.

• python==3.8.8
• pytorch==1.7.1
• torchvision==0.8.2
• cudatoolkit==10.1.243
• opencv-python==4.5.1.48
• numpy==1.19.2
• pillow==8.1.2
• cupy==9.0.0
• sk-video==1.1.10

Installation with anaconda:

conda create -n stmfnet python=3.8.8
conda activate stmfnet
conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 -c pytorch
conda install -c conda-forge cupy
pip install opencv-python==4.5.1.48
pip install sk-video==1.1.10

Model

Preparing datasets

Training sets:

[Vimeo-90K] | [BVI-DVC quintuplets]

Test sets:

[UCF101] | [DAVIS] | [SNU-FILM] | [VFITex]

The dataset folder names should be lower-case and structured as follows.

└──── <data directory>/
    ├──── ucf101/
    |   ├──── 0/
    |   ├──── 1/
    |   ├──── ...
    |   └──── 99/
    ├──── davis90/
    |   ├──── bear/
    |   ├──── bike-packing/
    |   ├──── ...
    |   └──── walking/
    ├──── snufilm/
    |   ├──── test-easy.txt
    |   ├──── test-medium.txt
    |   ├──── test-hard.txt
    |   ├──── test-extreme.txt
    |   └──── data/SNU-FILM/test/...
    ├──── vfitex/
    |   ├──── beach02_4K_mitch/
    |   ├──── bluewater_4K_pexels/
    |   ├──── ...
    |   └──── waterfall_4K_pexels/
    ├──── bvidvc/quintuplets
    |   ├──── 00000/
    |   ├──── 00001/
    |   ├──── ...
    |   └──── 17599/
    └──── vimeo_septuplet/
        ├──── sequences/
        ├──── readme/
        ├──── sep_testlist.txt
        └──── sep_trainlist.txt

Downloading the pre-trained model

Download the pre-trained ST-MFNet from here.

Evaluation (on test sets)

python evaluate.py \
--net STMFNet \
--data_dir <data directory> \
--checkpoint <path to pre-trained model (.pth file)> \
--out_dir eval_results \
--dataset <dataset name>

where <dataset name> should be the same as the class names defined in data/testsets.py, e.g. Snufilm_extreme_quintuplet.

Evaluation (on videos)

python interpolate_yuv.py \
--net STMFNet \
--checkpoint <path to pre-trained model (.pth file)> \
--yuv_path <path to input YUV file> \
--size <spatial size of input YUV file, e.g. 1920x1080>
--out_fps <output FPS, e.g. 60>
--out_dir <desired output dir>

See more details in interpolate_yuv.py. Note the script provided is for up-sampling .yuv files. To process .mp4 files, one can modify the frame reading parts of the script, or simply convert mp4 to yuv using ffmpeg then use this script.

Training

Feel free to experiment with other options, but here is an example:

python train.py \
--net STMFNet \
--data_dir <data directory> \
--out_dir ./train_results \
--epochs 70 \
--batch_size 4 \
--loss 1*Lap \
--patch_size 256 \
--lr 0.001 \
--decay_type plateau \
--gamma 0.5 \
--patience 5 \
--optimizer ADAMax

Example results

Citation

@InProceedings{Danier_2022_CVPR,
    author    = {Danier, Duolikun and Zhang, Fan and Bull, David},
    title     = {ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {3521-3531}
}

Acknowledgement

Lots of code in this repository are adapted/taken from the following repositories:

• AdaCoF-pytorch
• softmax-splatting
• pytorch-pwc

We would like to thank the authors for sharing their code.