Skip to content

Efficient Feature Extraction for High-resolution Video Frame Interpolation (BMVC 2022)

License

Notifications You must be signed in to change notification settings

visinf/fldr-vfi

Repository files navigation

Efficient Feature Extraction for High-resolution Video Frame Interpolation

License Framework

This is the official repository accompanying the BMVC 2022 paper:

Efficient Feature Extraction for High-resolution Video Frame Interpolation
Moritz Nottebaum, Stefan Roth and Simone Schaub-Mayer
BMVC 2022. [paper (open access)] [supplemental] [example results] [talk video] [preprint (arXiv)]

This repository contains the training and test code along with the trained weights to reproduce our results, and our test datasets Inter4K-S and Inter4K-L (subsets of Inter4K).

Installation

The following steps will set up a local copy of the repository.

  1. Create conda environment:
conda create --name fldrnet
conda activate fldrnet
  1. Install PyTorch:
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
  1. Install the dependencies:
pip install -r requirements.txt
  1. Handle CuPy problems (Optional):
pip uninstall cupy-cuda116
conda install -c conda-forge cupy

If CuPy throws importing errors during execution, a conda installation can fix it.

Training

The following command will train a model from scratch:

python train_it.py --x_train_data_path /to/xtrain/train  --toptim

The flag --x_train_data_path /to/xtrain/train contains the location of the training data. We used X-Train from Sim et al. The --toptim flag is optional and adds a post-training optimization of the temperature parameter as described in our paper in the occlusion estimation section.

Testing

  1. Download the respective testset from the following links:
Dataset Link
X-Test XVFI repository
Xiph Xiph benchmark
Inter4K Our subset (Licence)
  1. Use the file path for the test sets accordingly or update them in main.py:
--x_test_data_path /to/your/location
--xiph_data_path /to/your/location
--inter4k_data_path /to/your/location
  1. The following line will evaluate the provided checkpoint "fLDRnet_X4K1000FPS_exp1_best_PSNR.pt" on all four testsets (X-Test, Xiph-4K, Inter4K-S, Inter4K-L):
python main.py --exp_num 1 --gpu 0 --papermodel --test5scales 

By adding the option --testsets you can choose on which data you want to evaluate (options are "Inter4K-S", "Inter4k-L", "X-Test","Xiph-4K"). The option --papermodel ensures all preferences are set according to the model of the paper. The option --test5scales adapts args.fractions, args.scales, args.phase, args.S_tst and args.moreTstSc to allow for additional scales for testing.

Run Model

In run_on_your_images.py you have the run_on_images(...) function, which simplifies to test the model on your own data. In the main() function you can see examplary code to test on X-Test.

Acknowledgements

We thank the contributors of the following repositories for using parts of their publicly available code and 4K datasets, which were necessary to adequately train and evaluate our method:

Citation

We hope you find our work useful. If you would like to acknowledge it in your project, please use the following citation:

@inproceedings{Nottebaum:2022:EFE,
  author    = {Nottebaum, Moritz and Roth, Stefan and Schaub-Meyer, Simone},
  title     = {Efficient Feature Extraction for High-resolution Video Frame Interpolation},
  booktitle = {British Machine Vision Conference  {BMVC}},
  year      = {2022}
}

About

Efficient Feature Extraction for High-resolution Video Frame Interpolation (BMVC 2022)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •  

Languages