This is the official repository accompanying the BMVC 2022 paper:
Efficient Feature Extraction for High-resolution Video Frame Interpolation
Moritz Nottebaum, Stefan Roth and Simone Schaub-Mayer
BMVC 2022. [paper (open access)] [supplemental] [example results] [talk video] [preprint (arXiv)]
This repository contains the training and test code along with the trained weights to reproduce our results, and our test datasets Inter4K-S and Inter4K-L (subsets of Inter4K).
The following steps will set up a local copy of the repository.
- Create conda environment:
conda create --name fldrnet
conda activate fldrnet
- Install PyTorch:
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
- Install the dependencies:
pip install -r requirements.txt
- Handle CuPy problems (Optional):
pip uninstall cupy-cuda116
conda install -c conda-forge cupy
If CuPy throws importing errors during execution, a conda installation can fix it.
The following command will train a model from scratch:
python train_it.py --x_train_data_path /to/xtrain/train --toptim
The flag --x_train_data_path /to/xtrain/train
contains the location of the training data. We used X-Train from Sim et al.
The --toptim
flag is optional and adds a post-training optimization of the temperature parameter as described in our paper in the occlusion estimation section.
- Download the respective testset from the following links:
Dataset | Link |
---|---|
X-Test | XVFI repository |
Xiph | Xiph benchmark |
Inter4K | Our subset (Licence) |
- Use the file path for the test sets accordingly or update them in
main.py
:
--x_test_data_path /to/your/location
--xiph_data_path /to/your/location
--inter4k_data_path /to/your/location
- The following line will evaluate the provided checkpoint "fLDRnet_X4K1000FPS_exp1_best_PSNR.pt" on all four testsets (X-Test, Xiph-4K, Inter4K-S, Inter4K-L):
python main.py --exp_num 1 --gpu 0 --papermodel --test5scales
By adding the option --testsets
you can choose on which data you want to evaluate (options are "Inter4K-S"
, "Inter4k-L"
, "X-Test"
,"Xiph-4K"
).
The option --papermodel
ensures all preferences are set according to the model of the paper. The option --test5scales
adapts args.fractions
, args.scales
, args.phase
, args.S_tst
and args.moreTstSc
to allow for additional scales for testing.
In run_on_your_images.py
you have the run_on_images(...)
function, which simplifies to test the model on your own data. In the main()
function you can see examplary code to test on X-Test.
We thank the contributors of the following repositories for using parts of their publicly available code and 4K datasets, which were necessary to adequately train and evaluate our method:
- XVFI
- Softmax-splatting
- Inter4K: Our Inter4K testset for video frame interpolation is a subset of the images occuring in the Inter4K dataset shared under CC BY-NC 4.0.
We hope you find our work useful. If you would like to acknowledge it in your project, please use the following citation:
@inproceedings{Nottebaum:2022:EFE,
author = {Nottebaum, Moritz and Roth, Stefan and Schaub-Meyer, Simone},
title = {Efficient Feature Extraction for High-resolution Video Frame Interpolation},
booktitle = {British Machine Vision Conference {BMVC}},
year = {2022}
}