Mimicked Alignment for Real-World SR - ACCV 2024

Enhanced Super-Resolution Training via Mimicked Alignment for Real-World Scenes

Omar Elezabi, Zongwei Wu^*, and Radu Timofte

Computer Vision Lab, CAIDAS & IFI, University of Würzburg

^* Corresponding authors

Latest

• 02/12/2024: Code & ckpt & Data release. Google Drive
• 07/10/2024: Technical report released on arxiv.
• 20/09/2024: Paper has been accepted at ACCV 2024! 🎉

Method:

Abstract

Image super-resolution methods have made significant strides with deep learning techniques and ample training data. However, they face challenges due to inherent misalignment between low-resolution (LR) and high-resolution (HR) pairs in real-world datasets. In this study, we propose a novel plug-and-play module designed to mitigate these misalignment issues by aligning LR inputs with HR images during training. Specifically, our approach involves mimicking a novel LR sample that aligns with HR while preserving the degradation characteristics of the original LR samples. This module seamlessly integrates with any SR model, enhancing robustness against misalignment. Importantly, it can be easily removed during inference, therefore without introducing any parameters on the conventional SR models. We comprehensively evaluate our method on synthetic and real-world datasets, demonstrating its effectiveness across a spectrum of SR models, including traditional CNNs and state-of-the-art Transformers.

Mimicked Alignment:

Results:

Main Results

Visual Comparison

Install

installation for Cuda 12.2 and Pytorch 2.4
Create a conda enviroment:

ENV_NAME="Mim_Ali"
conda create -n $ENV_NAME python=3.10.*
conda activate $ENV_NAME

Install the dependencies:

pip3 install torch torchvision torchaudio
pip3 install -r requirements.txt
python setup.py develop --no_cuda_ext

Usage

Pre-trained checkpoints and visual results can be downloaded here.

In options you can find the corresponding config files for reproducing our experiments.

Datasets

• SR-RAW: we preprocessed the original SR-RAW dataset according to the code. We slightly changed the code to generate RGB to RGB data instead of RAW to RGB. You can download the dataset from this link. We used the same split for training/testing.
• RealSR: We use the data capture used the Canon camera for our experiments. You can download the dataset from this link. We used the same training/testing split.
• Synthetic datasets: For our synthetic benchmark we used Set5, Set14, BSD100, Urban100, Manga109, and DIV2KRK datasets. You can download the datasets from this link and this link for DIV2KRK dataset.

Testing

Synthetic Benchmark

We used the evaluation code and datasets from DAT for our Synthetic Benchmark. You can dowenload the dataset from this link

Realistic Data Benchmark

For testing the pre-trained SR model with our Alignment please use following commands To generate output. Replace [TEST OPT YML] with the path to the corresponding option file.

python basicsr/test.py -opt [TEST OPT YML]

Measure the No Reference metric on the output images.

python scripts/no_ref_quality.py --metric [Metric Name (niqe,nrqm,pi)] --imgs_path [Images Folder]

Training

For single-GPU training use the following commands. Replace [TRAIN OPT YML] with the path to the corresponding option file.

python basicsr/train.py -opt [TRAIN OPT YML] 

Citation

If you find our work helpful, please consider citing the following paper and/or ⭐ the repo.

@article{elezabi2024enhanced,
  title={Enhanced Super-Resolution Training via Mimicked Alignment for Real-World Scenes},
  author={Elezabi, Omar and Wu, Zongwei and Timofte, Radu},
  journal={arXiv preprint arXiv:2410.05410},
  year={2024}
}

Acknowledgements

This code is built on NAFNet.