Enhanced Hyperspectral Image Super-Resolution via RGB Fusion and TV-TV Minimization

Python 3.x code to reproduce the experiments in

Enhanced Hyperspectral Image Super-Resolution via RGB Fusion and TV-TV Minimization

Marija Vella, Bowen Zhang, Wei Chen, João F. C. Mota
Accepted to ICIP 2021

Overview

Hyperspectral (HS) images contain detailed spectral information that has proven crucial in applications like remote sensing, surveillance, and astronomy. However, because of hardware limitations of HS cameras, the captured images have low spatial resolution. To improve them, the low-resolution hyperspectral images are fused with conventional high-resolution RGB images via a technique known as fusion based HS image super-resolution. Currently, the best performance in this task is achieved by deep learning (DL) methods. Such methods however, cannot guarantee that the input measurements are satisfied in the recovered image, since the learned parameters by the network are applied to every test image. Conversely, model-based algorithms can typically guarantee such measurement consistency. Inspired by these observations, we propose a framework that integrates learning and model based methods. Experimental results show that our method produces images of superior spatial and spectral resolution compared to the current leading methods, whether model- or DL-based.

Proposed Method

Diagram of our method:

Given a HRMS image, LRHS image and the corresponding super-resolved output W from a MS/HS image fusion HS super-resolution method, the proposed post-processing step guarantees data consistency and improves the quality of W. It achieves that by solving TV-TV minimization:

where:
W and X̂ (HRHS) ∈ M₀.N₀ x S₀
Y (HRMS) ∈ M₀.N₀ x S
Z (LRHS) ∈ M.N x S₀

Demo

This repo runs TV-TV minimization on 3 sample images from the CAVE dataset and uses MHF as the base-method. To run this demo use the following commands

git clone https://github.com/marijavella/hs-sr-tvtv
cd hs-sr-tvtv
python main.py

Base methods and datasets

This repo only considers 3 sample images. To test more images, place the images in the respective sub-folder of the data folder: HRMS in Y, LRHS in Z and HRHS (from base method) in W. The ground-truth image in sub-folder X is needed to compute the quality metrics and the CSR function (R) is also required. All the images were read from a .mat file.

In our paper we considered 4 base methods: MHF, uSDN, CNMF, NLSTF, and two datasets CAVE and Harvard. To generate the results from each base method, keep the code/settings provided by the respective authors and place the images in the correct folder as explained above.

Results

Result obtained on one of the sample images.

Dependencies

To install the required libraries

pip install -r requirements.txt

Contents

There is one main folder:

• data: contains a .mat file with the CSR function and the following subfolders:
  • W - outputs from base method (automatically created once the demo is run)
  • X - HRHS (ground truth)
  • Y - HRMS
  • Z - LRHS

The images provided in this demo were generated according to the specificiations of MHF but these can be replaced with any other fusion based HS image super-resolution method.

The main scripts are:

• main.py: script that runs the solver, saves and shows results of the images in the data folder.

• utils.py: utility functions.

• TVTVHS_Solver.py: TV-TV solver code.

Citation

Vella, M. Zhang, B., Chen, W. & Mota, J. F. C. Enhanced Hyperspectral Image Super-Resolution via RGB Fusion and TV-TV Minimization, ICIP 2021.

Contact

The code is provided to be able to replicate our research. If you encounter and errors or would like to give any suggestions kindly contact me directly on mv37@hw.ac.uk.