Haiyang Mei, Bo Dong, Wen Dong, Jiaxi Yang, Seung-Hwan Baek, Felix Heide, Pieter Peers, Xiaopeng Wei, Xin Yang
[Paper] [Project Page]
Transparent and semi-transparent materials pose significant challenges for existing scene understanding and segmentation algorithms due to their lack of RGB texture which impedes the extraction of meaningful features. In this work, we exploit that the light-matter interactions on glass materials provide unique intensity-polarization cues for each observed wavelength of light. We present a novel learning-based glass segmentation network that leverages both trichromatic (RGB) intensities as well as trichromatic linear polarization cues from a single photograph captured without making any assumption on the polarization state of the illumination. Our novel network architecture dynamically fuses and weights both the trichromatic color and polarization cues using a novel global-guidance and multi-scale self-attention module, and leverages global cross-domain contextual information to achieve robust segmentation. We train and extensively validate our segmentation method on a new large-scale RGB-Polarization dataset (RGBP-Glass), and demonstrate that our method outperforms state-of-the-art segmentation approaches by a significant margin.
- Python 3.8.10
- PyTorch == 1.10.0
- TorchVision == 0.11.0
- CUDA 11.4
- tqdm
- timm
Lower version should be fine but not fully tested :-)
Download Conformer_base_patch16.pth, resnet18-5c106cde.pth, and pre-trained model PGSNet.pth at project homepage, then run infer.py.
Figure 1: Qualitative comparison results.
Table 1: Quantitative comparison results.
-
Results of our PGSNet can be found in this download link.
-
Performance of competing methods can be found in this download link.
Overview of our RGBP-Glass dataset.
RGBP-Glass
├── train
│ ├── aolp
│ | ├── 20210717001004_aolp.tiff
│ | └── ...
│ ├── dolp
│ | ├── 20210717001004_dolp.tiff
│ | └── ...
│ ├── edge
│ | ├── 20210717001004_edge.png
│ | └── ...
│ ├── image
│ | ├── 20210717001004_rgb.tiff
│ | └── ...
│ ├── mask
│ | ├── 20210717001004_mask.png
│ | └── ...
│ ├── reflection
│ | ├── 20210717001004_reflection.png
│ | └── ...
│ └── instances_train.json
├── test
│ ├── aolp
│ | ├── 20210717001002_aolp.tiff
│ | └── ...
│ ├── dolp
│ | ├── 20210717001002_dolp.tiff
│ | └── ...
│ ├── image
│ | ├── 20210717001002_rgb.tiff
│ | └── ...
│ ├── mask
│ | ├── 20210717001002_mask.png
│ | └── ...
│ └── instances_test.json
└── LICENSE
We also provide the reflection and edge maps of the training set, which are required for training some glass segmentation methods (our PGSNet is out of this case).
Please refer to the Project Page.
Please cite our paper if you find the work useful::
@InProceedings{Mei_2022_CVPR,
author = {Mei, Haiyang and Dong, Bo and Dong, Wen and Yang, Jiaxi and Baek, Seung-Hwan and Heide, Felix and Peers, Pieter and Wei, Xiaopeng and Yang, Xin},
title = {Glass Segmentation Using Intensity and Spectral Polarization Cues},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2022},
pages = {12622-12631}
}
Please see LICENSE
E-Mail: Haiyang Mei (haiyang.mei@outlook.com)