Semantic Guided Single Image Reflection Removal
Yunfei Liu Yu Li Shaodi You Feng Lu
cs.CV














0000-0001-6898-0058

  State Key Laboratory of Vitrtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University

  China
  100191

lyunfei@buaa.edu.cn


0000-0003-1865-8276

  International Digital Economy Academy
  Shenzhen
  China
yul@Illinois.edu


0000-0001-8973-645X

  University of Amsterdam
  Amsterdam
  The Netherlands

s.you@uva.nl


0000-0001-9064-7964
Feng Lu is the corresponding author. 

	State Key Laboratory of Vitrtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University

	China
	100191


Peng Cheng Laboratory
Shenzhen
China

lufeng@buaa.edu.cn










acmcopyright
TOMM
2022 1 1 1 1 15.0010.1145/3510821





  Reflection is common when we see through a glass window, which not only is a visual disturbance but also influences the performance of computer vision algorithms. Removing the reflection from a single image, however, is highly ill-posed since the color at each pixel needs to be separated into two values belonging to the clear background and the reflection respectively. To solve this, existing methods use additional priors such as reflection layer smoothness, double reflection effect, and color consistency to distinguish the two layers. However, these low-level priors may not be consistently valid in real cases. In this paper, inspired by the fact that human beings can separate the two layers easily by recognizing the objects and understanding the scene, we propose to use the object semantic cue, which is high-level information, as the guidance to help reflection removal. Based on the data analysis, we develop a multi-task end-to-end deep learning method with a semantic guidance component, to solve reflection removal and semantic segmentation jointly. Extensive experiments on different datasets show significant performance gain when using high-level object-oriented information. We also demonstrate the application of our method to other computer vision tasks.




	<ccs2012>
	<concept>
	<concept_id>10010147.10010178.10010224</concept_id>
	<concept_desc>Computing methodologies Computer vision</concept_desc>
	<concept_significance>500</concept_significance>
	</concept>
	<concept>
	<concept_id>10010147.10010178.10010224.10010226.10010236</concept_id>
	<concept_desc>Computing methodologies Computational photography</concept_desc>
	<concept_significance>300</concept_significance>
	</concept>
	</ccs2012>


[500]Computing methodologies Computer vision
[300]Computing methodologies Computational photography










Semantic Guided Single Image Reflection Removal
    Feng Lu
    December 30, 2023
===============================================





§ INTRODUCTION
 

When taking a photo of objects behind the glass window, unwanted reflection frequently appears, which not only is visually disturbing but may also affect the performance of other computer vision algorithms (, classification <cit.>, object detection <cit.>, scene parsing <cit.>, ).  
To solve this problem, reflection removal has been explored by a number of existing works <cit.>.
Single image reflection removal is a challenging problem, , it only takes one single image as the input and aims to separate it into two outputs, the clear background and the reflection.
More formally, given an input image with reflection, denoted as 𝐈, we need to separate it into background 𝐁 and reflection 𝐑 <cit.>:

    𝐈 = 𝐁 + 𝐑.





Apparently,  Eqn. (<ref>) is ill-posed due to there are two unknowns (B and R) to be divided from only one known (I). 
Without any constraints, there are numerously possible solutions. Earlier methods propose different priors as constraints (, relative smoothness <cit.>, chromaticity consistency <cit.>, ghost effect <cit.>, ) to get a meaningful solution.  
However, such low-level priors are not general enough in real cases. 
Fan  <cit.> are the first to train a deep neural network for this task with losses on colors and edges.
Following <cit.>, other methods have been proposed in the direction of new network design <cit.>, new loss functions like perceptual loss <cit.> and GAN loss <cit.>, or attention modules <cit.>.
All of them use a network to learn a mapping from a blended reflection image to a clear background. 
While adversarial learning using GAN loss and contextual attention may provide information beyond low-level features, the object-level semantic information has never been explored.
We have observed that using only the low-level information (even equips with adversarial learning) is insufficient for the reflection separation due to the high ambiguity in low-level appearance.

Fig. <ref> shows a case, though <cit.> introduced high-level constraints, they still cannot handle this case well. 
In this paper, we use a simple intuition from human cognition, which we humans can easily separate the mixed visual appearance into reflection and background layers by object level priors. 
One can first recognize the internal objects and quickly assign them to the background layer. Take Fig. <ref> as an example, we understand that the background is a person sitting on a chair so that all human parts and the chair need to be in the background layer. This enables us to recover the background correctly and meanwhile recover the reflection layer clearly.

However, implementing such an idea is not trivial, because it is not guaranteed to get the robust semantics from the input image with reflection, while a cleaner image is of many benefits to the estimation of semantic segmentation. To enable this, we assume the intensity of the background image is stronger than the reflection, which is the same as all existing reflection removal methods. Therefore, the background's semantics is more significant than that in the reflection layer in this situation.
We then propose a novel multi-task learning method for single image reflection removal, named Semantic Guided Reflection Removal Network (SGR^2N). This means the semantic estimation and reflection removal are learned and optimized simultaneously.
Furthermore, we design the semantic guidance block, which explicitly utilizes the semantic to guide the reflection removal. 

To evaluate the effectiveness of SGR^2N, we conducted systematical experiments on three datasets.
Experiments report consistent and significant performance improvement on all three datasets. Ablation study and analytical experiments also show the effectiveness of our method.

Contributions. We summary the contributions as follows:



	
  * To the best of our knowledge, it is the first method to use pixel-level semantic information as guidance for reflection removal, which is an explicit way to represent and use semantic information. And our method jointly solves the semantic segmentation and reflection removal from a single image.
	
	
  * We propose a novel multi-task and end-to-end network with a novel semantic guidance block to achieve single simultaneous image reflection removal (main-task) and semantic analysis (sub-task). 
	
	
  * We demonstrate the effectiveness of the method through comprehensive experiments on both three datasets.   





§ RELATED WORK


Multiple-input methods. 
Many works solve Eqn. (<ref>) with multiple inputs. Methods <cit.> assume the reflection and background layer are at different depth planes which can be separated by multi-view depth estimation.
To align multiple inputs, optical flow is adopted to achieve reflection removal <cit.>. 
Another group of work uses a pair of with/without flash points to make reflection removal like <cit.>. 
Schechner <cit.> using a group of images with different focal lengths, remove reflections by solving the depth 
of different layers. 
Punnappurath <cit.> use dual pixel sensors to capture two sub-aperture views of the scene, and then find defocus disparity clues for reflection removal.
Many works <cit.> explore the polarization and take multiple images to solve the optimal separation through angle filter.
Lei <cit.> propose to use a pair of flash and no-flash images as input. Then they exploit flash-only cues for reflection removal.
Niklaus <cit.> propose a learning-based dereflection algorithm that uses stereo images as input. They find the cues for reflection removal from two views.

Non-CNN single-image methods.
Eqn. (<ref>) is not directly solvable for a single image.	To tackle this 
Li <cit.> assume that the reflection layer is more blurry than the background layer and model these as two different gradient distributions in the two layers for the separation. 
Shih <cit.> explore the ghost effect in the reflection layer and design a GMM model to make reflection removal.
Arvanitopoulos <cit.> make reflection suppression through the relative gradient prior between two different layers. 
Sandhan <cit.> use the symmetry in the human face to remove the reflections on glasses. 
Yun <cit.> propose an algorithm to remove virtual points in large-scale 3D points clouds using the conditions of the reflection symmetry and the geometric similarity.
Yang  <cit.> propose a relatively fast reflection suppression algorithm via convex optimization. 

CNN-Based single image methods. 
Fan <cit.> propose the Cascade Edge and Image Learning Network (CEIL Net) for reflection removal, in which the background's edge is predicted at first and then is used to guide the reflection separation.
Wan utilize existing prior information, design a benchmark <cit.> for reflection removal, and then train an end-to-end model called CRRN <cit.> to separate layers. Wan <cit.> adopt multi-scale gradient information for reflection removal.
Inspired from image-to-image translation, Liu <cit.> proposed an unsupervised method for reflection removal. Kim <cit.> extend the deep image prior to unsupervised reflection removal. 
Zhang <cit.> propose perceptual loss, which is extracted from the first layers of VGG, later they combine feature loss, adversarial loss, and exclusive loss together.
The main difference between perceptual loss 
and ours is that we explicitly utilize high-level semantic information to guide 
reflection removal during training. 

To suit the method for more challenging scenarios, Wen <cit.> propose to train one CNN to synthesize the reflection contaminated image beyond linearity.
Wei <cit.> train ERRNet to remove reflections from little misaligned image pairs.
More recently, Ma  <cit.> introduce the entanglement and disentanglement mechanisms to reflection removal.
Hong  <cit.> study the reflection removal on panoramic image, which includes a partial view of reflection scenes. Zheng  <cit.> explore the absorption effect and design a two-stage framework for reflection removal.
There are also many methods that try to tackle reflection removal in a multi-stage manner <cit.>.
Differ from the aforementioned methods, 
we explicitly use the pixel-level semantic information for reflection removal task. Specifically, we adopt intermediate features from the semantic segmentation task, which contain meaningful object attentions, as guidance signals for reflection removal.

Semantic information in other low-level vision tasks.
Semantic information has been used in many low-level vision tasks. For instance, 
Baslamisli  <cit.> propose to use semantic information to guide albedo computation and design cascade CNNs for joint learning of intrinsic images and semantic segmentation.
Liu  <cit.> propose a deep neural network solution that cascades two modules for image denoising and high-level tasks, respectively.
Wen  <cit.> propose to incorporate global semantic priors as input to regularize the image dehazing. 
Wang  <cit.> propose a three-stage model for rainy image restoration. In which a semantic consistency loss is adopted for details recovery. 
These methods usually adopt multi-stage designs to estimate the semantics and low-level vision task successively. Unlike these problems, we proposed a novel unified framework with a semantic guidance module that jointly learns semantic segmentation and reflection removal.





§ SEMANTIC GUIDED REFLECTION REMOVAL




 §.§ Study on Semantic Information with Reflection Interference
 



Our motivation is that if the background layer semantic information is provided or can b estimated, we can use it to help separate the background layer from the input image.  
In other low-level vision tasks to recover the clean image from the interference, the interference layer to be removed is usually largely different from the image layer and contains no semantic object information, , noise, hazy volume, rain streaks. Unlike these tasks, in reflection removal, the reflection layer to be removed is also a reflected scene, which causes difficulty in estimating the background semantics. Therefore, it is important to see how the reflection affects the estimation of background semantics.

We try an empirical study to test the robustness of semantic segmentation estimation against different intensity levels of reflection.
We randomly sample images from the Pascal VOC dataset <cit.>, where the ground truth of semantic label is provided in 20 categories. 
Based on it, we synthesize the image with reflections by linearly blending two images using 𝐈 = (1-α)𝐁 + α𝐑, 
where larger α can simulate the reflection layer with stronger intensity.
In total, we sample and generate 5000 × 9 sets of images with α=0.1, 0.2, …, 0.9.
Fig. <ref> illustrates the relationship between semantic segmentation quality and the weight of reflection layer. 

Statistical results have shown that semantic estimation is relatively robust when
the weight of reflection layer is below 50%. 
When the weight is bigger than 50%, such reflections would be taken as extreme cases, which might contain 
1) reflection layer is too strong or 
2) background layer contains no salient object (and leads texture-less transmission layer). 
In this work, following all previous methods  <cit.>, we make a similar assumption and address the cases with moderate reflection strength and the background.
The extreme cases with intense reflections are considered as limitations of our method and leave such cases as our future directions. 




 §.§ Multi-task Learning for Simultaneous Reflection Removal and Semantic Estimation




Given an input image with reflection interference, we perform two tasks: (1) Extracting background
semantic map 𝐒_B from input 𝐈, and 
(2) Recovering background layer 𝐁
(also reflection layer 𝐑) from input 𝐈 along with the semantic information (𝐒_𝐁) obtained in the first task.
To this end, we propose the semantic guide reflection removal network (SGR^2N), in which a novel semantic guidance module is proposed to link semantic segmentation and reflection removal together. Our method can work adaptively using semantic information as guidance for reflection removal. 



  §.§.§ Architecture design.


Our SGR^2N layout is illustrated in Fig. <ref>, containing a Shared Encoder, a Semantic segmentation module, and a Separation module with Semantic guidance. 
Given the input image, Shared Encoder aims to extract rich features from different stages of the backbone for semantic segmentation task and reflection removal task. Following <cit.>, we use ResNet-101 <cit.> as the backbone, in which intermediate features from different stages (res 1-5) are taken as outputs.
Semantic segmentation module estimates the semantic map of the input. We follow the DeepLabv3+ <cit.> and move its ASPP module to the decoder. We use the semantic classifier to predict the semantic map of the input image. 
The semantic classifier is constructed by a 1× 1 convolutional layer and a softmax activation layer. It maps the feature into the semantic predictions (with the shape of [B, N, H, W], where N is the number of semantic labels).
The intermediate features before/after the classifier are used as guidance for reflection separation.  

Separation module utilizes the fused features as input and tries to recover the corresponding background and reflection with semantic guidance. In this module, we use contextual attention to get richer information firstly. This module is proved effective in low-level vision tasks in <cit.>. We bring a similar architecture from <cit.> but with fewer channel attention layers.	

Semantic guidance for reflection removal. 
We carefully design a semantic guidance module to use semantic information as guidance for reflection removal.
As shown in Fig. <ref>, the detailed design is as follows.
(1) Selection of guidance features. We choose the mid-level feature is because it contains the most informative features from the semantic segmentation module. Specifically, there are two intermediate features (, denoted as A and B in Fig. <ref>) that are used for guiding the separation network. As illustrated in Fig. <ref>, we adopt the intermediate features (A and B) of semantic segmentation module. These intermediate features ensure the guidance contains the soft object's attentions for reflection removal. In detail, 
	a) A contains all information about high-level information (generated by ASPP module) and low-level feature (from the output the second residual block of the encoder). 
	b) B contains all the features of the segmentation result.
(2) Fusing strategy of guidance. The underlying design principle is that we use semantic information to make the network focus on meaningful objects for reflection separation. One of the best ways is to use an attention mechanism to introduce semantic information across channels <cit.>. We use the commonly used attention operation for implementation. Based on the standard attention layer, we further get a richer structure by making it within a residual block, which can also avoid a dramatic increase of the trainable parameters. 

Here we use two semantic guidance blocks to use mid-level and high-level semantic features to guide reflection removal.
Specifically, let F_R denote original features produced by the former network layer, F_S denote the semantic feature map produced by the semantic estimation module. F_S mainly provides contextual attention to the image. Mathematically, the semantic guidance module uses semantic feature F_S as guidance for original features F_R as follows:

    F_out = V⊗F_S ⊕F_R,   V = τ(w_v1*τ(w_v2*K + b_v2) + b_v1),   K =  F_R ⊙F_S,

where τ is ReLU activation function, w_v1, b_v1, w_v2, b_v2 are learnable parameters.
For the regions with no semantic information, the semantic guidance module will yield the F_R directly thanks to the residual structure. This design can provide relatively robust guidance for separation.
 
The output layer of the reflection removal module is a convolution layer with 6 channels. The former 3 channels construct the background layer B, and the remaining channels construct the reflection layer  R.



  §.§.§ Loss function


As we are jointly performing two tasks, the final multi-task learning (MTL) loss functions are built on reflection removal task and semantic segmentation together.

Reflection removal loss. We use three terms of loss functions for regularizing ℒ_B and ℒ_R, which is similar to previous methods <cit.>.

1) Feature loss. Inspired by <cit.>, we define the feature loss based on the features from the Rich Encoder. This design will not introduce extra resources. Take image X as the input of encoder, let ϕ_l being the feature from the l-th stage of Rich Encoder, the feature loss is defined as: 

    l_feat = ∑_l λ_l∥ϕ_l(X) - ϕ_l(X̂)∥_1,

where {λ_l} denotes balancing weight. 

2) Pixel loss. Following <cit.>, we compute the differences between the prediction and the ground-truth in pixel-level and gradient level as follow:

    l_pix = ∥X - X̂∥_1 + ∥∇X - ∇X̂∥_1,

where ∇ is the gradient operator.  X̂ and X are the network prediction and corresponding ground-truth.

3) Adversarial loss. We further add adversarial loss to make the produced background and reflection look more realistic. We use a discriminator network D_θ and minimize the adversarial loss <cit.>, which is defined as 

    l_adv = -log (D_θ(X, X̂)),

where the loss for discriminator network is l_adv^D = -log (1- D_θ(X, X̂)) - log (D_θ (X, X̂)).

To summarize, the loss for background (ℒ_B) and reflection (ℒ_R) are

    ℒ_B    = w_1 l_feat (B̂, B) + w_2 l_pix (B̂, B) + w_3 l_adv (B̂, B), 
    ℒ_R    = w_1 l_feat (R̂, R) + w_2 l_pix (R̂, R) + w_3 l_adv (R̂, R),

where we empirically set w_1=0.1, w_2=1 and w_3=0.01 in our experiments. 

Semantic segmentation loss. 
For semantic loss ℒ_S, we use cross-entropy as the penalization.

    ℒ_S = ∑_j=1^K∑_i=1^M(-y_jilogŷ_ji - (1-y_ji)log(1-ŷ_ji)),

where K is the size of training batch, M is the summation over classes total number of semantic categories. M=21 in our synthetic dataset, ŷ is the prediction, the ground truth label is y.	

Total loss. 
The total loss is the combination of reflection removal loss and segmentation loss.

    ℒ = αℒ_B + βℒ_R + ℒ_S,

where the hyper-parameters are empirically set as α=0.5, β=0.25.



  §.§.§ Implementation details.
 The proposed method is implemented via PyTorch <cit.>, which is available at Github[<https://github.com/DreamtaleCore/SGRRN>]. To make our model converge faster and save training time, we employ ResNet-101 with pre-trained parameters on ImageNet <cit.>, whose effectiveness has been proved in <cit.>.
Convolution weights are initialized as CZ18 <cit.>. Input images are resized to 256× 256 before feeding into the network.
We train the SGR^2N with 500 epochs using the Momentum optimizer <cit.>, which is employed with momentum = 0.99, where the cycle learning rate is initially set as 0.007 and decay in every 30000 iterations until 0.0001. The batch size during training is empirically set to 5.




§ EXPERIMENTS
 

We first synthesis the semantically annotated dataset. 
Then we perform rigorous ablation studies to analyze the effect of different parts of the SGR^2N. We evaluate our semantic guided reflection removal method quantitatively and qualitatively on single image reflection removal against previous methods. Next, we make additional experiments on how the intensity of reflection layer affects the final performance of the semantic segmentation and the reflection removal task. Finally, we show applications of our model and make a discussion on failure cases.



 §.§ Semantic Reflection Dataset Generation
  



To train the SGR^2N, a training set with semantic annotations is needed. However, existing datasets <cit.> for reflection removal are all lack semantic annotations. On the other hand, it's labor-consuming to label semantic annotations for the existing reflection removal dataset. Therefore, we generate the reflection removal dataset with semantic labels. 
In detail, there are four types of aligned data in each group of the proposed dataset: (1) image with reflection 𝐈, (2) clear background  𝐁, (3) reflection 𝐑, and most importantly (4) semantic labels 𝐒_B.

To render the realistic reflections, we follow the generation strategy in <cit.>, besides, we add ghost effect <cit.> for more abundant and realistic cases.
For semantic labels, we generate the dataset on the basis of Pascal VOC <cit.>, which provides semantic annotations. 
The visual samples of the generated dataset are illustrated in Fig. <ref>. 
The generated dataset contains 20 categories of objects, and each image contains 2.93 foreground objects with semantic labels on average. Additionally, the comparison with the existing datasets is listed in Table <ref>.	





 §.§ Experimental Setups


We train the SGR^2N on our synthetic training set, and then we evaluate the performance on the different test sets. Specifically, we use 29503 images from our generated dataset as the training data. Then we evaluate the trained SGR^2N on the synthetic test set, 20 real images from Berkeley dataset <cit.> and 463 real images from SIR^2 dataset <cit.>.
Image is cropped with the size of 224 × 224 randomly and is flipped randomly as data augmentation. For the compared methods, we use the same dataset for training, unless specifically stated. For evaluation, we use PSNR, SSIM <cit.> to measure the quality of reflection removal. 
Although these two metrics are most widely used in low-level vision tasks, they have their limitations during the test. It is mainly because SSIM and PSNR are both sensitive to the intensity variance. After all, they leverage luminance and contrast similarity between two images. Therefore, we further use histogram alignment techniques to better calculate SSIM and PSNR in Sec.4.4.
Following recent and popular semantic segmentation papers <cit.>, we use mean Intersection over Union (mIoU) and pixel accuracy of different categories for semantic segmentation. 



 §.§ Ablation Study
 

In this section, we evaluate the effectiveness of 1) our multi-task learning architecture, analyze the 2) semantic guidance module and 3) loss functions.

Architecture.
To evaluate the effectiveness of our semantic guidance architecture, we remove the multi-task scheme or semantic guidance block and re-train the network under the same condition. We first reconstruct our model with training reflection separation solely and take this version as the baseline, as shown in Fig <ref> (a). In this case, the architecture of baseline is similar to previous state-of-the-art methods <cit.>.

 

We find that reflection cannot be removed in Fig. <ref>-top (c), and even unpleasant color-shift appears at the area of reflections. Meanwhile, as shown in the secondthird row of Fig. <ref>-top (c), the reflection layer separated from the input contains nothing meaningful. The reason is that the baseline is a reflection removal network with no semantic awareness. 

  

Semantic guidance block. 
To evaluate the effectiveness of the semantic guidance block, we conduct additional experiments by adding semantic guidance into the baseline, as illustrated in Fig. <ref> (b).
Specifically, we train the semantic segmentation network at first, then freeze the semantic segmentation branch and use its decoder to guide reflection separation through semantic guidance block.
Table <ref> shows the ablation study for architecture. This operation can improve the performance of reflection separation (background/reflection) by 3.03/0.58 PSNR and 0.007/0.035 SSIM. The visual results at the top of Fig. <ref> (d) show that parts of reflection can be separated from the input, but there are still many residuals that remain on the girl's faces. The reason why reflection separation cannot significantly benefit from semantic guidance is that semantic information is not accurate enough when it is extracted from the image contaminated directly.

MTL scheme. 
To verify the contribution of MTL, we use a Basic MTL scheme to jointly optimize semantic segmentation and reflection separation and make these two tasks share the same encoder, which is illustrated in Fig. <ref> (c).
This effective operation can promote the performance of reflection separation (background/reflection) improved by 3.33/1.93 PSNR and 0.038/0.116 SSIM. The visual comparison is illustrated in Fig <ref>-top (d) showing that the background layer and reflection layer are separated effectively. However, it can be found that the arm of the left girl is mistakenly separated into the reflection layer. As a result, there are many residuals of background on the reflection layer. In the meantime, some parts with semantic meaning of the background are missing.

The performance of reflection separation is further improved by merging semantic guidance with MTL, as illustrated in Fig. <ref> (d), which is our final model. Fig <ref>-top (f) shows clearer background at the region where semantic information guides. More specifically, persons in the background layer become clearer without artifacts on the arm (highlighted with dashed box). Meanwhile, the residuals are removed totally from the reflection layer. The PSNR of background/reflection is improved from 25.22/17.47 to 26.02/18.51 and SSIM is improved from 0.859/0.538 to 0.875/0.592. The semantic segmentation achieves better results by simultaneously learning to reflection removal. 

We further explore the relationship between semantic segmentation and different architectures. The quantitative comparison is reported in the 4-th column of Table <ref> and one group's results are illustrated in Fig. <ref>-top. 
Without multi-task learning, we find that the bottle in the input image cannot be recognized and separated correctly due to the reflection, which is shown in Fig. <ref>-top (d) and (f). 
The MTL can improve the performance of semantic segmentation by 0.051 mIoU. The mIoU can be further improved from 0.546 to 0.584 by adding semantic guidance. 
It can be seen that the proposed MTL and semantic guidance not only can consistently improve the reflection removal performance on three datasets but also improve the performance of semantic segmentation (MTL: ↑ 0.151 mIoU, Final version: ↑ 0.189 mIoU). These quantitative results show the effectiveness of our proposed architecture.



Analysis of different loss functions.
To analyze how each loss contributes to the final performance of our network on reflection separation, we conduct additional experiments on the feature loss Eqn. (<ref>), the pixel loss Eqn. (<ref>), and adversarial loss Eqn. (<ref>). 
To make a detailed ablation study on feature loss l_feat(B̂, B), we conduct ablation studies as follow:


	
  * Totally remove the loss term. We remove l_feat(R̂, R) from the total loss, then R̂ is generated through I - B̂.
	
  * Replace the loss term. We replace l_feat(B̂, B) with the L1 difference between B and B̂.


The l_feat(R̂, R) is useful to the performance of reflection removal. To demonstrate this, we have made a similar ablation study on it. The quantitative results are reported in Table <ref>. Note that the results of ablation studies on l_feat(R̂, R) are marked with †. 

When we remove the feature loss l_feat, there are many residuals of reflections on the background layer, and some parts of the man's face are taken as reflection mistakenly, as shown in Fig. <ref>-bottom (b). The reason can be that l_feat contains many perceptual constraints, which can constrain the low-level, mid-level, and high-level information of the background and reflection layers in the latent space <cit.>.  
When we remove the l_pix, some color-shifting occurs on the image, , the man's shirt becomes darker, and the man's skin looks slightly over-saturated; the reason is that the l_pix constrains the pixel-level appearance of the image.
The adversarial loss l_adv further helps recover more natural backgrounds and reflections, as shown in the bottom of Fig. <ref> (d) and (e).

The quantitative comparisons are shown in Table. <ref>. In the first row, we take the state-of-the-art method <cit.> as reference. Removing different loss terms from the total loss, we test the SGR^2N under the same condition. In the second row, we remove feature loss and adversarial loss and take this model as a baseline. It can be found that the reference method <cit.> performs better than our baseline. These results are reasonable because method <cit.> adopts various loss functions,  feature loss, adversarial loss, and exclusive loss.
Adding adversarial loss into the objectiveness, the performance of background/reflection improves 0.01/0.055 SSIM and 0.42/1.25 PSNR, and our method with these settings gets comparable results to the state-of-the-art. The 4-th line of Table <ref> shows that feature loss can improve 0.047/0.107 SSIM and 3.48/1.81 PSNR. 
When (a) l_feat(R̂, R) is removed, the SSIM/PSNR of the background layer drops from 0.875/26.02 to 0.861/25.58, and the SSIM/PSNR of the reflection layer drops from 0.592/18.51 to 0.450/16.93. As for changing the l_feat(R̂, R) to L1 loss, the performance is also decreased. For background layer, SSIM: 0.875→0.871, PSNR: 26.02→25.89. For the reflection layer, SSIM: 0.592→0.499, PSNR: 18.51→17.87. 
The performance drops steadily without pixel loss. The combination of feature loss and adversarial loss can further boost the performance to 0.875/0.592 SSIM and 26.02/18.51 PSNR.
As shown in the last row, numerical results on other real benchmark datasets also show the combination of different loss terms achieves the best performance comprehensively. Visual results in a Fig. <ref> also show the clearer background and reflection layers, especially in the region with dashed boxes.








 §.§ Comparison with the State-of-the-Art Methods
 

In this section, we compare our SGR^2N against state-of-the-art methods including optimization-based approaches (LB14 <cit.>, AN17 <cit.>, and YM19 <cit.>) and the learning-based methods (CEILNet <cit.>, CRRN <cit.>, BDN <cit.>, ZN18 <cit.>, ERRNet <cit.>, WT19 <cit.> and ICBLN <cit.>). For a fair comparison, we finetune these models on our training dataset if the training code has been published. We have also provided the numerical results of pretrained IBCLN (termed as IBCLN-p).

Synthetic images. 
Our synthetic test set contains 1969 {input, background, and reflection} triplets and corresponding semantic maps. We train all the existing methods on our synthetic training set if they were CNNs based and provided training code. Then we make quantitative and qualitative comparisons among these methods and ours.




The quantitative results are shown in Table <ref>.
The fine-tuned version of previous methods shows better numerical performance than the pre-trained version (Table <ref> IBCLN-p v.s IBCLN).
We show stronger quantitative performance over previous works on both synthetic data, meanwhile, our method reaches a relatively fast runtime. 

Fig. <ref> shows the qualitative results of different methods. It can be found that the state-of-the-art methods cannot remove reflection totally, while these reflections can be removed easily with semantic understanding via our SGR^2N. More specifically, WT19 <cit.> cannot remove reflection on the area which has obvious semantic meaning. CEILNet <cit.> can remove some reflections, however, it can be found other artifacts appear on the boy's face (the bottom of 5-th column of Fig. <ref>). ZN18 <cit.> can remove most reflections, but unpleasant color-shifting occurs on the girl's face. BDN <cit.> enhances the color of the input image and smooths the reflection area, but the reflections aren't removed apparently. YM19 <cit.> over-smooths the input image and outputs unrealistic results. ERRNet <cit.> can produce a relatively clean background layer, but there are still some reflection residuals on the output. To summarize, our method produces the most compelling results with correct removal and cleaner local details. Furthermore, the previous methods show obvious quantitative performance margins below the proposed method on synthetic data. It can be explained that plenty of semantic objects greatly boosted the performance of reflection removal through our method.



Real images. 
To make a fair comparison, if the previous methods' pretrained models are published, we use the published model directly for comparison. Otherwise, We fine-tune these models on the real images under the same setting. 
The test set of Berkeley real image contains 20 {input, background} pairs, which are collected behind a portable glass <cit.>. 	




We further conduct comparisons on SIR^2 <cit.>, which contains three sub-datasets. These three sub-datasets are captured under different conditions: (1) 20 controlled indoor scenes composed by solid objects; (2) 20 different controlled scenes on postcards; and (3) 55 wild scenes with ground truth. In the following, we compute the average PSNR/SSIM of these three sub-datasets for different methods.

Fig. <ref> shows the qualitative comparisons on the test set. Our method can handle images with complex semantic information. Furthermore, our method can generate cleaner background than other methods. 
The first row shows one visual comparison among state-of-the-art methods on Berkeley test set and the last two rows show the results on SIR^2 dataset. For the visual results on the first row, we observe that CEILNet <cit.>, ZN18 <cit.>, and ERRNet <cit.> can produce clean background. However, some details of color are missing, as illustrated in the red box. BDN <cit.> and YM19 <cit.> can preserve the color detail of the input image, but BDN cannot remove reflection totally (as shown in the green boxes). YN19 produces over-smoothed images which are not realistic. Our method can overcome such limitations through semantic understanding and can reconstruct clean and correct background images. ICBLN introduces additional artifacts on the image, such as the white table suffered seriously in the second row.

Table <ref> summarizes the results of all competing methods on these two real benchmarks Berkeley test and SIR^2 benchmark. Note that Berkeley test images only provide the ground-truth for backgrounds; thus it is not adaptable to evaluate the performance on reflection images. Although there are few semantic objects in the  𝒟_BKL, which leads to the results that our reflection removal module cannot get enough guidance from the semantic segmentation module, the SGR^2N still achieves comparable results.
We find in this dataset, there are semantic categories that are unseen in the training set. The semantic information cannot be used to guide reflection removal. In this situation, the performance of reflection removal decreases slightly. One possible way for this is to annotate more kinds of semantic labels. Even though, our SGR^2N still achieves comparable results.
Meanwhile, our model shows superior performance on SIR^2 benchmark. Because the codes of Ma  <cit.> and Kim  <cit.> have not been published yet, we compare the numerical results (which are from their published paper) on the subset of the SIR^2 (Wild Scenes). 
As reported in the SIR* in Table <ref>, our method shows consistently effective results. 



Our method also performs well for images with complex semantics. As illustrated in Fig. <ref> and Fig. <ref>, our method can handle images with complex semantic information. It can be found that even if there are more than 6 kinds of semantic objects in the input, our method can still perform well in this case. The semantic map is generated through DETR <cit.> for reference. 



Fig. <ref> illustrates more visual results on real images, which are provided by <cit.>. It can be observed that the proposed method generates clean backgrounds and clear reflections. Meanwhile, the semantic map is predicted correctly. These results show that our method can generalize well on various real scenes across different datasets.



Alleviate the intensity sensitivity of SSIM and PSNR.
We notice in the qualitative comparison that the resulting images given by some comparing approaches are slightly darker than the ground truth. It may affect the SSIM and PSNR result. 
Therefore, we align the lightness-histogram of all resulting images to be the same as the input such that the numerical comparison will not be biased by such as the brightness of the images. 

To handle this, we have conducted an additional experiment as follows:

1) Histogram alignment. We align the lightness-histogram of all resulting images to be the same as the input image. In detail, we first convert the resulting image and input from RGB space to HSV space. Since the V channel is highly related to the brightness, we align the V channel of the resulting image to the V channel of the input image. In the next, the original V channel of the resulting image is replaced with the result from the last step. Finally, we convert the modified resulting image back to RGB space. The visual results and alignment curves are illustrated in Figure <ref>. After histogram alignment, the resulting image becomes brighter.

2) Performance test. We then computed the SSIM and PSNR between the aligned image and the corresponding ground truth. The alignment results will affect the PSNR and SSIM, visual results are shown in Figure  <ref>. Quantitative results are shown in Table <ref>. As reported, the performances of all methods are slightly boosted in the synthetic dataset. Our method consistently generates better results over different datasets and layers.

In summary, after brightness alignment, our method still outperforms other competed methods, especially in 𝒟_BKL.








Inference time. 
We further test the running time of prior works and ours and present the result in the last column of Table <ref>. 
We test different approaches on 𝚄𝚋𝚞𝚗𝚝𝚞 18.04, with an Intel ®Core^TM i7-7700 CPU and a 𝙶𝚎𝙵𝚘𝚛𝚌𝚎 𝙶𝚃𝚇 1080 GPU card. 
The comprehensive comparison is illustrated in Fig. <ref>.
In detail, methods LM14 <cit.>, AN17 <cit.>. and YM19 <cit.> are tested on CPU with Mathlab 2017b <cit.>. 
The other methods are tested on GPU:
CEILNet <cit.> is implemented through Torch <cit.>.
YG18 <cit.>, BDN <cit.>, ERRNet <cit.>, and WT19 <cit.> are implemented via PyTorch <cit.>.
Method ZN18 <cit.> is implemented through Tensorflow.
We find our method decreased the computational cost, in terms of parameters and inference time. To demonstrate this, we have conducted additional comparisons between our methods and the other two state-of-the-art methods <cit.>. Detailed results are listed in Table <ref> supplemental material.






 §.§ Relationship Between the Semantic Segmentation and Reflection Removal


To understand how the quality of semantic information affects reflection removal, we conduct an additional experiment to more deeply explore the relationship between semantic guidance and reflection removal. 
The experiment is organized as follows:

	
  * Reflection removal with accurate semantics as guidance. We use the features from the semantic estimation branch of our final model. In this case, semantic features are correctly computed.
	
  * Reflection removal without semantic guidance. In this case, we remove the semantic features directly. 
	
  * Reflection removal with wrong semantics as guidance. To demonstrate the effect of wrong semantics, we pre-compute the semantic features of another image. Then we replace the input of the semantic guidance module with these features. In this way, wrong semantics are taken as guidance for reflection removal.   


The visual results are illustrated in Fig. <ref>. Then one can find the results that 

	
  * When accurate semantics are used as guidance, which is illustrated in Fig. <ref> (b), reflection can be well removed and the background tends to be clean and of high quality.
	
  * When the semantic information is removed, as shown in Fig. <ref> (c), the reflection can be still removed from the mixed image. But there may leave few contextual residuals of reflection, as illustrated in the dashed box.
	
  * When the wrong semantic features are used to guide reflection removal, as shown in Fig. <ref> (d)-(e), the results suffer some artifacts.

In summary, semantic information is important to reflection removal, and different semantics will affect the reflection separation performance. 
In addition, the quality and accuracy of the semantic information are jointly optimized with our method. As shown in Fig. <ref>, the semantic estimation is relatively robust in most cases.






 §.§ Exploration of Performance v.s the Reflection Strength


In this section, we make experiments on the relationship between the SGR^2N performance and the strength level of the reflection layer.
We generate a series of image quadruples of {𝐈_t, 𝐁_t, 𝐑_t, 𝐒_t}^N_t=1, where 𝐈 = (1-α)𝐁 + α𝐑, α=0.1, 0.2, … 0.9, where α is a reflection strength-relevant parameter. We set α=0.1, 0.2, …, 0.9 in this experiment to generate 9 groups of datasets, respectively. 
Here images are randomly selected from Pascal VOC <cit.>.
Each group contains 1000 quadruples (I, B, R, and S) for training and 200 quadruples for test. For each group, we train and test our method under the same condition.
Then we compare the reflection removal and semantic performance (pixel accuracy and mIoU) v.s α through different settings. 
We compare the final mIoU of DeeplabV3+ <cit.> (pretraiend on PascalVOC dataset) and the final PSNR of our baseline <cit.> on such images.

As presented in Fig. <ref>, the proposed SGR^2N performs a higher score than the baseline in most cases with different α values.
Furthermore, the SGR^2N performs a more robust result to different intensities of reflection layer, as illustrated in Fig. <ref>. We find that reflection influences semantic accuracy slightly when reflection strength is lower than 0.5. Otherwise, the reflection will influence semantic accuracy. Note that such scenarios are also hard cases for reflection removal because the input image is dominated by reflection scenes.






 §.§ Failure Cases and Discussion


Although the SGR^2N achieves the state-of-the-art on these three datasets, there are challenging cases illustrated in Fig. <ref>.
One of the challenging scenarios where the reflection in the input is too strong, the background is contaminated heavily and our model may not separate layers successfully.
The cascade update scheme ICBLN <cit.> can not separate the reflection correctly. As shown in the first row of Fig. <ref>, the low-frequency part of the input image is incorrectly separated into reflection layer.
Note that reflections cannot be totally removed by these methods, but still, our result is superior to <cit.> (, the person in the background is more distinguishable and the reflection layer is clearer to the other two state-of-the-art methods). 






 §.§ Extend Applications


We extend the proposed method on object detection.
To verify how reflection removal benefits object detection, we conduct additional experiments by using the state-of-the-art objection detection approach YOLO-v3 <cit.>. Specifically, we generate 500 groups of images with bounding box annotations based on Pascal VOC <cit.>.
We use mean average precision (mAP) over all 20 classes as evaluation metrics for objection detection.
Quantitative and qualitative comparisons are shown in Table. <ref> and Fig. <ref>, respectively. Table. <ref> shows that the mAP in reflection contaminated image drops over 12%, and the results are greatly improved after reflection removal.
The detection results on reflection-removed output B̂ hold higher numerical performance and better visual appearance. In detail, as shown in Fig. <ref>, the person can be predicted accurately when the reflection is removed. We find the detection results on our method are relatively accurate because our method produces more clean background than compared method.








§ CONCLUSION AND DISCUSSION


In this paper, we have presented an approach to use semantic clues for the task of single image reflection separation. 
We first explore the relationship between semantic segmentation and reflection removal and applied high-level guidance explicitly.
We design a deep encoder-decoder network for image feature extraction and use a semantic segmentation network in parallel. Then with the two kinds of information fused together, our separation network can correctly separate the background layer and reflection layer.
We evaluate our method with other prior works extensively on three different datasets. The comparison result shows that our approach can outperform the existing methods both quantitatively and visually on all three datasets. Like other methods, the extreme cases where the reflection layer is too strong or the background layer contains texture-less objects are considered as the limitation of our method and we leave such cases as our future directions.




ACM-Reference-Format






Online Appendix



§ MORE VISUAL RESULTS ON REAL BENCHMARK DATASETS


We show more visual results in Fig. <ref> and Fig. <ref> on real benchmark dataset, including 𝒟_BKL and 𝒟_SIR^2. 
Various scenarios with reflections are presented. Here we compare three state-of-the-art reflection removal methods, ZN18 <cit.>, ERRNet <cit.>, IBCLN <cit.>, SILS <cit.>, and RAGNet <cit.>. 
In Fig. <ref>, we find our method generates precise semantic segmentation (S). Also, compared with the other methods, our method yields favorable results. For instance, the reflections on the semantic objects, , vase(first row), the chair (last row), , are all removed properly by our method.


We have also presented the visual results on 𝒟_SIR^2 in Fig. <ref>. This figure contains three types of scenes. From top to down, they are solid object scenes, postcard scenes, and wild scenes. We find out method can generate comparable results when the semantic information is limited, especially in the postcards scenes. When there is some semantic information for guidance, our method produces more favorably visual results than other methods. These results show the flexibility and generalization ability of our framework and semantic guidance module.







§ MORE DETAILS ABOUT THE ARCHITECTURE OF DIFFERENT METHODS



As shown in Table <ref>, ZN18 and ERRNet use VGG-19 as encoder,
which uses much more parameters than that in our encoder, , 143.67M and 57.93M. Besides, the sum of parameters in the decoder of ZN18 is 3.90M, and ours is 4.46M, whereas ERRNet uses 18.95M parameters in the decoder. 
Therefore, our method has fewer parameters in the total model. Furthermore, Our method is also inference faster than these two state-of-the-art methods. More intuitive comparisons among different methods are illustrated in Fig. <ref>.



§ EXTEND APPLICATIONS ON LOW-LEVEL VISION TASKS







To further show the generalization ability of our method, we have conducted additional experiments on dehazing and deraining. Visual results have been shown in Fig. <ref>. Numerical results are reported in Table <ref>. Here we train our model on standard RESIDE dataset <cit.> for dehazing and PBRR dataset <cit.> for deraining, respectively. We generate pseudo semantic labels via DETR <cit.>. Here we conduct the recent dehazing method (MSBDN <cit.>) and deraining approach (Quan  <cit.>) as reference. Numerical results in Table <ref> demonstrate the effectiveness and generalization ability of our proposed framework.


