2021

[1,2]Keren Fu

1]College of Computer Science, Sichuan University

2]National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan
University

3]Institute of Artificial Intelligence, School of Computer Science, Wuhan
University

# Depth Quality-Inspired Feature Manipulation for Efficient RGB-D and Video
Salient Object Detection

Wenbo Zhang<EMAIL_ADDRESS><EMAIL_ADDRESS>Zhuo Wang
<EMAIL_ADDRESS>Ge-Peng Ji<EMAIL_ADDRESS>Qijun Zhao
<EMAIL_ADDRESS>[  [  [

###### Abstract

Recently CNN-based RGB-D salient object detection (SOD) has obtained
significant improvement on detection accuracy. However, existing models often
fail to perform well in terms of efficiency and accuracy simultaneously. This
hinders their potential applications on mobile devices as well as many real-
world problems. To bridge the accuracy gap between lightweight and large
models for RGB-D SOD, in this paper, an efficient module that can greatly
improve the accuracy but adds little computation is proposed. Inspired by the
fact that depth quality is a key factor influencing the accuracy, we propose
an efficient depth quality-inspired feature manipulation (DQFM) process, which
can dynamically filter depth features according to depth quality. The proposed
DQFM resorts to the alignment of low-level RGB and depth features, as well as
holistic attention of the depth stream to explicitly control and enhance
cross-modal fusion. We embed DQFM to obtain an efficient lightweight RGB-D SOD
model called _DFM-Net_ , where we in addition design a tailored depth backbone
and a two-stage decoder as basic parts. Extensive experimental results on nine
RGB-D datasets demonstrate that our _DFM-Net_ outperforms recent efficient
models, running at 20 FPS on CPU with only $\sim$8.5Mb model size, and
meanwhile being 2.9/2.4 times faster and 6.7/3.1 times smaller than the latest
best models A2dele and MobileSal. It also maintains state-of-the-art accuracy
when even compared to non-efficient models. Interestingly, further statistics
and analyses verify the ability of DQFM in distinguishing depth maps of
various qualities without any quality labels. Last but not least, we further
apply _DFM-Net_ to deal with video SOD (VSOD), achieving comparable
performance against recent efficient models while being 3/2.3 times
faster/smaller than the prior best in this field. Our code is available at
https://github.com/zwbx/DFM-Net.

###### keywords:

RGB-D saliency detection, RGB-D salient object detection, video salient object
detection, efficiency, neural networks, deep learning

##  1 Introduction

Salient object detection (SOD) aims to locate image regions that attract
master human visual attention. It is useful in many downstream tasks _, e.g.,_
object segmentation SaliencyAwareVO , medical segmentation fan2020pra ;
ji2021pnsnet ; ji2022vps , tracking 2019Non , video object segmentation
ji2021FSNet , and point cloud registration wan2021rgb . Owing to the powerful
representation ability of deep learning, great progresses have been made for
SOD in recent years, but most of them use only RGB images as input to detect
salient objects RGBsurvey ; deng2021re ; Jiang2013SalientOD ; He2015SuperCNNAS
; Tian2022LearningTD ; peng2021saliency . This unavoidably incurs challenges
in complex scenarios, such as cluttered or low-contrast background, where
colors hardly provide clear distinguishable cues.

With the popularity of depth sensors/devices, RGB-D SOD has become a hot
research topic BBSNet ; fu2021siamese ; UCNet-TPAMI ; HDFNet ;
zhang2021bilateral ; 2020ASIF ; 2020Going ; chen2021cnn , because additional
useful spatial information embedded in depth maps could serve as a
complementary cue for more robust detection RGBDsurvey ; Cong2019Review .
Meanwhile, depth data is already widely available on many mobile devices D3Net
_, e.g.,_ modern smartphones like Huawei Mate 40 Pro, iPhone 12 Pro, and
Samsung Galaxy S20+. This has opened up a new range of applications for the
RGB-D SOD task. Unfortunately, the time-space consumption of existing
approaches JLDCF ; D3Net ; S2MA ; UCNet ; SSF ; cmMS ; DPA ; CPFP ; DRMA  is
still too high, hindering their further applications on mobile devices and
real-world problems. Therefore, an efficient and accurate RGB-D SOD model is
highly desirable.

Figure 1: Average probability distributions (solid curves) of edge Dice
coefficients computed from “Good quality”, “Bad quality”, and “Mismatch”
subsets (10 times random sampling), as well as the probability distributions
(dash curves) of the whole STERE and SIP datasets. The right images show three
examples from the three subsets, respectively.

An efficiency-oriented model should aim to significantly decrease the number
of operations and memory needed while retaining decent accuracy mobilenetV2 .
In light of this, several efficiency-oriented RGB-D SOD models have emerged
with specific considerations. For example, A2dele ; CoNet ; zhao2021rgb  adopt
a depth-free inference strategy for fast speed, while depth cues are only
utilized in the training phase. DANet ; PGAR ; ATSA  choose to design
efficient cross-modal fusion modules DANet  or light depth backbones PGAR ;
ATSA . However, these methods seem to achieve high efficiency at the sacrifice
of state-of-the-art (SOTA) accuracy. The underlying challenge is that their
accuracy is likely to degrade when the corresponding models are simplified or
reduced. It is also worth noting that a recent work wu2021mobilesal , which is
concurrent with ours, proposes a MobileSal (MSal) network that explicitly
addresses efficient RGB-D SOD. Although very encouraging performance and speed
are reported in wu2021mobilesal , to guarantee efficiency, the authors have
discarded the popular multi-scale fusion manner and only merge RGB and depth
information at the coarsest level, which may result in less sufficient feature
utilization and fusion.

Motivation. We notice that unstable quality of depth is one key factor which
largely influences the accuracy, as mentioned in previous works BTSNet ; DPA ;
D3Net ; depthconfidence . However, very few RGB-D SOD models explicitly take
this issue into consideration, _hence not to mention for efficient models_
wu2021mobilesal ; PGAR ; A2dele . We also argue that depth quality is
difficult to determine solely according to a depth map itself depthconfidence
; DPA , because it is tough to judge whether a prominent region in the depth
map belongs to noise or a target object, as illustrated in Fig. 1 (b). Since
RGB-D SOD concerns two paired images as input, namely an RGB image and a depth
map, we observe that a high-quality depth map usually has some boundaries
_well-aligned_ to the corresponding RGB image. A similar phenomenon is also
discussed in NRDQA  for no-reference depth assessment. We call this assumption
“boundary alignment” (BA).

To validate such a BA assumption, we randomly choose 50 paired samples (tagged
as “Good quality” as shown in Fig. 1) from SIP D3Net  dataset, and also 50
“Bad quality” samples from STERE STERE  dataset. These two datasets are widely
used for RGB-D SOD, and are chosen here based on the general observations of
previous works SMAC ; D3Net ; JLDCF . Additionally, we construct a new set of
samples from the “Good quality” set, tagged as “Mismatch”, by randomly
mismatching the RGB and depth images of the “Good quality” set, to see if this
shuffling behavior can be reflected by BA. Note that this behavior actually
causes no changes to individual RGB or depth images, therefore having no
impact to a depth quality measurement that is dependant only on depth itself
(e.g., depthconfidence ). To quantify boundary alignment, an off-the-shelf
edge detector BDCN  is used to obtain two edge probability maps from RGB and
depth, respectively, and then we calculate their Dice coefficient VNet
($C_{Dice}$) as a measure of BA:

$\displaystyle C_{Dice}=\frac{2\sum_{i}(E_{RGB}(i)\times
E_{Depth}(i))}{\sum_{i}E_{RGB}^{2}(i)+\sum_{i}E_{Depth}^{2}(i)},$ (1)

where $E_{RGB}$ and $E_{Depth}$ denote RGB and depth edge maps, respectively,
and $i$ means the pixel index. $E_{RGB}(i)$/$E_{Depth}(i)$ is the $i$th pixel
value on the associated map. $C_{Dice}$ will equal to 1 if the two continuous
edge maps are perfectly matched. The probability distributions (average of 10
times random sampling) of Dice coefficients are shown in Fig. 1, where the
three sets of samples correspond to different colors. We can see that BA seems
a strong evidence for the depth quality issue, and meanwhile for the
“Mismatch” set, its Dice coefficients are generally lower than those of the
“Good quality” set.

\begin{overpic}[abs,scale={0.8},unit=1mm]{fig/Fig_vis_accuracy_vs_speed}
\par\put(7.0,22.3){ \tiny \begin{tabular}[]{l|c}Methods&Size(Mb)\\\
\hline\cr\hline\cr\hline\cr D3Net~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{D3Net}{\@@citephrase{(}}{\@@citephrase{)}}}&530\\\ JL-
DCF~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{JLDCF}{\@@citephrase{(}}{\@@citephrase{)}}}&520\\\
DCF~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{DCF}{\@@citephrase{(}}{\@@citephrase{)}}}&436\\\
cmMS~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{cmMS}{\@@citephrase{(}}{\@@citephrase{)}}}&430\\\
DSA2F~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{DSA2F}{\@@citephrase{(}}{\@@citephrase{)}}}&415\\\
S2MA~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{S2MA}{\@@citephrase{(}}{\@@citephrase{)}}}&330\\\
DMRA~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{DRMA}{\@@citephrase{(}}{\@@citephrase{)}}}&228\\\
CoNet~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{CoNet}{\@@citephrase{(}}{\@@citephrase{)}}}&167\\\
SSF~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{SSF}{\@@citephrase{(}}{\@@citephrase{)}}}&125\\\
ASTA~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{ATSA}{\@@citephrase{(}}{\@@citephrase{)}}}&123\\\
UCNet~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{UCNet}{\@@citephrase{(}}{\@@citephrase{)}}}&119\\\
DANet~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{DANet}{\@@citephrase{(}}{\@@citephrase{)}}}&102\\\ {{DFM-
Net*}}&{93}\\\ \hline\cr\hline\cr\hline\cr
PGAR~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{PGAR}{\@@citephrase{(}}{\@@citephrase{)}}}&62\\\
A2dele~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{A2dele}{\@@citephrase{(}}{\@@citephrase{)}}}&57\\\
MSal~{}\cite[cite]{\@@bibref{Authors
Phrase1YearPhrase2}{wu2021mobilesal}{\@@citephrase{(}}{\@@citephrase{)}}}&26\\\
{{DFM-Net}}&{8.5}\\\ \end{tabular}} \end{overpic} Figure 2: Performance
visualization. The vertical axis indicates the average $S_{\alpha}$ smeasure
over six widely used datasets (NJU2K NJU2K , NLPR NLPR , STERE STERE , RGBD135
RGBD135 , LFSDLFSD , SIP D3Net ). The horizontal axis indicates CPU speed
(FPS). The circle area is proportional to the model size. More details refer
to Tab. 2.

Inspired by the above fact that the alignment of low-level RGB and depth
features can somewhat reflect depth quality, we propose a new depth quality-
inspired feature manipulation (DQFM) process. The intuition behind DQFM is to
assign _lower weights_ to depth features if the quality of depth is _bad_ ,
effectively avoiding injecting noisy or misleading depth features to improve
detection accuracy for efficient models. We also augment DQFM with depth
holistic attention, in order to _enhance_ depth features when the depth
quality is judged to be _good_. With the help of DQFM, we explicitly control
and enhance the role of depth features during cross-modal fusion. Finally, we
embed DQFM into an encoder-decoder framework to obtain an efficient
lightweight model called _DFM-Net_  _(Depth Feature Manipulation Network)_ ,
where a tailored depth backbone and a two-stage decoder are designed for
further efficiency consideration. Fig. 2 gives an intuitive performance
visualization of the proposed methods _DFM-Net_ and _DFM-Net_ * (the latter
utilizes larger backbones for RGB and depth feature extraction), where 15 SOTA
methods and ours are plotted in terms of accuracy, inference speed on CPU, and
model size. It is clearly seen that, compared to SOTA techniques, _DFM-Net_
and _DFM-Net_ * can rank the most upper right with very small circles,
indicating that our methods perform better in terms of both efficiency and
accuracy.

In general, our contributions are five-fold:

  * •

To the best of our knowledge, we are the first to address efficient RGB-D SOD
from a depth quality perspective, which is in clear contrast to existing
methods PGAR ; A2dele  as well as the concurrent work MSal wu2021mobilesal
111Work wu2021mobilesal  is concurrent with our preliminary work
zhang2021depth2 ..

  * •

We propose an efficient depth quality-inspired feature manipulation (DQFM)
process, to explicitly control and enhance depth features during cross-modal
fusion. DQFM avoids injecting noisy or misleading depth features, and can
effectively improve detection accuracy with very little time-space addition.

  * •

Benefited from DQFM, we propose an efficient lightweight model _DFM-Net_
_(Depth Feature Manipulation Network)_ , which has a tailored depth backbone
and a two-stage decoder for further efficiency consideration.

  * •

Compared to 15 SOTA models on nine datasets, _DFM-Net_ is able to achieve
superior accuracy, meanwhile running in quasi real-time at 20 FPS on CPU with
only $\sim$8.5Mb model size. Such performance is 2.9$\times$ faster and
6.7$\times$ smaller than the prior best model A2dele A2dele , and also
2.4$\times$ faster and 3.1$\times$ smaller than MSal wu2021mobilesal .

  * •

Concerning video SOD (VSOD) is another saliency field that may demand high
efficiency, we apply _DFM-Net_ to VSOD and achieve quite competitive
performance. Meanwhile, we introduce a joint training strategy for lightweight
models to better adapt to VSOD task, which proves to be more effective than
the conventional pretraining strategy in this field.

The remainder of this paper is organized as follows. Sec. 2 discusses related
work on general RGB-D SOD, efficient RGB-D SOD, and depth quality analyses.
Sec. 3 describes the proposed _DFM-Net_ in detail. Experimental results,
performance evaluations, and ablation analysis, including the application of
_DFM-Net_ to video SOD, are given in Sec. 4. Finally, conclusions are drawn in
Sec. 5. The preliminary version of this work has appeared in zhang2021depth2 .

##  2 Related Work

The utilization of RGB-D data for SOD has been extensively explored for years.
Based on the goal of this paper, in this section, we review general RGB-D SOD
methods, as well as previous works on efficient models and depth quality
analyses.

###  2.1 General RGB-D SOD Methods

Traditional methods mainly rely on hand-crafted features RGBD135 ; 2013An ;
2012Context ; 2015Exploiting . Lately, deep learning-based methods have made
great progress and gradually become a mainstream PCF ; HDFNet ; UCNet ; DRMA ;
JLDCF ; SSF ; cmMS ; PDNet ; CPFP ; BBSNet ; MMCI ; CoNet ; D3Net ; DANet . Qu
et al. QuRGBD  first introduced CNNs to infer object saliency from RGB-D data.
Zhu et al. PDNet  designed a master network to process RGB data, together with
a sub-network for depth data, and then incorporated depth features into the
master network. Fu  et al. JLDCF  utilized a Siamese network for simultaneous
RGB and depth feature extraction, which discovers the commonality between
these two views from a model-based perspective. Zhang  et al. UCNet  proposed
a probabilistic network via conditional variational auto-encoders to model
human annotation uncertainty. Zhang  et al. SSF  proposed a complementary
interaction fusion framework to locate salient objects with fine edge details.
Liu  et al. S2MA  introduced a selective self-mutual attention mechanism that
can fuse attention learned from both modalities. Li  et al. CMWNet  designed a
cross-modal weighting network to encourage cross-modal and cross-scale
information fusion from low-, middle- and high-level features. Fan  et al.
BBSNet  adopted a bifurcated backbone strategy to split multi-level features
into student and teacher ones, in order to suppress distractors within low-
level layers. Pang  et al. HDFNet  provided a new perspective to utilize depth
information, in which the depth and RGB features are combined to generate
region-aware dynamic filters to guide the decoding in the RGB stream. Li  et
al. cmMS  proposed a cross-modality feature modulation module that enhances
feature representations by taking depth features as prior. Luo  et al. CAS-GNN
utilized graph-based techniques to design a network architecture for RGB-D
SOD. Ji  et al. CoNet  proposed a novel collaborative learning framework,
where multiple supervision signals are employed, yielding a depth-free
inference method. Zhao  et al. DANet  designed a single stream network to
directly take a depth map as the fourth channel of an RGB image, and proposed
a depth-enhanced dual attention module.

With the flourish in this research direction, other inspiring techniques are
also recently employed into the RGB-D SOD task, such as discrepant cross-
modality interaction zhang2021cross , triplet transformer embedding network
liu2021tritransnet , pure transformer network liu2021visual , neural
architecture search sun2021deep , mutual information minimization zhang2021rgb
, specificity-preserving architecture zhou2021specificity , hierarchical
cross-modal distillation chen2021cnn , cross-modal edge-guidance liu2021cross
, LSTM-based context-aware modules liang2021context . A relatively complete
survey on RGB-D SOD can be found in RGBDsurvey .

Despite that encouraging detection accuracy has been advanced by the above
RGB-D SOD methods, most of them have heavy models and are computationally
expensive.

###  2.2 Efficient RGB-D SOD Methods

Besides the above-mentioned methods, several recent methods attempt to take
model efficiency into consideration222The concept of “efficient model” is hard
to define. Here and also in the following experiment section, we consider that
efficient models should be less than 100Mb.. Specific techniques are used to
reduce high computation brought by multi-modal feature extraction and fusion.
Piao  et al. A2dele  employed knowledge distillation for a depth distiller,
which aims at transferring depth knowledge obtained from the depth stream to
the RGB stream, thus allowing a depth-free inference framework. Chen  et al.
PGAR  constructed a tailored depth backbone to extract complementary features.
Such a backbone is much more compact and efficient than classic backbones _,
e.g.,_ ResNet ResNet  and VGGNet vgg . Besides, the method adopts a coarse-to-
fine prediction strategy that simplifies the top-down refinement process. The
utilized refinement module is in a recurrent manner, further reducing model
parameters. More recently, wu2021mobilesal  introduces an efficient model
called MSal for this task, which utilizes an implicit depth restoration
technique to strengthen feature representation of the main RGB stream. To
ensure high efficiency, depth features are only merged into to the RGB ones at
the coarsest level in a small resolution.

###  2.3 Depth Quality Analyses in RGB-D SOD

Since the quality of depth often affects model performance, a few researchers
have considered the depth quality issue in RGB-D SOD, so as to alleviate the
impact of low-quality depth. As early attempts, some works proposed to conduct
depth quality assessment from a global perspective and obtained a quality-
inspired score. Cong et al.  depthconfidence  first proposed a no-reference
depth quality metric IQA  to alleviate the contamination of low-quality depth.
Later, Fan et al.  D3Net  employed triple networks sharing the same structure
to process RGB, RGB-D, and depth inputs individually. Depth quality is then
evaluated by comparing between the results from the later two sub-networks to
explicitly judge whether to introduce depth map information. Cong et al.  DPA
proposed to predict the depth quality score via a perceptron with high-level
RGB and depth features as input. Such a perceptron was trained by scores
calculated by comparing thresholded depth maps with ground truth. Ji et al.
ji2021calibrated  introduced a depth calibration and fusion framework to
explicitly address the depth quality issue. It learns to predict a reliability
score, based on which the raw depth map is calibrated by the estimated depth
map.

Instead of a single quality-inspired score, more recently, spatial quality
evaluation of depth maps was also considered, in order to find valuable depth
region. Wang et al.  DQSF  designed three hand-crafted features to excavate
depth following multi-scale methodology. Chen et al.  DQSD  proposed to locate
the “Most Valuable Depth Regions” of depth by comparing pseudo ground truth
(GT) generated from a sub-network with RGB-D as input, with two saliency maps
generated from two sub-networks with RGB/depth as input.

Different from the above existing methods that have high time-space
complexity, our depth quality assessment is much more efficient and is more
suitable to benefit a lightweight model. Besides, our quality module is end-
to-end trainable and is also unsupervised. In contrast, methods DPA ; DQSD ;
ji2021calibrated  are supervised for quality estimation.

Figure 3: Block diagram of the proposed _DFM-Net_. Best view in color.

##  3 Methodology

###  3.1 Overview

Fig. 3 shows the block diagram of the proposed _DFM-Net_ , which consists of
encoder, decoder, and supervision parts. For efficiency consideration, our
encoder part follows the design in BBSNet , where the RGB branch is
simultaneously responsible for RGB feature extraction and cross-modal fusion
between RGB and depth features. The decoder part, on the other hand, conducts
simple two-stage fusion to generate the final saliency map. More specifically,
the encoder consists of an RGB-related branch which is based on MobileNet-v2
ResNet , a depth-related branch which is a tailored efficient backbone, and
also the proposed DQFM. Both branches lead to five feature hierarchies, and
the output stride is 2 for each hierarchy except 1 for the last hierarchy. The
extracted depth features at a certain hierarchy, after passing through a DQFM
gate, are then fused into the RGB branch by simple element-wise addition
before being sent to the next hierarchy. Besides, in order to capture multi-
scale semantic information, we add a PPM (pyramid pooling module ppm ) at the
end of the RGB branch. Note that in practice, the DQFM gate contains two
successive operations, namely depth quality-inspired weighting (DQW) and depth
holistic attention (DHA).

Let the features from the five RGB/depth hierarchies be denoted as
$f_{m}^{i}~{}(m\in\\{r,d\\},i=1,...,5)$, the fused features be denoted as
$f_{c}^{i}~{}(i=1,...,5)$, and the features from PPM be denoted as
$f_{c}^{6}$. The aforementioned cross-modal feature fusion can be expressed
as:

$\displaystyle
f_{c}^{i}=f_{r}^{i}+(\alpha_{i}\otimes\beta_{i}\otimes{f}_{d}^{i}),$ (2)

where $\alpha_{i}$ and $\beta_{i}$ are computed by DQW and DHA to manipulate
depth features $f_{d}^{i}$ to be fused, and $\otimes$ denotes element-wise
multiplication333If the multipliers’ dimensions are different, before element-
wise multiplication, the one with less dimension will be replicated to have
the same dimension of the other(s).. After encoding as shown in Fig. 3,
$f_{c}^{i}~{}(i=1,...,5)$ and $f_{c}^{6}$ are fed to the subsequent decoder
part.

Figure 4: Architecture of DQW (depth quality-inspired weighting). The red line
arrows indicate the computation of Eq. 4. The dash line arrows indicate the
max-pooling with stride 2.

###  3.2 Depth Quality-Inspired Feature Manipulation (DQFM)

DQFM consists of two key components, namely DQW (depth quality-inspired
weighting) and DHA (depth holistic attention), which generate $\alpha_{i}$ and
$\beta_{i}$ in Eq. 2, respectively. $\alpha_{i}\in\mathbb{R}^{1}$ is a scalar
determining _“how much”_ depth features are involved, while
$\beta_{i}\in\mathbb{R}^{k\times k}$ ($k$ is the feature size at hierarchy
$i$) is a spatial attention map, deciding _“what regions”_ to focus in the
depth features. Below, the internal structures of DQW and DHA are introduced.

####  3.2.1 Depth Quality-Inspired Weighting (DQW)

Inspired by the aforementioned BA observation in Sec. 1, as shown in Fig. 4,
DQW learns the weighting term $\alpha_{i}$ adaptively from low-level features
$f_{r}^{1}$ and $f_{d}^{1}$, because such low-level features characterize
image edges/boundaries Amulet . To this end, we first apply convolutions to
$f_{r}^{1}$/$f_{d}^{1}$ to obtain transferred features
$f_{rt^{\prime}}$/$f_{dt^{\prime}}$, which are expected to capture more edge-
related activation:

$\displaystyle f_{rt^{\prime}}=\textbf{BConv}_{1\times
1}(f_{r}^{1}),~{}f_{dt^{\prime}}=\textbf{BConv}_{1\times 1}(f_{d}^{1}),$ (3)

where $\textbf{BConv}_{1\times 1}(\cdot{})$ denotes a $1\times 1$ convolution
with BatchNorm and ReLU activation. To evaluate low-level feature alignment
and inspired by the aforementioned Dice coefficient VNet , given edge
activation $f_{rt^{\prime}}$ and $f_{dt^{\prime}}$, the alignment feature
vector $V_{BA}$ that encodes the alignment between $f_{rt^{\prime}}$ and
$f_{dt^{\prime}}$ is computed as:

$\displaystyle V_{BA}=\frac{\textbf{GAP}(f_{rt^{\prime}}\otimes
f_{dt^{\prime}})}{\textbf{GAP}(f_{rt^{\prime}}+f_{dt^{\prime}})},$ (4)

where $\textbf{GAP}(\cdot{})$ denotes the global average pooling operation
that aggregates element-wise information, and $\otimes$ means element-wise
multiplication. Here, we empirically drop the square of each element in the
denominator of the Dice coefficient, which we find has better performance.
Such simplification has also been adopted by some previous works (_e.g._ ,
diceloss ).

Further, in order to make $V_{BA}$ robust against slight edge shifting, we
compute $V_{BA}$ at multi-scale and concatenate the results to generate an
enhanced vector. As shown in Fig. 4, such multi-scale calculation is conducted
by subsequently down-sampling the initial features
$f_{rt^{\prime}}$/$f_{dt^{\prime}}$ by max-pooling with stride 2, and then
computing $V_{BA}^{1},V_{BA}^{2}$ the same way as Eq. 4. Suppose $V_{BA}$,
$V_{BA}^{1}$, and $V_{BA}^{2}$ are the alignment feature vectors computed from
three scales as shown in Fig. 4, the enhanced vector $V_{BA}^{ms}$ is computed
as:

$\displaystyle V_{BA}^{ms}=[V_{BA},V_{BA}^{1},V_{BA}^{2}],$ (5)

where $[\cdot]$ denotes channel concatenation. Next, two fully connected
layers are applied to derive $\alpha\in\mathbb{R}^{5}$ from $V_{BA}^{ms}$ as
below:

$\displaystyle\alpha=\textbf{MLP}({V_{BA}^{ms}}),$ (6)

where $\textbf{MLP}(\cdot{})$ denotes a two-layer perceptron with the Sigmoid
function in the end. Thus the vector $\alpha$ obtained contains
$\alpha_{i}\in(0,1)~{}(i={1,2...,5})$ as its elements. Notably, here we adopt
different weighting terms for different hierarchies rather than an identical
one. The effectiveness of this multi-variable strategy is validated in Sec.
4.5.

Figure 5: Architecture of DHA (depth holistic attention).

Figure 6: Visualization of holistic attention maps $\beta_{1}\sim\beta_{5}$,
as well as the resulting saliency maps with and without DHA.

####  3.2.2 Depth Holistic Attention (DHA)

Depth holistic attention (DHA) enhances depth features spatially, by deriving
holistic attention map $\beta_{i}$ from the depth stream. Technically as in
Fig. 5, we first utilize the highest-level features $f_{d}^{5}$ from the depth
stream to locate coarse salient regions (with supervision signals imposed as
shown in Fig. 3). To facilitate subsequent pixel-wise operations, we compress
and then up-sample $f_{d}^{5}$ into $f_{dht}$, which has the same dimension as
$f_{r}^{1}$/$f_{d}^{1}$, formulated as:

$\displaystyle f_{dht}=\textbf{F}_{UP}^{8}(\textbf{BConv}_{1\times
1}(f_{d}^{5})),$ (7)

where $\textbf{F}_{UP}^{8}(\cdot)$ means 8$\times$ bilinear up-sampling. Then
we combine low-level RGB and depth features to recalibrate $f_{dht}$. Similar
to the computation in DQW, we first transfer $f_{r}^{1}$/$f_{d}^{1}$ to
$f_{rt^{\prime\prime}}$/$f_{dt^{\prime\prime}}$. The resulting features are
element-wisely multiplied to generate features $f_{ec}$, which emphasizes
common edge-related activation. To better model long-range dependencies across
low-level and high-level features (_i.e.,_ $f_{ec}$ and $f_{dht}$) while
maintaining efficiency for DHA, we employ the max-pooling operation and
dilated convolution to rapidly increase receptive fields. This recalibration
process is defined as:

$\displaystyle\textbf{F}_{rec}(f_{dht})=\textbf{F}_{UP}^{2}\Big{(}\textbf{DConv}_{3\times
3}\big{(}\textbf{F}_{DN}^{2}(f_{dht}+f_{ec})\big{)}\Big{)},$ (8)

where symbol $\textbf{F}_{rec}(\cdot)$ denotes once recalibration process of
the input. $\textbf{DConv}_{3\times 3}(\cdot{})$ denotes the $3\times 3$
dilated convolution with stride 1 and dilation rate 2, followed by BatchNorm
and ReLU activation. $\textbf{F}_{UP}^{2}(\cdot)/\textbf{F}_{DN}^{2}(\cdot)$
denotes bilinear up-sampling/down-sampling operation to $2/(\frac{1}{2})$
times the original size. As a trade-off between performance and efficiency, we
conduct recalibration twice as below:

$\displaystyle
f^{\prime}_{dht}=\textbf{F}_{rec}(f_{dht}),~{}f^{\prime\prime}_{dht}=\textbf{F}_{rec}(f^{\prime}_{dht}),$
(9)

where $f^{\prime}_{dht}$ and $f^{\prime\prime}_{dht}$ represent the features
recalibrated once and twice, respectively. Finally, we combine
$f^{\prime\prime}_{dht}$ and the previous $f_{ec}$ to achieve holistic
attention maps:

$\displaystyle\beta=\textbf{BConv}_{3\times
3}(f_{ec}+f^{\prime\prime}_{dht}).$ (10)

Note that here the ReLU activation in $\textbf{BConv}_{3\times 3}$ is replaced
by the Sigmoid one, in order to achieve attention outputs for $\beta$.
Finally, we have obtained five depth holistic attention maps
$\\{\beta_{1},\beta_{2},...,\beta_{5}\\}$ by down-sampling from $\beta$,
severing as spatial enhancement terms for those depth hierarchies. A visual
case of DHA is shown in Fig. 6. One can see that $\beta_{1}$ is likely to
highlight regions around edges, while $\beta_{5}$ focuses more on the dilated
entire object. Generally, irrelevant background clutters in depth features can
be somewhat suppressed by multiplying attention maps $\beta_{1}\sim\beta_{5}$.

Table 1: Detailed structure of the proposed tailored depth backbone (TDB), which is based on inverted residual bottleneck blocks (IRB) of MobileNet-V2 mobilenetV2 . About notations, $t$: expansion factor of IRB, $c$: output channels, $n$: times the block is repeated, and $s$: stride of hierarchy, which is applied to the first block of the repeating blocks. Input | Output | Block | $t$ | $c$ | $n$ | $s$
---|---|---|---|---|---|---
$256\times 256\times 1$ | $128\times 128\times 16$ | IRB | 3 | 16 | 1 | 2
$128\times 128\times 16$ | $64\times 64\times 24$ | IRB | 3 | 24 | 3 | 2
$64\times 64\times 24$ | $32\times 32\times 32$ | IRB | 3 | 32 | 7 | 2
$32\times 32\times 32$ | $16\times 16\times 96$ | IRB | 2 | 96 | 3 | 2
$16\times 16\times 96$ | $16\times 16\times 320$ | IRB | 2 | 320 | 1 | 1

###  3.3 Tailored Depth Backbone (TDB)

Usually, depth is less informative than RGB. Hence we consider using a
tailored depth backbone (TDB) to extract depth features, which is much lighter
than the off-the-shelf backbone such as MobileNet-V2 but able to maintain
comparable accuracy. Specifically, we base our TDB on the inverted residual
bottleneck blocks (IRB mobilenetV2 ), and construct a new smaller backbone
with reduced channel numbers and stacked blocks. IRB is the basic block of
MobileNet-V2, which distinguishes it from previous MobileNet mobilenet  and
ResNet ResNet . In contrast to the ResNet block, the intermediate layers of
IRB are designed to be wider than its input/output, so that the inverted
residual just connects to the thin bottleneck. Besides, similar to MobileNet,
IRB also utilizes depth-separable convolution, but removes the last bottleneck
non-linearity for improved performance. The detailed structure of IRB refers
to mobilenetV2 .

Based on IRB, the structure of TDB is detailed in Tab. 1. While output
channels $c$ and hierarchy stride $s$ are made aligned to the RGB branch, $t$
and $n$ are empirically designed inspired by mobilenetV2 . More specifically,
we reduce the $t$ of MobileNet-V2 by nearly half, and $n$ is determined by
referring to the corresponding layers of MobileNet-V2 (those with the same
output channels), meanwhile merging channels of certain adjacent layers to
maintain the overall bottleneck number (Ours: 15 _vs._ MobileNet-V2’s: 17).
This finally results in a {1,3,7,3,1} aligned sequential design of TDB. As a
result, our TDB is much lighter than previous lightweight backbones (Ours:
only 0.9Mb, ATSA’s ATSA : 6.7Mb, PGAR’s PGAR : 6.2Mb, MobileNet-V2: 6.9Mb),
and meanwhile, its performance is competitive and even slightly better than
MobileNet-V2 in our _DFM-Net_ implementation (see Sec. 4.5).

During training, we embed TDB into _DFM-Net_ with random initialization, and
supervision signals are imposed at the end of the backbone to enforce saliency
feature learning from depth, as shown in Fig. 3. The coarse prediction result
obtained from TDB is formulated as:

$\displaystyle S_{d}=\textbf{F}_{p}^{d}(f_{d}^{5}),$ (11)

where $S_{d}$ means the coarse prediction from TDB, which is supervised by
ground truth (GT). $\textbf{F}_{p}^{d}(\cdot)$ denotes a prediction head
consisting of a $1\times 1$ convolution followed by Sigmoid activation, and
also $16\times$ bilinear up-sampling to recover the original input size. The
effectiveness of the proposed TDB will be validated in Sec. 4.5 .

###  3.4 Two-Stage Decoder

Unlike the popular U-Net unet  which adopts the hierarchy-by-hierarchy top-
down decoding strategy, we propose a simplified two-stage decoder, including
pre-fusion and full-fusion, to further improve efficiency. The pre-fusion aims
to reduce feature channels and hierarchies, by channel compression and
hierarchy grouping, denoted as “CP” and “G” in Fig. 3. Based on the outputs of
pre-fusion, the full-fusion further aggregates low-level and high-level
hierarchies to generate the final saliency map. It is also worth noting that
in our decoder, depth-wise separable convolutions are mainly adopted for large
numbers of input channels, instead of ordinary convolutions.

####  3.4.1 Pre-fusion Stage

We first use $3\times 3$ depth-wise separable convolution mobilenet  with
BatchNorm and ReLU activation, denoted as $\textbf{DSConv}_{3\times
3}(\cdot)$, to compress the encoder features ($f_{c}^{i},i=1,2...6$) to a
unified channel 16. Such an operation is denoted as “CP” in Fig. 3. Then we
use the well-known channel attention operator senet  $\textbf{F}_{CA}(\cdot)$
to enhance features by weighting different channels, denoted as “CA” in Fig.
3. The above process can be described as:

$\displaystyle cf_{c}^{i}=\textbf{F}_{CA}(\textbf{DSConv}_{3\times
3}(f_{c}^{i})),$ (12)

where $cf_{c}^{i}$ denotes the compressed and enhanced features. To reduce
feature hierarchies, inspired by BBSNet , we group 6 hierarchies into two,
namely the low-level hierarchy and the high-level hierarchy, as below:

$\displaystyle
cf_{c}^{low}=\sum_{i=1}^{3}\textbf{F}_{UP}^{2^{i-1}}(cf_{c}^{i}),~{}cf_{c}^{high}=\sum_{i=4}^{6}cf_{c}^{i},$
(13)

where $\textbf{F}_{UP}^{i}$ is bilinear up-sampling to $i$ times the original
size.

####  3.4.2 Full-fusion Stage

Since in the pre-fusion stage, the channel numbers and hierarchies are already
reduced, in the full-fusion stage, we directly concatenate high-level and low-
level hierarchies, and then feed the concatenation to a prediction head to
achieve the final full-resolution prediction map, denoted as:

$\displaystyle
S_{c}=\textbf{F}_{p}^{c}\bigg{(}\big{[}cf_{c}^{low},\textbf{F}_{UP}^{8}(cf_{c}^{high})\big{]}\bigg{)},$
(14)

where $S_{c}$ is the final saliency map, and $\textbf{F}_{p}^{c}(\cdot)$
indicates a prediction head consisting of two $3\times 3$ depth-wise separable
convolutions (followed by BatchNorm layers and ReLU activation), a $3\times 3$
convolution with Sigmoid activation, as well as a $2\times$ bilinear up-
sampling to recover the original input size.

###  3.5 Loss Function

The overall loss $\mathcal{L}$ is composed of the final loss $\mathcal{L}_{c}$
and deep supervision for the depth branch loss $\mathcal{L}_{d}$, formulated
as:

$\displaystyle\mathcal{L}=\mathcal{L}_{c}(S_{c},G)+\mathcal{L}_{d}(S_{d},G),$
(15)

where $G$ denotes the ground truth (GT). Similar to previous works JLDCF ;
BBSNet ; HDFNet ; D3Net , we use the standard cross-entropy loss for
$\mathcal{L}_{c}$ and $\mathcal{L}_{d}$.

##  4 Experiments and Results

Table 2: Quantitative benchmark results. The model size, $T_{CPU}$, $S_{GPU}$,
and $S_{GPU}^{*}$ are measured by Mb/ms/FPS/FPS respectively.
$\uparrow$/$\downarrow$ for a metric denotes that a larger/smaller value is
better. Our results are highlighted in bold. The scores/numbers better than
ours are underlined (efficient and non-efficient models are labeled
separately).

| Metric | DMRA | D3Net | JL-DCF | UCNet | SSF | S2MA | CoNet | cmMS | DANet | ATSA | DCF | DSA2F |  _DFM-Net_ ∗ | A2dele | PGAR | MSal | _DFM-Net_
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| ICCV | TNNLS | CVPR | CVPR | CVPR | CVPR | ECCV | ECCV | ECCV | ECCV | CVPR | CVPR | Ours | CVPR | ECCV | TPAMI | Ours
| 2019 | 2020 | 2020 | 2020 | 2020 | 2020 | 2020 | 2020 | 2020 | 2020 | 2021 | 2021 | - | 2020 | 2020 | 2021 | -
| DRMA  | D3Net  | JLDCF  | UCNet  | SSF  | S2MA  | CoNet  | cmMS  | DANet  | ATSA  | DCF  | DSA2F  | - | A2dele  | PGAR  | wu2021mobilesal  | -
|  Model size (Mb) $\downarrow$ | 228 | 530 | 520 | 119 | 125 | 330 | 167 | 430 | 102 | 123 | 436 | 415 | 93 | 57 | 62 | 26 | 8.5
$T_{CPU}$ (ms) $\downarrow$ | 840 | 450 | 7326 | 679 | 358 | 1019 | 333 | 814 | 782 | 1800 | 652 | 620 | 150 | 146 | 374 | 120 | 50
$S_{GPU}$ (FPS) $\uparrow$ | 15 | 50 | 12 | 48 | 28 | 25 | 59 | 19 | 47 | 32 | 21 | 19 | 64 | 184 | 48 | 65 | 70
|  $S_{GPU}^{*}$ (FPS) $\uparrow$ | 15 | 69 | 17 | 129 | 100 | 31 | 108 | 27 | 57 | 46 | 51 | 25 | 180 | 225 | 84 | 260 | 370
SIP | $S_{\alpha}\uparrow$ | .806 | .860 | .879 | .875 | .874 | .878 | .858 | .867 | .878 | .864 | .876 | .862 | .885 | .829 | .875 | .873 | .883
$F_{\beta}^{\rm m}\uparrow$ | .821 | .861 | .885 | .879 | .880 | .884 | .867 | .871 | .884 | .873 | .884 | .875 | .890 | .834 | .877 | .883 | .887
$E_{\xi}^{\rm m}\uparrow$ | .875 | .909 | .923 | .919 | .921 | .920 | .913 | .091 | .920 | .911 | .922 | .912 | .926 | .889 | .914 | .920 | .926
$\mathcal{M}\downarrow$ | .085 | .063 | .051 | .051 | .053 | .054 | .063 | .061 | .054 | .058 | .052 | .057 | .049 | .070 | .059 | .053 | .051
NLPR | $S_{\alpha}\uparrow$ | .899 | .912 | .925 | .920 | .914 | .915 | .908 | .915 | .915 | .907 | .924 | .919 | .926 | .890 | .918 | .920 | .923
$F_{\beta}^{\rm m}\uparrow$ | .879 | .897 | .916 | .903 | .896 | .902 | .887 | .896 | .903 | .876 | .912 | .906 | .912 | .875 | .898 | .908 | .908
$E_{\xi}^{\rm m}\uparrow$ | .947 | .953 | .962 | .956 | .953 | .950 | .945 | .949 | .953 | .945 | .963 | .952 | .961 | .937 | .948 | .961 | .957
$\mathcal{M}\downarrow$ | .031 | .025 | .022 | .025 | .026 | .030 | .031 | .027 | .029 | .028 | .022 | .024 | .024 | .031 | .028 | .025 | .026
NJU2K | $S_{\alpha}\uparrow$ | .886 | .900 | .903 | .897 | .899 | .894 | .895 | .900 | .891 | .901 | .904 | .895 | .912 | .868 | .906 | .905 | .906
$F_{\beta}^{\rm m}\uparrow$ | .886 | .900 | .903 | .895 | .896 | .889 | .892 | .897 | .880 | .893 | .906 | .897 | .913 | .872 | .905 | .905 | .910
$E_{\xi}^{\rm m}\uparrow$ | .927 | .950 | .944 | .936 | .935 | .930 | .937 | .936 | .932 | .921 | .950 | .936 | .950 | .914 | .940 | .942 | .947
$\mathcal{M}\downarrow$ | .051 | .041 | .043 | .043 | .043 | .053 | .047 | .044 | .048 | .040 | .040 | .044 | .039 | .052 | .045 | .041 | .042
RGBD135 | $S_{\alpha}\uparrow$ | .900 | .898 | .929 | .934 | .905 | .941 | .910 | .932 | .904 | .907 | .905 | .917 | .938 | .884 | .894 | .929 | .931
$F_{\beta}^{\rm m}\uparrow$ | .888 | .885 | .919 | .930 | .883 | .935 | .896 | .922 | .894 | .885 | .894 | .916 | .932 | .873 | .879 | .924 | .922
$E_{\xi}^{\rm m}\uparrow$ | .943 | .946 | .968 | .976 | .941 | .973 | .945 | .970 | .957 | .952 | .951 | .954 | .973 | .920 | .929 | .970 | .972
$\mathcal{M}\downarrow$ | .030 | .031 | .022 | .019 | .025 | .021 | .029 | .020 | .029 | .024 | .024 | .023 | .019 | .030 | .032 | .021 | .021
LFSD | $S_{\alpha}\uparrow$ | .839 | .825 | .862 | .864 | .859 | .837 | .862 | .849 | .845 | .865 | .842 | .883 | .870 | .834 | .833 | .847 | .865
$F_{\beta}^{\rm m}\uparrow$ | .852 | .810 | .866 | .864 | .867 | .835 | .859 | .869 | .846 | .862 | .842 | .889 | .866 | .832 | .831 | .841 | .864
$E_{\xi}^{\rm m}\uparrow$ | .893 | .862 | .901 | .905 | .900 | .873 | .907 | .896 | .886 | .905 | .883 | .924 | .903 | .874 | .893 | .888 | .903
$\mathcal{M}\downarrow$ | .083 | .095 | .071 | .066 | .066 | .094 | .071 | .074 | .083 | .064 | .075 | .055 | .068 | .077 | .093 | .078 | .072
STERE | $S_{\alpha}\uparrow$ | .835 | .899 | .905 | .903 | .893 | .890 | .908 | .895 | .892 | .897 | .902 | .898 | .908 | .885 | .903 | .903 | .898
$F_{\beta}^{\rm m}\uparrow$ | .847 | .891 | .901 | .899 | .890 | .882 | .904 | .891 | .881 | .884 | .901 | .900 | .904 | .885 | .893 | .895 | .893
$E_{\xi}^{\rm m}\uparrow$ | .911 | .938 | .946 | .944 | .936 | .932 | .948 | .937 | .930 | .921 | .945 | .942 | .948 | .935 | .936 | .940 | .941
$\mathcal{M}\downarrow$ | .066 | .046 | .042 | .039 | .044 | .051 | .040 | .042 | .048 | .039 | .039 | .039 | .040 | .043 | .044 | .041 | .045
ReDWeb-S | $S_{\alpha}\uparrow$ | .592 | .689 | .734 | .713 | .595 | .711 | .696 | .656 | .693 | .679 | .709 | N/A | .715 | .641 | .656 | .652 | .705
$F_{\beta}^{\rm m}\uparrow$ | .579 | .673 | .727 | .710 | .558 | .696 | .693 | .632 | .679 | .673 | .714 | N/A | .700 | .603 | .632 | .610 | .686
$E_{\xi}^{\rm m}\uparrow$ | .721 | .768 | .805 | .794 | .710 | .781 | .782 | .749 | .778 | .758 | .789 | N/A | .788 | .672 | .749 | .722 | .783
$\mathcal{M}\downarrow$ | .188 | .149 | .128 | .130 | .189 | .139 | .147 | .161 | .142 | .155 | .135 | N/A | .132 | .160 | .161 | .162 | .136

Note the speed test results here differ from those in our preliminary version
zhang2021depth2 , because we later switched from Windows 10 to Ubuntu 16.04
and re-evaluated all models. We found the latter was more compatible with
model implementations, allowing us to better standardize the test environment.
We also found that generally the inference speed on Ubuntu was faster.

###  4.1 Datasets and Metrics

We conduct experiments on nine public datasets, including NJU2K NJU2K  (1,985
samples), NLPR NLPR  (1,000 samples), STERE STERE  (1,000 samples), RGBD135
RGBD135  (135 samples), LFSD LFSD  (100 samples), SIP D3Net  (929 samples),
DUT-RGBD DRMA  (1,200 samples), RedWeb-S S2MA  (1,000 samples, only the
testing set) as well as COME zhang2021rgb  (15,625 samples).

Four commonly used metrics are used for evaluation, including S-measure
($S_{\alpha}$) smeasure , maximum F-measure ($F_{\beta}^{\rm m}$) fmeasure ,
maximum E-measure ($E_{\xi}^{\rm m}$) emeasure ; Fan2018Enhanced , and mean
absolute error (MAE, $\mathcal{M}$) mae . For these four metrics, higher
$S_{\alpha}$, $F_{\beta}^{\rm m}$, $E_{\xi}^{\rm m}$, and lower $\mathcal{M}$
indicate better performance. For efficiency analysis, we report the model size
(Mb, Mega-bytes), inference time $T_{CPU}$ (ms, millisecond) on CPU, and
one-/maximum-batch inference speed $S_{GPU}/S_{GPU}^{*}$ (FPS, frame-per-
second) on GPU. The maximum-batch setting is previously adopted by Wu et al.
wu2021mobilesal  because it would better reflect the efficiency superiority of
light-weight models444This is because depth-wise separable convolutions-based
lightweight models often reach a bottleneck in read/write rate first prior to
reaching maximal GPU utilization Rooflinemodel , so their acceleration on GPU
and efficiency advantages cannot be fully reflected in the small batch
setting.. In the following, we illustrate how $S_{GPU}$ and $S_{GPU}^{*}$ are
measured. Let the inference speed $S$ be defined as:

$\displaystyle S=\frac{N\times B}{T},$ (16)

where $N$ means the number of total inference, $B$ denotes the batch size
used, and $T$ is the total consumed time (in seconds). $N$ is empirically set
to 100 in our evaluation, which is large enough for time statistics. When $B$
equals to 1, $S$ becomes $S_{GPU}$, and when $B$ increases to reach the
maximal (100%) GPU utilization, $S$ then becomes $S_{GPU}^{*}$.

###  4.2 Implementation Details

Experiments are conducted on a workstation with Intel Core i7-8700 CPU, NVIDIA
GTX 1080 Ti GPU. We implement _DFM-Net_ by Pytorch pytorch , and RGB and depth
images are both resized to $256\times 256$ for input. For training, in order
to generalize the network on limited training samples, following BBSNet , we
apply various data augmentation techniques _i.e.,_ random
translation/cropping, horizontal flipping, color enhancement and so on. We
train _DFM-Net_ for 300 epochs on a single 1080 Ti GPU, taking about 4 hours.
The initial learning rate is set as 1e-4 for Adam optimizer kingma2014adam ,
and the batch size is 10. The poly learning rate policy is used, where the
power is set to 0.9.

Figure 7: Qualitative comparisons of _DFM-Net_ and _DFM-Net_ * with SOTA RGB-D
SOD models. Efficient and non-efficient models are shown separately.

###  4.3 Comparisons to SOTAs

Since we cannot expect that an extremely lightweight model can always
outperform existing non-efficient models which are much larger, we have also
extended _DFM-Net_ to obtain a larger and more powerful model called _DFM-Net_
*, by replacing MobileNet-v2 used in the RGB branch with ResNet-34 ResNet .
Then to align the features from TDB to those from the RGB branch, slight
modifications are made by adjusting $c$, $t$, and $n$, finally yielding TDB*
of $\sim$2.7Mb model size.

Following classic RGB-D SOD benchmark setting JLDCF ; UCNet ; D3Net , we use
1,500 samples from NJU2K and 700 samples from NLPR for training, and test on
NJU2K, NLPR, STERE, RGBD135, LFSD, SIP as well as the newly released dataset
RedWeb-S. Results of _DFM-Net_ and _DFM-Net_ *, compared to 15 SOTA models
including DRMA DRMA , D3Net D3Net , JL-DCF JLDCF , UCNet UCNet , SSF SSF ,
S2MA S2MA , CoNet CoNet , cmMS cmMS , DANet DANet , ATSA ATSA , DCF DCF ,
DSA2F DSA2F , A2dele A2dele , PGAR PGAR , and MSal wu2021mobilesal  can be
found in Tab. 2, where the saliency maps of other models are from their
released results if available, otherwise they are computed by their released
code. Note that the parameter file of DSA2F DSA2F  and its results on RedWeb-S
are not available, so the associated results are denoted as “N/A”.

Table 3: Quantitative benchmark results on DUT-RGBD. Compared methods are those whose results on this dataset are publicly available, and include DMRA DRMA , S2MA S2MA , CoNet CoNet , DANet DANet , ATSA ATSA , DCF DCF , DSA2F DSA2F , A2dele A2dele , PGAR PGAR , and MSal wu2021mobilesal . Ours* and Ours refer to the proposed _DFM-Net_ * and _DFM-Net_ , respectively. Metric | DRMA  | S2MA  | CoNet  | DANet  | ATSA  | DCF  | DSA2F  | Ours* | A2dele  | PGAR  | wu2021mobilesal  | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---
$S_{\alpha}\uparrow$ | .888 | .905 | .919 | .889 | .883 | .925 | .921 | .924 | .885 | .919 | .895 | .913
$F_{\beta}^{\rm m}\uparrow$ | .897 | .911 | .926 | .894 | .891 | .930 | .932 | .931 | .891 | .923 | .918 | .917
$E_{\xi}^{\rm m}\uparrow$ | .932 | .943 | .955 | .930 | .928 | .956 | .956 | .956 | .928 | .948 | .949 | .948
$\mathcal{M}\downarrow$ | .047 | .059 | .032 | .046 | .050 | .030 | .030 | .031 | .043 | .034 | .044 | .037

As shown in Tab. 2, _DFM-Net_ surpasses existing efficient models MSal
wu2021mobilesal , A2dele A2dele , and PGAR PGAR  on detection accuracy and
model size. It runs at 50ms on CPU, which is the fastest among all the
contenders, with only $\sim$8.5Mb model size (3.1$\times$ smaller than the
concurrent work MSal wu2021mobilesal ). Regarding $S_{GPU}$, _DFM-Net_ exceeds
the others except for A2dele A2dele  whose model size is larger than ours.
This is because _DFM-Net_ is mainly built upon the depth-wise separable
convolutions, which cannot be fully accelerated on GPU in the small batch
setting. A similar issue also exists for other efficient models like
MobileNet-v2 and MSal. However, regarding $S_{GPU}^{*}$, we can see that _DFM-
Net_ is significantly faster than all the other models including MSal.
Comparing to MSal, ours is $1.4\times$ faster.

Table 4: Quantitative benchmark results on the recent COME. Compared methods are those whose results on this dataset are publicly available, and include JL-DCF JLDCF , UCNetUCNet , SSFSSF , S2MA S2MA , CoNet CoNet , DANet DANet , ATSA ATSA , DCF DCF , A2dele A2dele , and PGAR PGAR . Ours* and Ours refer to the proposed _DFM-Net_ * and _DFM-Net_ , respectively. | Metric | JLDCF  | UCNet  | SSF  | S2MA  | CoNet  | DANet  | ATSA  | DCF  | Ours* | A2dele  | PGAR  | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---|---
COME-E | $S_{\alpha}\uparrow$ | .895 | .891 | .852 | .876 | .837 | .884 | .807 | .892 | .894 | .831 | .887 | .887
$F_{\beta}^{\rm m}\uparrow$ | .891 | .890 | .847 | .865 | .838 | .877 | .812 | .892 | .895 | .838 | .888 | .888
$E_{\xi}^{\rm m}\uparrow$ | .932 | .930 | .900 | .912 | .878 | .921 | .852 | .933 | .926 | .889 | .925 | .925
$\mathcal{M}\downarrow$ | .041 | .042 | .060 | .059 | .073 | .048 | .085 | .038 | .045 | .060 | .050 | .050
COME-H | $S_{\alpha}\uparrow$ | .850 | .844 | .802 | .832 | .794 | .833 | .768 | .842 | .836 | .785 | .834 | .830
$F_{\beta}^{\rm m}\uparrow$ | .848 | .844 | .801 | .822 | .802 | .832 | .782 | .847 | .838 | .798 | .836 | .832
$E_{\xi}^{\rm m}\uparrow$ | .888 | .885 | .855 | .868 | .834 | .876 | .811 | .886 | .874 | .845 | .873 | .872
$\mathcal{M}\downarrow$ | .071 | .072 | .094 | .091 | .105 | .080 | .116 | .070 | .081 | .092 | .084 | .085

On the other hand, one can see that _DFM-Net_ * achieves SOTA performance when
compared to existing non-efficient models, with absolutely the fastest
$T_{CPU}$, $S_{GPU}$, $S_{GPU}^{*}$, and the smallest model size. As early
mentioned, Fig. 2 visualizes all models’ average performance across six out of
the seven datasets, demonstrating the high efficiency and effectiveness of the
proposed models.

Figure 8: Visualized examples of setting #3 (w/o DQW) and #4 (with DQW) for
good (a) and bad (b) depth quality cases. $\overline{\alpha}$ denotes the
average of $\alpha_{1}\sim\alpha_{5}$, which can be deemed as an indicator for
depth quality.

To intuitively compare with SOTA methods, visual comparisons are further shown
in Fig. 7, where our results are closer to GT. Note in Fig. 7, efficient and
non-efficient models are compared separately.

Table 5: Ablation analyses for DQFM, where the effectiveness of DQW and DHA is validated. The best results are highlighted in bold. # | DQW | DHA | Size | SIP D3Net  | NLPR NLPR  | NJU2K NJU2K  | RGBD135 RGBD135  | LFSD LFSD  | STERE STERE
---|---|---|---|---|---|---|---|---|---
(Mb) | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$
1 |  |  | 8.409 | .849 | .842 | .897 | .070 | .907 | .884 | .946 | .032 | .890 | .887 | .931 | .052 | .929 | .915 | .968 | .025 | .847 | .841 | .883 | .084 | .880 | .872 | .931 | .054
2 | ✓ |  | 8.416 | .878 | .884 | .922 | .054 | .918 | .902 | .957 | .027 | .898 | .898 | .938 | .047 | .932 | .922 | .971 | .022 | .857 | .856 | .896 | .078 | .894 | .886 | .939 | .047
3 |  | ✓ | 8.451 | .861 | .861 | .908 | .063 | .918 | .901 | .953 | .028 | .899 | .896 | .940 | .046 | .921 | .911 | .966 | .024 | .857 | .855 | .895 | .076 | .897 | .891 | .942 | .045
4 | ✓ | ✓ | 8.459 | .883 | .887 | .926 | .051 | .923 | .908 | .957 | .026 | .906 | .910 | .947 | .042 | .931 | .922 | .972 | .021 | .865 | .864 | .903 | .072 | .898 | .893 | .941 | .045

###  4.4 Additional Evaluation on DUT-RGBD and COME

For more comprehensive evaluation, we test our methods on two extra datasets,
namely DUT-RGBD DRMA  and COME zhang2021rgb . To evaluate on DUT-RGBD, we
follow the benchmark setting as previous works CoNet ; PGAR ; DANet ; ATSA ;
A2dele , where 800 samples from DUT-RGBD are also added for training besides
the training data from NJU2K and NLPR. Fairly, all the compared models are
trained under this setting and publicly provided by the authors. Testing is
conducted on 400 samples remained in DUT-RGBD, and the results are shown in
Tab. 3. It can be seen that without bells and whistles, _DFM-Net_ and _DFM-
Net_ * achieve better balance between accuracy and efficiency, and are
4.7/7.2$\times$ smaller than the most accurate efficient/non-efficient ones
(DSA2F /PGAR ) with little accuracy degradation.

Recently, Zhang et al.  zhang2021rgb  introduces a new RGB-D SOD benchmark
called COME, which officially splits out 8,025 samples for training and the
rest samples are divided into COME-E and COME-H for normal and hard cases.
COME-E and COME-H include 4,600 and 3,000 samples, respectively. We re-train
our methods on its training set, and the compared models are the re-trained
off-the-shelf versions provided by Zhang et al.  in their benchmark
zhang2021rgb . As shown in Tab. 4, the competitive performance of _DFM-Net_
and _DFM-Net_ * again illustrates their adaptability onto this brand new
benchmark dataset.

###  4.5 Ablation Studies

We conduct thorough ablation studies on six classic datasets including NJU2K,
NLPR, STERE, RGBD135, LFSD, and SIP, by removing or replacing components from
the full implementation of _DFM-Net_.

Figure 9: Saliency maps with and without DQW, and the holistic attention maps
$\beta_{1}\sim\beta_{5}$ achieved by DHA. $\overline{\alpha}$ achieved by DQW
is also shown. Note the somewhat inaccurate holistic attention maps generated
by DHA, which are derived from the low-quality depth maps.

####  4.5.1 Effectiveness of DQFM

DQFM consists of two key components, namely DQW and DHA. Tab. 5 shows
different configurations by ablating DQW/DHA. In detail, #1 denotes a baseline
model which has removed both DQW and DHA from _DFM-Net_. Configuration #2 and
#3 mean having either one component, while #4 means the full model of _DFM-
Net_. Basically, from Tab. 5 one can see that incorporating either DQW and DHA
into the baseline model #1 leads to consistent improvement on almost all
datasets. Besides, comparing #2/#3 to #4, we see that employing both DQW and
DHA can further enhance the results, demonstrating the synergy effect between
DQW and DHA. The underlying reason could be that, although DHA is able to
enhance potential target regions in the depth, it is unavoidable to make some
mistakes (_e.g._ , highlight wrong regions) especially in low depth quality
cases. Luckily, DQW somewhat relieves such a side-effect because in the
meantime it assigns low global weights to depth features. In all, these two
components can work cooperatively to improve the robustness of the network, as
we mentioned in Sec. 3.2. Last but not least, from the model sizes of
different configurations shown in Tab. 5, the introduction of DQW and DHA
brings merely $\sim$0.6% extra parameters compared to the baseline #1. This
rightly meets our goal and implies that they could potentially become
universal components for lightweight models in the future.

Table 6: Validation of $V_{BA}$ in DQW, including the proposed operation (Pro.), original Dice-like computation (Dice.), straightforward addition (Add.), and straightforward multiplication (Mul.). Note we have added the last “Average” group, which shows the average scores over those six benchmark datasets, in order for comparison convenience. The best results are highlighted in bold. # | $V_{BA}$ | SIP D3Net  | NLPR NLPR  | NJU2K NJU2K  | RGBD135 RGBD135  | LFSD LFSD  | STERE STERE  | Average
---|---|---|---|---|---|---|---|---
$S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$
4 | Pro. | .883 | .887 | .926 | .051 | .923 | .908 | .957 | .026 | .906 | .910 | .947 | .042 | .931 | .922 | .972 | .021 | .865 | .864 | .903 | .072 | .898 | .893 | .941 | .045 | .901 | .897 | .941 | .043
5 | Dice. | .877 | .881 | .922 | .053 | .921 | .907 | .957 | .026 | .904 | .904 | .942 | .044 | .926 | .917 | .969 | .023 | .866 | .861 | .901 | .070 | .897 | .891 | .942 | .045 | .898 | .893 | .939 | .044
6 | Add. | .870 | .873 | .916 | .057 | .921 | .909 | .959 | .026 | .904 | .904 | .944 | .044 | .922 | .911 | .968 | .025 | .859 | .856 | .895 | .074 | .900 | .893 | .942 | .044 | .896 | .891 | .937 | .045
7 | Mul. | .881 | .885 | .927 | .051 | .920 | .907 | .956 | .027 | .902 | .900 | .943 | .044 | .933 | .924 | .973 | .021 | .862 | .859 | .898 | .072 | .902 | .896 | .946 | .042 | .900 | .895 | .940 | .043

Figure 10: Distribution of the inferred $\overline{\alpha}$. (a) denotes the
distribution corresponding to the proposed operation for $V_{BA}$. (b)-(d)
denote different variants applied for $V_{BA}$, namely the original Dice
coefficient-like computation, straightforward addition, and straightforward
multiplication. Note the colors of the distributions on STERE and SIP are
emphasized for better comparison.

In Fig. 8, we show visual examples of setting #3 (namely without DQW) and #4.
We also visualize the magnitudes of $\alpha$, namely the mean value
$\overline{\alpha}$ of $\alpha_{1}\sim\alpha_{5}$. From Fig. 8 (a) and (b), we
can see that incorporating DQW does help improve detection accuracy, and
practically, DQW is able to function as expected, namely rendering the good
quality cases with higher weights (Fig. 8 (a)), and vice versa (Fig. 8 (b)).
In the first (row 1) good quality case of (a), it is difficult to distinguish
between the shadow and the man’s legs in the RGB view, but this can be done
easily in the depth view. Incorporating DQW to give more emphasis to depth
features, therefore, can help better separate the entire human body from the
shadow. In the first (also row 1) bad quality case of (b), although in the
depth view the cup handle is much blurry, the impact of misleading depth has
been alleviated, and the whole object still can be detected out accurately.

We also validate the synergy effect of combining DQW and DHA. Since DHA cannot
always highlight correct regions, what if DHA gives wrong response? We display
in Fig. 9 some cases where DHA generates misleading maps and show such effects
can be alleviated by DQW (with lower $\overline{\alpha}$ achieved). This
proves that DHA and DQW could cooperate with each other and enhance detection
robustness. That is, incorporating DQW into DHA (comparing between “with DQW”
and “w/o DQW” columns in Fig. 9) consistently improves the detection accuracy.

####  4.5.2 Delving into DQW

The above only give intuitive examples, but _does DQW really function as we
expected?_ To statistically validate the role of DQW, we plot the distribution
curves of the inferred $\overline{\alpha}$ on different datasets. As shown in
Fig. 10 (a), we can see that DQW can distinguish between SIP and STERE well by
the predicted $\overline{\alpha}$, as expected by our BA motivation in Fig. 1.
In addition to this, we also validate other possible variants of Eq. 4 in DQW,
including straightforward addition, multiplication, as well as the original
Dice coefficient-like computation (with square in the denominator). These
variants are denoted as #6, #7, #5, respectively. Tab. 6 compares the
performance of these variants and their $\overline{\alpha}$ are exhibited in
Fig. 10 (b)-(d).

From Tab. 6, one can see according to the average evaluation scores that the
proposed operation performs better than the other variants, including the
original Dice coefficient-like computation. We find that this phenomenon
should be attributed to the fact that the proposed operation results in a more
reasonable distribution of $\overline{\alpha}$, and can better indicate the
low-level alignment between RGB and depth. It is also interesting to note
that, #7 (“Mul.”) practically performs better than the remaining two variants,
and is very close to the performance of #4. The underlying reasons may be two-
fold: 1) Though without normalization by the denominator, using merely feature
multiplication still can somewhat excavate and capture the common low-level
edge features. 2) The obtained distribution of $\overline{\alpha}$ from #7,
namely Fig. 10 (d), also seems to be more reasonable and can better tell STERE
and SIP apart.

Table 7: Ablation analyses for times of recalibration process $\textbf{F}_{rec}$ in DHA. Zero means no recalibration is conducted. # | $\textbf{F}_{rec}$ | SIP D3Net  | NLPR NLPR  | NJU2K NJU2K  | RGBD135 RGBD135  | LFSD LFSD  | STERE STERE
---|---|---|---|---|---|---|---
$S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$
8 | 0 | .877 | .881 | .919 | .054 | .920 | .904 | .953 | .027 | .904 | .904 | .943 | .043 | .934 | .924 | .973 | .021 | .855 | .855 | .894 | .076 | .897 | .890 | .940 | .045
9 | 1 | .872 | .873 | .918 | .056 | .920 | .904 | .957 | .027 | .903 | .902 | .943 | .043 | .929 | .922 | .973 | .023 | .864 | .865 | .902 | .072 | .900 | .893 | .942 | .044
4 | 2 | .883 | .887 | .926 | .051 | .923 | .908 | .957 | .026 | .906 | .910 | .947 | .042 | .931 | .922 | .972 | .021 | .865 | .864 | .903 | .072 | .898 | .893 | .941 | .045
10 | 3 | .864 | .868 | .916 | .059 | .921 | .907 | .957 | .026 | .898 | .897 | .941 | .042 | .917 | .905 | .963 | .025 | .858 | .858 | .896 | .077 | .905 | .897 | .946 | .042

Figure 11: Saliency maps with and without DHA, and the holistic attention maps
$\beta_{1}\sim\beta_{5}$ achieved by DHA.

However, we notice that all the variants, including the proposed operation,
apparently fail on RGBD135, because the obtained $\overline{\alpha}$
distribution for RGBD135 is on the left of STERE. This means that the
individual model thinks RGBD135 presents worse depth quality than STERE.
However, according to the observation of human eyes and also previous works
SMAC ; D3Net ; JLDCF , the general depth quality of RGBD135 is not that bad
and should be better than that of STERE. This failure case of DQW on RGBD135
may also explain why DQW has the least improvement for this dataset when
comparing #2 with #1 in Tab. 5. We attribute this failure case to the
unsupervised learning mechanism of DQW, which means that without explicit
guidance, the rules implicitly learned from data may not be that accurate and
cannot generalize well to some unseen scenarios. Interestingly, we observe
improved performance on RGBD135 for _DFM-Net_ if we manually modify the
obtained $\alpha$ further to larger values during inference.

Finally, we notice that our inferred depth quality results of datasets, which
are indicated by $\overline{\alpha}$, are partially consistent with the
analyses in a recent work LSMA . In LSMA , the authors point out that the
descending order for depth map quality of existing datasets is: SIP, NLPR,
RGBD135, NJU2K, LFSD, and STERE. From Fig. 10 (a), we can derive that the
ranking statistic from our _DFM-Net_ is: SIP, NJU2K, LFSD, STERE, NLPR, and
RGBD135, which coincides with the former if NLPR and RGBD135 are ignored. We
believe such a coincidence, from the other aspect, validates the feasibility
and effectiveness of our proposed model.

Table 8: DQFM gating strategy: using identical (only one) $\alpha_{i}$ and $\beta_{i}$ _vs._ using multiple (five different) $\alpha_{i}$ and $\beta_{i}$.  # | Strategy | SIP D3Net  | NLPR NLPR  | NJU2K NJU2K  | RGBD135 RGBD135  | LFSD LFSD  | STERE STERE
---|---|---|---|---|---|---|---
$S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$
11 | Identical | .880 | .884 | .924 | .053 | .922 | .907 | .958 | .026 | .901 | .901 | .944 | .044 | .924 | .911 | .961 | .024 | .859 | .857 | .899 | .073 | .898 | .892 | .942 | .045
4 | Multiple | .883 | .887 | .926 | .051 | .923 | .908 | .957 | .026 | .906 | .910 | .947 | .042 | .931 | .922 | .972 | .021 | .865 | .864 | .903 | .072 | .898 | .893 | .941 | .045
Table 9: Performance of the proposed TDB (tailored depth backbone) against MobileNet-V2. Details are in Sec. 4.5. # | Depth | Size | SIP D3Net  | NLPR NLPR  | NJU2K NJU2K  | RGBD135 RGBD135  | LFSD LFSD  | STERE STERE
---|---|---|---|---|---|---|---|---
backbone | (Mb) | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$
12 | MoblieNet-V2 | 6.9 | .879 | .886 | .923 | .054 | .919 | .904 | .954 | .027 | .906 | .908 | .948 | .042 | .930 | .922 | .969 | .022 | .864 | .864 | .902 | .070 | .893 | .889 | .942 | .046
4 | Tailored | 0.9 | .883 | .887 | .926 | .051 | .923 | .908 | .957 | .026 | .906 | .910 | .947 | .042 | .931 | .922 | .972 | .021 | .865 | .864 | .903 | .072 | .898 | .893 | .941 | .045

####  4.5.3 Recalibration in DHA

As described in Sec. 3.2.2, in DHA, we utilize operation $\textbf{F}_{rec}$ to
recalibrate the coarse information from high-level depth features. To validate
the necessity of using $\textbf{F}_{rec}$, we experiment with different times
(from 0 to 3) of using $\textbf{F}_{rec}$. These variants are denoted as #5,
#6, #4, and #7 in Tab. 7. Note that #4 corresponds to the default
implementation of _DFM-Net_. From Tab. 7, we can see that #4 (recalibrate
twice) achieves the overall best performance. The underlying reason should be
that, appropriate usage of $\textbf{F}_{rec}$ can expand the coverage areas of
attention maps to make conservative filtering for object edges as well as some
inaccurately located objects, but too large receptive field leads to over-
dilated attention regions that are less informative. This is the reason why
when the times increase to 3, the performance starts to degenerate on most
datasets, except on STERE whose depth quality is generally low, which easily
leads to inaccurate attention location. In Fig. 11 we show more visual
examples by using/removing DHA modules to intuitively show its function.

####  4.5.4 DQFM Gating Strategy

As we mentioned in Sec. 3.2, we adopt a multi-variable strategy for
$\alpha_{i}$ and $\beta_{i}$. To validate this strategy, we compare it to the
single-variable strategy, namely using identical (only one) $\alpha_{i}$ and
$\beta_{i}$. Tab. 8 shows the results, from which it can be seen that our
proposed multi-variable strategy is better, because it somewhat increases the
network flexibility by rendering different hierarchies with different quality-
inspired weights and attention maps.

####  4.5.5 Tailored Depth Backbone

The effectiveness of TDB is validated by comparing it to MobileNet-V2. We
implement a configuration #9 by switching TDB directly to MobileNet-V2, while
maintaining all other settings unchanged. Evaluation results are shown in Tab.
9. We can see that our tailored depth backbone is much lighter (0.9Mb vs.
MobileNet-V2’s 6.9Mb) but able to achieve slightly better accuracy when
compared to MobileNet-V2 for _DFM-Net_. Although this phenomenon seems
counter-intuitive, it is not unique. As a recent large benchmark of model
robustness tang2021robustart  shows, for some lightweight model families such
as MobileNet-V2, MobileNet-V3, and ShuffleNet-V2, larger versions are not
necessarily more robust than the smaller ones. This sheds light on the
possibility of utilizing a lighter backbone to process depth data for
efficiency purpose.

Figure 12: Illustration of the common characteristic of optical flow and depth
maps, for video and RGB-D SOD.

###  4.6 On Efficient Video SOD

Inspired by the encouraging performance of _DFM-Net_ on RGB-D SOD, in this
section, we are motivated to adapt _DFM-Net_ to the VSOD task to answer the
following question: _Does DQFM still work on optical flow (OF) obtained from
video data, and can DFM-Net contribute to efficient VSOD?_ Note we believe
that efficient VSOD is also a promising direction since automatic and
efficient video analyses on mobile devices is very useful.

Table 10: Quantitative benchmark results on the VSOD task. The model size is measured by Mb. $\uparrow$/$\downarrow$ for a metric denotes that a larger/smaller value is better. Our results are highlighted in bold. The scores/numbers better than ours are underlined (efficient and non-efficient models are labeled separately). |  Metric | DLVS | MBN | FGRN | SCNN | SCOM | RSE | SRP | MESO | LTSI | SSAV | RCR | SPD | FSNet |  _DFM-Net_ * | PCSA | Reuse | _DFM-Net_
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| TIP | ECCV | CVPR | TCSVT | TIP | TCSVT | TIP | TMM | TIP | CVPR | ICCV | TMM | ICCV | Ours | AAAI | CVPR | Ours
| 2017 | 2018 | 2018 | 2018 | 2018 | 2019 | 2019 | 2019 | 2019 | 2019 | 2019 | 2020 | 2021 | - | 2020 | 2021 | -
| Wang2018VideoSO  | li2018unsupervised  | li2018flow  | tang2018weakly  | chen2018scom  | cong2019video  | cong2019video  | xu2019video  | chen2019improved  | fan2019shifting  | Yan2019SemiSupervisedVS  | li2019accurate  | ji2021FSNet  | - | Gu2020PyramidCS  | Park2021Learning  | -
| Model size (Mb) | 347.3 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 132 | 247.7 | 205.5 | 754 | 391.4 | 93 | 19.8 | 27.5 | 8.5
DAVIS | $S_{\alpha}\uparrow$ | .802 | .887 | .838 | .761 | .814 | .748 | .662 | .718 | .876 | .893 | .886 | .783 | .920 | .915 | .899 | .897 | .898
$F_{\beta}^{\rm m}\uparrow$ | .721 | .862 | .783 | .679 | .746 | .698 | .660 | .660 | .850 | .861 | .848 | .763 | .907 | .898 | .880 | .884 | .869
$E_{\xi}^{\rm m}\uparrow$ | .895 | .966 | .917 | .843 | .874 | .878 | .843 | .853 | .957 | .948 | .947 | .892 | .970 | .971 | .965 | .955 | .959
$\mathcal{M}\downarrow$ | .055 | .031 | .043 | .077 | .055 | .063 | .070 | .070 | .034 | .028 | .027 | .061 | .020 | .019 | .021 | .018 | .025
FBMS | $S_{\alpha}\uparrow$ | .794 | .857 | .809 | .794 | .794 | .670 | .648 | .635 | .805 | .879 | .872 | .691 | .890 | .899 | .853 | .744 | .889
$F_{\beta}^{\rm m}\uparrow$ | .759 | .816 | .767 | .762 | .797 | .652 | .671 | .618 | .799 | .865 | .859 | .686 | .888 | .895 | .810 | .683 | .880
$E_{\xi}^{\rm m}\uparrow$ | .861 | .892 | .863 | .865 | .873 | .790 | .773 | .767 | .871 | .926 | .905 | .804 | .935 | .943 | .916 | .745 | .937
$\mathcal{M}\downarrow$ | .091 | .047 | .088 | .095 | .079 | .128 | .134 | .134 | .087 | .040 | .053 | .125 | .041 | .031 | .049 | .074 | .034
MCL | $S_{\alpha}\uparrow$ | .682 | .755 | .709 | .730 | .569 | .682 | .689 | .477 | .768 | .819 | .820 | .685 | .864 | .864 | .754 | .754 | .834
$F_{\beta}^{\rm m}\uparrow$ | .551 | .698 | .625 | .628 | .422 | .576 | .646 | .144 | .667 | .773 | .742 | .601 | .821 | .810 | .683 | .679 | .768
$E_{\xi}^{\rm m}\uparrow$ | .810 | .858 | .817 | .828 | .704 | .657 | .812 | .730 | .872 | .889 | .895 | .794 | .924 | .935 | .838 | .836 | .906
$\mathcal{M}\downarrow$ | .060 | .119 | .044 | .054 | .204 | .073 | .058 | .102 | .044 | .026 | .028 | .069 | .023 | .020 | .038 | .037 | .027
DAVSOD-E | $S_{\alpha}\uparrow$ | .664 | .646 | .701 | .680 | .603 | .577 | .575 | .549 | .695 | .755 | .741 | .626 | .773 | .790 | .735 | .765 | .774
$F_{\beta}^{\rm m}\uparrow$ | .541 | .506 | .589 | .541 | .473 | .417 | .453 | .360 | .585 | .659 | .653 | .500 | .685 | .700 | .647 | .699 | .684
$E_{\xi}^{\rm m}\uparrow$ | .737 | .694 | .765 | .745 | .669 | .663 | .655 | .673 | .769 | .806 | .803 | .685 | .825 | .838 | .794 | .832 | .830
$\mathcal{M}\downarrow$ | .129 | .109 | .095 | .127 | .219 | .146 | .146 | .159 | .106 | .084 | .087 | .138 | .072 | .066 | .087 | .051 | .072

We first analyze the feasibility of this idea. As shown in Fig. 12, the
optical flow map plays a complementary role in the VSOD task as the depth map
does for RGB-D SOD, _i.e.,_ a depth map provides extra spatial information and
an optical flow map provides motion cues. Meanwhile, the optical flow map may
also present misleading appearances _, e.g.,_ broken structures, or erroneous
motion. This similar characteristic between depth and optical flow makes it
possible to transfer _DFM-Net_ to VSOD. In addition, we find that the widely
used _pretraining strategy_ in the VSOD task is less effective for lightweight
models, so we propose a simple but effective _joint training strategy_ , to
help better adapt to VSOD.

####  4.6.1 Joint Training Strategy

Most existing optical flow-based VSOD models ji2021FSNet ; fu2021siamese ;
Gu2020PyramidCS  adopt a pretraining strategy by utilizing extra RGB data for
the RGB branch to enhance generalisation, owing to the relatively small size
of VSOD datasets. However, almost no work in this field considers if this
strategy is suitable for lightweight models. We empirically find that it is
less effective for our lightweight model, as will be shown in Tab. 13. The
failure of pretraining for lightweight models has also been reported by
cheng2021highly ; poudel2019fast . The underlying reason is that, compared to
normal models, lightweight models are easier to converge and also to forget
prior information, making such pretraining less influential cheng2021highly .
To alleviate this problem and better leverage extra RGB data, we introduce a
strategy called _joint training_. Specifically, we match each RGB image with
an all-black image to form a pseudo RGB-flow pair. An all-black image can be
seen as an optical flow map for the still case where no motion takes place.
Finally, we train the model on the still dataset and VSOD datasets jointly,
where the two types of data are mixed up. This proposed strategy improves
model generalizability significantly according to Sec. 4.6.4.

Figure 13: Qualitative comparisons of _DFM-Net_ and _DFM-Net_ * with SOTA VSOD
models. Efficient and non-efficient models are compared separately.

####  4.6.2 Experimental details

Following previous works ji2021FSNet ; fu2021siamese , we use VSOD datsets
DAVIS (30 clips) perazzi2016benchmark , FBMS (29 clips) ochs2013segmentation ,
and also still RGB SOD dataset DUTS wang2017learning  (10,553 images) for
training. Tests are conducted on the validation set (20 clips) of DAVIS16, the
test set of FBMS (30 clips), the whole MCL Kim2015SpatiotemporalSD  (9 clips)
as well as the easy set of DAVSOD19 fan2019shifting  (35 clips).

For other details, we basically follow the settings for RGB-D SOD in Sec 4.2.
For the TDB network, we modify the input channel of the first IRB block from 1
to 3, to accommodate the change from one-channel depth to three-channel
optical flow. We obtain optical flow maps by using the recent RAFT
teed2020raft .

####  4.6.3 Quantitative and Qualitative Comparisons on VSOD

Table 11: Efficiency comparison with existing VSOD methods. The model size, $T_{CPU}$, $S_{GPU}$, and $S_{GPU}^{*}$ are measured by Mb/ms/FPS/FPS, respectively. $S_{GPU}^{*}$ of Reuse is not available for it only supports one-batch inference. The score better than ours is underlined. Method | Model size | $T_{CPU}$ | $S_{GPU}$ | $S_{GPU}^{*}$
---|---|---|---|---
DLVS Wang2018VideoSO  | TIP17 | 347.3 | 4600 | 3 | 85
RCR Yan2019SemiSupervisedVS  | ICCV19 | 205.5 | 975 | 18 | 25
FSNet ji2021FSNet  | ICCV21 | 391.4 | 2600 | 25 | 75
_DFM-Net_ * | Ours | 93 | 161 | 62 | 176
PCSA Gu2020PyramidCS  | AAAI20 | 19.8 | 176 | 111 | 120
Reuse Park2021Learning  | CVPR21 | 27.5 | 1750 | 35 | N/A
_DFM-Net_ | Ours | 8.5 | 53 | 60 | 330

As shown in Tab. 10, we compare _DFM-Net_ and _DFM-Net_ * quantitatively with
15 advanced methods. The accuracy and efficiency metrics follow the criteria
in RGB-D SOD experiments. Note that some of these works do not provide code or
report the number of parameters, so the model size of these methods are
labeled as “N/A”. As for inference latency, we compare with some
representative ones and show the results individually in Tab. 11.

Generally, although _DFM-Net_ and _DFM-Net_ * are not specially designed for
VSOD (no long-term temporal consideration), they both achieve competitive
accuracy while they are much smaller and faster than the contenders.
Specifically, _DFM-Net_ is much more efficient than the recent lightweight
models PCSA and Reuse, with 57% and 69% parameters decrease. This leads to
_DFM-Net_ ’ s best performance in terms of $T_{CPU}$ and $S_{GPU}^{*}$. As for
$S_{GPU}$, PCSA runs a little faster than _DFM-Net_ because PCSA has custom
CUDA implementation. Furthermore, according to Tab. 10 and Tab. 11, _DFM-Net_
* surpasses almost all the non-efficient competitors in terms of both
efficiency and accuracy. Qualitative results are exhibited in Fig. 13, our
models generate stable and better predictions under various optical flow
qualities.

####  4.6.4 Ablation Analyses on VSOD

Effectiveness of DQFM. To answer _whether DQFM still works for optical flow_ ,
we conduct an ablation experiment. As shown in Tab. 13, #13, #14, #15 denote
the variants of _DFM-Net_ that are obtained by removing DQFM, DHA, and DQW,
respectively, while #16 indicates the full _DFM-Net_. Generally, we can see
that on the VSOD task, inserting either DQW or DHA, or both, will lead to
notable improvements with only little parameter increases. With DQFM equipped,
the max F-measure is improved by 6.4% on DAVSOD. As for the synergy effect of
DQW and DHA, it works on almost all datasets except DAVIS.

Table 12: Ablation analyses for DQFM on four VSOD datasets. The best results are highlighted in bold. # | DQW | DHA | DAVIS perazzi2016benchmark  | FBMS ochs2013segmentation  | MCL Kim2015SpatiotemporalSD  | DAVSOD-E fan2019shifting
---|---|---|---|---|---|---
$S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$
13 |  |  | .885 | .851 | .949 | .030 | .861 | .862 | .920 | .050 | .814 | .730 | .883 | .032 | .723 | .610 | .781 | .094
14 | ✓ |  | .900 | .875 | .961 | .023 | .880 | .866 | .925 | .041 | .821 | .741 | .889 | .028 | .771 | .678 | .827 | .073
15 |  | ✓ | .900 | .875 | .961 | .024 | .880 | .871 | .929 | .039 | .829 | .765 | .900 | .027 | .774 | .683 | .827 | .075
16 | ✓ | ✓ | .898 | .869 | .959 | .025 | .889 | .880 | .937 | .034 | .834 | .768 | .906 | .027 | .774 | .684 | .830 | .072
Table 13: Performance of different training strategies. “w/o Pretraining” means direct training on VSOD datasets without DUTS. “Pretraining” means using DUTS for pretraining. “Joint training” means the proposed joint training strategy. # | Training | Datasets for training | DAVIS perazzi2016benchmark  | FBMS ochs2013segmentation  | MCL Kim2015SpatiotemporalSD  | DAVSOD-E fan2019shifting
---|---|---|---|---|---|---
Strategy | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$ | $S_{\alpha}$ | $F_{\beta}^{\rm m}$ | $E_{\xi}^{\rm m}$ | $\mathcal{M}$
17 | w/o Pretraining | DAVIS+FBMS | .890 | .856 | .959 | .027 | .868 | .860 | .914 | .048 | .830 | .753 | .913 | .029 | .729 | .618 | .799 | .091
18 | Pretraining | DAVIS+FBMS+DUTS | .885 | .853 | .953 | .030 | .879 | .866 | .926 | .042 | .819 | .753 | .885 | .031 | .744 | .637 | .802 | .090
16 | Joint training | DAVIS+FBMS+DUTS | .898 | .869 | .959 | .025 | .889 | .880 | .937 | .034 | .834 | .768 | .906 | .027 | .774 | .684 | .830 | .072

Effectiveness of Joint Training. To see _how the pretraining strategy performs
for lightweight models,_ we conduct several training experiments as shown in
Tab. 13. Model #16 is the proposed _DFM-Net_ applied to VSOD, which adopts the
“Joint training” strategy. #17 means direct training on VSOD datasets DAVIS
and FBMS, but without pretraining on DUTS. #18 is trained on DAVIS and FBMS
after pretraining on DUTS for 300 epochs (equal to that for training).
Comparing between #17 (w/o Pretraining) and #18 (with Pretraining) in Tab. 13,
one can see pretraining becomes less effective, with accuracy improvement on
FBMS and DAVSOD but decrease on DAVIS and MCL. Moreover, we have experienced
double-long training time for #18. This comparison justifies our motivation to
adopt the joint training strategy, resulting in model #16 to better utilize
extra RGB data. As can be seen, #16 achieves the best results, producing
stable and consistent improvements across all datasets, especially on DAVSOD.

##  5 Conclusion

In this paper, we introduce an efficient RGB-D SOD model called _DFM-Net_. It
is achieved by equipping a lightweight RGB-D SOD framework with an efficient
DQFM process, which can significantly boost the detection accuracy with little
time-space consumption increase. DQFM consists of two components namely DQW
and DHA. The DQW weighs the depth features according to the low-level
alignment between RGB and depth inputs, while DHA attends to the depth
features spatially through multiple holistic attention maps derived from the
depth stream and refined with low-level RGB features. Besides, we also design
a tailored depth backbone and a two-stage decoder as the basis of the
lightweight framework to further ensure efficiency. Interestingly, the
statistics of inferred weights from DQW somewhat reveal its underlying working
mechanism, showing that it is feasible to distinguish depth maps of various
qualities without any quality labels.

Extensive experiments on nine RGB-D datasets demonstrate that _DFM-Net_ breaks
records in terms of model size and inference latency while retaining very
competitive accuracy for RGB-D SOD. Furthermore, _DFM-Net_ shows strong
performance when applied to the VSOD task and achieves the highest efficiency.
We also propose and validate the joint training strategy concerned for
lightweight models. In the future, we believe it would be attractive to deploy
_DFM-Net_ onto embedding/mobile systems that process RGB-D and video data. Not
just limited to that, it is worth exploring whether DQFM can be applied to
more quality-inspired dense prediction tasks, such as RGB-D and video semantic
segmentation, as well as camouflage object detection.

## References

  *   * (1) Wang, W., Shen, J., Yang, R., Porikli, F.: Saliency-aware video object segmentation. IEEE T. Pattern Anal. Mach. Intell. 40(1), 20–33 (2018)
  * (2) Fan, D.-P., Ji, G.-P., Zhou, T., Chen, G., Fu, H., Shen, J., Shao, L.: Pranet: Parallel reverse attention network for polyp segmentation. Int. Conf. Med. Image Comput. Comput. Assist. Intervent. (2020)
  * (3) Ji, G.-P., Chou, Y.-C., Fan, D.-P., Chen, G., Jha, D., Fu, H., Shao, L.: Progressively normalized self-attention network for video polyp segmentation. In: Int. Conf. Med. Image Comput. Comput. Assist. Intervent. (2021)
  * (4) Ji, G.-P., Xiao, G., Chou, Y.-C., Fan, D.-P., Zhao, K., Chen, G., Van Gool, L.: Video polyp segmentation: A deep learning perspective. arXiv:2203.14291 (2022)
  * (5) Zhang, P., Liu, W., Wang, D., Lei, Y., Lu, H.: Non-rigid object tracking via deep multi-scale spatial-temporal discriminative saliency maps. Pattern Recognit. 100, 107130 (2019)
  * (6) Ji, G.-P., Fu, K., Wu, Z., Fan, D.-P., Shen, J., Shao, L.: Full-duplex strategy for video object segmentation. In: Int. Conf. Comput. Vis. (2021)
  * (7) Wan, T., Du, S., Cui, W., Yao, R., Ge, Y., Li, C., Gao, Y., Zheng, N.: Rgb-d point cloud registration based on salient object detection. IEEE T. Neural Netw. Learn. Syst. (2021)
  * (8) Wang, W., Lai, Q., Fu, H., Shen, J., Ling, H.: Salient object detection in the deep learning era: An in-depth survey. IEEE T. Pattern Anal. Mach. Intell. (2021)
  * (9) Fan, D.-P., Li, T., Lin, Z., Ji, G.-P., Zhang, D., Cheng, M.-M., Fu, H., Shen, J.: Re-thinking co-salient object detection. IEEE T. Pattern Anal. Mach. Intell. (2021)
  * (10) Jiang, H., Yuan, Z., Cheng, M.-M., Gong, Y., Zheng, N., Wang, J.: Salient object detection: A discriminative regional feature integration approach. Int. J. Comput. Vis. 123, 251–268 (2013)
  * (11) He, S., Lau, R.W.H., Liu, W., Huang, Z., Yang, Q.: Supercnn: A superpixelwise convolutional neural network for salient object detection. Int. J. Comput. Vis. 115, 330–344 (2015)
  * (12) Tian, X., Xu, K., Yang, X., Yin, B., Lau, R.W.H.: Learning to detect instance-level salient objects using complementary image labels. Int. J. Comput. Vis. 130, 729–746 (2022)
  * (13) Peng, P., Yang, K.-F., Luo, F.-Y., Li, Y.-J.: Saliency detection inspired by topological perception theory. Int. J. Comput. Vis. 129(8), 2352–2374 (2021)
  * (14) Fan, D.-P., Zhai, Y., Borji, A., Yang, J., Shao, L.: Bbs-net: Rgb-d salient object detection with a bifurcated backbone strategy network. In: Eur. Conf. Comput. Vis., pp. 275–292 (2020)
  * (15) Fu, K., Fan, D.-P., Ji, G.-P., Zhao, Q., Shen, J., Zhu, C.: Siamese network for rgb-d salient object detection and beyond. IEEE T. Pattern Anal. Mach. Intell. (2021)
  * (16) Zhang, J., Fan, D.-P., Dai, Y., Anwar, S., Saleh, F., Aliakbarian, S., Barnes, N.: Uncertainty inspired rgb-d saliency detection. IEEE T. Pattern Anal. Mach. Intell. (2021)
  * (17) Pang, Y., Zhang, L., Zhao, X., Lu, H.: Hierarchical dynamic filtering network for rgb-d salient object detection. In: Eur. Conf. Comput. Vis., pp. 235–252 (2020)
  * (18) Zhang, Z., Lin, Z., Xu, J., Jin, W.-D., Lu, S.-P., Fan, D.-P.: Bilateral attention network for rgb-d salient object detection. IEEE T. Image Process. 30, 1949–1961 (2021)
  * (19) Li, C., Cong, R., Kwong, S., Hou, J., Fu, H., Zhu, G., Zhang, D., Huang, Q.: Asif-net: Attention steered interweave fusion network for rgb-d salient object detection. IEEE T. Cybern. PP(99), 1–13 (2020)
  * (20) Cong, R., Lei, J., Fu, H., Hou, J., Huang, Q., Kwong, S.: Going from rgb to rgbd saliency: A depth-guided transformation model. IEEE T. Cybern. 50(8), 3627–3639 (2020)
  * (21) Chen, H., Li, Y., Deng, Y., Lin, G.: Cnn-based rgb-d salient object detection: Learn, select, and fuse. Int. J. Comput. Vis., 1–21 (2021)
  * (22) Zhou, T., Fan, D.-P., Cheng, M.-M., Shen, J., Shao, L.: Rgb-d salient object detection: A survey. Comput. Vis. Media 7, 37–69 (2020)
  * (23) Cong, R., Lei, J., Fu, H., Cheng, M.-M., Lin, W., Huang, Q.: Review of visual saliency detection with comprehensive information. IEEE T. Circuit Syst. Video Technol. 29, 2941–2959 (2019)
  * (24) Fan, D.-P., Lin, Z., Zhang, Z., Zhu, M., Cheng, M.-M.: Rethinking RGB-D salient object detection: Models, datasets, and large-scale benchmarks. IEEE T. Neural Netw. Learn. Syst. (2020)
  * (25) Fu, K., Fan, D.-P., Ji, G.-P., Zhao, Q.: Jl-dcf: Joint learning and densely-cooperative fusion framework for rgb-d salient object detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 3052–3062 (2020)
  * (26) et al., N.L.: Learning selective self-mutual attention for rgb-d saliency detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 13756–13765 (2020)
  * (27) Zhang, J., Fan, D.-P., Dai, Y., Anwar, S., Sadat Saleh, F., Zhang, T., Barnes, N.: Uc-net: Uncertainty inspired rgb-d saliency detection via conditional variational autoencoders. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 8582–8519 (2020)
  * (28) Zhang, M., Ren, W., Piao, Y., Rong, Z., Lu, H.: Select, supplement and focus for rgb-d saliency detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 3472–3481 (2020)
  * (29) Li, C., Cong, R., Yongri, P., Qianqian, X., Loy, C.C.: Rgb-d salient object detection with cross-modality modulation and selection. In: Eur. Conf. Comput. Vis., pp. 225–241 (2020)
  * (30) Chen, Z., Cong, R., Xu, Q., Huang, Q.: Dpanet: Depth potentiality-aware gated attention network for rgb-d salient object detection. IEEE T. Image Process. (2020)
  * (31) Zhao, J.-X., Cao, Y., Fan, D.-P., Cheng, M.-M., Li, X.-Y., Zhang, L.: Contrast prior and fluid pyramid integration for rgbd salient object detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 3927–3936 (2019)
  * (32) Piao, Y., Ji, W., Li, J., Zhang, M., Lu, H.: Depth-induced multi-scale recurrent attention network for saliency detection. In: Int. Conf. Comput. Vis., pp. 7254–7263 (2019)
  * (33) Sandler, M., Howard, G.A., Zhu, M., Zhmoginov, A., Chen, L.-C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 4510–4520 (2018)
  * (34) Piao, Y., Rong, Z., Zhang, M., Ren, W., Lu, H.: A2dele: Adaptive and attentive depth distiller for efficient rgb-d salient object detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 9060–9069 (2020)
  * (35) Ji, W., Li, J., Zhang, M., Piao, Y., Lu, H.: Accurate rgb-d salient object detection via collaborative learning. In: Eur. Conf. Comput. Vis., pp. 52–69 (2020)
  * (36) Zhao, Y., Zhao, J., Li, J., Chen, X.: Rgb-d salient object detection with ubiquitous target awareness. IEEE T. Image Process. 30, 7717–7731 (2021)
  * (37) Zhao, X., Zhang, L., Pang, Y., Lu, H., Zhang, L.: A single stream network for robust and real-time rgb-d salient object detection. In: Eur. Conf. Comput. Vis., pp. 520–538 (2020)
  * (38) et al., S.C.: Progressively guided alternate refinement network for rgb-d salient object detection. In: Eur. Conf. Comput. Vis., pp. 520–538 (2020)
  * (39) Zhang, M., Fei, X.S., Liu, J., Xu, S., Piao, Y., Lu, H.: Asymmetric two-stream architecture for accurate rgb-d saliency detection. In: Eur. Conf. Comput. Vis., pp. 374–390 (2020)
  * (40) Wu, Y.-H., Liu, Y., Xu, J., Bian, J.-W., Gu, Y.-C., Cheng, M.-M.: Mobilesal: Extremely efficient rgb-d salient object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)
  * (41) Zhang, W., Jiang, Y., Fu, K., Zhao, Q.: Bts-net: Bi-directional transfer-and-selection network for rgb-d salient object detection. In: Int. Conf. Multimedia and Expo (2021)
  * (42) Cong, R., Lei, J., Zhang, C., Huang, Q., Cao, X., Hou, C.: Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion. IEEE Signal Process. Lett. 23, 819–823 (2016)
  * (43) Xiang, S., Yu, L., Chen, C.W.: No-reference depth assessment based on edge misalignment errors for t \+ d images. IEEE T. Image Process. 25(3), 1479–1494 (2016)
  * (44) Niu, Y., Geng, Y., Li, X., Liu, F.: Leveraging stereopsis for saliency analysis. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 454–461 (2012)
  * (45) Liu, N., Zhang, N., Shao, L., Han, J.: Learning selective mutual attention and contrast for rgb-d saliency detection. arXiv:2010.05537 (2020)
  * (46) He, J., Zhang, S., Yang, M., Shan, Y., Huang, T.: Bi-directional cascade network for perceptual edge detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 3823–3832 (2019)
  * (47) Milletari, F., Navab, N., Ahmadi, S.-A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. 3D Vis., 565–571 (2016)
  * (48) Ji, W., Li, J., Yu, S., Zhang, M., Piao, Y., Yao, S., Bi, Q., Ma, K., Zheng, Y., Lu, H., Cheng, L.: Calibrated rgb-d salient object detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 9471–9481 (2021)
  * (49) Sun, P., Zhang, W., Wang, H., Li, S., Li, X.: Deep rgb-d saliency detection with depth-sensitive attention and automatic multi-modal fusion. In: IEEE Conf. Comput. Vis. Pattern Recog. (2021)
  * (50) Chen, M.-M., Fan, D.-P.: Structure-measure: A new way to evaluate foreground maps. Int. J. Comput. Vis. (2021)
  * (51) Ju, R., Ge, L., Geng, W., Ren, T., Wu, G.: Depth saliency based on anisotropic center-surround difference. In: IEEE Int. Conf. Image Process., pp. 1115–1119 (2014)
  * (52) eng, H., Li, B., Xiong, W., Hu, W., Ji, R.: Rgbd salient object detection: A benchmark and algorithms. In: Eur. Conf. Comput. Vis., pp. 92–109 (2014)
  * (53) Cheng, Y., Fu, H., Wei, X., Xiao, J., Cao, X.: Depth enhanced saliency detection method. In: ICIMCS, pp. 23–27 (2014)
  * (54) Li, N., Ye, J., Ji, Y., Ling, H., Yu, J.: Saliency detection on light field. IEEE T. Pattern Anal. Mach. Intell. 39, 1605–1616 (2014)
  * (55) Zhang, W., Ji, G.-P., Wang, Z., Fu, K., Zhao, Q.: Depth quality-inspired feature manipulation for efficient rgb-d salient object detection. In: ACM Int. Conf. Multimedia, pp. 731–740 (2021)
  * (56) Ciptadi, A., Hermans, T., Rehg, J.: An in depth view of saliency. In: Brit. Mach. Vis. Conf. (2013)
  * (57) Goferman, S., Zelnik-Manor, L., Tal, A.: Context-aware saliency detection. IEEE T. Pattern Anal. Mach. Intell. 34, 1915–1926 (2012)
  * (58) Ren, J., Gong, X., Yu, L., Zhou, W., Ying Yang, M.: Exploiting global priors for rgb-d saliency detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 25–32 (2015)
  * (59) Chen, H., Li, Y.: Progressively complementarity-aware fusion network for rgb-d salient object detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 3051–3060 (2018)
  * (60) Zhu, C., Cai, X., Huang, K., Li, T.H., Li, G.: Pdnet: Prior-model guided depth-enhanced network for salient object detection. In: Int. Conf. Multimedia and Expo, pp. 199–204 (2019)
  * (61) Chen, H., Li, Y., Su, D.: Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for rgb-d salient object detection. Pattern Recognition 86, 376–385 (2019)
  * (62) Qu, L., He, S., Zhang, J., Tian, J., Tang, Y., Yang, Q.: Rgbd salient object detection via deep fusion. IEEE T. Image Process. 26, 2274–2285 (2017)
  * (63) Li, G., Liu, Z., Ye, L., Wang, Y., Ling, H.: Cross-modal weighting network for rgb-d salient object detection. In: Eur. Conf. Comput. Vis., pp. 665–681 (2020)
  * (64) Luo, A., Li, X., Yang, F., Jiao, Z., Cheng, H., Lyu, S.: Cascade graph neural networks for rgb-d salient object detection. In: Eur. Conf. Comput. Vis., pp. 346–364 (2020)
  * (65) Zhang, C., Cong, R., Lin, Q., Ma, L., Li, F., Zhao, Y., Kwong, S.: Cross-modality discrepant interaction network for rgb-d salient object detection. In: ACM Int. Conf. Multimedia, pp. 2094–2102 (2021)
  * (66) Liu, Z., Wang, Y., Tu, Z., Xiao, Y., Tang, B.: Tritransnet: Rgb-d salient object detection with a triplet transformer embedding network. In: ACM Int. Conf. Multimedia, pp. 4481–4490 (2021)
  * (67) Liu, N., Zhang, N., Wan, K., Shao, L., Han, J.: Visual saliency transformer. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 4722–4732 (2021)
  * (68) Sun, P., Zhang, W., Wang, H., Li, S., Li, X.: Deep rgb-d saliency detection with depth-sensitive attention and automatic multi-modal fusion. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 1407–1417 (2021)
  * (69) Zhang, J., Fan, D.-P., Dai, Y., Yu, X., Zhong, Y., Barnes, N., Shao, L.: Rgb-d saliency detection via cascaded mutual information minimization. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 4338–4347 (2021)
  * (70) Zhou, T., Fu, H., Chen, G., Zhou, Y., Fan, D.-P., Shao, L.: Specificity-preserving rgb-d saliency detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 4681–4691 (2021)
  * (71) Liu, Z., Wang, K., Dong, H., Wang, Y.: A cross-modal edge-guided salient object detection for rgb-d image. Neurocomputing 454, 168–177 (2021)
  * (72) Liang, F., Duan, L., Ma, W., Qiao, Y., Miao, J., Ye, Q.: Context-aware network for rgb-d salient object detection. Pattern Recognit. 111, 107630 (2021)
  * (73) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 25–32 (2016)
  * (74) Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Int. Conf. Learn. Represent. (2015)
  * (75) Mittal, A., Moorthy, A.K., Bovik, A.: No-reference image quality assessment in the spatial domain. IEEE T. Image Process. 21, 4695–4708 (2012)
  * (76) Ji, W., Li, J., Yu, S., Zhang, M., Piao, Y., Yao, S., Bi, Q., Ma, K., Zheng, Y., Lu, H., et al.: Calibrated rgb-d salient object detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 9471–9481 (2021)
  * (77) Wang, X., Li, S., Chen, C., Hao, A., Qin, H.: Depth quality-aware selective saliency fusion for rgb-d image salient object detection. Neurocomputing 432, 44–56 (2021)
  * (78) Chen, C., Wei, J., Peng, C., Qin, H.: Depth-quality-aware salient object detection. IEEE T. Image Process. PP(99), 1–1 (2021)
  * (79) Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 6230–6239 (2017)
  * (80) Zhang, P., Wang, D., Lu, H., Wang, H., Ruan, X.: Amulet: Aggregating multi-level convolutional features for salient object detection. In: Int. Conf. Comput. Vis., pp. 202–211 (2017)
  * (81) Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., Pal, C.: The importance of skip connections in biomedical image segmentation. Deep learning and data labeling for medical applications, 179–187 (2016)
  * (82) Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 (2017)
  * (83) Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Int. Conf. Med. Image Comput. Comput. Assist. Intervent., pp. 234–241 (2015)
  * (84) Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: IEEE Conf. Comput. Vis. Pattern Recog. (2018)
  * (85) Achanta, R., Hemami, S., Estrada, F., Süsstrunk, S.: Frequency-tuned salient region detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 1597–1604 (2009)
  * (86) Fan, D.-P., Gong, C., Cao, Y., Ren, B., Cheng, M.-M., Borji, A.: Enhanced-alignment Measure for Binary Foreground Map Evaluation. In: Int. Jt. Conf. Artif. Intell., pp. 698–704 (2018)
  * (87) Fan, D.-P., Ji, G.-P., Qin, X., Cheng, M.-M.: Cognitive vision inspired object segmentation metric and loss function (in chinese). SCIENTIA SINICA Informationis (2021)
  * (88) Borji, A., Cheng, M.-M., Jiang, H., Li, J.: Salient object detection: A benchmark. IEEE T. Image Process. 24, 5706–5722 (2015)
  * (89) Williams, S., Waterman, A., Patterson, D.: Roofline: an insightful visual performance model for multicore architectures. Commun. ACM 52, 65–76 (2009)
  * (90) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Adv. Neural Inform. Process. Syst. (2019)
  * (91) Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv:1412.6980 (2014)
  * (92) Liu, N., Zhang, N., Shao, L., Han, J.: Learning selective mutual attention and contrast for rgb-d saliency detection. IEEE T. Pattern Anal. Mach. Intell., 1–1 (2021)
  * (93) Tang, S., Gong, R., Wang, Y., Liu, A., Wang, J., Chen, X., Yu, F., Liu, X., Song, D., Yuille, A., et al.: Robustart: Benchmarking robustness on architecture design and training techniques. arXiv:2109.05211 (2021)
  * (94) Wang, W., Shen, J., Shao, L.: Video salient object detection via fully convolutional networks. IEEE T. Image Process. 27, 38–49 (2018)
  * (95) Li, S., Seybold, B., Vorobyov, A., Lei, X., Kuo, C.-C.J.: Unsupervised video object segmentation with motion-based bilateral networks. In: Eur. Conf. Comput. Vis., pp. 207–223 (2018)
  * (96) Li, G., Xie, Y., Wei, T., Wang, K., Lin, L.: Flow guided recurrent neural encoder for video salient object detection. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 3243–3252 (2018)
  * (97) Tang, Y., Zou, W., Jin, Z., Chen, Y., Hua, Y., Li, X.: Weakly supervised salient object detection with spatiotemporal cascade neural networks. IEEE T. Circuit Syst. Video Technol. 29(7), 1973–1984 (2018)
  * (98) Chen, Y., Zou, W., Tang, Y., Li, X., Xu, C., Komodakis, N.: Scom: Spatiotemporal constrained optimization for salient object detection. IEEE T. Image Process. 27(7), 3345–3357 (2018)
  * (99) Cong, R., Lei, J., Fu, H., Porikli, F., Huang, Q., Hou, C.: Video saliency detection via sparsity-based reconstruction and propagation. IEEE T. Image Process. 28(10), 4819–4831 (2019)
  * (100) Xu, M., Liu, B., Fu, P., Li, J., Hu, Y.H.: Video saliency detection via graph clustering with motion energy and spatiotemporal objectness. IEEE T. Multimedia 21(11), 2790–2805 (2019)
  * (101) Chen, C., Wang, G., Peng, C., Zhang, X., Qin, H.: Improved robust video saliency detection based on long-term spatial-temporal information. IEEE T. Image Process. 29, 1090–1100 (2019)
  * (102) Fan, D.-P., Wang, W., Cheng, M.-M., Shen, J.: Shifting more attention to video salient object detection. In: IEEE Conf. Comput. Vis. Pattern Recog. (2019)
  * (103) Yan, P., Li, G., Xie, Y., Li, Z., Wang, C., Chen, T., Lin, L.: Semi-supervised video salient object detection using pseudo-labels. Int. Conf. Comput. Vis., 7283–7292 (2019)
  * (104) Li, Y., Li, S., Chen, C., Hao, A., Qin, H.: Accurate and robust video saliency detection via self-paced diffusion. IEEE T. Multimedia 22(5), 1153–1167 (2019)
  * (105) Gu, Y., Wang, L., Wang, Z., Liu, Y., Cheng, M.-M., Lu, S.-P.: Pyramid constrained self-attention network for fast video salient object detection. In: AAAI Conf. Art. Intell. (2020)
  * (106) Park, H., Yoo, J., Jeong, S., Venkatesh, G., Kwak, N.: Learning dynamic network using a reuse gate function in semi-supervised video object segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 8401–8410 (2021)
  * (107) Cheng, M.-M., Gao, S., Borji, A., Tan, Y.-Q., Lin, Z., Wang, M.: A highly efficient model to study the semantics of salient object detection. IEEE T. Pattern Anal. Mach. Intell. (2021)
  * (108) Poudel, R.P., Liwicki, S., Cipolla, R.: Fast-scnn: Fast semantic segmentation network. arXiv:1902.04502 (2019)
  * (109) Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 724–732 (2016)
  * (110) Ochs, P., Malik, J., Brox, T.: Segmentation of moving objects by long term video analysis. IEEE T. Pattern Anal. Mach. Intell. 36(6), 1187–1200 (2013)
  * (111) Wang, L., Lu, H., Wang, Y., Feng, M., Wang, D., Yin, B., Ruan, X.: Learning to detect salient objects with image-level supervision. In: IEEE Conf. Comput. Vis. Pattern Recog., pp. 136–145 (2017)
  * (112) Kim, H.-U., Kim, Y., Sim, J.-Y., Kim, C.-S.: Spatiotemporal saliency detection for video sequences based on random walk with restart. IEEE T. Image Process. 24, 2552–2564 (2015)
  * (113) Teed, Z., Deng, J.: Raft: Recurrent all-pairs field transforms for optical flow. In: Eur. Conf. Comput. Vis., pp. 402–419 (2020). Springer