# Denoising neural networks for magnetic resonance spectroscopy

###### Abstract

In many scientific applications, measured time series are corrupted by noise
or distortions. Traditional denoising techniques often fail to recover the
signal of interest, particularly when the signal-to-noise ratio is low or when
certain assumptions on the signal and noise are violated. In this work, we
demonstrate that deep learning-based denoising methods can outperform
traditional techniques while exhibiting greater robustness to variation in
noise and signal characteristics. Our motivating example is magnetic resonance
spectroscopy, in which a primary goal is to detect the presence of short-
duration, low-amplitude radio frequency signals that are often obscured by
strong interference that can be difficult to separate from the signal using
traditional methods. We explore various deep learning architecture choices to
capture the inherently complex-valued nature of magnetic resonance signals. On
both synthetic and experimental data, we show that our deep learning-based
approaches can exceed performance of traditional techniques, providing a
powerful new class of methods for analysis of scientific time series data.

Index Terms—  Deep learning, signal denoising, complex-valued neural networks,
nuclear quadrupole resonance

##  1 Introduction

In this work, we consider deep learning-based time-domain denoising of single-
sensor complex-valued (CV) signals with expected variation in signal and noise
properties during deployment. Our motivating application is magnetic resonance
spectroscopy, a chemical analysis technique that can be used to identify
materials and characterize the structure and chemical environment of molecules
in the sample [1]. We focus specifically on nuclear quadrupole resonance (NQR)
spectroscopy, which can capture unique signatures arising from transitions of
nuclei in the absence of an applied magnetic field and can thus provide a
sensitive standoff detection capability for some classes of substances [2].
Our ultimate goal is to extract clean signals from noisy NQR recordings of a
given substance for downstream tasks such as signal classification.

Fig. 1: Simulated clean signal (dark lines), with real (orange) and imaginary
(blue) components, overlaid on noisy signal (light lines) generated by
combining actual recordings of radio frequency interference with the clean
signal. Noise and distortions of the signal based on environmental conditions
often complicate signal detection and characterization.

In practice, many substances of interest produce very low-amplitude, short-
duration radio frequency (RF) signals that are difficult to detect due to
noise and environmental interference, including commonly-occurring RF
interference (illustration: Fig. 1). In addition, expected signal
characteristics may vary during deployment; for example, temperature changes
can shift the expected signal frequency [3] and crystalline samples with
strong dipolar coupling can deviate from theoretical exponential decay models
[4]. Existing signal processing methods (e.g., wavelet decomposition, time-
domain deconvolution, or subspace methods) have been found to work well when
there is a high signal-to-noise ratio (SNR), but we show that their
performance degrades in more challenging situations (e.g., low SNR, non-white
noise, or overlapping signal and noise frequency components).

Neural networks (NNs) have been used to isolate signals of interest across a
variety of applications, including image denoising [5] and speech separation
[6]. We adapt such approaches to the task of NQR denoising, incorporating CV
weights and activation functions to avoid losing information in the CV
signals. We show that NN-based approaches outperform existing denoising
methods, particularly in low SNR scenarios, with CV-NNs providing more robust
results across changes in noise structure and signal power.

##  2 RELATION TO PRIOR WORK

NQR denoising methods seek to recover clean signals from noisy signals for
improved chemical detection and characterization; we consider a single-sensor
setup in which it can be particularly difficult to separate noise and RF
interference from the signal. In addition, the short-duration signals often
exhibit low SNR, and signals cannot be expected to conform to simple physical
signal models (such as complex exponential decay with a known frequency).
Previous approaches for signal denoising in NQR include both time-domain and
frequency-domain denoising methods and interference cancellation (see [7] for
a review), with some methods leveraging specialized collection techniques with
multiple sensors. For the single-sensor scenario, we investigate automated
versions of three existing methods: wavelet denoising (e.g., [8]), statistical
curve fitting of assumed signal models (e.g., [9]), and subspace separation
methods (e.g., [10]). Recently, real-valued (RV) denoising autoencoders (AEs)
were shown to improve signal detection when applied to low SNR NQR data for
denoising prior to classification [11].

Our work differs from [11] in two key ways. First, we make use of CV-NNs,
allowing native processing of CV signals and improving learning by exploiting
structured relationships between real and imaginary components [12]. Several
recent works suggest CV-NNs can outperform RV-NNs (e.g., [13]), though this
may be context-dependent [14]. Second, we compare denoising AEs to a more
sophisticated and specialized architecture designed for speech separation,
ConvTasNet (CTN) [6]. This fully-convolutional network utilizes an encoder and
decoder, similar to an AE, but explicitly learns to separate multiple sources
(here, signal and noise) with a masking module.

##  3 Proposed denoising neural networks

We seek to compare RV and CV variants of two NN approaches, AE and CTN,
against existing denoising methods, and to understand which NNs provide more
accurate and robust results in this application. Full implementation details
are available in our GitHub repository [15].

###  3.1 Neural network (NN) architectures

Our CTN implementation [15] is based on the torchaudio implementation [16],
but modified to include three-layer deep encoder/decoder modules [17] and to
accommodate CV data (Section 3.2). We fixed most CTN hyperparameters to
default values in the torchaudio implementation, but select among three window
lengths (32, 64, or 128 samples) based on test set performance. For the AE
model, we use a fully-connected network with encoder consisting of one
intermediate hidden layer (dimension 10) that maps to a five-dimensional
latent representation. Models are trained using signal reconstruction error.
For each considered model, we train an ensemble of models in which the weights
are randomly initialized with different random seeds [18]; this provides a
metric of uncertainty and the opportunity to average across model predictions
for higher accuracy.

###  3.2 Complex-valued (CV) and real-valued (RV) NNs

We explore two main approaches to training NNs for CV data: one that relies on
an RV-NN to operate on separated real and imaginary components with learnable
RV weights representing relationships between the components (Fig. 2, left
side) and one that uses a CV-NN to directly operate on CV inputs (Fig. 2,
right side). In [15], we explore alternative RV approaches, but find they
perform worse than the presented RV approach. For the CV-NN approach, we
mirror the RV-NN architecture, but with CV activation functions and
convolutions suitable for complex data.

Complex Data Real Component Imaginary Component RV-NN Real Component Imaginary
Component Complex Reconstruction

Complex DataCV-NNComplex Reconstruction

Fig. 2: The first approach to NNs for CV data (left) jointly processes real
and imaginary components of the data with a real-valued neural network (RV-
NN), while the second approach (right) operates directly on complex data with
a complex-valued neural network (CV-NN).

##  4 Experimental Results

To evaluate performance, we generate a set of synthetic NQR datasets with
variation in SNR, signal model, and noise source (random white or actual
measured RF interference). We compare the NNs to each other and to three
baseline methods via a scaled test set $R^{2}$ metric, then evaluate
robustness of the NNs to out-of-distribution data. Finally, we apply the best-
performing methods to experimental data.

###  4.1 Datasets

For NN training and quantitative evaluation, we utilize synthetic signals
generated according to a physics-based model. Specifically, time-varying
signals $y(t)$ are generated as a noisy decaying complex exponential (Voigt
profile [4]),

$\displaystyle
y(t)=A\exp\left(-\frac{t^{2}}{2\sigma^{2}}-\frac{t}{T_{2}}+i[2\pi
wt+\phi]\right)+\epsilon(t),$ (1)

where $i$ is the imaginary unit, $A$ is the initial amplitude, $\sigma$ and
$T_{2}$ control the rate of decay, $\phi$ is the phase offset, $w$ is the
frequency, and $\epsilon$ is additive noise. We draw $\sigma$ and $T_{2}$ from
$\text{Unif}(0.001,0.01)$ to emulate decay rate variation relative to
recording length that might be observed in practice. The parameter
$\sigma\ll\infty$ models expected deviations from simple exponential decay. We
generate eight training datasets by selecting from two noise distributions,
two signal frequency bands centered at $f_{0}=0$Hz and $f_{0}=F=1600$Hz, and
two SNR regimes (low and high). Varying the frequency as
$\text{Unif}(f_{0}-150,f_{0}+150)$ emulates temperature-dependent signal
frequency variation that might be encountered in practice. The SNRs range
between $[-36\text{dB},-7\text{dB}]$ and $[-10\text{dB},3\text{dB}]$ for low
and high SNR cases, respectively, with average low and high SNRs at
approximately -17dB and -3dB. The noise distributions are random Gaussian
white noise and measured RF noise from a laboratory instrument with no signal
present; we note that $F=1600$Hz was chosen because the measured noise
features elevated power at this frequency, leading to a more challenging noise
separation task. For each dataset, we train an independent ensemble of NNs
using 6,000 noisy signals for training and two independent sets of 2,000 noisy
signals for validation and testing. We also apply our methods to
representative experimental data exhibiting relatively high SNR and signal
frequency near 0Hz (due to complex demodulation at a known signal frequency).
For more complete descriptions of each dataset, see [15].

###  4.2 Comparison of NN architectures

First, we compare test set ensemble performance across the eight datasets, two
model architectures (AE and CTN), and two approaches to CV data (CV-NN and RV-
NN) using a scaled $R^{2}$ metric,
$R^{2}=100\times\left(1-\frac{\sum_{i}\sum_{t}(y_{i}(t)-\hat{y}_{i}(t))^{2}}{\sum_{i}\sum_{t}y_{i}(t)^{2}}\right)$,
where $y_{i}(t)$ is the true clean signal instance $i$ at time $t$ and
$\hat{y}_{i}(t)$ is the corresponding denoised signal prediction. The best
possible performance would occur at $R^{2}=100.0$, with lower (or even
negative) values indicating poor performance; the metric can be interpreted as
measuring improvement of a prediction above a simple prediction of “no signal
present”. Table 1 shows the results for the best performing AE and CTN models;
overall, CV-CTN outperforms both RV-CTN and the AEs, though CV-CTN and RV-CTN
achieve similar performance in high SNR. Interestingly, in low SNR, CV-CTN
performs better on measured noise than white noise, perhaps because it learns
some relevant structure in the noise.

(a) High SNR regime NN scaled $R^{2}$ values. | White noise | Measured noise
---|---|---
| $f\sim 0$ | $f\sim F$ | $f\sim 0$ | $f\sim F$
Model | CV | RV | CV | RV | CV | RV | CV | RV
AE | 96.0 | 98.1 | 96.1 | 95.5 | 95.5 | 98.1 | 95.2 | 95.8
CTN | 99.4 | 99.4 | 99.5 | 99.5 | 99.5 | 99.6 | 99.5 | 99.5

(b) Low SNR regime NN scaled $R^{2}$ values. | White noise | Measured noise
---|---|---
| $f\sim 0$ | $f\sim F$ | $f\sim 0$ | $f\sim F$
Model | CV | RV | CV | RV | CV | RV | CV | RV
AE | 67.1 | 76.9 | 65.9 | 75.8 | 72.2 | 77.6 | 65.5 | 70.0
CTN | 81.7 | 81.1 | 81.8 | 81.2 | 85.5 | 84.9 | 83.7 | 82.9

Table 1: Scaled $R^{2}$ (best possible value: 100.0) of NN ensemble test set
predictions under different training and model scenarios. Within each SNR, AE
and CTN results are split across noise distributions (white or measured),
central signal frequency $f$, and complex-valued (CV) or real-valued (RV) NNs,
with best results in each case bolded. Overall, CV-CTN performs the best
within each noise, frequency, and SNR setting, particularly in low SNR.

###  4.3 Comparison to baseline methods

Next, we compare CV-CTN to three baseline methods: time-domain deconvolution
(DC; in which we fit the signal model of Eq. 1 to noisy data via nonlinear
least squares [9, 10]), singular spectrum analysis (SSA; a subspace-based
method [19, 20, 10]), and wavelet decomposition (W; [8]). Similar to NN-based
approaches, each method uses automated optimization or thresholding rather
than human intervention. As shown in Table 2(a), in the high SNR regime, many
of the baseline methods perform nearly as well as CV-CTN, although SSA
degrades under measured noise and wavelets degrade when $f\sim F$. We note
that initial values for DC are based on the known training distribution;
performance of DC in practice is likely to be worse if poor initial values are
used. In low SNR (Table 2(b)), all of the baseline methods are considerably
worse than CV-CTN. These results suggest that compared to baseline methods,
CV-CTN achieves good performance in low SNR conditions, with little dependence
on the generating noise or signal properties.

(a) High SNR regime scaled $R^{2}$ values. | White noise | Measured noise
---|---|---
Method | $f\sim 0$ | $f\sim F$ | $f\sim 0$ | $f\sim F$
CV-CTN | 99.4 | 99.5 | 99.5 | 99.5
DC | 99.2 | 99.1 | 99.0 | 99.5
SSA | 98.6 | 98.6 | 52.4 | 52.6
W | 97.8 | 87.0 | 98.4 | 89.0

(b) Low SNR regime scaled $R^{2}$ values. | White noise | Measured noise
---|---|---
Method | $f\sim 0$ | $f\sim F$ | $f\sim 0$ | $f\sim F$
CV-CTN | 81.7 | 81.8 | 85.5 | 83.7
DC | 57.7 | 65.5 | 68.0 | 62.4
SSA | -42.6 | -41.7 | -1970.2 | -1970.2
W | 21.5 | -38.1 | 51.5 | -10.9

Table 2: Scaled $R^{2}$ of CV-CTN compared to time-domain deconvolution (DC),
singular spectrum analysis (SSA), and wavelets (W). CV-CTN generally
outperforms the other methods, and is consistent across signal and noise
distributions. In contrast, the baseline methods are competitive with CV-CTN
only in high SNR, and performance can depend on signal/noise characteristics
(e.g., for SSA and W).

###  4.4 Out-of-distribution performance

Out-of-distribution (OOD) refers to the situation in which a machine learning
model is applied to data that differs in distribution from the training data;
in our application, OOD data could occur due to changes in noise/interference
or changes in the signal during deployment. While deep learning models
generally do not perform as well on OOD data, ideally, useful NN denoising
models would be somewhat robust to OOD data, or would at least provide
uncertainty metrics to alert users when applied to OOD data. We use two OOD
tests: one based on changing the test set noise distribution relative to the
training noise distribution, and one based on changing the test set signal
frequency band relative to the training signal frequency band. We omit full
description of results due to space constraints (see [15]). Briefly, we find
that applying to test data with a different noise distribution does not
appreciably degrade performance or increase the ensemble uncertainty. However,
significant changes to the signal distribution (e.g., training on $f=0$Hz and
applying to $f=300$Hz) does increase error. Here, the CV-CTN ensemble
uncertainty increases under high SNR, suggesting measures of ensemble
uncertainty could be used to identify OOD data, but the same pattern does not
hold in low SNR. We conclude that for a given application, a suitable training
set should cover the space of signal variations expected in practice.

###  4.5 Application to experimental data

We apply the best-performing NNs (CV-CTN and RV-AE) to experimental data that
exhibits signal frequency near zero and high SNR, suggesting networks trained
on $f\sim 0$, high SNR data are suitable. Fig. 3(a) compares the NN
predictions to the best high SNR baseline method (DC). While no ground truth
is available for this data, we note that DC and CV-CTN give very similar
signal predictions, while RV-AE gives generally noisier predictions with
distorted signal shape. To better illustrate differences between the methods,
we also show results for synthetic low SNR data with $f\sim 0$ (Fig. 3(b)).
Here, both RV-AE and DC deviate significantly from the true signal, while CV-
CTN recovers it. These results suggest that CV-CTN accurately recovers the
signal in both SNRs.

(a) Experimental data high SNR denoising results.

(b) Synthetic data low SNR denoising results.

Fig. 3: Three methods (RV-AE, CV-CTN, and DC) applied to (a) representative
experimental data and (b) realistic, but low SNR, synthetic data. In (a), CV-
CTN and DC agree closely, while RV-AE slightly deviates. In (b), CV-CTN
recovers the ground truth signal shape more accurately than RV-AE or DC.

##  5 CONCLUSIONS

In this work, we explored real- and complex-valued versions of two NN
architectures for denoising of CV data. We found that among NNs, CV-CTN
achieved the best denoising results. However, some RV-NNs capable of learning
relationships between real and imaginary components reached similar
performance with reduced computational complexity. We also demonstrated that
NN approaches outperformed baseline denoising methods, particularly in low
SNR. Our results also suggest that NN approaches are robust to some signal and
noise variation, important for our intended application to magnetic resonance
spectroscopy, where measurements are often corrupted by noise, interference,
and distortions due to environmental conditions. In future work, we intend to
explore other NN architectures, including recurrent NNs, and the influence of
the training set on the robustness of the resulting denoising network. In
addition, we plan to investigate the impact of our denoising approaches on
downstream tasks, such as classification. Additional results and details are
available on GitHub [15].

##  6 Acknowledgements

This project was supported by the Laboratory Directed Research and Development
program of Los Alamos National Laboratory under project number
LDRD-20220086DR. This manuscript has been approved for unlimited release and
has been assigned LA-UR-22-30980. This work has been authored by an employee
of Triad National Security, LLC which operates Los Alamos National Laboratory
under Contract No. 89233218CNA000001 with the U.S. Department of
Energy/National Nuclear Security Administration. The publisher, by accepting
the article for publication, acknowledges that the United States Government
retains a non-exclusive, paid-up, irrevocable, world-wide license to publish
or reproduce the published form of the manuscript, or allow others to do so,
for United States Government purposes.

## References

  * [1] Joel B Miller,  “Nuclear quadrupole resonance detection of explosives,”  in Counterterrorist Detection Techniques of Explosives, pp. 157–198. Elsevier, 2007.
  * [2] Allen N Garroway, Michael L Buess, Joel B Miller, Bryan H Suits, Andrew D Hibbs, Geoffrey A Barrall, Robert Matthews, and Lowell J Burnett,  “Remote sensing by nuclear quadrupole resonance,”  IEEE Transactions on Geoscience and Remote Sensing, vol. 39, no. 6, pp. 1108–1118, 2001.
  * [3] Alexander A Koukoulas and MA Whitehead,  “Observations in nuclear quadrupole resonance frequency temperature dependence,”  Chemical Physics Letters, vol. 167, no. 5, pp. 379–382, 1990.
  * [4] K Ramani, S Ganapathy, and R Srinivasan,  “A Fourier transform approach to Voigt profile analysis and its application to nuclear magnetic resonance,”  Journal of Magnetic Resonance (1969), vol. 24, no. 2, pp. 231–237, 1976.
  * [5] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou,  “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.,”  Journal of Machine Learning Research, vol. 11, no. 12, 2010.
  * [6] Yi Luo and Nima Mesgarani,  “Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 8, pp. 1256–1266, 2019.
  * [7] Cristian Monea and Nicu Bizon,  Signal Processing and Analysis Techniques for Nuclear Quadrupole Resonance Spectroscopy,  Springer Nature, 2021.
  * [8] Qi Shan,  “EMD and wavelet methods for interference cancellation in quadrupole resonance detection,”  Journal of Research in Science and Engineering (JRSE), vol. 4, no. 5, pp. 149–158, 2022.
  * [9] Samuel D Somasundaram, Andreas Jakobsson, and Naveed R Butt,  “Countering radio frequency interference in single-sensor quadrupole resonance,”  IEEE Geoscience and Remote Sensing Letters, vol. 6, no. 1, pp. 62–66, 2008.
  * [10] K Thulasiram Varma, Rangababu Peesapati, Ch V Rama Rao, AK Rajarajan, Makarand Dixit, and Gopal Joshi,  “A novel SSA-NLLSF approach for denoising NQR signals,”  Applied Magnetic Resonance, vol. 52, no. 11, pp. 1601–1613, 2021\.
  * [11] Cristian Monea,  “Enhancing deep learning nuclear quadrupole resonance detection using transfer learning and autoencoders,”  Expert Systems with Applications, vol. 207, pp. 118093, 2022.
  * [12] C Trabelsi, O Bilaniuk, Y Zhang, D Serdyuk, S Subramanian, JF Santos, S Mehri, N Rostamzadeh, Y Bengio, and CJ Pal,  “Deep complex networks,”  arXiv preprint arXiv:1705.09792, 2017.
  * [13] Karn N Watcharasupat, Thi Ngoc Tho Nguyen, Woon-Seng Gan, Shengkui Zhao, and Bin Ma,  “End-to-end complex-valued multidilated convolutional neural network for joint acoustic echo cancellation and noise suppression,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 656–660.
  * [14] Akira Hirose and Shotaro Yoshida,  “Generalization characteristics of complex-valued feedforward neural networks in relation to signal coherence,”  IEEE Transactions on Neural Networks and Learning Systems, vol. 23, no. 4, pp. 541–551, 2012.
  * [15] Natalie Klein and Amber Day,  “SplitML,” https://github.com/lanl/SplitML, 2022,  Accessed: 2022-10-19.
  * [16] Yao-Yuan Yang, Moto Hira, Zhaoheng Ni, Artyom Astafurov, Caroline Chen, Christian Puhrsch, David Pollack, Dmitriy Genzel, Donny Greenberg, Edward Z Yang, et al.,  “Torchaudio: Building blocks for audio and speech processing,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6982–6986.
  * [17] Vinith Kishore, Nitya Tiwari, and Periyasamy Paramasivam,  “Improved speech enhancement using TCN with multiple encoder-decoder layers,”  in Interspeech, 2020, pp. 4531–4535.
  * [18] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell,  “Simple and scalable predictive uncertainty estimation using deep ensembles,”  Advances in Neural Information Processing Systems, vol. 30, 2017\.
  * [19] David S Broomhead and Gregory P King,  “Extracting qualitative dynamics from experimental data,”  Physica D: Nonlinear Phenomena, vol. 20, no. 2-3, pp. 217–236, 1986\.
  * [20] Haifeng Wang, Jian-Feng Cai, Tianming Wang, and Ke Wei,  “Fast Cadzow’s algorithm and a gradient variant,”  Journal of Scientific Computing, vol. 88, no. 2, pp. 1–21, 2021\.

##  Appendix A Appendix for “Denoising neural networks for magnetic resonance
spectroscopy”

Authors: Natalie Klein⋆ Amber J. Day⋆† Harris Mason⋆ Michael W. Malone⋆ Sinead
A. Williamson†
⋆Los Alamos National Laboratory, Los Alamos, NM 87545, USA
†Department of Statistics and Data Sciences, University of Texas at Austin,
Austin, TX 78712, USA

###  A.1 Dataset details

When generating synthetic data based on Equation 1, we use a time vector with
1,024 points at a sample spacing of $1.8\times 10^{-5}$ seconds to correspond
to the experimental data. We draw parameters $T_{2}$ and $\sigma$
independently according to a Uniform($1\times 10^{-3},1\times 10^{-2})$
distribution. Phase shift $\phi$ is drawn according to a Uniform($-\pi,\pi$)
distribution. The noise is drawn from one of two distributions: white Gaussian
noise (‘white’) and experimental noise measurements from a nuclear quadrupole
resonance (NQR) instrument with no relevant signal present (‘measured’). The
measured noise consists of 4,096 independent measurements of noise at the same
sampling rate with duration 16,384 samples; we first split the independent
measurements into training, validation, and test sets before dividing into
pieces of length 1,024 to ensure no single measurement vector was reused in
the validation or test sets. We consider signals centered at frequency $f\sim
0$Hz (to emulate complex-demodulated data) and signals centered at a higher
frequency $f\sim F$, with $F=1600$Hz (chosen to have signal power in a
frequency band for which the measured noise has elevated power). Given a
choice of $f$, the frequency of an individual signal in the data set is drawn
as Uniform($f-150,f+150$). We control the signal-to-noise ratio (SNR) of the
signals by varying $A$ relative to standardized noise (variance equal to one),
with a low-SNR regime, $A\sim\text{Uniform}(0.1,1.0)$, and a high-SNR regime,
$A\sim\text{Uniform}(2.1,3.0)$. The experimental data comes from nuclear
magnetic resonance measurements of NaNO2. Complex demodulation at the expected
resonance frequency was performed during data collection, resulting in
complex-valued time series data expected to be centered at $f\sim 0$Hz. It
consists of the same time vector as the synthetically generated signal and
upon inspection was seen to have high SNR, with visible exponential decay
trend in the noisy signals.

###  A.2 SNR across datasets

Signal-to-noise ratio (SNR) is defined as the ratio of the power of a signal
($P_{signal}$) to the power of background noise ($P_{noise}$),
$\text{SNR}=\frac{P_{signal}}{P_{noise}}$. We reference the logarithmic
decibel scale of the SNR (SNRdB) for ease of comparison:
$\text{SNR}_{dB}=10\log_{10}\left(\frac{P_{signal}}{P_{noise}}\right)=10\log_{10}(\text{SNR})$.
Since the SNRdB values are approximately the same across noise distributions
and between the in-distribution and out-of-distribution datasets, we have only
reported the values for the white-noise and in-distribution datasets.

SNR Type | Frequency | Minimum SNRdB | Average SNRdB | Maximum SNRdB
---|---|---|---|---
low | $f\sim 0$ | -35.6 | -17.7 | -7.3
low | $f\sim 1600$ | -35.2 | -17.0 | -7.2
high | $f\sim 0$ | -10.0 | -3.1 | 2.5
high | $f\sim 1600$ | -9.7 | -2.4 | 2.6
Table A3: Observed minimum, average, and maximum SNR (dB) in the synthetic
datasets.

###  A.3 Complex- and real-valued NN approaches

In the main text, we focused on CV-NNs compared to one RV-NN approach working
on concatenated real and imaginary components. For the fully-connected AE, the
real and imaginary components were directly concatenated into a single vector;
for the convolutional CTN, the real and imaginary components were treated as
two channels. This approach allows the NN to learn relationships between the
real and imaginary components of the signal. We also investigated two other
RV-NN approaches (Fig. A4), but found the performance was consistently worse.
In particular, we considered one RV-NN applied independently to the real and
imaginary components, or separate RV-NNs applied independently to the real and
imaginary components. Neither of these approaches considers relationships
between the real and imaginary components, which could explain the poorer
performance.

Complex DataCV-NNComplex Reconstruction

Complex DataReal ComponentImaginary ComponentRV-NNReal ReconstructionImaginary
ReconstructionComplex Reconstruction

Complex DataReal ComponentImaginary ComponentRV-NNRV-NNReal
ReconstructionImaginary ReconstructionComplex Reconstruction

Complex Data Real Component Imaginary Component RV-NN Real Component Imaginary
Component Complex Reconstruction

Fig. A4: CV-NN and RV-NN architectures. From left to right, ComplexNet,
DualReal1, DualReal2, DualReal1C. ComplexNet operates directly on complex data
with complex-valued activation and loss functions. DualReal1 splits data into
real and imaginary but processes them with a single real-valued network.
DualReal2 splits into real and imaginary and processes them with two separate
real-valued networks. DualReal1C concatenates the real and imaginary
components into a single real-valued vector (or two-channel vector) and then
processes them with a single real-valued network which allows the model to
learn unconstrained relationships between real and imaginary components. The
other models are either unable to learn these relationships at all (DualReal1
and DualReal2) or have some symmetries imposed on the relationships
(ComplexNet).

###  A.4 Autoencoder implementation details

Four different undercomplete autoencoder architectures were implemented (Fig.
A4). Each of these four architectures were trained in the following way. The
full 1024 dimensional space was mapped down to 10 dimensions, then 5
dimensions, before being mapped back to 10 dimensions and finally the full
1024 dimensions. Training was performed using the Adam optimizer. Each epoch
trained on the full data set. The maximum number of training epochs was 1000
with early stopping after 5 consecutive increases in validation loss. The loss
was calculated as denoising loss, meaning the squared loss was calculated
between the reproduction from the model and the clean version of the signal. A
learning rate of 0.01 was used. The complex-valued and real-valued MSE loss
function and ReLU activation function were implemented since they had the best
overall performance when tested against complex ReLU, ReLU, complex cardioid,
complex phase TanH, Hardtanh for activation functions and complex MSE, MSE,
complex logMSE, logMSE, complex sdr, L1Loss, SNRloss for loss functions.

###  A.5 ConvTasNet implementation details

We based our implementation on the torchaudio implementation of ConvTasNet but
with two important modifications: first, we used a deep rather than single
layer encoder/decoder, and second, we developed an adaptation to use complex-
valued convolutions and activation functions. We set the depth of the encoder
and decoder to three layers. The key variation we explored in deviating from
default torchaudio hyperparameter settings was changing the window size to 32,
64, or 128 samples. For the complex-valued CTN, we used complex convolutions
and complex PReLU activation functions to mirror the real-valued CTN [12]. Due
to GPU memory constraints, we used minibatch training with a batch size of 256
signals. We optimized the networks for a maximum of 50 epochs with the Adam
optimizer (learning rate: $3\times 10^{-3}$), keeping the model with the
highest validation error as a form of early stopping. We found that using the
log of the squared reconstruction error resulted in more stable training for
CTN; this loss function is similar to an SNR-based loss.

###  A.6 Baseline denoising method details

The time-domain deconvolution method uses least_squares from the Python
scipy.optimize package to fit the generating signal model to the noisy data.
We used knowledge of the data generating distribution to set the scale and
initial values of the parameters to be optimized, so that the method would
have in a sense the best performance possible, but we note that in practice,
the procedure can be very sensitive to these generally unknown values. The SSA
method was implemented using the Fast Gradient Cadzow approach [20]; it works
by creating a lagged time series matrix from each signal, then performing SVD,
and iteratively thresholding to determine which components correspond to
signal. The wavelets baseline method processes the amplitude of the raw NQR
signal using the Python package PyWavelets with sym4 wavelet, 12
decompositions, Bayes denoising, and soft thresholding [8].

###  A.7 Complex- and real-valued autoencoder comparisons

Comparison of the performance of four different autoencoders, ComplexNet,
DualReal1, DualReal2, and DualReal1C across different random initialization
providing insight into each model’s sensitivity to initialization. It can be
seen that DualReal1C generally has the lowest variability across seeds and the
lowest MSE. We believe it performs best due to the density of its layers, as
it has the highest number of weights, and because of the flexibility in
learning the relationships between the real and imaginary components of the
data. While ComplexNet is over-all outperformed by DualReal1C, more often than
not, it outperforms the more naïve RV-NNs, DualReal1, and DualReal2. We
believe this is due to its ability to learn symmetrically constrained
relationships between the real and imaginary components of the data, which
DualReal1 and DualReal2 lack. Other things made apparent by A4 is that the
sensitivity to initialization is much higher for all autoencoders across the
low-SNR data sets and is much lower across the high-SNR data sets.

| White noise | Measured noise
---|---|---
| $f\sim 0$ | $f\sim F$ | $f\sim 0$ | $f\sim F$
Model | Mean | SD | Mean | SD | Mean | SD | Mean | SD
ComplexNet | 50.6 | 29.8 | 51.4 | 31.7 | 48.1 | 23.6 | 53.3 | 28.4
DualReal1 | 61.8 | 23.5 | 77.2 | 21.3 | 60.9 | 22.3 | 81.4 | 22.9
DualReal2 | 61.4 | 23.3 | 72.7 | 14.7 | 58.0 | 22.2 | 74.2 | 17.6
DualReal1C | 33.0 | 13.0 | 36.1 | 10.8 | 32.1 | 12.1 | 41.9 | 11.5

(a) low SNR

| White noise | Measured noise
---|---|---
| $f\sim 0$ | $f\sim F$ | $f\sim 0$ | $f\sim F$
Model | Mean | SD | Mean | SD | Mean | SD | Mean | SD
ComplexNet | 6.03 | 0.87 | 5.80 | 0.67 | 6.51 | 0.96 | 6.94 | 1.28
DualReal1 | 3.16 | 0.44 | 7.95 | 0.92 | 2.99 | 0.42 | 7.70 | 1.28
DualReal2 | 4.78 | 0.37 | 9.96 | 0.48 | 4.65 | 0.44 | 10.4 | 1.34
DualReal1C | 4.96 | 0.46 | 2.30 | 0.34 | 5.19 | 0.43 | 2.30 | 0.23

(b) high SNR

Table A4: Comparing mean $\times 10^{-3}$ and standard deviation (SD $\times
10^{-3}$) of MSE from four different autoencoders.

###  A.8 Complex- and real-valued ConvTasNet comparisons

For a fixed window size (128; results similar for other window sizes), we
compare the different complex- and real-valued ConvTasNet models across
different seeds. Table A5 shows the mean and SD of the rescaled MSE of each
model in the ensemble for the different complex- and real-valued networks
across different training regimes; boxplots below the tables show the
distributions of rescaled MSE across each ensemble. In low SNR, the complex-
valued CTN appears to outperform the DualReal1C CTN with smaller ensemble
variation; the other real-valued approaches (DualReal1 and DualReal2) perform
significantly worse. In high SNR, the DualReal1C model appears to perform
slightly better than the complex CTN.

| White noise | Measured noise
---|---|---
| $f\sim 0$ | $f\sim F$ | $f\sim 0$ | $f\sim F$
Model | Mean | SD | Mean | SD | Mean | SD | Mean | SD
ComplexNet | 19.38 | 0.23 | 19.24 | 0.16 | 15.43 | 0.21 | 17.25 | 0.10
DualReal1 | 24.15 | 0.28 | 33.59 | 0.74 | 19.34 | 0.24 | 27.62 | 0.38
DualReal2 | 24.19 | 0.30 | 33.29 | 0.65 | 19.14 | 0.23 | 27.18 | 0.18
DualReal1C | 19.80 | 0.13 | 19.74 | 0.27 | 16.04 | 0.30 | 18.07 | 0.40

(a) low SNR

| White noise | Measured noise
---|---|---
| $f\sim 0$ | $f\sim F$ | $f\sim 0$ | $f\sim F$
Model | Mean | SD | Mean | SD | Mean | SD | Mean | SD
ComplexNet | 0.67 | 0.02 | 0.66 | 0.01 | 0.56 | 0.02 | 0.62 | 0.01
DualReal1 | 0.77 | 0.02 | 1.01 | 0.01 | 0.61 | 0.02 | 0.77 | 0.01
DualReal2 | 0.84 | 0.02 | 1.13 | 0.01 | 0.66 | 0.01 | 0.86 | 0.01
DualReal1C | 0.64 | 0.01 | 0.62 | 0.01 | 0.52 | 0.01 | 0.57 | 0.02

(b) high SNR

Table A5: Comparing mean $\times 10^{-3}$ and standard deviation (SD $\times
10^{-3}$) of MSE from four different ConvTasNet.

###  A.9 ConvTasNet window selection

Based on previous work, we determined that varying the window size in
ConvTasNet would be the manipulation most likely impact model performance, so
we trained models with window lengths of 32, 64, and 128 (in number of
samples). Because Table 1 suggests that the complex-valued ConvTasNet was
better overall than the real-valued ConvTasNet across all window sizes, we
here investigate how window size impacted the performance of the complex-
valued ConvTasNet. Table 6(a) shows results for low SNR, where the table gives
the mean and standard deviation of the rescaled test set MSE across the
ensemble of models, with corresponding boxplots below the table showing the
distribution of rescaled MSE values for each window length (across the
ensemble members). In this case, it appears that the longest window length
performed best across all training data sets. Interestingly, the opposite
appeared to be true for high SNR (Table 6(b)), which suggests that smaller
window sizes performed better, though the differences between window sizes
were less pronounced than low SNR. These results suggest that in high SNR, the
window length is less impactful on the model performance, while in low SNR,
the longer window length tends to provide better performance, perhaps because
more total signal and noise information is contained in each window.

| White noise | Measured noise
---|---|---
| $f\sim 0$ | $f\sim F$ | $f\sim 0$ | $f\sim F$
Model | Mean | SD | Mean | SD | Mean | SD | Mean | SD
32 | 20.60 | 0.36 | 21.15 | 0.36 | 16.50 | 0.26 | 19.02 | 0.20
64 | 20.74 | 0.37 | 20.79 | 0.54 | 16.87 | 0.23 | 18.67 | 0.26
128 | 19.38 | 0.23 | 19.24 | 0.16 | 15.43 | 0.21 | 17.25 | 0.10

(a) low SNR

| White noise | Measured noise
---|---|---
| $f\sim 0$ | $f\sim F$ | $f\sim 0$ | $f\sim F$
Model | Mean | SD | Mean | SD | Mean | SD | Mean | SD
32 | 0.67 | 0.04 | 0.61 | 0.02 | 0.52 | 0.03 | 0.55 | 0.02
64 | 0.68 | 0.04 | 0.65 | 0.03 | 0.55 | 0.01 | 0.56 | 0.01
128 | 0.67 | 0.02 | 0.66 | 0.01 | 0.56 | 0.02 | 0.62 | 0.01

(b) high SNR

Table A6: Comparing mean $\times 10^{-3}$ and standard deviation (SD $\times
10^{-3}$) of MSE from different complex ConvTasNet window sizes.

###  A.10 Out-of-distribution data and results

We generated out-of-distribution (OOD) test sets in two ways. First, we
applied trained neural networks (NNs) to test sets that had different noise
distributions (trained on white noise, applied to data with measured noise, or
vice versa) with all other data characteristics held fixed. To measure
performance, we calculated a standardized mean squared error (MSE), in which
each true signal and prediction was first rescaled by the true signal
amplitude $A$. As a measure of ensemble uncertainty, we calculated the
standard deviation (SD) in the individual model MSE values. Table A7 shows
that the NN performance in terms of ensemble mean squared error (MSE) remains
stable when the noise distribution changes, and that the ensemble standard
deviation (SD) also remains stable. These results suggest that the networks
are robust to some changes in the noise distribution at deployment.

SNR | $f$ | w $\xrightarrow{}$ w | w $\xrightarrow{}$ m | m $\xrightarrow{}$ m | m $\xrightarrow{}$ w
---|---|---|---|---|---
high | 0 | 0.54(0.04) | 0.44(0.03) | 0.44(0.03) | 0.56(0.03)
high | $F$ | 0.50(0.02) | 0.48(0.02) | 0.45(0.02) | 0.51(0.02)
low | 0 | 17.9(0.23) | 14.6(0.19) | 14.1(0.21) | 18.3(0.28)
low | $F$ | 17.8(0.16) | 17.1(0.26) | 15.8(0.10) | 18.2(0.17)
(a) Ensemble MSE(SD) $\times 10^{-3}$ for CTN (CV) models.

SNR | $f$ | w $\xrightarrow{}$ w | w $\xrightarrow{}$ m | m $\xrightarrow{}$ m | m $\xrightarrow{}$ w
---|---|---|---|---|---
high | 0 | 1.86(0.25) | 1.81(0.16) | 1.89(0.13) | 1.95(0.12)
high | $F$ | 4.36(0.22) | 4.28(0.23) | 4.10(0.35) | 4.17(0.34)
low | 0 | 22.5(5.68) | 22.3(0.06) | 21.8(0.87) | 22.4(8.44)
low | $F$ | 23.6(4.83) | 23.4(5.28) | 29.3(6.34) | 29.5(5.75)
(b) Ensemble MSE(SD) $\times 10^{-3}$ for AE (DualReal1C) models.

Table A7: Ensemble MSE(SD) $\times 10^{-3}$ for NN models trained under four
SNR/central frequency conditions (rows); columns indicate training and test
noise distributions, where ‘w’ is white noise and ‘m’ is measured noise. Here,
w $\rightarrow$ w indicates that the model was trained with white noise and
applied to test data with white noise, while w $\rightarrow$ m indicates the
model was trained with white noise and applied to test data with measured
noise (OOD), and similar for the remaining two columns. Generally, moving from
in-distribution to OOD test sets does not increase MSE or ensemble
uncertainty, indicating robustness to noise distribution.

Second, we generated test sets that had different signal frequency
distributions ($f\sim 300$Hz or $f\sim 1900$Hz) with the frequency of an
individual example drawn as Uniform($f-150,f+150$). This means that during
test time the signal frequencies are shifted so that they do not overlap the
frequencies seen during training, but are still relatively close to the
training frequencies. Table A8 shows that performance degrades substantially
in this case. In high SNR, the CTN ensemble SD increases, suggesting that the
ensemble uncertainty could help indicate OOD data, but this pattern does not
hold in low SNR. This result suggests that training sets should be designed to
contain the range of frequencies anticipated during deployment.

| White noise | Measured noise
---|---|---
Trained | $f\sim 0$ | $f\sim 1600$ | $f\sim 0$ | $f\sim 1600$
Tested | $f\sim 0$ | $f\sim 300$ | $f\sim 1600$ | $f\sim 1900$ | $f\sim 0$ | $f\sim 300$ | $f\sim 1600$ | $f\sim 1900$
AE | 1.86(0.25) | 29.5(0.32) | 4.36(0.22) | 35.0(0.18) | 1.89(0.13) | 29.7(0.09) | 4.10(0.35) | 34.7(0.15)
CTN | 0.54(0.04) | 16.3(1.26) | 0.50(0.02) | 21.5(3.44) | 0.44(0.03) | 14.7(1.42) | 0.45(0.02) | 18.1(4.09)
(a) Relative MSE for out of distribution high-SNR data.

| White noise | Measured noise
---|---|---
Trained | $f\sim 0$ | $f\sim 1600$ | $f\sim 0$ | $f\sim 1600$
Tested | $f\sim 0$ | $f\sim 300$ | $f\sim 1600$ | $f\sim 1900$ | $f\sim 0$ | $f\sim 300$ | $f\sim 1600$ | $f\sim 1900$
AE | 22.5(5.68) | 47.1(1.41) | 23.6(4.83) | 48.6(1.13) | 21.8(0.87) | 47.1(1.32) | 29.3(6.34) | 48.4(1.16)
CTN | 17.9(0.23) | 65.2(0.24) | 17.8(0.16) | 65.9(0.45) | 14.1(0.21) | 59.5(0.26) | 15.8(0.10) | 63.2(0.67)
(b) Relative MSE for out of distribution low-SNR data.

Table A8: Ensemble MSE(SD) $\times 10^{-3}$ for NN models trained with central
frequencies near 0 and 1600Hz applied to test data at 0 or 1600Hz (in-
distribution) or 300 or 1900Hz (OOD). Moving from in-distribution to OOD
signals results in higher MSE across both NN models. In high SNR, the ensemble
uncertainty increases when moving to OOD data, suggesting it could be used as
an indicator of OOD data during deployment. However, the same pattern does not
hold for low SNR data.


latexFont shape latexfontFont shape latexfontSize substitutions

# SAM Struggles in Concealed Scenes –
Empirical Study on “Segment Anything”

Ge-Peng Ji, Deng-Ping Fan, Peng Xu, Ming-Ming Cheng, Bowen Zhou, Luc Van Gool
Ge-Peng Ji is with the College of Engineering, Computing & Cybernetics, ANU,
Canberra, Australia. Deng-Ping Fan and Luc Van Gool are with the Computer
Vision Lab (CVL), ETH Zurich, Zurich, Switzerland. Peng Xu and Bowen Zhou are
with the Department of Electronic Engineering, Tsinghua University, Beijing,
China. Ming-Ming Cheng is with the Nankai University, Tianjin, China.

###### Abstract

Segmenting anything is a ground-breaking step toward artificial general
intelligence, and the Segment Anything Model (SAM) greatly fosters the
foundation models for computer vision. We could not be more excited to probe
the performance traits of SAM. In particular, exploring situations in which
SAM does not perform well is interesting. In this report, we choose three
concealed scenes, i.e., camouflaged animals, industrial defects, and medical
lesions, to evaluate SAM under unprompted settings. Our main observation is
that SAM looks unskilled in concealed scenes.

###### Index Terms:

Segment Anything, SAM, Camouflaged, Concealed Scene Understanding, Concealed
Object Segmentation.

##  1 Introduction

Large models open up new opportunities for artificial intelligence. In the
past few months, there has been a boom in training foundation models on the
vast linguistic corpus to produce amazing applications, e.g.,
ChatGPT111https://chat.openai.com, GPT-4222https://openai.com/research/gpt-4.
Both natural language processing and multimodal learning communities have been
revolutionized. Large models’ capacity for generalization and emergent makes
it easy for users to believe that large models can solve anything.

Last week, the “Segment Anything” [1] project was released, and its Segment
Anything Model (SAM) is a large ViT-based model trained on the large visual
corpus (SA-1B). This is a ground-breaking step toward artificial general
intelligence, as SAM demonstrates promising segmentation capabilities in
various scenarios and the great potential of the foundation models for
computer vision. Like all computer vision researchers, we cannot wait to probe
the performance traits of SAM to help the community to comprehend it further.
Moreover, it is interesting to explore the situations in which SAM does not
work well.

In this paper, we compare SAM quantitatively with cutting-edge models on
camouflaged object segmentation task, and present diversified visualization
results in three concealed scenes333There is a newly released survey on deep
concealed scene understanding [2] (Project link:
https://github.com/DengPingFan/CSU). [2], i.e., camouflaged animals,
industrial defects, and medical lesions. Our main observation is that SAM
looks not skillful in concealed scenes.

##  2 Experiment

We use three frequently-used camouflaged object segmentation (COS) benchmarks
to evaluate SAM. Next, we describe our mask selection strategy (§2.1) and
evaluation protocols (§2.2), compare SAM with the state-of-the-art models of
COS (§2.3), and provide qualitative visualisations on three concealed
scenarios (§2.4).

###  2.1 Mask Selection Strategy

If under the unprompted setting, SAM generates multiple binary masks and can
pop out several potential objects within an input. For a fair evaluation of
interesting regions in a specific segmentation task, we take a strategy to
select the most appropriate mask based on its ground-truth mask. Formally,
given $N$ binary predictions $\\{\mathbf{P}_{n}\\}_{n=1}^{N}$ and the ground-
truth $\mathbf{G}$ for an input image, we calculate intersection over union
(IoU) scores for each pair to generate a set of evaluation scores
$\\{\text{IoU}_{n}\\}_{n=1}^{N}$. We finally select the mask with the highest
IoU score from this set.

###  2.2 Evaluation Protocols

Our protocols are following the standard practice as in [3].

$\bullet$ Datasets. We use three commonly-used COS benchmarks in our
experiments, including CAMO [4] (250 samples), COD10K [5] (2,026 samples), and
NC4K [6] (4,121 samples).

$\bullet$ Models. To ensure a fair comparison with SAM, we choose the current
top-performing COS models using transformer architecture, i.e.,
CamoFormer-P/-S [7], HitNet [8].

$\bullet$ Metrics. We use five commonly-used metrics for the evaluation:
structure measure ($S_{\alpha}$) [9], enhanced-alignment measure ($E_{\phi}$)
[10], F-measure ($F_{\beta}$) [11, 12], weighted F-measure ($F_{\beta}^{w}$)
[13], and mean absolute error ($M$). According to different thresholding
strategies, the adaptive/mean/maximum values of F-measure and E-measure are
reported. We denote different $E_{\phi}$ scores as $E_{\phi}^{ad}$,
$E_{\phi}^{mn}$, and $E_{\phi}^{mx}$.

###  2.3 Quantitative Evaluation

We report the quantitative comparison in Table I, SAM demonstrates significant
improvements as model capabilities increase from ViT-B to ViT-L, with an
increase in $F_{\beta}^{w}$ score from $0.353$ to $0.655$ on CAMO. However,
the improvement is limited when the model becomes larger, increasing only from
$0.655$ (ViT-L) to $0.700$ (ViT-H). Moreover, we observe that there remains a
large gap between SAM even with ViT-H and current top-performing COS models on
three benchmarks. For example as presented in Table I, the difference of
$E_{\phi}^{mx}$ score between SAM (ViT-H) and CamoFormer-S [7] reaches 13.8%
on COD10K dataset, 25.6% on CAMO dataset, and 16.9% on NC4K dataset. This gap
indicates that the perception ability of SAM needs further improvement for
concealed scenes.

TABLE I: Quantitative comparison on three popular COS benchmarks. The symbols $\uparrow$/$\downarrow$ indicate that a higher/lower score is better. The highest scores are marked in bold. $\Delta$ represents the difference between SAM and the highest score achieved by current cutting-edge COS models. | Model | Pub/Year | Backbone | $S_{\alpha}\uparrow$ | $F_{\beta}^{w}\uparrow$ | $M\downarrow$ | $E_{\phi}^{ad}\uparrow$ | $E_{\phi}^{mn}\uparrow$ | $E_{\phi}^{mx}\uparrow$ | $F_{\beta}^{ad}\uparrow$ | $F_{\beta}^{mn}\uparrow$ | $F_{\beta}^{mx}\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---|---
COD10K [5] |  CamoFormer-P [7] |  arXiv23 |  PVTv2-B4 [14] | 0.869 | 0.786 | 0.023 | 0.931 | 0.932 | 0.939 | 0.794 | 0.811 | 0.829
CamoFormer-S [7] |  arXiv23 |  Swin-B [15] | 0.862 | 0.772 | 0.024 | 0.932 | 0.931 | 0.941 | 0.780 | 0.799 | 0.818
HitNet [8] |  AAAI23 |  PVTv2-B2 [14] | 0.871 | 0.806 | 0.023 | 0.936 | 0.935 | 0.938 | 0.818 | 0.823 | 0.838
SAM [1] | arXiv23 |  ViT-B [16] | 0.585 | 0.353 | 0.108 | 0.535 | 0.533 | 0.535 | 0.423 | 0.422 | 0.423
Difference ($\Delta$) | -28.6% | -45.3% | +8.5% | -40.1% | -40.2% | -40.3% | -39.5% | -40.1% | -41.5%
ViT-L [16] | 0.751 | 0.655 | 0.065 | 0.766 | 0.764 | 0.766 | 0.718 | 0.716 | 0.718
Difference ($\Delta$) | -12% | -15.1% | +4.2% | -17% | -17.1% | -17.2% | -10% | -10.7% | -12%
ViT-H [16] | 0.781 | 0.700 | 0.054 | 0.800 | 0.798 | 0.800 | 0.756 | 0.754 | 0.756
Difference ($\Delta$) | -9% | -10.6% | +3.1% | -13.6% | -13.7% | -13.8% | -6.2% | -6.9% | -8.2%
| Model | Pub/Year | Backbone | $S_{\alpha}\uparrow$ | $F_{\beta}^{w}\uparrow$ | $M\downarrow$ | $E_{\phi}^{ad}\uparrow$ | $E_{\phi}^{mn}\uparrow$ | $E_{\phi}^{mx}\uparrow$ | $F_{\beta}^{ad}\uparrow$ | $F_{\beta}^{mn}\uparrow$ | $F_{\beta}^{mx}\uparrow$
CAMO [4] |  CamoFormer-P [7] |  arXiv23 |  PVTv2-B4 [14] | 0.872 | 0.831 | 0.046 | 0.931 | 0.929 | 0.938 | 0.853 | 0.854 | 0.868
CamoFormer-S [7] |  arXiv23 |  Swin-B [15] | 0.876 | 0.832 | 0.043 | 0.935 | 0.930 | 0.938 | 0.856 | 0.856 | 0.871
HitNet [8] |  AAAI23 |  PVTv2-B2 [14] | 0.849 | 0.809 | 0.055 | 0.910 | 0.906 | 0.910 | 0.833 | 0.831 | 0.838
SAM [1] | arXiv23 |  ViT-B [16] | 0.462 | 0.238 | 0.219 | 0.402 | 0.401 | 0.402 | 0.312 | 0.312 | 0.312
Difference ($\Delta$) | -41.4% | -59.4% | +17.6% | -53.3% | -52.9% | -53.6% | -54.4% | -54.4% | -55.9%
ViT-L [16] | 0.630 | 0.534 | 0.162 | 0.628 | 0.626 | 0.628 | 0.617 | 0.615 | 0.617
Difference ($\Delta$) | -24.6% | -29.8% | +11.9% | -30.7% | -30.4% | -31% | -23.9% | -24.1% | -25.4%
ViT-H [16] | 0.677 | 0.594 | 0.136 | 0.682 | 0.680 | 0.682 | 0.670 | 0.668 | 0.670
Difference ($\Delta$) | -19.9% | -23.8% | +9.3% | -25.3% | -25% | -25.6% | -18.6% | -18.8% | -20.1%
| Model | Pub/Year | Backbone | $S_{\alpha}\uparrow$ | $F_{\beta}^{w}\uparrow$ | $M\downarrow$ | $E_{\phi}^{ad}\uparrow$ | $E_{\phi}^{mn}\uparrow$ | $E_{\phi}^{mx}\uparrow$ | $F_{\beta}^{ad}\uparrow$ | $F_{\beta}^{mn}\uparrow$ | $F_{\beta}^{mx}\uparrow$
NC4K [6] |  CamoFormer-P [7] |  arXiv23 |  PVTv2-B4 [14] | 0.892 | 0.847 | 0.030 | 0.941 | 0.939 | 0.946 | 0.863 | 0.868 | 0.880
CamoFormer-S [7] |  arXiv23 |  Swin-B [15] | 0.888 | 0.840 | 0.031 | 0.941 | 0.937 | 0.946 | 0.857 | 0.863 | 0.877
HitNet [8] |  AAAI23 |  PVTv2-B2 [14] | 0.875 | 0.834 | 0.037 | 0.928 | 0.926 | 0.929 | 0.854 | 0.853 | 0.863
SAM [1] | arXiv23 |  ViT-B [16] | 0.544 | 0.334 | 0.166 | 0.494 | 0.493 | 0.494 | 0.403 | 0.403 | 0.403
Difference ($\Delta$) | -34.8% | -51.3% | +13.6% | -44.7% | -44.6% | -45.2% | -46% | -46.5% | -47.7%
ViT-L [16] | 0.728 | 0.643 | 0.101 | 0.735 | 0.733 | 0.735 | 0.706 | 0.704 | 0.706
Difference ($\Delta$) | -16.4% | -20.4% | +7.1% | -20.6% | -20.6% | -21.1% | -15.7% | -16.4% | -17.4%
ViT-H [16] | 0.763 | 0.696 | 0.087 | 0.777 | 0.775 | 0.777 | 0.752 | 0.750 | 0.752
Difference ($\Delta$) | -12.9% | -15.1% | +5.7% | -16.4% | -16.4% | -16.9% | -11.1% | -11.8% | -12.8%
Figure 1: SAM [1] fails to perceive the animals that are visually “hidden” in
their natural surroundings. All the samples are from COD10K dataset [5].
Figure 2:  SAM [1] is unskilled in detecting concealed defects in industrial
scenes. These samples are taken from KolektorSDD [17], MagneticTile [18], and
MVTecAD [19] datasets. Figure 3:  SAM [1] fails to detect these lesion regions
in various medical modalities. These samples cover the RGB colour modality
from CVC-300 [20] (1st column); the MRI modality from BraTS2021 [21] (2nd
column); the CT modalities from COVID-SemiSeg [22] (3rd column) and MSD [23]
(from 4th to 7th columns).

###  2.4 Qualitative Comparison

We further qualitatively evaluate SAM in three concealed scenarios, and
several interesting findings are as follows. All the visualisation results are
generated by the online demo of SAM. a) Camouflaged animal. As presented in
Fig. 1, it is difficult for SAM to detect concealed animals in their natural
habitat. SAM fails to segment a mantis crouching on a leaf (in the second
column) and a seahorse swimming in an orange coral (in the last column). In
these two cases, SAM struggles to distinguish the target semantics from their
surroundings because the foreground and background share similar appearances
of shape and colour. As a result, SAM becomes more dependent on unreliable
pixel intensity changes along the boundaries. b) Industrial defect. In this
scenario, given a product, it is usually collected by the close-range shots,
occupying a large area in the image, and SAM’s behaviour appears to segment
the main part of the object, such as the screw in the 3rd column and bottle in
the 5th column of Fig. 2. Furthermore, we notice that it is difficult for SAM
to distinguish defective areas from the textured background. For instance,
products with surface cracks (in 1st column) and leather with a poke (in the
6th column) are challenging to identify accurately. This phenomenon is not
surprising because SAM is trained on natural objects with standard sizes and
high-contrast attributes. c) Medical lesion. As illustrated in 1st column of
Fig. 3, we observe that SAM does not handle medical data with concealed
patterns well, such as benign colorectal polys that share similar colours with
the surrounding tissues. The remaining samples in Fig. 3 are grayscale slices
from three-dimensional MRI and CT scans. SAM can roughly segment the organ
regions since they have distinct boundaries, but it does not perform well in
recognizing amorphous lesion regions, e.g., cancer, vessels, and tumours. This
suggests that SAM lacks the medical domain knowledge of these anatomical and
pathological cases. To alleviate this limitation, the intrinsic relationships
and semantics of anatomical structures can be injected into SAM, such as the
assumption that liver tumours should be inside the liver, rather than the
brain.

###  2.5 Discussion

From the above empirical analyses, our conclusions are:

$\bullet$ We observe that SAM often segments an occluded object into multiple
separated masks, indicating that its semantic capabilities in concealed scenes
can be improved.

$\bullet$ Unlike self-supervised large language models, SAM employs supervised
training, in our experiments its emergent and reasoning abilities have not
been observed. Thus, it would be interesting to try if more challenging
training tasks improve its performance.

$\bullet$ Considering the practical open-set problem, now the granularity and
uncertainty are the bottlenecks of SAM, limiting its applications to the
scenes that require high accuracy, e.g., autonomous driving and clinical
diagnosis. To alleviate this issue, one potential solution is to support the
model with prior knowledge.

$\bullet$ SAM’s great success is demonstrating the power of data-centric AI in
the large model era. We see a significant trend that human feedback-based
learning and large foundation models bring new opportunities for the vision
community.

##  3 Conclusion

In summary, this paper presents an empirical study of SAM. Firstly, we
quantitatively evaluate SAM using cutting-edge models on the camouflaged
object segmentation task. Secondly, we present several failure cases in three
concealed scenarios: camouflaged animals, industrial defects, and medical
lesions. We expect that this paper helps the readers to comprehend SAM’s
performance traits in concealed scenes and brings new ideas to computer vision
researchers.

## References

  * [1] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo _et al._ , “Segment anything,” _arXiv preprint arXiv:2304.02643_ , 2023.
  * [2] D.-P. Fan, G.-P. Ji, P. Xu, M.-M. Cheng, C. Sakaridis, and L. Van Gool, “Advances in deep concealed scene understanding,” _arXiv preprint arXiv:2304.11234_ , 2023.
  * [3] D.-P. Fan, G.-P. Ji, M.-M. Cheng, and L. Shao, “Concealed object detection,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 44, no. 10, pp. 6024–6042, 2022.
  * [4] T.-N. Le, T. V. Nguyen, Z. Nie, M.-T. Tran, and A. Sugimoto, “Anabranch network for camouflaged object segmentation,” _Comput. Vis. Image Underst._ , vol. 184, pp. 45–56, 2019.
  * [5] D.-P. Fan, G.-P. Ji, G. Sun, M.-M. Cheng, J. Shen, and L. Shao, “Camouflaged object detection,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 2777–2787.
  * [6] Y. Lv, J. Zhang, Y. Dai, A. Li, B. Liu, N. Barnes, and D.-P. Fan, “Simultaneously localize, segment and rank the camouflaged objects,” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2021, pp. 11 591–11 601.
  * [7] B. Yin, X. Zhang, Q. Hou, B.-Y. Sun, D.-P. Fan, and L. Van Gool, “Camoformer: Masked separable attention for camouflaged object detection,” _arXiv preprint arXiv:2212.06570_ , 2023.
  * [8] X. Hu, D.-P. Fan, X. Qin, H. Dai, W. Ren, Y. Tai, C. Wang, and L. Shao, “High-resolution iterative feedback network for camouflaged object detection,” in _AAAI Conf. Art. Intell._ , 2023.
  * [9] D.-P. Fan, M.-M. Cheng, Y. Liu, T. Li, and A. Borji, “Structure-measure: A new way to evaluate foreground maps,” in _Int. Conf. Comput. Vis._ , 2017, pp. 4548–4557.
  * [10] D.-P. Fan, G.-P. Ji, X. Qin, and M.-M. Cheng, “Cognitive vision inspired object segmentation metric and loss function,” _SCIENTIA SINICA Informationis_ , vol. 6, p. 6, 2021.
  * [11] A. Borji, M.-M. Cheng, H. Jiang, and J. Li, “Salient object detection: A benchmark,” _IEEE Trans. Image Process._ , vol. 24, no. 12, pp. 5706–5722, 2015.
  * [12] M. Zhuge, D.-P. Fan, N. Liu, D. Zhang, D. Xu, and L. Shao, “Salient object detection via integrity learning,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [13] R. Margolin, L. Zelnik-Manor, and A. Tal, “How to evaluate foreground maps?” in _IEEE Conf. Comput. Vis. Pattern Recog._ , 2014, pp. 248–255.
  * [14] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pvt v2: Improved baselines with pyramid vision transformer,” _Computational Visual Media_ , vol. 8, no. 3, pp. 415–424, 2022.
  * [15] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 10 012–10 022.
  * [16] A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16-16 words: Transformers for image recognition at scale,” in _Int. Conf. Learn. Represent._ , 2021.
  * [17] D. Tabernik, S. Šela, J. Skvarč, and D. Skočaj, “Segmentation-based deep-learning approach for surface-defect detection,” _J. Intell. Manuf._ , vol. 31, no. 3, pp. 759–776, 2020.
  * [18] Y. Huang, C. Qiu, and K. Yuan, “Surface defect saliency of magnetic tile,” _The Visual Computer_ , vol. 36, no. 1, pp. 85–96, 2020.
  * [19] P. Bergmann, K. Batzner, M. Fauser, D. Sattlegger, and C. Steger, “The mvtec anomaly detection dataset: a comprehensive real-world dataset for unsupervised anomaly detection,” _Int. J. Comput. Vis._ , vol. 129, no. 4, pp. 1038–1059, 2021.
  * [20] J. Bernal, J. Sánchez, and F. Vilarino, “Towards automatic polyp detection with a polyp appearance model,” _Pattern Recognition_ , vol. 45, no. 9, pp. 3166–3182, 2012.
  * [21] U. Baid, S. Ghodasara, S. Mohan, M. Bilello, E. Calabrese, E. Colak, K. Farahani, J. Kalpathy-Cramer, F. C. Kitamura, S. Pati _et al._ , “The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification,” _arXiv preprint arXiv:2107.02314_ , 2021.
  * [22] D.-P. Fan, T. Zhou, G.-P. Ji, Y. Zhou, G. Chen, H. Fu, J. Shen, and L. Shao, “Inf-net: Automatic covid-19 lung infection segmentation from ct images,” _IEEE Trans. Med. Imag._ , vol. 39, no. 8, pp. 2626–2637, 2020.
  * [23] M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze, O. Ronneberger, R. M. Summers _et al._ , “The medical segmentation decathlon,” _Nature communications_ , vol. 13, no. 1, p. 4128, 2022.