††thanks: Correspondence<EMAIL_ADDRESS>(L.-S. B.)

# Nuclear Induction Lineshape Modeling via Hybrid SDE and MD Approach

Mohamad Niknam Department of Chemistry and Biochemistry, University of
California Los Angeles, 607 Charles E. Young Drive East, Los Angeles, CA
90095-1059, USA Center for Quantum Science and Engineering, UCLA Louis-S.
Bouchard Department of Chemistry and Biochemistry, University of California
Los Angeles, 607 Charles E. Young Drive East, Los Angeles, CA 90095-1059, USA
Center for Quantum Science and Engineering, UCLA<EMAIL_ADDRESS>
###### Abstract

The temperature dependence of the nuclear free induction decay in the presence
of a magnetic-field gradient was found to exhibit motional narrowing in gases
upon heating, a behavior that is opposite to that observed in liquids. This
has led to the revision of the theoretical framework to include a more
detailed description of particle trajectories, since decoherence mechanisms
depend on histories. In the case of free diffusion and single component, the
new model yields the correct temperature trends. Inclusion of boundaries in
the current formalism is not straightforward. We present a hybrid SDE-MD
(stochastic differential equation - molecular dynamics) approach whereby MD is
used to compute an effective viscosity and the latter is fed to the SDE to
predict the lineshape. The theory is in agreement with experiments. This two-
scale approach, which bridges the gap between short (molecular collisions) and
long (nuclear induction) timescales, paves the way for the modeling of complex
environments with boundaries, mixtures of chemical species and intermolecular
potentials.

††preprint: APS/123-QED

##  I Introduction

During the mid 20th century, when the first nuclear magnetic resonance (NMR)
experiments were performed in liquids, magnetic-field inhomogeneity was the
limiting factor that determined the properties of nuclear induction. Rapid
signal decay is problematic on so many levels, especially because it limits
the spectral resolution. This led to the development of the Hahn echo [1],
where the effects of external magnetic field inhomogeneity are removed by a
$\pi$-rotation about the $x$ axis in a time-reversal experiment. Spin echoes
enabled more accurate studies of intrinsic spin-spin relaxation mechanisms and
molecular structures. Years later, the deliberate creation and modulation of
magnetization by magnetic-field gradients formed the basis of modern
techniques in magnetic resonance imaging (MRI) [2, 3].

Given the ubiquitous nature and uses of magnetic-field gradients in modern
magnetic resonance experiments, it is imperative that the nuclear response
function and its dependence on the sample under study in the presence of
applied gradients be properly understood. A correct understanding of the true
limits and capabilities of the experiment will enable accurate interpretation
of experimental results, as well as inform future developments and
applications. In organic and biological chemistries, for example, NMR
spectroscopy is routinely used as a tool to analyze solution content and
composition for chemical species’ identities and relative content. Limiting
factors that compromise linewidth must be minimized or removed so that spectra
can be obtained of sufficiently high resolution to elucidate molecular
structures. The diffusion of molecules can cause undesirable effects such as
altering lineshape or be used advantageously, for example, by helping probe
transport phenomena. While diffusion can sometimes lead to sharper lines
(e.g., increased spectral resolution due to diffusional averaging of the
intermolecular magnetic dipole interaction), the presence of residual
gradients does create a complex relationship between nuclear induction and
sample properties and geometry that obscures the interpretation of results
that rely on lineshape analysis. Diffusion-weighted readouts have enabled
imaging of gases in lungs [4], operando monitoring of chemical reaction [5, 6,
7] as well as biophysical or mechanical properties of gels and biological
tissues [8, 9]. In materials science and engineering, diffusion processes are
essential to fabrication and synthesis.

In light of these various effects, the observation that in gases lines become
narrow with temperature was rather surprising [7, 10, 11] given that the
conventional theory predicted a broadening instead [1, 12, 13, 14]. In
liquids, a broadening is observed experimentally, and this is in agreement
with theory. This led to a revision of the lineshape theory in gases
accounting for the time history of rapid molecular motions below the timescale
of the NMR experiment [15]. It is critical to account for the history of
molecular motions when the process involves motional averaging due to
collisions. Conventional theory ignores the histories of molecular diffusion
and instead assumes a Gaussian probability distribution for the accumulated
phase of the spins. The Gaussian assumption on phase accumulation, which
neglects the details of molecular trajectories, only appears justified in
liquids, where many collisions occur on the NMR timescale. But in gases the
average separation between molecules is much larger and this assumption is
more difficult to justify. A drawback of the new theory is its increased
complexity. While it provides good agreement between theory and experiments in
the case of free diffusion, it is unclear how one should model the effects of
realistic boundaries. Stochastic differential equations (SDE) can be used to
model molecular trajectories but this still requires validation against
experiments.

In this study, we bridge the gap using molecular dynamics (MD) simulations and
coupling it to an SDE in order to describe the nuclear induction. By modeling
viscosity of fluids using MD, the lineshape is then derived from the viscosity
via its connection to the SDE. Our simulations of particle collisions and
trajectories using Lennard-Jones interaction reveals opposite trends in liquid
and gas particle viscosity coefficients as function of temperature. This in
turn can be used to describe the lineshape trends in gases and liquids. This
work establishes feasibility of the hybrid SDE-MD approach and paves the way
for modeling of complex interactions with boundaries and other components. The
advantages of modeling viscosity by MD include the possibility of including
realistic boundaries, multiple components or varying the details of
intermolecular interactions. Indeed, models of effective viscosity have
already been developed for such situations [16, 17, 18, 19].

###  I.1 Review of Conventional Theory

Assume that an ensemble of spins is placed in an external magnetic field and a
coherent superposition between the states $\ket{\uparrow}$ and
$\ket{\downarrow}$ is created. In an inhomogeneous field, the phase
accumulation for each spin is proportional to the local field experienced by
the spin over time. Phase accrued by nuclear spins can be refocused by
application of a $\pi$ pulse which inverts the direction of spin rotations, or
simulating a “time-reversal”, forming a Hahn echo [1, 20, 21, 22, 23]. If the
spins are fixed (frozen) in space, evolution is described with a unitary
propagator and the entropy remains constant. With diffusion (e.g. liquids and
gases) not all initial phases within a spin ensemble can be recovered and the
Hahn echo signal will be smaller. Assuming that the molecules undergo a random
walk, it has been postulated that they sample random phase increments from a
Gaussian distribution [1]. In the presence of a magnetic-field gradient
$\mathbf{g}$ the phase accrued by a spin from time $0$ to $t$ is:

$\Delta\phi(t)=\int_{0}^{t}\omega(t^{\prime})dt^{\prime}=\gamma_{n}\int_{0}^{t}\mathbf{g\cdot
r}(t^{\prime})dt^{\prime},$ (1)

where $\omega(t)$ is the time-dependent frequency of a moving spin,
$\gamma_{n}$ is the nuclear gyromagnetic ratio, and $\mathbf{r}(t)$ is the
time-dependent position of the diffusing particle. The nuclear induction
signal is weighted by the ensemble average of these phase factors, written
with probability distribution $P(\Delta\phi)$

$\left<\exp(i\Delta\phi(t))\right>=\int_{-\infty}^{\infty}P(\Delta\phi)\exp(i\Delta\phi)d(\Delta\phi).$
(2)

To alleviate the notation we will drop $t$ from the notation and write
$\Delta\phi(t)=\Delta\phi$. Assuming a Gaussian distribution for
$P(\Delta\phi)$ with variance $\langle(\Delta\phi)^{2}\rangle$ proportional to
$t$, we get

$S(t)=\left<\exp(i\Delta\phi)\right>=\exp(-\left<(\Delta\phi)^{2}\right>/2).$

Thus for the free diffusion under steady magnetic field of a linear gradient
$\mathbf{g}$ and duration $t$, attenuation of the NMR signal $S(t)$ was found
to be

$S(t)=\exp{[(-1/3)\gamma_{n}^{2}g^{2}Dt^{3}]},$ (3)

where $D$ is the self-diffusion coefficient and the $t^{3}$ decay arises from
self-diffusion in presence of a field gradient [23, 24]. This equation shows
that for unrestricted diffusion, as $D$ gets larger a wider range of
$\Delta\phi$ is sampled resulting in faster damping of the signal. In the Hahn
echo experiment, $90^{\circ}-\tau-180^{\circ}-\tau$, at $t=2\tau$ the decay
function is similar:

$S(2\tau)=\exp(-\frac{2}{3}\gamma_{n}^{2}g^{2}D\tau^{3}).$ (4)

The $t^{3}$ dependence has been validated experimentally for liquids. This
form of signal can be used to describe a Carr-Purcell-Meiboom-Gill (CPMG)
experiment with $n$ refocusing $\pi$ pulse and interpulse delay of $2\tau$
[20, 25]. The signal at $t=2n\tau$ decays according to

$S(2n\tau)=\exp\left(-\frac{2}{3}\gamma_{n}^{2}g^{2}D\tau^{2}(n\tau)\right).$
(5)

As a result, the signal attenuation at time $t=2n\tau$ can be reduced by
decreasing time delay between the pulses $\tau$ while increasing number of
pulses $n$. Equations (4) and (5) have been extensively validated in
experiments for the case of liquids. The assumption of Gaussian-distributed
phase increments is reasonable for liquids because of the short mean-free
paths and large number of collisions during time intervals of duration $\tau$.

###  I.2 Problems With Conventional Theory

In the case of gas molecules, which are characterized by longer mean free
paths, the assumption of randomly distributed phase increments becomes more
difficult to justify [10, 26]. A simple example can be used to illustrate the
need for a more sophisticated model. Take the Ornstein-Uhlenbeck process,
which is used to model Brownian particle diffusion. The velocity process obeys
the SDE:

$dv=-\gamma vdt+\frac{\Gamma_{f}}{M}dW(t),$

where $\gamma^{-1}$ is a damping time constant, $dW(t)$ is the increment of
the Wiener process $W(t)$ at time $t$ ($dW(t):=W(t)-W(t-dt)$). The strength of
fluctuations is $\Gamma_{f}=\sqrt{2\gamma Mk_{B}T}$ (fluctuation-dissipation
theorem), where $M$ is the mass of diffusing particle and $k_{B}$ is the
Boltzmann constant. After integration

$v(t)=\langle v(t)\rangle+\sqrt{\frac{2\gamma k_{B}T}{M}}e^{-\gamma
t}\int_{0}^{t}e^{\gamma t^{\prime}}dW(t^{\prime}),$

where $\langle v(t)\rangle=\langle v_{0}\rangle e^{-\gamma t}$. The position
process is the time integral of the velocity process, $dx(t)=v(t)dt$, whose
solution is (setting $x(0)=0$):

$x(t)=\frac{v_{0}}{\gamma}(1-e^{-\gamma t})+\sqrt{\frac{2\gamma
k_{B}T}{M}}\int_{0}^{t}dt^{\prime}e^{-\gamma
t^{\prime}}\int_{0}^{t^{\prime}}dW(t^{\prime\prime})e^{\gamma
t^{\prime\prime}}.$

Defining a thermal velocity $v_{T}=\sqrt{k_{B}T/M}$, the distribution of
$x(t)$ is (again, assuming $x(0)=0$),

$p(x,t)=\frac{1}{\sqrt{2\pi\sigma_{x}^{2}(t)}}\exp\left[-\frac{(x-v_{T}(1-e^{-\gamma
t})/\gamma)^{2}}{2\sigma_{x}^{2}(t)}\right]$

where

$\sigma_{x}^{2}(t):=\frac{2v_{T}^{2}}{\gamma}t-\frac{v_{T}^{2}}{\gamma^{2}}(3-4e^{-\gamma
t}+e^{-2\gamma t})$

at short $t$ is not the usual Gaussian with variance $t$ characteristic of
long-time particle diffusion (i.e. the Einstein-Fick limit). Only at long
times ($t\gg\gamma^{-1}$) does the distribution become Gaussian with variance
$t$:

$p(x,t)\rightarrow\frac{1}{\sqrt{4\pi
Dt}}\exp\left[-\frac{(x-v_{T}/\gamma)^{2}}{4Dt}\right]$

with diffusion coefficient $D=k_{B}T/M\gamma$. To compute the accrued phase,
which is a time integral of $x(t)$ (c.f. Equation 1), it appears prudent to
model the behavior at short times ($t\lesssim\gamma^{-1}$). The timescale
$\gamma^{-1}$ of the friction coefficient is associated with molecular
collisions.

It has been shown [10] that Eq. (4) or (5) does not describe the nuclear
induction decay in gases. Instead, it was observed that in the CPMG
experiment:

$\left<\exp(i\Delta\phi)\right>=\exp(-\gamma^{2}g^{2}\kappa(2n\tau)).$ (6)

Here the decay constant $\kappa$ is a decreasing function of temperature $T$,
damping rate $\gamma$, and particle mass $M$, Eq. (17). Notice the different
powers of $\tau$ inside the argument of the exponential function ($\tau^{3}$
vs $\tau$). This difference was first noticed by observing the linewidth
dependence on temperature [7] and subsequently investigated more in-depth in
Ref. [10, 26]. The conclusion from these studies is that a more detailed
description of diffusion effects is needed. Obviously, exact solutions to the
$N$-body problem are not possible; hence we seek suitable models.

###  I.3 Generalized Langevin Equation (GLE)

The GLE is a well-established model of particle diffusion that accounts for
memory effects as the molecules undergo diffusion via collisions. In the gas
and liquid phase, nuclear spin degrees of freedom are fairly well isolated
from spatial degrees of freedom, at least to first order. Consequently,
individual molecular collisions, do not completely depolarize the spins.
Instead, decoherence and depolarization processes, which are second-order
processes, take place over much longer periods of time. Decoherence is
described by a characteristic decay time $T_{2}$. The motional part of the NMR
signal thus can be written as the expectation value of the spin phase factors
[c.f. Equation 1, with $\omega(t)=\gamma_{n}g\cdot x(t)$ in 1-D] which depend
on their position $x(t)$ [24, 10]. In Ref. [10] it is assumed that $x(t)$ is a
Gaussian random process that is stationary in the wide sense. The expectation
value

$S(t)=\left<\exp\left(i\int_{0}^{t}\omega(t^{\prime})dt^{\prime}\right)\right>$
(7)

takes the form:

$\exp\left[i\gamma_{n}g\int_{0}^{t}\langle x(t^{\prime})\rangle
dt^{\prime}-\gamma_{n}^{2}g^{2}\int_{0}^{t}\langle
x(t^{\prime})x(0)\rangle(t-t^{\prime})dt^{\prime}\right].$

The second term determines the lineshape of the signal decay. For liquids,
displacements are small $x(t)\approx x(0)$ and this leads to the Einstein-Fick
limit $\langle x(t)x(0)\rangle\approx\langle x(t)x(t)\rangle=2Dt$, where $D$
is a diffusion coefficient, and the classic Hahn result (3) is recovered.

We note that there is no easy way to compute the integral over the function
$\langle x(t^{\prime})x(0)\rangle$ from first principles in the general case
of arbitrary physical conditions. Even MD simulations fail because $\langle
x(t^{\prime})x(0)\rangle$ decays on short time scales (tens to hundreds of
picoseconds) before reaching a nonzero steady state value, after which the
time integral is weighted by the monotonically increasing weight
$(t-t^{\prime})dt^{\prime}$ over much longer time scales (microseconds to
seconds). The initial decay, and any numerical errors associated with it, is
then amplified by orders of magnitude. Instead, a model is needed that bridges
the two widely different timescales by analytically solving the integral and
reducing it to a function of transport coefficients that are easier to
compute. Such a transformation was first presented in Ref. [7]. The main steps
of its derivation, although well known to the fluid dynamics community, are
recapped below for convenience.

In the “weak collision” regime (see Ref. [10]) a GLE with memory kernel
describes particle dynamics

$M\dot{v}+\int_{0}^{t}\Gamma(t-t^{\prime})v(t^{\prime})dt^{\prime}=\eta_{f}(t)$
(8)

where $M$ is the mass of diffusing particle, $v(t)=\dot{x}(t)$ and $\dot{v}$
are particle velocity and acceleration. $\eta_{f}(t)$ on the right-hand side,
represents a time-dependent stochastic force. The memory kernel $\Gamma(t)$,
is convoluted with the particle velocity to describe the friction asserted by
the viscous dynamics. Projecting both sides of the GLE equation with the inner
product $\langle v(0),\cdot\rangle$ we get

$M\left<v(0)\dot{v}(t)\right>+\int_{0}^{t}\Gamma(t-t^{\prime})\left<v(0)v(t^{\prime})\right>dt^{\prime}=0.$
(9)

Fluctuation-dissipation,
$\langle\eta_{f}(0)\eta_{f}(t)\rangle=k_{B}T\Gamma(t)$, and equipartition,
$\left<v(0)v(0)\right>=k_{B}T/M$, theorems are invoked. Here, $\nu(t)$ is the
integral of the velocity autocorrelation function,
$\int_{0}^{t}\left<v(0)v(t^{\prime})\right>dt^{\prime}$. Then,

$M\dot{\nu}(t)+\int_{0}^{t}\Gamma(t-t^{\prime})\nu(t^{\prime})dt^{\prime}=k_{B}T.$
(10)

For the memory kernel one often invokes the Ornstein-Uhlenbeck process to
model the delayed response of surrounding fluid

$\Gamma(t)=(\gamma^{2}/m)\exp(-\gamma t/m),$ (11)

where $\gamma$ is a friction coefficient proportional to the viscosity of the
fluid and $m$ represents a mass attributed to solvent particles. The solution
using $\zeta_{\mp}=\frac{\gamma}{2m}(1\mp\sqrt{1-4m/M})$ is

$\nu(t)=\frac{k_{B}T}{M}\left(\frac{\gamma}{m\zeta_{-}\zeta_{+}}+\frac{1}{\zeta_{+}-\zeta_{-}}\left[(1-\frac{\gamma}{m\zeta_{+}})\exp({-\zeta_{+}t})\right.\right.\\\
\left.\left.-(1-\frac{\gamma}{m\zeta_{-}})\exp(-\zeta_{-}t)\right]\right).$
(12)

For the position auto-correlation function one finds

$\left<x(t)x(0)\right>=\frac{k_{B}T}{M(\zeta_{+}-\zeta_{-})}\left[\zeta_{+}^{-1}(1-\frac{\gamma}{m\zeta_{+}})\exp({-\zeta_{+}t})\right.\\\
\left.\zeta_{-}^{-1}(1-\frac{\gamma}{m\zeta_{-}})\exp({-\zeta_{-}t})\right].$
(13)

In the case that $\gamma t/m$ is sufficiently large, the Ornstein-Uhlenbeck
kernel rapidly decays. At this point, it is possible to introduce a dependence
on viscosity by invoking Stokes’ drag law $\gamma=6\pi\eta R$ where $R$ is the
radius of the “Brownian particle” and $\eta$ is the shear viscosity. The
validity of Stokes’ law is predicated on the assumption that the Brownian
particle is much larger than the solvent particles ($M\gg m$). In the case of
self-diffusion all particles are identical. Hence, the Stokes’ law appears
unjustified. However, the proportionality between $\gamma$ and $\eta$ is
always correct [27]. In fact, Einstein developed the concept of “effective
viscosity”, which has been used to describe the effective viscosity of
lubricants [28, 29, 27, 30].

We are now in a position to replace the integral by a function of the
viscosity, a transport coefficient that is easily computed from MD simulations
with good accuracy. Viscosity is essentially a coarse-grained quantity
describing the relaxation of momentum density to its equilibrium value, after
a perturbation. It is directly related to the velocity autocorrelation
function. Stokes’ law has been extended to the frequency domain to describe
dynamical and dissipative effects in rheology [31, 32, 33]. The relationship
between the Fourier representation of frequency dependent viscosity and the
frequency-dependent friction coefficient $\tilde{\gamma}(s)$ is

$\tilde{\eta}(s)=\frac{\tilde{\gamma}(s)}{6\pi R}$ (14)

where $s$ is the complex frequency in Laplace domain and
$\tilde{\gamma}(\omega)=\int_{0}^{\infty}\Gamma(t)\exp(i\omega t)$ [31]. This
relationship, which has been termed the “correspondence principle” [33],
provides a direct link between the memory function and viscosity. The memory
kernel itself encodes the response of particles at all frequencies to
collisions and boundary conditions. Therefore, we expect the viscosity to be
sensitive to boundary conditions and intermolecular potentials as well. In
particular, we expect that changes in the lineshape of the NMR signal, c.f.
Eq. (7), as a function of temperature, to be reflected in the memory function
as well as viscosity coefficient.

The case of gases. A model for the viscosity and its temperature dependence is
needed [34]. Dynamic viscosity of gases was described by Sutherland as

$\eta=\frac{\mu_{0}(T_{0}+C)}{T+C}(\frac{T}{T_{0}})^{3/2}\sim\frac{T^{3/2}}{T+C},$
(15)

where $C$ is Sutherland’s constant, and $\mu_{0}$ is the viscosity at
temperature $T_{0}$. At high temperatures viscosity grows with the square root
of the temperature, $\eta\sim T^{1/2}$, while at low temperature $\eta\sim
T^{3/2}$. Using Stokes’ drag law, we rewrite the nuclear induction signal as
[10]

$S(t)=\exp(-\gamma_{n}^{2}g^{2}\kappa t),$ (16)

with

$\kappa(T)=\frac{kT(-m\zeta_{-}^{2}\zeta_{+}-m\zeta_{-}\zeta_{+}^{2}+\zeta_{-}^{2}\gamma+\zeta_{-}\zeta_{+}\gamma+\zeta_{+}^{2}\gamma)}{mM\zeta_{-}^{3}\zeta_{+}^{3}}.$
(17)

Note that $\kappa$ determines the linewidth $\Delta f$, and it shows two
distinct trends with temperature

$\Delta f\sim\begin{cases}T^{-7/2}&T<C\\\ T^{-1/2}&T>C.\end{cases}$ (18)

Figure 1 represents the decay of gas phase NMR signal in an echo experiment
for a range of temperatures. The sample is a sealed tube of liquid
tetramethylsilane (TMS), prepared using freeze-pump-thaw method. The sample
tube is heated above $25^{\circ}$ C to evaporate the TMS. Measurements were
performed on a Bruker AV 600 MHz NMR spectrometer equipped with variable
temperature and pulsed gradient capabilities. In the presence of a magnetic-
field gradient, the line narrows with increasing temperature, a surprising
result that was first observed and explained in [7, 10, 11]. Data points are
fitted to exponential decay functions according to Eq. (16) and the resulting
linewidths are plotted in the inset of Fig. 1(b). Their temperature trend is
in agreement with Eq.( 18).

Figure 1: a) Shows the decay of NMR echo signal for TMS in gas phase, for
variable temperature at $g=0.01$ G/cm. The decay becomes slower at higher
temperatures. Panel b) shows the Fourier transformed data in the frequency
domain, illustrating that the NMR spectra narrows at higher temperatures. The
inset indicates the linewidth $\Delta f\propto\kappa(T)$, extracted by fitting
the time domain data to an exponential decay function Eq. (16). The blue line
shows the best fit using $\kappa(T)=cT^{-1/2}$, where $c$ is a constant.

The case of liquids. To model the viscosity of liquids an empirical equation
of the form

$\eta=A\exp(B/T)$ (19)

where $A$ and $B$ are constants can be substituted into the expression for the
nuclear induction decay (c.f. Eq. 16). Recalling that $\kappa(T)$ is
proportional to $T\zeta^{-3}$, we find:

$\kappa(T)\propto T\exp(-3B/T)=T(1-3B/T+\tfrac{9}{2}B^{2}/T^{2}+\dots).$

The overall temperature dependence of the linewidth is a line broadening with
increasing temperature [7, 10, 11]. In any case, these equations are only
models for $\eta$. They do not account for boundaries; yet, it is known that
boundaries alter the effective viscosity [28, 29, 27, 30]. Thus, the problem
has been reduced to the modeling of an effective viscosity.

##  II MD results

MD simulations of particle trajectories have been used to model gas diffusion
in systems of gas mixtures in complex geometries [35, 36, 37]. In particular,
for liquids and non-ideal gases in which transport properties are difficult to
describe, MD simulations have proven very successful [16, 17, 18, 19].

Figure 2:  Green-Kubo correlation functions for liquid and gas simulations are
evaluated at each temperature for both densities. Here the simulation results
at $300$ is presented (left). These correlation are integrated to the
saturation point to evaluate the viscosity coefficient eq.( 21) (right). Note
that the time axis is three orders of magnitude larger for gas simulations.

At each time step, intermolecular forces between nearest neighbors are
enforced to recreate realistic particle trajectories [38, 39]. To obtain such
trajectories over a large number of particles we used the open source software
“Large-scale Atomic/Molecular Massively Parallel Simulator” (LAMMPS) [40].
LAMMPS results were then used to compute viscosity coefficients in liquid and
gaseous xenon (Xe). We used the Lennard-Jones (LJ) pairing,
$U(r)=4\epsilon[(\sigma/\ r)^{12}-(\sigma/\ r)^{6}]$, were $\epsilon=1.77$
kJ/mol is the depth of the potential well and $\sigma=4.1~{}\textup{\AA}$ is
the distance where the potential is zero. Simulations were performed for 1,000
Xe atoms in a box with periodic boundary conditions. The simulations ran for a
constant number of particles, volume and temperature in the canonical ensemble
($NVT$). The equilibrium time correlation function approach, a.k.a. Green-Kubo
autocorrelation function, was used to derive the viscosity [41, 39, 42, 34,
19]

$\eta=\lim_{t\rightarrow\infty}\eta_{GK}(t)$ (20)

with

$\eta_{GK}(t)=\frac{V}{3k_{B}T}\int_{0}^{t}\sum_{\alpha<\beta}C_{\alpha\beta}(\tau)d\tau,$
(21)

where $\alpha,\beta\in\\{x,y,z\\}$, $V$ is the volume and $T$ is the
temperature.
$C_{\alpha\beta}(\tau)=\left<p_{\alpha\beta}(\tau)p_{\alpha\beta}(0)\right>$
is the autocorrelation function of non-diagonal elements of the pressure
tensor, e.g.

$p_{xy}(t)=\frac{1}{V}\Bigl{\\{}\sum_{j}m_{j}v_{jx}(t)v_{jy}(t)+\frac{1}{2}\sum_{i\neq
j}r_{ijx}(t)f_{ijy}(t)\Bigr{\\}}.$

Here $f_{ijy}$ represents the $y$-component of the force between two particles
$i$ and $j$. The first term on the right hand side, is the kinetic
contribution to the pressure tensor while the second term indicates the
potential contribution. Other components $p_{\alpha\beta}$ of the pressure
tensor are defined analogously.

Simulations were performed for 21 different temperature values in the range
200-400 K, using a high density ($\rho=N/V$) of particles compatible with
liquid phase, and a low density for gas phase. Particle trajectories were
evolved for $10^{6}$ femtoseconds to equilibrate. We evaluated
$C_{\alpha\beta}$ correlation functions 60 times in each simulation, recording
them well beyond their saturation point (see Fig. 2). They were then used to
evaluate the Green-Kubo integral Eq. (20). For this approach to work, it is
critical that the correlation functions decay with similar rates so the
resulting integrals converge to an average value.

The shear viscosity coefficient for the liquid state was found to decrease
with temperature (see Fig. 3). This behavior is expected and well understood
for liquids, and is in agreement with the line broadening observed in the
liquid state NMR experiments [7, 10]. The verification of simulated viscosity
coefficients against experimental values is complicated by the fact that Xe is
not in the liquid form above 270 K whereas below that temperature,
measurements report viscosity vs temperature but across different densities
[43, 44]. Nevertheless, a matching data point exists: at 220 K our simulated
viscosity coefficient lies within $5\%$ of the experimentally measured value
[43]. Our simulation results are similar to those from Ref. [45] where the
authors performed a general study of LJ potential dynamics vs $\rho$ and $T$
and found that the resulting viscosity coefficients are within $6\%$ of
experimental values. The decay of $\eta(T)$ with $T$ was found to be
exponential and was fitted to Eq. (19).

Figure 3: Viscosity in liquids drops as the temperature increases. Error bars
indicate the standard deviation for the set of 60 accumulated correlation
functions. The solid line is a fit to Eq. (19) with $A=0.38\pm 0.01$ and
$B=135\pm 7$. Figure 4: Viscosity in gas molecules grows as the temperature
increases. Red circles indicate results of MD simulation and black circles are
experimental data. Error bars are the standard deviation for the set of 60
accumulated correlation functions. The solid line shows a fit to Eq. (15) with
$C=258\pm 69$.

The gas phase simulations were performed by first equilibrating the system
over a period of $10^{6}$ picoseconds. Comparing to the liquid case, the
correlation functions were found to decay slower as the frequency of particle
collision is smaller (see Fig. 2). The simulated viscosity values were found
to be two orders of magnitude smaller as well. Moreover, viscosity
coefficients in gases show the opposite trend to those of liquids, i.e. shear
viscosity increases with temperature. A comparison to the available
experimental data [46, 47] shows that the simulated viscosity coefficients
overlap with them. The calculated viscosity coefficients are fitted to Eq.
(15). We find that the Sutherland’s constant ($C=258\pm 69$) is in agreement
with the literature value for Xe gas ($C=252$) [46].

Our MD results confirmed that as the temperature of gases increases, the
higher frequency of molecular collisions results in greater resistance and
larger viscosities. Viscosity results are in agreement with previous studies
and experimental results, as expected. On the other hand, what is new here is
our ability to predict the correct lineshape for the nuclear induction signal
from MD results, and this lineshape is predicted based on the effective
viscosity. Equation (17) suggests that the decay rate of the nuclear induction
signal is inversely proportional to the viscosity coefficient $\eta$ and as a
result, the linewidth narrows at higher temperatures. A depiction of such
behavior for the experiment with $g=0.01$ G/cm, is shown in Fig. 5 where the
simulated viscosity coefficients were used to predict the lineshape in gases.
The resulting linewidths are very close to the experimental values of Fig. 1.
A side-by-side comparison of experiments and theory is shown in the Inset of
Fig. 5.

Figure 5: Dependence of nuclear induction lineshape for Xe gas on temperature.
MD-derived viscosity coefficients were used together with Eq. (17). The inset
compares experimental linewidths with simulated ones.

##  III Conclusion

We have shown the feasibility of using MD simulations to model transport
coefficients in liquids and gases; these transport coefficients are then used
together with a GLE model to obtain the nuclear induction lineshape. In
particular, our MD results correctly indicate that gas phase spectra exhibit
narrower lines at higher temperatures. (This is in contrast to the
conventional formalism [12, 13, 14], which incorrectly predicts the opposite
trend.) This work opens the door for the prediction of lineshapes in more
complex geometries and boundary conditions. Our work differs from previous
studies of diffusion [48] in that memory effects can be explicitly modeled
using our method. Future directions may include explicit modeling of
boundaries and gas types using appropriate memory kernels, and ultimately,
solving the inverse problem of computing molecular and pore parameters from
experimentally-measured nuclear induction decays.

##  IV Acknowledgment

This work used computational and storage services associated with the Hoffman2
Shared Cluster provided by UCLA Institute for Digital Research and Education’s
Research Technology Group. L.-S. B. acknowledges funding from NSF award
CHE-2002313.

## References

  * [1] Hahn, E.  Spin echoes.  _Phys. Rev._ 80, 580–594 (1950).
  * [2] Mansfield, P.  Multi-planar image formation using nmr spin echoes.  _Journal of Physics C: Solid State Physics_ 10, L55 (1977).
  * [3] Lauterbur, P. C.  Image formation by induced local interactions: examples employing nuclear magnetic resonance.  _nature_ 242, 190–191 (1973).
  * [4] Yablonskiy, D. A., Sukstanskii, A. L. & Quirk, J. D.  Diffusion lung imaging with hyperpolarized gas mri.  _NMR in Biomedicine_ 30, e3448 (2017).
  * [5] Lysova, A. A. & Koptyug, I. V.  Magnetic resonance imaging methods for in situ studies in heterogeneous catalysis.  _Chemical Society Reviews_ 39, 4585–4601 (2010).
  * [6] Gladden, L. F.  Applications of in situ magnetic resonance techniques in chemical reaction engineering.  _Topics in Catalysis_ 8, 87–95 (1999).
  * [7] Jarenwattananon, N. _et al._ Thermal maps of gases in heterogeneous reactions.  _Nature_ 502, 537–540 (2013).
  * [8] van Dusschoten, D., Moonen, C. T., De Jager, P. A. & Van As, H.  Unraveling diffusion constants in biological tissue by combining carr-purcell-meiboom-gill imaging and pulsed field gradient nmr.  _Magnetic resonance in medicine_ 36, 907–913 (1996).
  * [9] Qin, E. C. _et al._ Combining mr elastography and diffusion tensor imaging for the assessment of anisotropic mechanical properties: a phantom study.  _Journal of Magnetic Resonance Imaging_ 37, 217–226 (2013).
  * [10] Jarenwattananon, N. & Bouchard, L.-S.  Motional averaging of nuclear resonance in a field gradient.  _Phys. Rev. Lett._ 114, 197601 (2015).
  * [11] Jarenwattananon, N. & Bouchard, L.-S.  Breakdown of carr-purcell meiboom-gill spin echoes in inhomogeneous fields.  _The Journal of Chemical Physics_ 149, 084304 (2018).
  * [12] Herzog, B. & Hahn, E. L.  Transient nuclear induction and double nuclear resonance in solids.  _Phys. Rev._ 103, 148–166 (1956).
  * [13] Pfitsch, D., McDowell, A. & Conradi, M. S.  What are the conditions for exponential time-cubed echo decays?  _Journal of Magnetic Resonance_ 139, 364–370 (1999).
  * [14] Slichter, C.  _Principles of Magnetic Resonance_ (Springer, 1990).
  * [15] Li, T. & Raizen, M.  Brownian motion at short time scales.  _Annalen der Physik_ 525, 281–295 (2013).
  * [16] Starov, V. M. & Zhdanov, V. G.  Effective viscosity and permeability of porous media.  _Colloids and Surfaces A: Physicochemical and Engineering Aspects_ 192, 363–375 (2001).
  * [17] Breugem, W.-P.  The effective viscosity of a channel-type porous medium.  _Physics of Fluids_ 19, 103104 (2007).
  * [18] V. Ya. Rudyak, A. A. B.  Molecular dynamics simulation of fluid viscosity in nanochannels.  _Nanosystems: Physics, Chemistry, Mathematics_ 9, 349–355 (2013).
  * [19] Rizk, F., Gelin, S., Biance, A.-L. & Joly, L.  Microscopic origins of the viscosity of a lennard-jones liquid.  _Phys. Rev. Lett._ 129, 074503 (2022).
  * [20] Carr, H. & Purcell, E.  Effects of diffusion on free precession in nuclear magnetic resonance experiments.  _Phys. Rev._ 94, 630–638 (1954).
  * [21] Torrey, H.  Bloch equations with diffusion terms.  _Phys. Rev._ 104, 563–565 (1956).
  * [22] Callaghan, P.  _Principles of Nuclear Magnetic Resonance Microscopy_ (Oxford, 1991).
  * [23] Stejskal, E. & Tanner, J.  Spin diffusion measurements: Spin echoes in the presence of a time‐dependent field gradient.  _The Journal of Chemical Physics_ 42, 288–292 (1965).
  * [24] Callaghan, P.  _Translational Dynamics & Magnetic Resonance_ (Oxford, 2011).
  * [25] Meiboom, S. & Gill, D.  Modified spin‐echo method for measuring nuclear relaxation times.  _Review of Scientific Instruments_ 29, 688–691 (1958).
  * [26] Jarenwattananon, N. & Bouchard, L.-S.  Jarenwattananon and bouchard reply:.  _Phys. Rev. Lett._ 117, 249702 (2016).
  * [27] Peppin, S.  Effective hard-sphere model of diffusion in aqueous polymer solutions (2021).
  * [28] Haines, B. & Mazzucato, A.  A proof of einstein’s effective viscosity for a dilute suspension of spheres.  _SIAM Journal on Mathematical Analysis_ 44, 2120–2145 (2012).
  * [29] Duerinckx, M. & Gloria, A.  On einstein’s effective viscosity formula.  _arXiv preprint arXiv:2008.03837_ (2020).
  * [30] Breki, A. & Nosonovsky, M.  Einstein’s viscosity equation for nanolubricated friction.  _Langmuir_ 34, 12968–12973 (2018).
  * [31] Zwanzig, R. & Bixon, M.  Hydrodynamic theory of the velocity correlation function.  _Physical Review A_ 2, 2005 (1970).
  * [32] Mason, T. G. & Weitz, D. A.  Optical measurements of frequency-dependent linear viscoelastic moduli of complex fluids.  _Physical review letters_ 74, 1250 (1995).
  * [33] Córdoba, A., Indei, T. & Schieber, J. D.  Elimination of inertia from a generalized langevin equation: applications to microbead rheology modeling and data analysis.  _Journal of Rheology_ 56, 185–212 (2012).
  * [34] Brush, S.  Theories of liquid viscosity.  _Chemical reviews._ 62 (1962-12-01).
  * [35] Acosta, R. _et al._ Diffusion in binary gas mixtures studied by NMR of hyperpolarized gases and molecular dynamics simulations.  _Phys. Chem. Chem. Phys._ 8, 4182–4188 (2006).
  * [36] Yeh, I.-C. & Hummer, G.  System-size dependence of diffusion coefficients and viscosities from molecular dynamics simulations with periodic boundary conditions.  _The Journal of Physical Chemistry B_ 108, 15873–15879 (2004).
  * [37] Hazelbaker, E. D. _et al._ Combined application of high-field diffusion nmr and molecular dynamics simulations to study dynamics in a mixture of carbon dioxide and an imidazolium-based ionic liquid.  _The Journal of Physical Chemistry B_ 116, 9141–9151 (2012).
  * [38] Frenkel, D. & Smit, B.  Chapter 4 - molecular dynamics simulations.  In Frenkel, D. & Smit, B. (eds.) _Understanding Molecular Simulation (Second Edition)_ , 63–107 (Academic Press, San Diego, 2002), second edition edn.
  * [39] Rapaport, D. C.  _The Art of Molecular Dynamics Simulation_ (Cambridge University Press, 2004), 2 edn.
  * [40] Thompson, A. P. _et al._ LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales.  _Comp. Phys. Comm._ 271, 108171 (2022).
  * [41] McQuarrie, D.  _Statistical mechanics / Donald A. McQuarrie_ (Harper & Row New York, 1975).
  * [42] Todd, B. D. & Daivis, P. J.  _Nonequilibrium Molecular Dynamics: Theory, Algorithms and Applications_ (Cambridge University Press, 2017).
  * [43] Hanley, H. J., McCarty, R. D. & Haynes, W. M.  The viscosity and thermal conductivity coefficients for dense gaseous and liquid argon, krypton, xenon, nitrogen, and oxygen.  _Journal of Physical and Chemical Reference Data_ 3, 979–1017 (1974).
  * [44] Grisnik, S.  Measurement of xenon viscosity as a function of low temperature and pressure.  In _34th AIAA/ASME/SAE/ASEE Joint Propulsion Conference and Exhibit_ , 3498 (1998).
  * [45] Rowley, R. & Painter, M.  Diffusion and viscosity equations of state for a lennard-jones fluid obtained from molecular dynamics simulations.  _International journal of thermophysics_ 18, 1109–1121 (1997).
  * [46] Haynes, W., Lide, D. & Bruno, T.  _CRC HANDBOOK of Chemistry and Physics_ (CRC Press, 2016).
  * [47] Mountain, R. D.  Molecular dynamics calculation of the viscosity of xenon gas.  _International journal of thermophysics_ 28, 259–267 (2007).
  * [48] Grebenkov, D.  Paradigm shift in diffusion-mediated surface phenomena.  _Physical Review Letters_ 125, 078102 (2020).


# Salient Object Detection via Bounding-box Supervision

###### Abstract

The success of fully supervised saliency detection models depends on a large
number of pixel-wise labeling. In this paper, we work on bounding-box based
weakly-supervised saliency detection to relief the labeling effort. Given the
bounding box annotation, we observe that pixels inside the bounding box may
contains extensive labeling noise. However, as the large amount of background
is excluded, the foreground bounding box region contains less complex
background, making it possible to perform handcrafted features based saliency
detection with only the cropped foreground region. As the conventional
handcrafted features are not representative enough, leading to noisy saliency
maps, we further introduce structure-aware self-supervised loss to regularize
the structure of the prediction. Further, we claim that pixels outside
bounding box should be background, thus partial cross-entropy loss function
can be used to accurately localize the accurate background region.
Experimental results on six benchmark RGB saliency dataset illustrate
effectiveness of our model.

Index Terms—  Weakly supervised learning, Bounding-box annotation, Structure-
aware self-supervised loss

##  1 Introduction

Salient object detection aims to localize the full scope of the salient
foreground, which is usually defined as a binary segmentation task. Most of
the conventional techniques are fully-supervised [1, 2, 3], where the pixel-
wise annotations are needed as supervision to train a mapping from input image
space to output saliency space. We find that the strong dependency of pixel-
wise labeling pose both efficiency and budget challenges for existing fully-
supervised saliency detection techniques. To relief the labeling effort, some
un/weakly-supervised learning based salient object detection techniques [4, 5,
6, 7, 8, 9, 10, 11, 12, 13, 14] are introduced, which aim to learn saliency
from easy-to-obtain labeling, including image-level labels [5], scribbles [4,
13], bounding box supervisions [12] or noisy labeling [9, 7, 6, 8, 14]. In
this paper, we present a new bounding box supervision based weakly-supervised
saliency detection model.

|  |
---|---|---
Image | bndBox | GrabCut
Fig. 1: Given image (“Image”) and it’s bounding box annotation (“bndBox”),
GrabCut leads to an pseudo segmentation result (“GrabCut”), serving as pseudo
label for the proposed weakly supervised saliency detection model.

The advantage of bounding-box supervision is that the background of bounding-
box annotation is pure background. Although there still exists background
region within the bounding-box foreground annotation, the significantly
reduced background region makes it possible to perform conventional
handcrafted feature based saliency detection methods on the cropped foreground
region. In Fig. 1, we show the generated pseudo label by combining bounding-
box annotation with the conventional handcrafted feature based method, which
clearly show it’s superiority.

As a deep neural network can fit any types of noise [15], directly using the
generated pseudo label as ground truth will lead to biased model, over-fitting
on the noisy pseudo label. To further constrain the structure of the
prediction, we present structure-aware self-supervised loss function. The main
goal of this loss function is constrain the prediction with structure well-
aligned with the input image. In this way, we aim to obtain structure accurate
predictions (see Fig. 3).

##  2 Related Work

Fully-supervised Saliency models: The main focus of fully-supervised saliency
detection models is to achieve effective feature aggregation [16] [17] [18].
Due to the use of stride operation, the resulting saliency maps are usually
with low resolution. To produce structure accurate saliency prediction, some
method use edge supervision to learn more feature about the object boundary to
refine the saliency predictions using better structure of the object [19] [20]
[21].

---
Fig. 2: Given bounding box annotation $y_{i}$, we first generate pseudo
saliency maps $g_{i}$ via GrabCut. Then, we design saliency prediction
network(SPN) to regress the saliency map with the above pseudo saliency maps
as supervision ($\mathcal{L}_{spn}$). Due to labeling noise, we introduce a
foreground structure-aware loss $\mathcal{L}_{fore}$, namely the smoothness
loss [22] to further constrain the structure of the salient foreground
(regions within the bounding box area). As the background of the pseudo
saliency map is accurate, we introduce background loss $\mathcal{L}_{back}$
using the partial cross-entropy loss to constrain model prediction in the
background bounding box area.

Weakly-supervised Saliency models: The goal of weakly-supervised saliency
detection is to relief the labeling effort [4, 13, 5]. Given bounding box
supervision, They use mask to create the persudo mask,such as GrabCut [23].
[24] use the crossing line of the bounding box as supervision prior.

##  3 Our Method

###  3.1 Overview

In this paper, we study weakly-supervised saliency detection with bounding-box
supervision. Specifically, given bounding-box supervision of the training
dataset, model is trained to accurately produce saliency maps for testing
images. Let’s define the training data set as: $D=\\{x_{i},y_{i}\\}_{i=1}^{N}$
of size $N$, where $x_{i}$ and $y_{i}$ are the input RGB image and it’s
corresponding bounding box supervision, and $i$ indexes the images. To
generate $y_{i}$, each salient non-overlap salient instance is annotated with
separated bounding box, and we generate one single bounding box for the
overlapped salient instances. In this way, the foreground of $y_{i}$ contains
different level of noise depending on the position and shape of the bounding
box, and the background of $y_{i}$ is accurate background.

Four main steps are included in our method: (1) a persudo saliency map
generator which use the Grabcut to generate pseudo saliency maps given
bounding box supervision; (2) a saliency prediction network (SPN) to produce a
saliency map, which is supervised with above pseudo saliency maps; (3)
structure-aware loss to optimize the foreground predictions; (4) partial-cross
entropy based background loss to optimize the background prediction.

###  3.2 Persudo Saliency Map Generator via GrabCut

Given bounding box supervision, we first generate pseudo saliency maps with
Grabcut (see Fig. 1). Compared with the direct bounding box supervision
$y_{i}$, the generated pseudo saliency map $g_{i}$ is more accurate in
structure, making it suitable to serve as pseudo label for saliency
prediction. Specifically, we repeat the Grabcut operation several times until
we obtain a reasonably accurate pseudo label.

###  3.3 Saliency Prediction Network

Following the conventional fully-supervised saliency detection, we design a
“saliency prediction network” to generate saliency maps. Specifically, we take
the ResNet50 as backbone. Given the backbone feature
$f_{\theta_{1}}(x)=\\{s_{k}\\}_{k=1}^{4}$ ($\theta_{1}$ is parameters of the
encoder (the backbone model), the saliency prediction network aims to generate
saliency map $s=f_{\theta}(x)$, where $\theta=\\{\theta_{1},\theta_{2}\\}$
with $\theta_{2}$ as parameters of the decoder. Specifically, we feed each
$s_{k}$ to a $3\times 3$ convolutional layer to generate new backbone feature
$\\{s^{\prime}_{k}\\}_{k=1}^{4}$ of same channel size $C=64$. Then, we adopt
decoder from [25], which takes $\\{s^{\prime}_{k}\\}_{k=1}^{4}$ as input and
generate $s=f_{\theta_{2}}(\\{s^{\prime}_{k}\\}_{k=1}^{4})$. Note that, we
define parameters of the above 4 copies of $3\times 3$ convolutional layers as
part of parameters within $\theta_{1}$.

Given saliency prediction $s_{i}$, we first design task related loss function,
aiming to regress the pseudo salienncy map $g_{i}$ from GrabCut. Specifically,
we adopt the symmetric cross-entropy loss [26], which is proven relative
robust to labeling noise:

$\mathcal{L}_{spn}=\alpha\mathcal{L}_{ce}(s,g)+\beta\mathcal{L}_{ce}(g,s),$
(1)

where $\mathcal{L}_{ce}$ is the cross-entropy loss, $\alpha$ and $\beta$ are
used to weigh the contribution of each loss, and empirically we set
$\alpha=\beta=1$ in this paper. $s$ and $g$ are the model prediction and
pseudo saliency map with GrabCut.

###  3.4 Foreground Structure Constrain

Although GrabCut can generate relatively better pseudo label compared with the
original bounding box supervision, the complex saliency foreground leads to
noisy supervision after GrabCut. To constrain the prediction within the
foreground bounding box region, we adopt smoothness loss [22] to produce
structure accurate prediction.

The smoothness loss is defined as:

$\mathcal{L}_{fore}=\sum_{u,v}\sum_{d\in{\overrightarrow{x},\overrightarrow{y}}}\Psi(|\partial_{d}s_{u,v}|e^{-\alpha|\partial_{d}(y\cdot
I_{u,v})|}),$ (2)

where $\Psi$ is defined as $\Psi(s)=\sqrt{s^{2}+1e^{-6}}$ to avoid calculating
the square root of zero, $I_{u,v}$ is the image intensity value at pixel
$(u,v)$, $d$ indicates the partial derivatives on the $\overrightarrow{x}$ and
$\overrightarrow{y}$ directions. Different with the conventional smoothness
loss in [22], we introduce gate $y$ to the calculation of smoothness loss to
pay attention to the bounding box foreground region inspired by [4].

Table 1: Performance comparison with benchmark saliency detection models. |  DUTS [5] |  ECSSD [27] |  DUT [28] |  HKU-IS [29] |  PASCAL-S [30] |  SOD [31]
---|---|---|---|---|---|---
Method | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$ | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$ | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$ | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$ | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$ | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$
SCRN [32] | .885 | .833 | .900 | .040 | .920 | .910 | .933 | .041 | .837 | .749 | .847 | .056 | .916 | .894 | .935 | .034 | .869 | .833 | .892 | .063 | .817 | .790 | .829 | .087
F3Net [33] | .888 | .852 | .920 | .035 | .919 | .921 | .943 | .036 | .839 | .766 | .864 | .053 | .917 | .910 | .952 | .028 | .861 | .835 | .898 | .062 | .824 | .814 | .850 | .077
ITSD [34] | .886 | .841 | .917 | .039 | .920 | .916 | .943 | .037 | .842 | .767 | .867 | .056 | .921 | .906 | .950 | .030 | .860 | .830 | .894 | .066 | .836 | .829 | .867 | .076
PAKRN [35] | .900 | .876 | .935 | .033 | .928 | .930 | .951 | .032 | .853 | .796 | .888 | .050 | .923 | .919 | .955 | .028 | .859 | .856 | .898 | .068 | .833 | .836 | .866 | .074
MSFNet [36] | .877 | .855 | .927 | .034 | .915 | .927 | .951 | .033 | .832 | .772 | .873 | .050 | .909 | .913 | .957 | .027 | .849 | .855 | .900 | .064 | .813 | .822 | .852 | .077
CTDNet[37] | .893 | .862 | .928 | .034 | .925 | .928 | .950 | .032 | .844 | .779 | .874 | .052 | .919 | .915 | .954 | .028 | .861 | .856 | .901 | .064 | .829 | .832 | .858 | .074
VST[38] | .896 | .842 | .918 | .037 | .932 | .911 | .943 | .034 | .850 | .771 | .869 | .058 | .928 | .903 | .950 | .030 | .873 | .832 | .900 | .067 | .854 | .833 | .879 | .065
GTSOD [39] | .908 | .875 | .942 | .029 | .935 | .935 | .962 | .026 | .858 | .797 | .892 | .051 | .930 | .922 | .964 | .023 | .877 | .855 | .915 | .054 | .860 | .860 | .898 | .061
SSAL [4] | .803 | .747 | .865 | .062 | .863 | .865 | .908 | .061 | .785 | .702 | .835 | .068 | .865 | .858 | .923 | .047 | .798 | .773 | .854 | .093 | .750 | .743 | .801 | .108
WSS [5] | .748 | .633 | .806 | .100 | .808 | .774 | .801 | .106 | .730 | .590 | .729 | .110 | .822 | .773 | .819 | .079 | .701 | .691 | .687 | .187 | .698 | .635 | .687 | .152
C2S [40] | .805 | .718 | .845 | .071 | - | - | - | - | .773 | .665 | .810 | .082 | .869 | .837 | .910 | .053 | .784 | .806 | .813 | .130 | .770 | .741 | .799 | .117
SCWS [13] | .841 | .818 | .901 | .049 | .879 | .894 | .924 | .051 | .813 | .751 | .856 | .060 | .883 | .892 | .938 | .038 | .821 | .815 | .877 | .078 | .782 | .791 | .833 | .090
Ours | .796 | .715 | .821 | .070 | .856 | .838 | .873 | .069 | .792 | .695 | .812 | .072 | .855 | .826 | .882 | .057 | .748 | .760 | .772 | .147 | .740 | .704 | .758 | .119
|  |  |  |  |  |
---|---|---|---|---|---|---
Image | GT | F3Net [1] | MSFNet [36] | CTDNet[37] | SSAL [4] | Ours
Fig. 3: Visual comparison with benchmark saliency detection models, where
F3Net [1], MSFNet [36] and CTDNet[37] are fully-supervised saliency detection
models, and SSAL [4] is a scribble-supervised weakly-supervised saliency
detection model.

###  3.5 Background Accuracy Constrain

As background of the bounding box supervision is accurate background for
saliency prediction, we adopt partial cross-entropy loss to constrain accuracy
of prediction within the background bounding box region. Specifically, given
bounding box annotation $y$ and model prediction from the saliency prediction
network $s$, we define the background loss as:

$\mathcal{L}_{back}=\gamma\mathcal{L}_{ce}(s*(1-y),\mathbf{0}),$ (3)

where $\gamma=\frac{H*W}{H*W-z}$ ($H$ and $W$ represent image size, $z$ is the
number of pixels that are covered in the foreground bounding box),
$\mathbf{0}$ is an all-zero matrix of the same size as $s$.

###  3.6 Training the Model

With both the saliency regression loss $\mathcal{L}_{spn}$, foreground
structure loss $\mathcal{L}_{fore}$ and background accuracy loss
$\mathcal{L}_{back}$, our final loss function is defined as:

$\mathcal{L}_{f}=\mathcal{L}_{spn}+\lambda_{1}\mathcal{L}_{fore}+\lambda_{2}\mathcal{L}_{back},$
(4)

where $\lambda_{1}$ and $\lambda_{2}$ is used to control the contribution of
the foreground loss and the background loss. Empirically, we set
$\lambda_{1}=\lambda_{2}=1$.

We train our final model for 40 epochs. The saliency prediction network is
initialized with ResNet50 pre-trained on ImageNet. The initial learning rate
is 2.5e-4, and the decay epoch is 20 while the decay rate is 0.9. As the
training batch size be set up to 16 on a PC with an NVIDIA GeForce GTX 1080ti
GPU, the training process of the network takes 8 hours.

##  4 Experimental Results

###  4.1 Setup

Data set: We train our models using the DUTS training dataset [5]
$D=\\{x_{i},y_{i}\\}_{i=1}^{N}$ of size $N=10,553$, and test on six other
widely used datasets: the DUTS testing dataset, ECSSD [27], DUT [28], HKU-IS
[29], PASCAL-S [30] and the SOD testing dataset [31]. The supervision $y_{i}$
in our case is bounding box annotation.

Table 2: Performance of ablation study related experiments.  |  DUTS [5] |  ECSSD [27] |  DUT [28] |  HKU-IS [29] |  PASCAL-S [30] |  SOD [31]
---|---|---|---|---|---|---
Method | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$ | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$ | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$ | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$ | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$ | $S_{\alpha}\uparrow$ | $F_{\beta}\uparrow$ | $E_{\xi}\uparrow$ | $\mathcal{M}\downarrow$
BndBox | .642 | .514 | .673 | .167 | .683 | .620 | .712 | .185 | .671 | .533 | .701 | .149 | .683 | .597 | .722 | .167 | .621 | .608 | .669 | .232 | .625 | .552 | .669 | .203
GCut | .795 | .707 | .814 | .071 | .856 | .834 | .871 | .069 | .787 | .681 | .800 | .075 | .852 | .820 | .874 | .059 | .745 | .751 | .762 | .151 | .733 | .689 | .747 | .120
FGCut | .787 | .715 | .823 | .069 | .854 | .846 | .883 | .068 | .778 | .687 | .806 | .070 | .851 | .835 | .892 | .055 | .733 | .749 | .760 | .157 | .724 | .694 | .750 | .124
Ours | .796 | .715 | .821 | .070 | .856 | .838 | .873 | .069 | .792 | .695 | .812 | .072 | .855 | .826 | .882 | .057 | .748 | .760 | .772 | .147 | .740 | .704 | .758 | .119

Evaluation Metrics: Four evaluation metrics are used, including Mean Absolute
Error (MAE $\mathcal{M}$), Mean F-measure ($F_{\beta}$), mean E-measure
($E_{\xi}$) and the S-measure $S_{\alpha}$ [41]

###  4.2 Performance Comparison

Quantitative comparison: We show performance of our model in Table 1, where
models in the top two blocks are fully supervised models (models in the middle
block are transformer [42] based), and models in the last block (not “Ours”)
are weakly supervised models. Performance comparison show competitive
performance of our model, leading to an alternative weakly supervised saliency
detection model.

Qualitative comparison: We compare predictions of our model with four
benchmark models and show results in Fig. 3, which further explain that with
both the foreground and background constrains, our weakly supervised model can
obtain relative structure accurate predictions.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
Image | GT | BndBox | GCut | FGCut | Ours
Fig. 4: Visual comparison of the ablation study related experiments, where
each model (“BndBox”, “GCut” and “FGCut”) is introduced in the ablation study
section.

Running time comparison: As both the foreground and background related loss
are only used during training. At test time, we can produce saliency maps with
the saliency prediction network, leading to an average inference time of
0.02s/image, which is comparable with existing techniques.

###  4.3 Ablation Study

We conducted further experiments to explain the contribution of each component
of the proposed model, and show performance of the related experiments in
Table 2.

Training directly with bounding box supervision $y_{i}$: Given bounding box
supervision, a straight-forward solution is training directly with the binary
bounding box as supervision. We show it’s performance as “BndBox”.

Training directly with pseudo label from GrabCut: With the refined pseudo
label $g_{i}$ using GrabCut, we can train another model with $g_{i}$ as pseudo
label directly. The performance is shown as “GCut”.

Contribution of foreground structure loss: We further add the foreground
structure loss to “GCut”, leading to “FGCut”.

Analysis: As shown in Table 2, directly training with bounding box supervision
yields unsatisfactory results, where the model learns to regress the bounding
box region (see “BndBox” in Fig. 4). Although the pseudo saliency map with
GrabCut is noisy, the model based on it can still generate reasonable
predictions (see “GCut” in both Table 2 and Fig. 4.). Then, with the proposed
foreground and background loss as constrains ($\mathcal{L}_{fore}$ and
$\mathcal{L}_{back}$), we obtain better performance with more accurte
structures.

###  4.4 Model Analysis

We analyse the model further to explain the advantage and limitations of the
proposed method.

Impact of the maximum training epoch: We set the maximum epoch as 40 in this
paper. As proven in existing noisy labeling literature that longer training
time is harmful for noisy labeling setting. We further performed experiments
with both longer and shorter training epochs, and observed similar conclusion.
We will investigate the optimal training epochs for better performance.

Impact of the feat channel $C$ for new backbonf feature generation in the
“saliency prediction network”: For dimension reduction, we feed the backbone
feature $\\{s_{k}\\}_{k=1}^{4}$ to four different $3\times 3$ convolutional
layers to generate the new backbone feature $\\{s^{\prime}_{k}\\}_{k=1}^{4}$
of channel size $C=64$. We find that model performance is influenced with $C$.
The larger $C$ leads to better overall performance. However, the size of the
model is also significantly enlarged. To achieve trade off between model
performance and training/testing time, we set $C=64$. We will investigate the
optimal $C$ in the future.

Edge detection as auxiliary task for prediction structure recovery: [4]
introduced auxiliary edge detection module to their weakly supervised learning
framework for structure recovery. We have tried the same strategy and observed
no significant performance improvement in our setting. As a multi-task
learning framework, the convergence rate of each task is especially important
for the final performance, and more sophisticated multi-task learning solution
is important to fully explore the contribution of auxiliary edge detection for
weakly supervised learning.

##  5 Conclusion

In this paper, we introduce a bounding box based weakly supervised saliency
detection model. Due to the different accuracy of foreground and background,
we introduce two sets of loss functions to constrain the predictions within
the foreground and background bounding box regions. Experimental results
explain the superiority of the proposed solution, leading to an alternative
for weakly supervised saliency detection. However, we observe that model
performance is sensitive to the maximum training epochs. Longer training will
lead to over-fitting on noisy labeling, while shorter training may lead to
less effective model. More research on optimal training mechanism for noisy
label learning should be investigated.

## References

  * [1] Qingming Huang Jun Wei, Shuhui Wang,  “F3net: Fusion, feedback and focus for salient object detection,”  in AAAI, 2020.
  * [2] Bo Wang, Quan Chen, Min Zhou, Zhiqiang Zhang, Xiaogang Jin, and Kun Gai,  “Progressive feature polishing network for salient object detection.,”  in AAAI, 2020, pp. 12128–12135.
  * [3] Jun Wei, Shuhui Wang, Zhe Wu, Chi Su, Qingming Huang, and Qi Tian,  “Label decoupling framework for salient object detection,”  in CVPR, June 2020.
  * [4] Jing Zhang, Xin Yu, Aixuan Li, Peipei Song, Bowen Liu, and Yuchao Dai,  “Weakly-supervised salient object detection via scribble annotations,”  in CVPR, 2020.
  * [5] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan,  “Learning to detect salient objects with image-level supervision,”  in CVPR, 2017, pp. 136–145.
  * [6] Duc Tam Nguyen, Maximilian Dax, Chaithanya Kumar Mummadi, Thi-Phuong-Nhung Ngo, Thi Hoai Phuong Nguyen, Zhongyu Lou, and Thomas Brox,  “Deepusps: Deep robust unsupervised saliency prediction via self-supervision,”  in NeurIPS, 2019, pp. 204–214.
  * [7] Jing Zhang, Tong Zhang, Yuchao Dai, Mehrtash Harandi, and Richard Hartley,  “Deep unsupervised saliency detection: A multiple noisy labeling perspective,”  in CVPR, 2018, pp. 9029–9038.
  * [8] Jing Zhang, Jianwen Xie, and Nick Barnes,  “Learning noise-aware encoder-decoder from noisy labels by alternating back-propagation for saliency detection,”  in ECCV, 2020.
  * [9] Dingwen Zhang, Junwei Han, and Yu Zhang,  “Supervision by fusion: Towards unsupervised learning of deep salient object detector,”  in ICCV, 2017, pp. 4048–4056.
  * [10] Guanbin Li, Yuan Xie, and Liang Lin,  “Weakly supervised salient object detection using image labels,”  in AAAI, 2018.
  * [11] Yu Zeng, Yunzhi Zhuge, Huchuan Lu, Lihe Zhang, Mingyang Qian, and Yizhou Yu,  “Multi-source weak supervision for saliency detection,”  in CVPR, 2019, pp. 6074–6083.
  * [12] Yuxuan Liu, Pengjie Wang, Ying Cao, Zijian Liang, and Rynson W. H. Lau,  “Weakly-supervised salient object detection with saliency bounding boxes,”  TIP, pp. 4423–4435, 2021.
  * [13] Siyue Yu, Bingfeng Zhang, Jimin Xiao, and Eng Gee Lim,  “Structure-consistent weakly supervised salient object detection with local saliency coherence,”  in AAAI, 2021, pp. 3234–3242.
  * [14] Jing Zhang, Yuchao Dai, Tong Zhang, Mehrtash Harandi, Nick Barnes, and Richard Hartley,  “Learning saliency from single noisy labelling: A robust model fitting perspective,”  TPAMI, vol. 43, no. 8, pp. 2866–2873, 2020.
  * [15] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger,  “On calibration of modern neural networks,”  in ICML, 2017, pp. 1321–1330.
  * [16] Zuyao Chen, Qianqian Xu, Runmin Cong, and Qingming Huang,  “Global context-aware progressive aggregation network for salient object detection,”  in AAAI, 2020, pp. 10599–10606.
  * [17] Xiaoning Zhang, Tiantian Wang, Jinqing Qi, Huchuan Lu, and Gang Wang,  “Progressive attention guided recurrent network for salient object detection,”  in CVPR, 2018, pp. 714–722.
  * [18] Youwei Pang, Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu,  “Multi-scale interactive network for salient object detection,”  in CVPR, 2020, pp. 9413–9422.
  * [19] Siyue Yu, Bingfeng Zhang, Jimin Xiao, and Eng Gee Lim,  “Structure-consistent weakly supervised salient object detection with local saliency coherence,”  in AAAI, 2021.
  * [20] Wenguan Wang, Shuyang Zhao, Jianbing Shen, Steven CH Hoi, and Ali Borji,  “Salient object detection with pyramid attention and salient edges,”  in CVPR, 2019, pp. 1448–1457.
  * [21] Jia-Xing Zhao, Jiang-Jiang Liu, Deng-Ping Fan, Yang Cao, Jufeng Yang, and Ming-Ming Cheng,  “Egnet: Edge guidance network for salient object detection,”  in ICCV, 2019, pp. 8779–8788.
  * [22] Yang Wang, Yi Yang, Zhenheng Yang, Liang Zhao, Peng Wang, and Wei Xu,  “Occlusion aware unsupervised learning of optical flow,”  in CVPR, 2018, pp. 4884–4893.
  * [23] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake,  “” grabcut” interactive foreground extraction using iterated graph cuts,”  ACM transactions on graphics (TOG), vol. 23, no. 3, pp. 309–314, 2004.
  * [24] Cheng-Chun Hsu, Kuang-Jui Hsu, Chung-Chi Tsai, Yen-Yu Lin, and Yung-Yu Chuang,  “Weakly supervised instance segmentation using the bounding box tightness prior,”  in NeurIPS, 2019, vol. 32.
  * [25] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun,  “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,”  TPAMI, 2020.
  * [26] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey,  “Symmetric cross entropy for robust learning with noisy labels,”  in ICCV, 2019.
  * [27] Qiong Yan, Li Xu, Jianping Shi, and Jiaya Jia,  “Hierarchical saliency detection,”  in CVPR, 2013, pp. 1155–1162.
  * [28] C. Yang, L. Zhang, H. Lu, X. Ruan, and M. Yang,  “Saliency detection via graph-based manifold ranking,”  in CVPR, 2013, pp. 3166–3173.
  * [29] Guanbin Li and Yizhou Yu,  “Visual saliency based on multiscale deep features,”  in CVPR, 2015, pp. 5455–5463.
  * [30] Yin Li, Xiaodi Hou, Christof Koch, James M Rehg, and Alan L Yuille,  “The secrets of salient object segmentation,”  in CVPR, 2014, pp. 280–287.
  * [31] Vida Movahedi and James H Elder,  “Design and perceptual validation of performance measures for salient object segmentation,”  in CVPR Workshop, 2010, pp. 49–56.
  * [32] Zhe Wu, Li Su, and Qingming Huang,  “Stacked cross refinement network for edge-aware salient object detection,”  in ICCV, 2019, pp. 7264–7273.
  * [33] Jun Wei, Shuhui Wang, and Qingming Huang,  “F3net: Fusion, feedback and focus for salient object detection,”  in AAAI, 2020, pp. 12321–12328.
  * [34] Huajun Zhou, Xiaohua Xie, Jian-Huang Lai, Zixuan Chen, and Lingxiao Yang,  “Interactive two-stream decoder for accurate and fast saliency detection,”  in CVPR, 2020, pp. 9141–9150.
  * [35] Binwei Xu, Haoran Liang, Ronghua Liang, and Peng Chen,  “Locate globally, segment locally: A progressive architecture with knowledge review network for salient object detection,”  in AAAI, 2021, pp. 3004–3012.
  * [36] Miao Zhang, Tingwei Liu, Yongri Piao, Shunyu Yao, and Huchuan Lu,  “Auto-msfnet: Search multi-scale fusion network for salient object detection,”  in ACM Int. Conf. Multimedia, 2021, pp. 667–676.
  * [37] Zhirui Zhao, Changqun Xia, Chenxi Xie, and Jia Li,  “Complementary trilateral decoder for fast and accurate salient object detection,”  in ACM Int. Conf. Multimedia, 2021, pp. 4967–4975.
  * [38] Nian Liu, Ni Zhang, Kaiyuan Wan, Ling Shao, and Junwei Han,  “Visual saliency transformer,”  in ICCV, 2021, pp. 4722–4732.
  * [39] Jing Zhang, Jianwen Xie, Nick Barnes, and Ping Li,  “Learning generative vision transformer with energy-based latent space for saliency prediction,”  in NeurIPS, 2021, vol. 34.
  * [40] Xin Li, Fan Yang, Hong Cheng, Wei Liu, and Dinggang Shen,  “Contour knowledge transfer for salient object detection,”  in ECCV, 2018, pp. 355–370.
  * [41] Deng-Ping Fan, Ge-Peng Ji, Xuebin Qin, and Ming-Ming Cheng,  “Cognitive vision inspired object segmentation metric and loss function,”  SCIENTIA SINICA Informationis, 2021.
  * [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in NeurIPS, 2017.