# Elucidating the roles of collision energy and photon momentum transfer in
the formation of ultralong-range Rydberg molecules

C. Wang  Department of Physics and Astronomy, Rice University, Houston, TX
77005-1892, USA  Smalley-Curl Institute, Rice University, Houston, TX
77005-1892, USA  Y. Lu  S. K. Kanungo  Department of Physics and Astronomy,
Rice University, Houston, TX 77005-1892, USA  F. B. Dunning  T. C. Killian
Department of Physics and Astronomy, Rice University, Houston, TX 77005-1892,
USA  Smalley-Curl Institute, Rice University, Houston, TX 77005-1892, USA  S.
Yoshida  Institute for Theoretical Physics, Vienna University of Technology,
Vienna, Austria, EU

###### Abstract

Spectroscopic measurements of the rotational distribution of 84Sr and 86Sr
$~{}5sns~{}^{1}S_{0}$ ultralong-range Rydberg molecular dimers created via
photoassociation in a cold gas are reported. The dimers are produced by two-
photon excitation via the 5s5p ${}^{1}P_{1}$ intermediate state. The use of
singlet states permits detailed study of the roles that the initial atom-atom
interaction, photon momentum transfer during Rydberg excitation, and sample
temperature play in determining the spectral lineshape and final dimer
rotational distribution. The results are in good agreement with the
predictions of a model that includes these effects. The present work further
highlights the sensitivity of ultralong-range Rydberg molecule formation to
the state of the initial cold gas.

##  I Introduction

Interest in ultralong-range Rydberg molecules (ULRRMs), which comprise a
Rydberg atom in whose electron cloud are embedded one (or more) weakly-bound
ground-state atoms, has increased steadily over the years [1, 2, 3, 4, 5].
Such molecules, which represent a new molecular class, have provided a
valuable microscale laboratory in which to study low-energy electron-atom
scattering at energies not readily accessible using alternate techniques [6,
7], have furnished a powerful probe of non-local spatial correlations in cold
quantum gases [8, 9], illuminating the important role played by quantum
statistics, and have, through measurements of ULRRM formation in dense Bose-
Einstein condensates, provided the opportunity to study many-body phenomena,
such as the creation of Rydberg polarons [10, 11, 12, 13].

ULRRMs are bound by scattering of the Rydberg electron from the ground state
atom which, as demonstrated in earlier studies, can frequently be described
using a zero-range Fermi pseudo-potential [14, 15]. The resulting molecular
Born-Oppenheimer potential can support multiple vibrational states. In the
case of strontium $ns$ Rydberg states, the ground $v=0$ vibrational state is
of particular interest because it is strongly localized near the outer
classical turning point at an internuclear separation, $R_{n}\sim
1.8(n-\delta)^{2}a_{0}$, where $a_{0}$ is the Bohr radius, and $\delta$ is the
quantum defect.

The vibrational structure of ULRRMs has been examined in a number of earlier
studies. Recent work, however, extended measurements to include the study of
their rotational structure [15]. These studies employed 86Sr 5sns
${}^{3}S_{1}$ dimers because the scattering length for 86Sr-86Sr collisions is
unusually large, $a_{s}=811a_{0}$, and is comparable in size to the
internuclear separation for $v=0$ strontium ULRRM dimers with values of $n\sim
25$. Since the value of $a_{s}$ defines the position of a node in the atom-
atom scattering wavefunction, the probability of finding an atom pair with
initial separations close to this value is reduced. Thus, for values of $n$
near 25, the Franck-Condon overlap between the initial scattering state and
the final ($v$=0) molecular state is reduced, thereby suppressing dimer
formation via the $s$-wave channel. This, in turn, allows the effects of
higher-partial-wave scattering, which can lead to the creation of
rotationally-excited dimers, to be more visible. Furthermore, in the formation
of ULRRM dimers only one atom in the initial ground-state pair acquires photon
momentum through its excitation which, given the large size of the product
dimer, can lead to significant angular momentum transfer to the atom pair. The
earlier experimental results for ${}^{3}S_{1}$ dimers, however, were not in
good agreement with the predictions of a theoretical model which incorporated
the effects of both the scattering length and the recoil photon momentum. The
interaction of the electron spins with the molecular rotational motion was
estimated to be small and was neglected. In this paper photoassociation
spectra for Rydberg dimers created using singlet $5sns$ ${}^{1}S_{0}$ Rydberg
states are analyzed to examine if the discrepancies observed in the previous
study are peculiar to strontium triplet Rydberg states.

The 86Sr 5sns ${}^{3}S_{1}~{}(m=+1)$ Rydberg dimers employed in the earlier
experiments [15] were created by two-photon excitation via the intermediate
5s5p ${}^{3}P_{1}$ state using lasers operating at 689 and 320 nm. The role of
photon momentum transfer could not be clearly identified because, even when
the laser beams counter-propagate, significant momentum transfer occurs. Here
we remove this limitation by studying the formation of 5sns ${}^{1}S_{0}$
dimers created by two-photon excitation via the intermediate 5s5p
${}^{1}P_{1}$ state, which requires lasers operating at 461 and 413 nm. Thus,
if the lasers counter-propagate, the net photon momentum transfer is very
small, thereby providing a benchmark against which to identify the effects of
photon momentum transfer. To examine the effects of photon momentum transfer,
co-propagating laser beams are employed.

The effects of the initial atom-atom interaction are explored through
comparative measurements using 84Sr for which the $s$-wave scattering length,
$a_{s}=123$$a_{0}$, is much less than that for 86Sr.

The experimental measurements are compared with the results of model
calculations that include the effects of both the initial atom-atom
interaction and of photon momentum transfer. The theoretical predictions are
in good agreement with experiment and highlight the important role that these
factors, together with the initial cold gas temperature, play in determining
the rotational distribution of the product ULRRMs.

##  II Experimental method

The experimental methods are described in detail elsewhere [16, 17, 18].
Briefly, strontium atoms are laser-cooled to a few microKelvin and loaded into
a “pancake”-shaped optical dipole trap (ODT) formed from two crossed
elliptical 1064 nm laser beams. Evaporative cooling is used to further lower
and control the atom temperature. The Rydberg molecules are created by two-
photon excitation via the intermediate 5s5p${}^{1}P_{1}$ state. The necessary
radiation at 413 and 461 nm is provided by frequency-doubled diode laser
systems that are stabilized to high-finesse ultralow-expansion optical
cavities. To limit trap loss and heating from photon scattering, the 461 nm
laser was blue-detuned by $\sim 18$ GHz. The photoexcitation lasers are
typically applied for $\sim 20~{}\mu$s. Whereas the laser linewidths are
small, typically a few kHz, the effective overall linewidth, measured by
observing the excitation of atomic Rydberg states using counter-propagating
beams, is larger $\sim 50$ kHz, and is limited by transform broadening (when
using counter-propagating laser beams Doppler broadening is minimal.) It is,
however, a significant improvement in linewidth as compared to that in earlier
measurements, $\sim 120-140$ kHz [15], and is sufficiently narrow as to allow
detailed spectroscopic studies of molecular photoassociation.

The ODT was loaded with $\sim 5-10\times 10^{5}$ atoms, resulting in trap
densities of up to $\sim 1\times 10^{13}$ cm-3. The final atom number, and
temperature, were determined through absorption imaging on the
$5s^{2}~{}^{1}S_{0}\rightarrow 5s5p~{}^{1}P_{1}$ transition following release
of the atoms from the ODT and a fall time of $\sim 20$ ms.

The number of Rydberg molecules produced is determined through ionization in a
pulsed electric field [19, 20]. The resulting electrons are directed to a
dual-microchannel-plate for detection and counted. Experimental limitations on
the size of the ionizing field that could be generated in the experimental
volume limited measurements to states with $n\geq 29$. Excitation rates were
kept low, typically $\lesssim 0.3$ per experimental cycle, to minimize
possible effects due to Rydberg-Rydberg interactions. The need to maintain a
constant sample temperature and density during a set of measurements limited
the number of experimental cycles that could be undertaken using a single
cold-atom sample to $\sim 200$.

##  III Theoretical analysis

The present results are interpreted using a theoretical model that has been
described in detail previously [15], and which includes the effects of ground-
state atom-atom interactions in the initial scattering channel, together with
the recoil momentum associated with photon momentum transfer.

Since a Rydberg dimer is formed by a weak perturbation to isolated atoms, the
two atoms in the molecule can be treated as composite, identical bosonic
particles as they are in the initial scattering state. Thus, the Hamiltonian
of the interacting atoms is invariant with an exchange of both electron and
ion coordinates simultaneously, and the wavefunctions are symmetric with
respect to the overall exchange. However, for an initial scattering state in
which both atoms are in the same electronic state (the $5s^{2}$ ground state),
the wavefunction becomes symmetric with respect to only the ion coordinates.
When a partial-wave representation $|k,N^{\prime},M^{\prime}_{N}\rangle$ is
adopted for the relative motion of the colliding pair, where $N^{\prime}$ is
the rotational quantum number, $M_{N}^{\prime}$ its projection on the
quantization axis, and $\hbar$k is the relative momentum, only even partial
waves are allowed for the initial scattering state. Thus, when the effects of
recoil photon momentum are negligible, only even, $N=0,2,4,...$, rotational
states of the Rydberg dimer can be created. Since the total orbital angular
momentum of the valence electrons in strontium vanishes, the total mechanical
angular momentum of the product dimer is completely characterized by the
quantum numbers $N$ and $M_{N}$. The transition amplitude is given by an
inelastic form factor that may be written

$\displaystyle F_{v,N,M_{N}}(k,N^{\prime},M_{N}^{\prime})$
$\displaystyle=\frac{1}{\sqrt{2}}\langle
v,N,M_{N}|[e^{-i(\vec{\kappa}/2)\cdot\vec{R}}+(-1)^{N}e^{i(\vec{\kappa}/2)\cdot\vec{R}}]|k,N^{\prime},M^{\prime}_{N}\rangle$

where $v$ denotes the final vibrational state and $\vec{\kappa}$ is the sum of
the wave vectors associated with the laser fields. Here we assume no mixing
between vibrational and rotational levels for the ground vibrational level
since the energy difference between the vibrational levels $v=0$ and 1 is
typically much larger than the rotational energies. This form factor is
similar to that used in molecular spectroscopy except that it includes the
effects of photon recoil momentum. The form factor is evaluated using
spherical representations of both the initial asymptotic scattering state of
the atomic pair and the final state of the Rydberg molecule whence the
transition amplitude can be approximated by

$F_{v,N,M_{N}}(k,N^{\prime},M^{\prime}_{N})\propto j_{\lambda}(\kappa
R_{n}/2)\tilde{j}_{N^{\prime}}(kR_{n}),$ (2)

where $\lambda\simeq|N-N^{\prime}|$ is the change in rotational quantum number
and

$\tilde{j}_{N^{\prime}}(kR)=\left\\{\begin{array}[]{ll}\frac{\sin[k(R-a_{s})]}{kR}&N^{\prime}=0,\\\
j_{N^{\prime}}(kR)&N^{\prime}>0\end{array}\right.$ (3)

where $j_{N^{\prime}}(kR)$ is the spherical Bessel function. This form factor
resembles the Franck-Condon (FC) overlap between the initial scattering state
and final molecular state. The Franck-Condon principle, however, argues that
electronically- and vibrationally-excited states are formed by “vertical”
transitions at a given internuclear separation, and that rotational angular
momentum will be conserved during photoassociation. The non-diagonal
transitions with $N-N^{\prime}\neq 0$ seen here result from the interplay of
photon momentum recoil and the large molecular size. The angular momentum
transferred to the dimer, $\sim\kappa R_{n}/2$, when using counter-propagating
beams, $\sim 0.04$ a.u., is very much less than when using co-propagating
beams, $\sim 0.9$ a.u..

The final excitation strength, $f(\omega)$, is obtained by averaging over the
Boltzmann distribution of the relative momenta for atoms in the trap and the
effective laser linewidths,

$\displaystyle f(\omega)$ $\displaystyle\propto$
$\displaystyle\sqrt{\frac{2}{\pi}}\left(\frac{\hbar^{2}}{\mu
k_{B}T}\right)^{3/2}\sum_{v,N,M_{N}}\int dkk^{2}e^{-h^{2}k^{2}/(2\mu k_{B}T)}$
(4)
$\displaystyle\times\sum_{N^{\prime},M^{\prime}_{N}}|F_{v,N,M_{N}}(k,N^{\prime},M_{N}^{\prime})|^{2}L_{v,N}(k,\omega),$

where $L_{v,N}$ is a normalized line-shape function which, for the present
experimental data, can be well approximated by a distribution including the
Doppler broadening induced by the center of mass momentum $\hbar\vec{k}_{\rm
CM}$ of the dimer,

$\displaystyle L_{v,N}(k,\omega)$
$\displaystyle=\frac{1}{\pi}\left(\frac{\hbar^{2}}{2\pi
Mk_{B}T}\right)^{3/2}\int d\vec{k}_{\rm CM}e^{-h^{2}k_{\rm
CM}^{2}/(2Mk_{B}T)}$
$\displaystyle\times\frac{2\Gamma}{4(\omega-\hbar\vec{\kappa}\cdot\vec{k}_{\rm
CM}/M+\hbar k^{2}/(2\mu)-E_{v,N}/\hbar)^{2}+\Gamma^{2}}$ (5)

where $M$ is the total mass of the dimer and $\mu=M/4$ the reduced mass,
$\hbar^{2}k^{2}/2\mu$ the initial relative kinetic energy of the collision
pair, $\omega$ the two-photon detuning from atomic resonance for an atom at
rest, and $\Gamma$ the full-width-at-half-maximum (FWHM) of the effective
instrumental linewidth. The peak position in Eq. 5 is determined by the
initial relative kinetic energy of the collision pair, i.e.,
$E_{v,N}\simeq\hbar\omega+\hbar^{2}k^{2}/(2\mu)$ on resonance, and energy
conservation during excitation. Note that thermal averaging, i.e., the
convolution of Lorentzians with the Boltzmann distributions contained in Eqs.
4 and 5 leads to an increased linewidth and asymmetric non-Lorentzian overall
line profiles [21] that can be calculated using the molecular binding energies
$E_{v,N}$ and the Franck-Condon overlap evaluated from the molecular
Hamiltonian.

In order to extract rotational energies $E_{v=0,N}$ and the contribution from
each rotational level to the photoassociation spectra, the measured spectra
are fit using the expression

$\displaystyle f_{\rm fit}(\omega)$ $\displaystyle=$
$\displaystyle\left(\frac{2\hbar^{2}}{\pi\mu
k_{B}T}\right)^{3/2}\sum_{N}C_{N}$ $\displaystyle\times\int
dk\,k^{2}\frac{|{\cal F}_{N}(k)|^{2}~{}\Gamma e^{-h^{2}k^{2}/(2\mu
k_{B}T)}}{4(\omega+\hbar k^{2}/(2\mu)-E_{v=0,N}/\hbar)^{2}+\Gamma^{2}}$

and the measured sample temperatures $T$ while treating the effective
linewidth $\Gamma$, $E_{v=0,N}$, and weights, $C_{N}$, of the contributions
from each rotational level as adjustable parameters. The Franck-Condon overlap
is approximated using Eq. 2 as

${\cal F}_{N}(k)=\sum_{N^{\prime}}j_{\lambda}(\kappa
R_{n}/2)\tilde{j}_{N^{\prime}}(kR_{n})$ (7)

with $\lambda=|N-N^{\prime}|=0$ for even $N$ and 1 for odd $N$. In fitting the
data for counter-propagating beams, i.e. in the limit of small $\kappa$, it is
further assumed that $j_{\lambda}(\kappa
R_{n}/2)\sim\delta_{\lambda,0}=\delta_{N,N^{\prime}}$. Sizable uncertainties
in the fitted values, however, can result if the different contributions are
not well-resolved. As will be demonstrated, the excellent quality of the fits
obtained using the predicted lineshapes, i.e., Eq. LABEL:eq:fit, provides
strong support for the present model.

##  IV Results and discussion

Figure 1 shows photoassociation spectra recorded when creating

Figure 1: Photoassociation spectra recorded when creating ground-state $v=0$
86Sr $5s29s^{1}$S0 ULRRM dimers using counter-propagating laser beams and the
sample temperatures indicated. To emphasize the changes in spectral lineshape
that occur, the results in each panel are normalized to the same peak height.
Fits obtained using the sum of contributions from just two states are included
(see text). Note that the horizontal axis in this, and later, figures is
expressed in units of frequency, i.e., $\omega/2\pi$.

86Sr $n=29$ ${}^{1}S_{0}$ dimers in the ground, $v=0$, vibrational state using
counter-propagating laser beams and a number of representative sample
temperatures. To enable a more direct comparison of the spectral profiles,
each data set is normalized to the same peak height. (As the sample
temperature is reduced, however, the number of atoms remaining in the trap is
reduced, resulting in lower signal levels.) As shown in Fig. 1, each measured
spectrum can be well fit using Eq. LABEL:eq:fit and the sum of contributions
from just two product states whose separations, $\Delta E$, are observed to be
essentially independent of sample temperature.

These observations point to the formation of just two

Rydberg | levels | level spacings, kHz
---|---|---
state |  | A | B | C | D
5s29s${}^{1}S_{0}$ | N=0 to N=2 | 166 | 177 | 167$\pm$12 | 173$\pm$12
5s30s${}^{1}S_{0}$ | N=0 to N=2 | 142 | 152 | 140$\pm$11 | 141$\pm$11
5s31s${}^{1}S_{0}$ | N=0 to N=2 | 122 | 132 | 127$\pm$10 | 123$\pm$10
5s32s${}^{1}S_{0}$ | N=0 to N=2 | 105 | 114 | 106$\pm$9 | 113$\pm$9
5s33s${}^{1}S_{0}$ | N=0 to N=2 | 92 | 100 | 94$\pm$8 | 92$\pm$8
5s29s${}^{1}S_{0}$* | N=0 to N=1 | 55 | 59 | 54$\pm$7 |
| N=0 to N=2 | 166 | 177 | 179$\pm$25 |
Table 1:  Calculated and fitted level spacings for the transitions indicated.
The asterisk denotes measurements using co-propagating rather than counter-
propagating, laser beams. Column A shows rotational level spacings derived
from the eigenenergies $E_{v=0,N}$ of the Rydberg dimer Hamiltonian. Column B
lists the approximate values obtained when assuming the molecule behaves as a
rigid rotor (Eq. 8). Columns C and D show level separations deduced from fits
to measured 86Sr and 84Sr spectra, respectively, and represent the average of
measurements taken at several different sample temperatures and trap operating
conditions.

different final molecular states. The energy separations, $\Delta E$,
extracted from fitting the measured data are shown in Table 1. Also included
in Table 1 are the calculated values of $\Delta E$ derived from the
eigenenergies of the Rydberg dimer Hamiltonian together with those predicted
by assuming the molecule behaves as a rigid rotor, in which event the
rotational levels can be approximated by

$E_{v,N}\simeq E_{v,N=0}+\frac{\hbar^{2}N(N+1)}{2\mu R_{n}^{2}}$ (8)

where $R_{n}$ is the internuclear separation, $\sim 1.8(n-\delta)^{2}a_{0}$,
for the ground vibrational state. As seen in Table 1, the separations in the
calculated eigenenergies between the $N=0$ and $N=2$ levels are very similar
to those seen in the experimental measurements, pointing to the creation of
$N=0$ and $N=2$ rotational states. We note that the separation between the
peaks in the photoassociation spectra that correspond to the production of
$N=0$ and 2 states is typically smaller by an amount comparable to the thermal
energy, $k_{B}T\sim 20-40$ kHz, as compared to their actual energy separations
(see Table 1) due to the thermal averaging.

Figure 2 shows, for

Figure 2:  Model predictions of the temperature dependence of the fractional
contributions from $N=1$ and 2 states to the total Sr $n=29^{1}S_{0}$ dimer
signal for both (a) counter- and (b) co-propagating laser beams. (a) includes
results for both 84Sr and 86Sr dimers, (b) results for only 86Sr dimers.
Values obtained from fits to the experimentally-measured spectra are also
shown (symbols).

$n=29$, the fractional contributions from $N=1$ and 2 states to the total
dimer signal as a function of sample temperature when using co- and counter-
propagating laser beams. Values derived from fits to the measured spectra,
obtained by integrating their separate contributions to the total signal, are
shown by symbols. The model predictions (lines) were calculated using the
numerically-integrated form factor (Eq.1). The calculated fractional
contributions from $N=2$ states increase steadily with sample temperature
reflecting the fact that the contributions from $N=2$ partial waves increase
with increasing collision energy in this temperature range (Eq. 2).

For counter-propagating beams (Fig. 2(a)) the model predictions are in good
agreement with the measured values. The small discrepancies seen at high
temperatures for 86Sr may be partly due to the fitting error as the overlap
between the $N=0$ and 2 features becomes larger. No significant production of
$N=1$ states is predicted, or seen, consistent with the near-zero net photon
momentum transfer.

Figure 3 compares photoassociation spectra recorded when creating $n=29$ to 31
86Sr

Figure 3:  (a)-(c) Photoassociation spectra recorded when creating ground-
state $v$=0 86Sr $n=29$ to 31 ${}^{1}S_{0}$ dimers using counter-propagating
laser beams and a sample temperature of $\sim 1.3\mu$K. The results are
normalized for small changes in trap density, laser powers, and oscillator
strengths. Fits obtained using the sum of contributions from just two states
(see text), are also shown. (d) $n$-dependence of the total dimer signal.

dimers at a fixed sample temperature (1.3 $\mu$K). These data were recorded on
a single day to ensure similar trap conditions and are normalized for small
variations in trap density, laser powers, and the $n$-dependence in the
oscillator strengths for their creation [8]. As $n$ decreases towards $n=25$
(for which $R_{n}$ nearly matches $a_{s}$), the overlap between the atomic
scattering wavefunction and the the ground-state molecular vibrational
wavefunction decreases. Since for the present sample temperatures $s$-wave
scattering typically dominates, this reduced overlap leads to a decrease in
total dimer signal. As seen in Fig. 3, the (normalized) total dimer signal
does decrease significantly as $n$ decreases from $n=31$ to $n=29$.
(Experiment shows that the dimer signal continues to grow as $n$ increases
beyond 31.) Figure 3 also includes two-component fits to the data. The
resulting level separations are included in Table 1 and are in very good
agreement with the model predictions. As $n$ increases, the separation between
the rotational levels also decreases, ($\hbar^{2}N(N+1)/(2\mu R_{n}^{2})\sim
n^{-4}$), and for $n\geq 34$ their associated features can no longer be
reliably resolved. Note that, as the sample temperature rises, so too does the
fractional contribution of $N=2$ states to the total dimer signal. Figure 4

Figure 4:  Model predictions of the fractional contribution of $N=2$ states to
the total dimer signal as a function of $n$ for both 84Sr and 86Sr together
with the results of experimental measurements. The results are for counter-
propagating laser beams and a sample temperature of $1.3~{}\mu$K.

shows the calculated fractional contribution from $N=2$ states as a function
of $n$ for a sample temperature of $1.3~{}\mu$K together with the fitted
values, which are again in reasonable agreement with model predictions.

To further illustrate the importance of $s$-wave suppression in the case of
86Sr dimers, Fig. 5 shows photoassociation

Figure 5: Photoassociation spectra recorded when creating 84Sr ${}^{1}S_{0}$
$v$=0 5s29s dimers using counter-propagating laser beams and the sample
temperatures indicated. Fits to the experimental data are also included. To
emphasize the changes in the spectral profiles that occur, the results are
normalized to the same peak height.

spectra recorded when creating 84Sr ULRRM dimers. For 84Sr, the $s$-wave atom-
atom scattering length is small, $a_{s}\sim 123a_{0}$, and there is no node in
the scattering wavefunction at the internuclear separations explored in the
present work. In consequence, dimer production via the $s$-wave scattering
channel is no longer suppressed. However, little change is expected in the
higher-partial-wave channels between 84Sr and 86Sr due to the presence of a
long-range centrifugal barrier. For 84Sr, therefore, $s$-wave scattering will
be much stronger as compared to higher-partial-wave scattering. The
measurements do, however, reveal a small contribution to the total 84Sr dimer
signal at a detuning that is consistent with that for creation of $N=2$ states
and that increases with sample temperature.

As seen in Fig. 4, measurements show that the fractional contribution of $N=2$
states to the total dimer signal also increases as $n$ increases, behavior
that is mirrored in the model predictions. The agreement between theory and
experiment is again very good, further validating the present model by
demonstrating it produces reliable predictions even when considering an atom
pair with very different $s$-wave scattering characteristics.

Given that the trap operating conditions for 84Sr are rather different to
those for 86Sr, it is difficult to normalize data sets obtained for each
species to one another to compare their absolute relative dimer production
rates. Nonetheless, attempts at such normalization suggest that, for $n=29$,
the dimer production rate for 84Sr is some ten times larger than that for
86Sr. As $n$ increases, however, this difference is seen to decrease due to
the decreasing $s$-wave suppression for 86Sr.

Consider now the case of co-propagating laser beams for which photon momentum
transfer is maximized.

Figure 6:  Photoassociation spectra recorded when creating 86Sr
$n=29~{}^{1}S_{0}$ dimers using co-propagating laser beams and the sample
temperatures indicated. Also included are fits obtained using the sum of three
states.

Figure 6 shows photoassociation spectra recorded using co-propagating laser
beams when creating $n=29~{}^{1}S_{0}$ dimers. Whereas for counter-propagating
beams Doppler broadening should be very small, that for co-propagating beams
will be significant, which is reflected in the increased widths of the
spectral features. As a result the spectral features are less well resolved
than for counter-propagating beams. Attempts to obtain a good fit to the
measured spectra assuming contributions from just two product states with
separations corresponding to that expected between the $N=0$ and $N=2$ levels
proved unsuccessful. However, as demonstrated in Fig. 6, the spectra can be
well fit by the sum of three separate contributions. The separations between
these three features are included in Table 1 and are consistent with those
predicted between the $N=0$ and $N=1$ and 2 features, pointing to the
formation of a mixture of all three.

Figure 2 includes the results of model simulations of the fractional
contributions from the $N=1$ and 2 rotational levels for co-propagating beams.
The predicted fractions of $N=2$ states, which are reasonably well resolved,
are in good agreement with experimental observations. The relative production
of $N=1$ states inferred from the fits to the data, however, is somewhat
higher than suggested by the model (see Fig. 2), which might be attributed to
uncertainties introduced by the considerable overlap of the $N=0$ and $N=1$
features. Nonetheless, even though these features are not well resolved, the
quality of the fits is such that the $N=1$ contribution and position (see
Table 1) are reasonably well determined. Clearly, $N=1$ states provide a
sizable contribution to the dimer signal demonstrating that photon momentum
transfer is very important in the creation of rotationally-excited dimers.

Earlier studies using $n=29~{}^{3}S_{1}$ dimers and co-propagating laser beams
(which results in the same total photon momentum transfer as for ${}^{1}S_{0}$
dimers) suggested the additional production of $N=3$ dimers [15]. However, no
evidence of their creation (at an expected detuning of $\sim 350$ kHz) was
seen in the present study of ${}^{1}S_{0}$ dimers. Furthermore, the present
model (see Fig. 2), does not predict the formation of significant numbers of
such states. This suggests that the electron spin angular momentum might play
a significant role in determining the final rotational distribution. However,
direct coupling between the electron spin $S$ and the rotational motion $N$ is
estimated be negligible. Further work, therefore, will be required to examine
any other mechanisms by which such spin to rotational angular momentum
transfer might occur.

##  V Conclusions

The present measurements demonstrate the important roles played by atom-atom
interactions, photon momentum transfer, and sample temperature in the
production of rotationally-excited ULRRMs and validate the model described in
Section III which includes their effects. In future studies with 86Sr it will
be interesting, for example, to see if similar rotational structure can be
resolved when creating the less-well-localized vibrationally-excited states,
or even ground-state trimers, as well as exploring effects associated with
reduced system mass by creating heteronuclear dimers in a two-component cold
gas containing 86Sr and another alkali or alkaline earth element. In addition,
direct comparisons to detailed measurements using 86Sr $n^{3}S_{1}$ dimers
will help unravel the role played by spin angular momentum in determining the
final distribution of rotational states.

###### Acknowledgements.

Research supported by the National Science foundation under Grants No.
PHY-1904294 and PHY-2110596, and the FWF(Austria) under Grant No.
FWF-P35539-N, and Doctoral College FWF W 1243(Solids4fun).

## References

  * Shaffer  _et al._ [2018] J. P. Shaffer, S. T. Rittenhouse, and H. R. Sadeghpour, Ultracold Rydberg molecules, Nat. Comm. 9, 1965 (2018).
  * Fey  _et al._ [2020] C. Fey, F. Hummel, and P. Schmelcher, Ultralong-range Rydberg molecules, Mol. Phys. 118, e1679401 (2020).
  * Saßmannshausen  _et al._ [2016] H. Saßmannshausen, J. Deiglmayr, and F. Merkt, Long-range Rydberg molecules, Rydberg macrodimers and Rydberg aggregates in an ultracold Cs gas, The European Physical Journal Special Topics 225, 2891 (2016).
  * Lippe  _et al._ [2019] C. Lippe, T. Eichert, O. Thomas, T. Niederprum, and H. Ott, Excitation of Rydberg molecules in ultracold quantum gases, Phys. Status Solidi (B) 256, 1800654 (2019).
  * Eiles [2019] M. T. Eiles, Trilobites, butterflies, and other exotic specimens of long-range Rydberg molecules, J. Phys. B At. Mol. Opt. Phys. 52, 113001 (2019).
  * Schlagmüller  _et al._ [2016] M. Schlagmüller, T. C. Liebisch, H. Nguyen, G. Lochead, F. Engel, F. Böttcher, K. M. Westphal, K. S. Kleinbach, R. Löw, S. Hofferberth, T. Pfau, J. Pérez-Ríos, and C. H. Greene, Probing an electron scattering resonance using Rydberg molecules within a dense and ultracold gas, Phys. Rev. Lett. 116, 053001 (2016).
  * Engel  _et al._ [2019] F. Engel, T. Dieterle, F. Hummel, C. Fey, P. Schmelcher, R. Löw, T. Pfau, and F. Meinert, Precision spectroscopy of negative-ion resonances in ultralong-range Rydberg molecules, Phys. Rev. Lett. 123, 073003 (2019).
  * Whalen  _et al._ [2019] J. D. Whalen, S. K. Kanungo, R. Ding, M. Wagner, R. Schmidt, H. R. Sadeghpour, S. Yoshida, J. Burgdörfer, F. B. Dunning, and T. C. Killian, Probing nonlocal spatial correlations in quantum gases with ultra-long-range Rydberg molecules, Phys. Rev. A 100, 011402 (2019).
  * Kanungo  _et al._ [2023] S. K. Kanungo, Y. Lu, F. B. Dunning, S. Yoshida, J. Burgdörfer, and T. C. Killian, Measuring nonlocal three-body spatial correlations with Rydberg trimers in ultracold quantum gases, Phys. Rev. A 107, 033322 (2023).
  * Camargo  _et al._ [2018] F. Camargo, R. Schmidt, J. D. Whalen, R. Ding, G. Woehl, S. Yoshida, J. Burgdörfer, F. B. Dunning, H. R. Sadeghpour, E. Demler, and T. C. Killian, Creation of Rydberg polarons in a Bose gas, Phys. Rev. Lett. 120, 083401 (2018).
  * Schmidt  _et al._ [2018] R. Schmidt, J. D. Whalen, R. Ding, F. Camargo, G. Woehl, S. Yoshida, J. Burgdörfer, F. B. Dunning, E. Demler, H. R. Sadeghpour, and T. C. Killian, Theory of excitation of Rydberg polarons in an atomic quantum gas, Phys. Rev. A 97, 022707 (2018).
  * Tempere  _et al._ [2009] J. Tempere, W. Casteels, M. K. Oberthaler, S. Knoop, E. Timmermans, and J. T. Devreese, Feynman path-integral treatment of the BEC-impurity polaron, Phys. Rev. B 80, 184504 (2009).
  * Scheuing and Pérez-Ríos [2023] T. Scheuing and J. Pérez-Ríos, Quasi-static lineshape theory for Rydberg excitations in high-density media, Atoms 11, 6 (2023).
  * Thomas  _et al._ [2018] O. Thomas, C. Lippe, T. Eichert, and H. Ott, Photoassociation of rotating ultra-long range Rydberg molecules, Journal of Physics B: Atomic, Molecular and Optical Physics 51, 155201 (2018).
  * Lu  _et al._ [2022] Y. Lu, J. D. Whalen, S. K. Kanungo, T. C. Killian, F. B. Dunning, S. Yoshida, and J. Burgdörfer, Resolving rotationally excited states of ultralong-range Rydberg molecules, Phys. Rev. A 106, 022809 (2022).
  * Stellmer  _et al._ [2014] S. Stellmer, F. Schreck, and T. C. Killian, Annual Review of Cold Atoms and Molecules (World Scientific, 2014) Chap. 1.
  * de Escobar  _et al._ [2009] Y. N. M. de Escobar, P. G. Mickelson, M. Yan, B. J. DeSalvo, S. B. Nagel, and T. C. Killian, Bose-einstein condensation of ${}^{84}\mathrm{Sr}$, Phys. Rev. Lett. 103, 200402 (2009).
  * DeSalvo  _et al._ [2015] B. J. DeSalvo, J. A. Aman, F. B. Dunning, T. C. Killian, H. R. Sadeghpour, S. Yoshida, and J. Burgdörfer, Ultra-long-range Rydberg molecules in a divalent atomic system, Phys. Rev. A 92, 031403 (2015).
  * Stebbings and Dunning [1983] R. F. Stebbings and F. B. Dunning, eds.,  _Rydberg States of Atoms and Molecules_ (Cambridge University Press, Cambridge, 1983).
  * Gallagher [1994] T. F. Gallagher,  _Rydberg Atoms_ (Cambridge University Press, Cambridge, 1994).
  * Jones  _et al._ [1999] K. M. Jones, P. D. Lett, E. Tiesinga, and P. S. Julienne, Fitting line shapes in photoassociation spectroscopy of ultracold atoms: A useful approximation, Phys. Rev. A 61, 012501 (1999).


11institutetext: National-Regional Key Technology Engineering Laboratory for
Medical Ultrasound, School of Biomedical Engineering, Health Science Center,
Shenzhen University, China
11email<EMAIL_ADDRESS>22institutetext: Medical Ultrasound Image Computing
(MUSIC) Lab, Shenzhen University, China 33institutetext: Marshall Laboratory
of Biomedical Engineering, Shenzhen University, China 44institutetext: School
of Biomedical Engineering and Information, Nanjing Medical University, China
55institutetext: Centre for Computational Imaging and Simulation Technologies
in Biomedicine (CISTIB), University of Leeds, UK 66institutetext: Leeds
Institute of Cardiovascular and Metabolic Medicine, University of Leeds, UK
77institutetext: Medical Imaging Research Center (MIRC), KU Leuven, Belgium

# Self Context and Shape Prior for Sensorless Freehand 3D Ultrasound
Reconstruction

Mingyuan Luo Mingyuan Luo and Xin Yang contribute equally to this work.112233
Xin Yang 0 112233 Xiaoqiong Huang 112233 Yuhao Huang 112233 Yuxin Zou 112233
Xindi Hu 44 Nishant Ravikumar 5566 Alejandro F Frangi 11556677 Dong Ni (✉)
112233

###### Abstract

3D ultrasound (US) is widely used for its rich diagnostic information.
However, it is criticized for its limited field of view. 3D freehand US
reconstruction is promising in addressing the problem by providing broad range
and freeform scan. The existing deep learning based methods only focus on the
basic cases of skill sequences, and the model relies on the training data
heavily. The sequences in real clinical practice are a mix of diverse skills
and have complex scanning paths. Besides, deep models should adapt themselves
to the testing cases with prior knowledge for better robustness, rather than
only fit to the training cases. In this paper, we propose a novel approach to
sensorless freehand 3D US reconstruction considering the complex skill
sequences. Our contribution is three-fold. First, we advance a novel online
learning framework by designing a differentiable reconstruction algorithm. It
realizes an end-to-end optimization from section sequences to the
reconstructed volume. Second, a self-supervised learning method is developed
to explore the context information that reconstructed by the testing data
itself, promoting the perception of the model. Third, inspired by the
effectiveness of shape prior, we also introduce adversarial training to
strengthen the learning of anatomical shape prior in the reconstructed volume.
By mining the context and structural cues of the testing data, our online
learning methods can drive the model to handle complex skill sequences.
Experimental results on developmental dysplasia of the hip US and fetal US
datasets show that, our proposed method can outperform the start-of-the-art
methods regarding the shift errors and path similarities.

###### Keywords:

Self Context Shape Prior Freehand 3D Ultrasound

##  1 Introduction

Ultrasound (US) imaging is one of the main diagnostic tools in clinical due to
its safety, portability, and low-cost. 3D US is increasingly used in clinical
diagnosis [8, 9, 6] because of its rich context informations which are not
offered in 2D US. However, 3D US probe is constrained by the limited field of
view and poor operability. Thus, exploring 3D freehand US reconstruction from
a series of 2D frames has great application benefits [11]. However, this kind
of reconstruction is non-trivial due to the complex in-plane and out-plane
shifts among adjacent frames, which are caused by the diverse scan skills and
paths. In this paper, as the illustration in Fig. 1, besides the basic linear
scan, we also consider three typical scan skills and their hybrid cases which
are rarely handled in previous studies, including loop scan, fast-and-slow
scan, and sector scan.

Figure 1: Complex scan sequences considered in this work.

External sensor-based solutions can build tracking signal for 3D
reconstruction [5, 7]. But they are facing magnetic influences, optical
occlusion, and expensive costs. Therefore, US volume reconstruction from
sensorless freehand scans plays an important role and has tremendous potential
applications. Most previous researches are based on the non-deep learning
methods [10, 12]. Prevost et al. [15] pioneered the deep learning based
estimation of relative motion between US images, and later they extended their
works by introducing the extra optical flow [2] and integrating a sensor
source [14]. Guo et al. [3] proposed a deep contextual learning network for
reconstruction, which utilizes 3D convolutions over US video clips for feature
extraction. However, these methods mainly handle the basic cases, rather than
the complex scan sequences, such as the loop scan and fast-slow scan as shown
in Fig. 1. In addition, previous models rely on the training data heavily and
ignore the reconstruction robustness on the complex sequences during the test
phase. Recently though, self-supervised learning (SSL) and adversarial
learning (ADL) strategies in deep learning have been widely proven to learn an
enriched representation from image itself and improve method robustness [18,
1, 4].

Figure 2: Overview of our proposed online learning framework.

For the limitations described above, we propose a novel approach to
effectively address the difficulty of sensorless freehand 3D US reconstruction
under complex skill sequences. Our contribution is three-fold. First, a novel
online learning framework is advanced by devising a differentiable algorithm
to approximate the reconstruction process. It enables an end-to-end, recurrent
optimization from section sequence to the whole reconstructed volume. Second,
based on the differentiable design, a SSL scheme is developed to explore the
context cues contained in the reconstructed volumes as pseudo supervisions,
and hence regularize the prediction of future frames. Third, online ADL is
introduced to strengthen the representation learning of anatomical shape
priors, and thereby avoid irregular reconstructions. SSL and ADL in the online
form are general. They can enable our method to generate plausible visual
reconstructions when facing the difficult but practical scan cases that are
shown in Fig. 1.

##  2 Methodology

Fig. 2 shows the proposed online learning framework, including the recurrent
Convolutional LSTM (ConvLSTM) backbone [16] and the online learning framework
to adapt to complex skill sequences. During the training process, the
$N$-length sequence $S=\left\\{s_{i}|i=1,\dots,N\right\\}$ is input into the
ConvLSTM to predict the 3D relative transformation parameters
$\widehat{\Theta}=\\{\widehat{\theta}_{i}|i=1,\dots,N-1\\}$ among all adjacent
frames, where $\widehat{\theta}_{i}$ (or $\theta_{i}$) represents a rigid
transformation containing 3 translation and 3 rotation degrees. During the
test, SSL and ADL are combined to form an online learning process to optimize
the model’s estimation of complex skill sequences.

###  2.1 Online Self-Supervised Learning for Context Consistency

As shown in Fig. 2, for each frame in a sequence, its 3D transformation
estimation aggregates the context of its neighboring frames. Therefore, there
is an inherent context consistency constraint for each frame. When applying
the estimated transformation of each frame to extract the slice from the
volume reconstructed by the rest of the sequence frames, we should get a slice
with very similar content as the frame itself. This is a typical self-
supervision. Specifically, during the test phase, the estimated relative
position $\widehat{p}_{i}$ of any frame $s_{i}$ with respect to the first
frame $s_{1}$ is calculated according to the estimated relative transform
parameter $\widehat{\Theta}$. In the scan sequence $S$, we firstly uniformly
select a certain proportion (we set 0.5) of frames $S_{reco}\subsetneqq S$. A
volume $V_{reco}$ is then reconstructed by using $S_{reco}$ and its
corresponding estimated relative position
$\widehat{P}_{reco}=\left\\{\widehat{p}_{i}|s_{i}\in S_{reco}\right\\}$
through a differentiable reconstruction approximation. For the remaining
frames $S^{-}=S-S_{reco}$, we perform slice operation in $V_{reco}$ according
to the corresponding estimated relative position
$\widehat{P}^{-}=\left\\{\widehat{p}_{i}|s_{i}\in S^{-}\right\\}$ to get the
generated slice $\widehat{S}^{-}$. The difference between $S^{-}$ and
$\widehat{S}^{-}$ should be small to reflect the context consistency and is
used to optimize the ConvLSTM.

###  2.2 Online Adversarial Learning for Shape Constraint

Shape prior is a strong constraint to regularize the volume reconstruction.
Exposing the models to shape prior under the online learning scheme, rather
than the offline training, provides us more chances to better explore the
constraint for more generalizability, especially in handling the complex scan
sequences where unplausible reconstructions often occur. Specifically, as
shown in Fig. 2, we propose to encode the shape prior with an adversarial
module. The volume reconstruction $V_{r}$ from the training set is randomly
selected as a sample with real anatomical structures. While the volume $V_{f}$
reconstructed from the testing sequence $S$ with relative estimated position
parameters $\widehat{P}$ is taken as a sample of fake structure. A classifier
$C$ pre-trained to distinguish between the volume $V_{r}$ and $V_{f}$
reconstructed from all training sequences serves as the adversarial
discriminator. It adversarially tunes the structure contained in the
reconstructed volume via the proposed differentiable reconstruction
approximation.

###  2.3 Differentiable Reconstruction Approximation

The end-to-end optimization of our online learning framework is inseparable
from the differentiable reconstruction. In general, most of the unstructured
interpolation operations in the reconstruction algorithm (such as the Delaunay
algorithm) are not differentiable and make our online learning blocked. As a
foundation of our online design, we firstly propose a differentiable
reconstruction approximation to mimic the interpolation. As shown in Fig. 3,
the volume $V$ is reconstructed using the $N$ slices
$\left\\{s_{j}|j=1,\dots,N\right\\}$. For any pixel $v_{i}\in V$, the distance
$d_{ij}$ from any slice $s_{j}$ and the gray value $G_{ij}$ of the projection
points on any slice $s_{j}$ are calculated. Then the reconstructed gray value
$G_{v_{i}}$ at pixel $v_{i}$ is calculated as:

$G_{v_{i}}=\sum_{j}{W(d_{ij})G_{ij}}$ (1)

Among them, $W\left(\cdot\right)$ is a weight function. Its purpose is to
encourage small $d_{ij}$. The core formulation is softmax operation on the
reciprocal of $d_{ij}$:

$W(d_{ij})=\frac{\exp(1/(d_{ij}+\epsilon))}{\sum_{j}{\exp(1/(d_{ij}+\epsilon}))},$
(2)

where $\epsilon$ prevents division by $0$. If only the nearest slice is
weighted, the approximation can be expressed as follows:

$W(d_{ij})=\frac{1}{Z}(\frac{1}{d_{ij}+\epsilon}\frac{\exp(1/(d_{ij}+\epsilon))}{\sum_{j}{\exp(1/(d_{ij}+\epsilon}))}),$
(3)

where $1/Z$ is the normalization coefficient such that $\sum_{j}W(d_{ij})=1$.

Fig. 3 shows two reconstruction examples. The left volume of each example
directly puts the slice into the volume according to the position without
interpolation, and the right is the result of differentiable reconstruction
approximation.

Figure 3: Two reconstruction examples of differentiable reconstruction
approximation.

###  2.4 Loss Function

In the training phase, the ConvLSTM predicts relative transform parameters
$\widehat{\Theta}$ between all adjacent frames in the skill sequence. Its loss
function includes two items, the first item is the mean absolute error (MAE)
loss (i.e., L1 normalization), and the second item is the case-wise
correlation loss from [3], which is beneficial to improve the generalization
performance.

$L=\|\widehat{\Theta}-\Theta\|_{1}+(1-\frac{\textbf{Cov}(\widehat{\Theta},\Theta)}{\sigma(\widehat{\Theta})\sigma(\Theta)})$
(4)

where $\left\|\cdot\right\|_{1}$ indicates L1 normalization, Cov gives the
covariance, and $\sigma$ calculates the standard deviation. The classifier $C$
uses cross-entropy loss to distinguish whether the input volume is
reconstructed or real volume.

In the test phase, the SSL and the ADL are jointly iteratively optimized. The
first two items of $L_{d}$ are the adversarial loss used to optimize the
classifier $C$, and the third item is the quadratic potential divergence from
[17], which helps to prevent the gradient vanishing without additional
Lipschitz constraint. The first term of $L_{g}$ is the adversarial loss used
to optimize the ConvLSTM, and the second term is the self-supervised loss.

$L_{d}=\mathbb{E}_{V_{f}\sim\mathbb{P}_{V_{f}},V_{r}\sim\mathbb{P}_{V_{r}}}[C(V_{f})-C(V_{r})+\frac{\|C(V_{f})-C(V_{r})\|_{2}^{2}}{2\|V_{f}-V_{r}\|_{1}}]$
(5)
$L_{g}=-\mathbb{E}_{V_{f}\sim\mathbb{P}_{V_{f}}}[C(V_{f})]+\|\widehat{S}^{-}-S^{-}\|_{1}$
(6)

where $\left\|\cdot\right\|_{1}$ and $\left\|\cdot\right\|_{2}$ indicate L1
and L2 normalization, respectively.

##  3 Experiments

##### Materials and Implementation.

Our experiments involve two datasets, i.e., 3D developmental dysplasia of the
hip (DDH) US dataset and 3D fetus US dataset. The DDH dataset contains 101 US
volumes from 14 volunteers. Its average volume size is $370\times 403\times
398$ with voxel resolution $0.1\times 0.1\times 0.1$ mm. The fetal dataset
contains 78 fetal US volumes from 78 volunteers. The gestational age ranges
from 10 to 14 weeks. Its average volume size is $402\times 535\times 276$, the
voxel resolution is $0.3\times 0.3\times 0.3$ mm. All data collection is
anonymous. The collection and use of data are approved by the local IRB.

It is time-consuming and expensive to collect a large number of real freehand
sequences with complex scan skills. In order to verify the proposed framework
on the complex skill sequences, the real 3D volumes are used to generate
massive complex skill sequences for our training and evaluation. Multiple
complex scan sequences are dynamically simulated with diverse scan skills from
each US volume to form our final corpus. Specifically, the scan sequences are
a complex combination of loop scan, fast-and-slow scan and sector scan with
the aim to simulate the loop movement, uneven speed, and anisotropic movement
of probe.

For the 3D DDH and fetal US dataset, we randomly divide the dataset into 85/15
and 65/13 volumes for training/testing according to the patients, and the
sequence length is 120 and 90, respectively. The number of generated sequences
is 100 and 10 for training and testing, respectively. All generated slices are
with size $300\times 300$ pixel. In the training phase, the Adam optimizer is
used to iteratively optimize our ConvLSTM. The epoch and batchsize are 200 and
4, respectively. The learning rate is $10^{-3}$, and the learning rate is
reduced by half every 30 epochs. The ConvLSTM has 1 layer and convolutional
kernel size is 3. For classifier $C$, the epoch, batchsize and learning rate
is 50, 1 and $10^{-4}$, respectively. During the test phase, for each testing
sequence, we iterate the online learning with adversarial and self-supervised
losses for 30 iterations, and the learning rate is set as $10^{-6}$. All codes
are implemented in PyTorch [13].

##### Quantitative and Qualitative Analysis.

The current commonly used evaluation indicator is the final drift [14, 3],
which is the drift of final frame of a sequence, and the drift is the distance
between the center points of the frame according to the real relative position
and the estimated relative position. On this basis, a series of indicators are
used to evaluate the accuracy of our proposed framework in estimating the
relative transform parameters among adjacent frames. Final drift rate (FDR) is
the final drift divided by the sequence length. Average drift rate (ADR) is
the cumulative drift of all frames divided by the length from the frame to the
starting point of the sequence, and finally, the average value is calculated.
Maximum drift (MD) is the maximum accumulated drift of all frames. Sum of
drift (SD) is the sum of accumulated drift of all frames. The bidirectional
Hausdorff distance (HD) emphasizes the worst distances between the predicted
positions (accumulatively calculated by the relative transform parameters) and
the real positions of all the frames in the sequence.

Table 1: The mean (std) results of different methods on the DDH sequences.

Methods | FDR(%) | ADR(%) | MD(mm) | SD(mm) | HD(mm)
---|---|---|---|---|---
CNN [14] | 16.25(10.45) | 44.58(8.71) | 21.14(4.11) | 1798.71(433.18) | 21.10(4.10)
DCL-Net [3] | 12.78(14.87) | 25.46(23.48) | 9.07(7.99) | 363.68(345.06) | 7.43(5.43)
ConvLSTM | 11.15(3.97) | 22.08(9.03) | 7.61(2.99) | 498.84(197.32) | 6.63(2.77)
ConvLSTM+SSL | 10.89(4.05) | 21.93(9.03) | 7.52(2.94) | 493.44(197.70) | 6.58(2.76)
ConvLSTM+ADL | 6.21(3.09) | 13.89(6.95) | 4.70(2.30) | 291.01(142.08) | 4.10(1.88)
ConvLSTM+SSL+ADL | 5.44(3.03) | 13.47(6.47) | 4.44(2.29) | 274.14(136.89) | 3.91(1.89)

Table 2: The mean (std) results of different methods on the fetus sequences.

Methods | FDR(%) | ADR(%) | MD(mm) | SD(mm) | HD(mm)
---|---|---|---|---|---
CNN [14] | 16.59(9.67) | 31.82(14.02) | 24.50(10.43) | 1432.89(698.88) | 19.16(8.21)
DCL-Net [3] | 12.47(8.49) | 30.58(16.05) | 17.74(8.56) | 882.00(452.00) | 15.88(7.71)
ConvLSTM | 16.71(5.22) | 31.33(12.68) | 20.57(8.33) | 938.42(391.39) | 15.79(6.78)
ConvLSTM+SSL | 15.86(5.10) | 30.78(12.20) | 20.01(8.27) | 909.95(382.43) | 15.59(6.53)
ConvLSTM+ADL | 10.49(4.86) | 28.15(9.86) | 17.49(7.68) | 762.79(292.92) | 14.18(5.99)
ConvLSTM+SSL+ADL | 9.94(4.41) | 27.08(9.31) | 16.84(7.56) | 730.11(303.20) | 14.12(5.89)

Table 1 & 2 summarize the overall comparison of the proposed online learning
framework with other existing methods and ablation frameworks on simulated DDH
or fetus complex skill sequences, respectively. “CNN” [14] and “DCL-Net” [3]
are considered in this study for comparison. “SSL” and “ADL” are our proposed
online self-supervised learning and adversarial learning method, respectively.

It should be noted that we have updated the results of “DCL-Net” in the MICCAI
version. Considering that the mean parameter in the “DCL-Net” setting is not
suitable for scan sequences with large path variations, we removed the mean
parameter in the updated experiments and only used the relative transformation
parameters between all adjacent frames as the ground truth to obtain the final
results. Compared with “CNN”, all indicators of “DCL-Net” have been
significantly improved.

As can be seen from Table 1 & 2, our proposed methods get consistent and
better improvements both on DDH and fetal complex skill sequences. The
comparison with the ablation frameworks fully proves the necessity of
introducing online learning in the test phase. In particular, our proposed
online learning methods help the ConvLSTM improve the accuracy by 5.71%/2.72mm
for DDH sequences, and 6.77%/1.67mm for fetal sequences in terms of the FDR/HD
indicators. Under the complementary effects of mining context consistency and
shape constraint, the proposed framework has significantly robust the
estimation accuracy of complex skill sequences.

Figure 4: Indicator declining curves for online iterative optimization on DDH
(the first row) and fetus (the second row) US datasets. Figure 5: Specific
case of online iterative optimization. The first row and second row shows the
DDH and fetus cases, and the yellow arrows indicate the anatomic structure.

Fig. 4 shows the indicator declining curves for online iterative optimization
of the ConvLSTM on DDH and fetus US datasets. In each subfigure of Fig. 4, the
abscissa and ordinate represent the number of iterations and the extent of
indicators decline, respectively. The blue dot curve is the average indicator
declining curve of all data. It is obvious that the indicators curve of almost
all single sequences is declining significantly, and the downward trend is
from rapid decline to gentle convergence. Among the various indicators, FDR
has the largest decline, which is reduced to $48.78\%$ and $59.48\%$ of the
ConvLSTM on the DDH and fetus US dataset, respectively. Fig. 5 shows four
specific cases of online iterative optimization on DDH and fetus complex skill
sequences, respectively. In each subfigure of Fig. 5, the orange and red box
represent the first and final frame respectively and the pipeline that change
color represent the scan path. It can be observed that through online
iterative optimization, the final reconstruction result is significantly
improved compared with the ConvLSTM, with smaller drift and more stable
trajectory estimation.

##  4 Conclusion

In this paper, we proposed the first research about freehand complex scan 3D
US volume reconstruction. Benefiting from self context, shape prior and
differentiable reconstruction approximation, the proposed online learning
framework can effectively solve the challenge of 3D ultrasound reconstruction
under complex skill sequences. Experiments on two different US datasets prove
the effectiveness and versatility of the proposed framework. Future research
will focus on extending this framework to more scanning targets and even
different modalities.

#### Acknowledgements

This work was supported by the National Key R&D Program of China (No.
2019YFC0118300), Shenzhen Peacock Plan (No. KQTD20160
-53112051497, KQJSCX20180328095606003), Royal Academy of Engineering under the RAEng Chair in Emerging Technologies (CiET1919/19) scheme, EPSRC TUSCA (EP/V04799X/1) and the Royal Society CROSSLINK Exchange Programme (IES/NSFC/201380).

## References

  * [1] Chen, L., Bentley, P., Mori, K., Misawa, K., Fujiwara, M., Rueckert, D.: Self-supervised learning for medical image analysis using image context restoration. Medical image analysis 58, 101539 (2019)
  * [2] Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Scandinavian conference on Image analysis. pp. 363–370. Springer (2003)
  * [3] Guo, H., Xu, S., Wood, B., Yan, P.: Sensorless freehand 3d ultrasound reconstruction via deep contextual learning. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 463–472. Springer (2020)
  * [4] Hendrycks, D., Mazeika, M., Kadavath, S., Song, D.: Using self-supervised learning can improve model robustness and uncertainty. In: Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019)
  * [5] Hennersperger, C., Karamalis, A., Navab, N.: Vascular 3d+ t freehand ultrasound using correlation of doppler and pulse-oximetry data. In: International Conference on Information Processing in Computer-Assisted Interventions. pp. 68–77. Springer (2014)
  * [6] Huang, Y., Yang, X., Li, R., Qian, J., Huang, X., Shi, W., Dou, H., Chen, C., Zhang, Y., Luo, H., et al.: Searching collaborative agents for multi-plane localization in 3d ultrasound. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 553–562. Springer (2020)
  * [7] Lang, A., Mousavi, P., Gill, S., Fichtinger, G., Abolmaesumi, P.: Multi-modal registration of speckle-tracked freehand 3d ultrasound to ct in the lumbar spine. Medical image analysis 16(3), 675–686 (2012)
  * [8] Liu, S., Wang, Y., Yang, X., Lei, B., Liu, L., Li, S.X., Ni, D., Wang, T.: Deep learning in medical ultrasound analysis: A review. Engineering 5, 261–275 (2019)
  * [9] Looney, P., Stevenson, G.N., Nicolaides, K.H., Plasencia, W., Molloholli, M., Natsis, S., Collins, S.L.: Fully automated, real-time 3d ultrasound segmentation to estimate first trimester placental volume using deep learning. JCI insight 3(11) (2018)
  * [10] Mercier, L., Langø, T., Lindseth, F., Collins, D.L.: A review of calibration techniques for freehand 3-d ultrasound systems. Ultrasound in Medicine & Biology 31, 449–471 (2005)
  * [11] Mohamed, F., Siang, C.V.: A survey on 3d ultrasound reconstruction techniques. Artificial Intelligence—Applications in Medicine and Biology (2019)
  * [12] Mozaffari, M.H., Lee, W.S.: Freehand 3-d ultrasound imaging: A systematic review. Ultrasound in Medicine & Biology 43, 2099–2124 (2017)
  * [13] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019)
  * [14] Prevost, R., Salehi, M., Jagoda, S., Kumar, N., Sprung, J., Ladikos, A., Bauer, R., Zettinig, O., Wein, W.: 3d freehand ultrasound without external tracking using deep learning. Medical image analysis 48, 187–202 (2018)
  * [15] Prevost, R., Salehi, M., Sprung, J., Ladikos, A., Bauer, R., Wein, W.: Deep learning for sensorless 3d freehand ultrasound imaging. In: International conference on medical image computing and computer-assisted intervention. pp. 628–636. Springer (2017)
  * [16] Shi, X., Chen, Z., Wang, H., Yeung, D.Y., Wong, W.k., Woo, W.c.: Convolutional lstm network: A machine learning approach for precipitation nowcasting. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 28. Curran Associates, Inc. (2015)
  * [17] Su, J.: Gan-qp: A novel gan framework without gradient vanishing and lipschitz constraint (2018)
  * [18] Yi, X., Walia, E., Babyn, P.: Generative adversarial network in medical imaging: A review. Medical image analysis 58, 101552 (2019)