# A complementary experimental study of epitaxial La0.67Sr0.33MnO3 to identify
morphological and chemical disorder

Michael Verhage  Molecular Materials and Nanosystems (M2N) - Department of
Applied Physics - Eindhoven University of Technology, Eindhoven, Netherlands
Emma van der Minne  MESA+ Institute for Nanotechnology, Faculty of Science and
Technology, University of Twente, Enschede, Netherlands  Ellen M. Kiens  MESA+
Institute for Nanotechnology, Faculty of Science and Technology, University of
Twente, Enschede, Netherlands  Lucas Korol  Department of Physics &
Engineering Physics, University of Saskatchewan, Saskatoon, Canada  Raymond J.
Spiteri  Department of Computer Science, University of Saskatchewan,
Saskatoon, Canada  Gertjan Koster  MESA+ Institute for Nanotechnology, Faculty
of Science and Technology, University of Twente, Enschede, Netherlands  Robert
J. Green  Stewart Blusson Quantum Matter Institute, University of British
Columbia, Vancouver, Canada  Christoph Baeumer  MESA+ Institute for
Nanotechnology, Faculty of Science and Technology, University of Twente,
Enschede, Netherlands<EMAIL_ADDRESS>Kees Flipse  Molecular Materials
and Nanosystems (M2N) - Department of Applied Physics - Eindhoven University
of Technology, Eindhoven, Netherlands<EMAIL_ADDRESS>
###### Abstract

Gaining insight into the characteristics of epitaxial complex oxide films is
essential to control the behavior of devices and catalytic processes. It is
known that substrate-induced strain, doping, and layer growth can affect the
electronic and magnetic properties of the film’s bulk. In this study, we
demonstrate a clear distinction between the bulk and surface of thin films of
La0.67Sr0.33MnO3 in terms of chemical composition, electronic disorder, and
surface morphology. We employed a combined experimental approach of X-ray
based characterization methods and scanning probe microscopy. X-ray
diffraction and resonant X-ray reflectivity revealed surface non-stochiometry
in the strontium and lanthanum, as well as an accumulation of oxygen
vacancies. Scanning tunneling microscopy showed a staggered growth surface
morphology accompanied by an electronic phase separation (EPS) related to this
non-stochiometry. The EPS is likely responsible for the temperature-dependent
resistivity transition and is a cause of a proposed mixed-phase ferromagnetic
and paramagnetic state near room temperature in these thin films.

Contributed equally Contributed equally Department of Physics & Engineering
Physics, University of Saskatchewan, Saskatoon, Canada Peter Gruenberg
Institute and JARA-FIT, Forschungszentrum Juelich GmbH, Juelich, Germany

##  1 Introduction

Manganite complex oxides are considered to be attractive materials for spin-
based devices 1, 2, 3 due to a half-metallic character 4 and high spin
polarization 5. Doping the parent material LaMnO3 with Sr in La0.67Sr0.33MnO3
(LSMO) introduces insulator-to-metal and para-to-ferromagnetic transitions
driven by complex interplay of structure, charge, and spin degrees of freedom
6, 7. LSMO possesses an advantageous Curie temperature of
$370\text{\,}\mathrm{K}$ 8 that potentially enables spin control at room
temperature. The control of these properties is of interest in fundamental
electrochemical studies 9, 10 of manganites. LSMO has been used as a model
catalyst to study the oxygen reduction and evolution reactions (ORR and OER,
respectively) due to the tunability of the active manganese site and its
promising bifunctional activity 11, 12, 13, 14. A recent experiment has shown
that active strain can be used to regulate the activity of LSMO thin films in
the OER 15. This, combined with its high spin polarization, could be a useful
tool for exploring the influence of electron spin on the OER 16.

In addition to catalysis, and closely related to spin, LSMO has been
extensively studied for phenomena such as colossal magnetoresistance 17, 18
and electroresistance 19. These phenomena are understood within the framework
of electronic phase separation (EPS) and magnetic phase separation 20. Reports
of chemical 21, 20 and pressure 22 induced EPS of manganites are associated
with bulk disorder and are essential to this family of metal oxides. A great
deal of research has been done on LSMO thin films, exploring how changes in
parameters such as substrate-induced strain (tensile or compressive), film
thickness, substrate termination, 23 and growth conditions affect structural,
electronic, and magnetic behavior 24, 25, 26, 27. However, there are still
insights to be gained in the discrepancy between the surface structure and
chemical composition and that of the bulk. To address this, we conducted both
nanoscale imaging and spectroscopic analysis on a LSMO thin film to link
nanoscale surface characteristics 27 with those of the bulk to gain a
comprehensive view of our LSMO films.

This work presents evidence that, for LSMO epitaxial thin films, both
structural and chemical disorder persist at the surface, as revealed by high-
resolution scanning probe microscopy (SPM) imaging. We observe nanometer-scale
corrugation of the surface 28, consisting of the formation of a staggered
growth of small islands 29, that deviates from an atomically smooth
interpretation. We define an atomically smooth film as a surface roughness
variation on the scale of atomic sizes with a correlation length on the order
of the widths of the vicinal steps. Pandya et al. 30 have reported the
emergence of such staggered growth or “wedding cake" structures, which are
controlled by substrate-induced strain. Similarly, Kelley et al. 31 have
observed these islands in LSMO.

Furthermore, we used area-averaged chemical mapping by resonant X-ray
reflectometry (RXR) to investigate a gradient in the chemical composition from
a stochiometric bulk to an excess of the Sr dopant near the surface. The
chemical disorder is accompanied by a partial change in the valence state of
Mn from Mn3+ to Mn2+ and an increase in oxygen vacancies 32. Sr segregation in
the surface region was previously identified in Ref. 33 and linked to
morphology restructuring 28.

Using scanning tunneling spectroscopy (STS), we detect an EPS across the
surface that is related to the nature of the staggered growth and chemical
disorder. This relation between rough surface morphology and EPS has been
reported for Ca-doped LaCaMnO3 (LCMO) films 34, and thus appears to be a
common feature of the manganite perovskite family.

The EPS is likely to be the cause of a magnetic phase separation in LSMO.
Previous studies of Ca-doped manganites have reported the presence of
ferromagnetic (FM) and paramagnetic (PM) areas that coexist and lead to
percolation 22, 35. By studying the temperature-dependent resistance, a
phenomenological model of phase separation was constructed between the FM and
PM states 35, with a dominant FM phase at lower temperatures and a mixture of
FM and PM at higher temperatures. A deeper understanding of the LSMO surface
enables insights into its applications in devices and catalysis.

##  2 Results and Discussion

LSMO films of $13\text{\,}$ unit cell (u.c.) thickness were grown with pulsed
laser deposition (PLD) on SrTiO3 (STO) 001 substrates; further experimental
details are described in Methods. We use reflection high-energy electron
diffraction (RHEED) to monitor growth in situ. The presence of clear RHEED
intensity oscillations indicates layer-by-layer growth, as can be observed in
Fig. 1a. A schematic illustration of the crystal structure is shown in the
inset of Fig. 1a, and the growth on STO introduces tensile strain. A decrease
in peak amplitude over time and faint streaks observed along with sharp
diffraction spots in Fig. 1b point toward the presence of disorder and
roughness 36 of the LSMO surface. The equal distance between the diffraction
spots in the RHEED pattern, Fig. 1b, indicates that the in-plane lattice
parameters of the film match those of the substrate, implying the presence of
tensile strain 37. The film quality was further verified by X-ray diffraction
(XRD), indicating a pure (001) oriented LSMO phase as seen in Fig. 1c.

Figure 1: In-situ growth characterization and structure of 13 u.c. LSMO films.
(a) In-situ RHEED intensity oscillations for a 13 unit cell (u.c.) thick LSMO
film grown on STO. Inset: schematic illustration of the atomic structure. (b)
RHEED patterns of the substrate (top) and the film (bottom). The red dashed
lines are a guide for the eye and indicate that the distance between the
diffraction spots does not change. (c) Wide angle $2\Theta$-$\omega$ scan,
indicating single (001)-oriented LSMO. From the position of the LSMO 002 peak
a c-axis parameter of $0.385\text{\,}\mathrm{nm}$ is obtained.  Figure 2: XRD
and RXR analysis of the $13\text{\,}\mathrm{u}$.c. LSMO film. (a) Reciprocal
space map around the STO 103 peak. (b) XRD $2\Theta$-$\omega$ scans around the
STO 002 peak along with dynamical simulated diffactrograms. The simulations
were performed without or with a lower crystalline layer of
$0.9\text{\,}\mathrm{nm}$. The large deviations of the fringe at 42∘ are
probably induced by the broadening of the experimental substrate peak, which
is not taken into account in the simulations. Simulations were done with the
Stephanov X-ray server 38. The arrows indicate large deviations from the data
for the simulation without a top layer. (c) XRR data for 13 u.c. LSMO film.
(d) The Fourier transform of c, which indicates an LSMO film thickness of
5.37±0.01 nm. (e) Atomic depth profiles of La, Ti, Sr, Mn2+, Mn3.3+ and O
obtained using resonant X-ray reflectivity (RXR) at 300K.

Further XRD analysis on the LSMO 002 peaks was performed to investigate the
crystallinity and strain state of the thin film. The reciprocal space map
around the STO 103 peak in Fig. 2a confirms that the bulk of the film exhibits
a coherent tensile strain. The clear Laue fringes in Fig. 2b indicate good
crystallinity in the bulk of the film. However, comparing these measured
fringes with an expected diffractogram of a 13 u.c. thick film (S1 - Fig. 6)
indicates a measured film thickness below 13 u.c. Furthermore, the measured
crystalline layer is thinner than the thickness obtained from the Fourier
transform of the XRR data in Fig. 2c, d of 5.37±0.01 nm, which is close to the
expected value for a 13 u.c. thin film. This indicates, as discussed in more
detail below, that a layer of low crystallinity, but with an electronic
density similar to that of the LSMO bulk, is present. See Methods, XRD for the
modeling details of the crystallinity variability in the film. Correlating
this with the resonant X-ray reflectivity (RXR) results, see below, indicates
that this layer of lower crystallinity is positioned at or near the surface.
The existence of such a layer is further confirmed by the asymmetric period
around the STO 002 peak, which indicates asymmetry in the film. The data can
be better fitted when a layer of approximately $0.9\text{\,}\mathrm{nm}$ with
low crystallinity is considered on the film surface, as shown in Fig. 2b.

Due to the crystalline dissimilarity between the bulk and the surface, we
turned to RXR to study the chemical disorder 39. With RXR, subnanometer depth-
resolution of the elemental composition can be obtained. 40. The depth profile
of the LSMO film shown in Fig. 2e indicates non-stochiometric regions near the
substrate/film interface and the film surface. At the buried interface, the
intermixing of the STO substrate with the LSMO film persists for more than
$1\text{\,}\mathrm{nm}$, inducing a La:Sr ratio that deviates from the
stochiometric value of 2:1. The non-stochiometry at the buried interface is a
likely cause of the reported magnetic dead layers of LSMO 41. Furthermore, we
observe Mn2+ at the buried interface, which could be an effect of the
intermixing of Ti4+ atoms. Near the surface, we observe Sr segregation and La
deficiency and the formation of Mn2+ species. The latter could be induced by
the presence of oxygen vacancies in the surface layer 32, 42. Surface non-
stochiometry is unlikely to be introduced by beam damage because with
repeating scans no changes in stochiometry were noted. On the basis of these
results, the non-stochiometry in the surface of the film could be a reason for
the lower crystallinity in the surface compared to that of the bulk.

###  2.1 Surface imaging

From the X-ray analysis, we identified that the surface of our LSMO films
deviates from the high-crystalline, chemically stochiometric bulk. We turned
to atomic force microscopy (AFM) to study the surface structure of 13 u.c.
LSMO films on STO in further detail. First, we imaged the surface with a
rather large diameter tip. Our custom AFM probe has a pyramidal tip 43 with a
radius of around $30\text{\,}\mathrm{nm}$, similar to commercial Si
cantilevers. We used Frequency-Modulation (FM-AFM) feedback in repulsive mode.
In this mode, the tip gently touches the sample and is lifted again during
each oscillation cycle, similar to tapping-mode AFM. The benefit of this
approach is that the AFM is most sensitive to short-range forces, increasing
the resolution 44. However, the obtainable resolution is mainly limited by the
radius of the tip 45. See Methods for further experimental details. As
expected from the vicinal cut STO, the epitaxial thin film LSMO demonstrated
flat plateaus and step edges,  Fig. 3a. These stepped film surfaces have often
been reported in the literature for a wide range of transition metal oxide
perovskites and layer thicknesses 46, 47. The steps give rise to significant
contrast in the AFM due to the convolution of the tip and the lateral friction
when the side of the tip hits the atomic edge 48. To show that this contrast
emerges from repulsive force or friction, even in FM-AFM when the side of the
tip hits the step edge, we performed contact AFM. The topographic results are
given in S2 Fig. 7, where a clear contrast near the edge of the step is
observed. Hence, step edges can dominate the AFM signal due to tip-sample
friction during contact and morphological details of the plateau surfaces can
be overlooked. The conformation of the surface is slightly bulged upward near
the step edge (S3 Fig. 8). However, with the resolution obtained here, only a
faint hint at nanometer-scale features of the scale of less than 1 u.c. height
variation could be identified as highlighted with the black arrows.

In order to improve lateral imaging resolution, we switched to a diamond tip
with a radius of $5\text{\,}\mathrm{nm}$; see Methods. Fig. 3b shows a
completely different surface morphology, despite it being the same sample as
in Fig. 3a. The surface is densely populated with semispherical objects with
an average roughness of $450\text{\,}\mathrm{pm}$ and a mean roughness
($S_{a}$) of $140\text{\,}\mathrm{pm}$. These features are more closely packed
and slightly taller near the step edge, as highlighted in blue, which explains
why AFM with a lower imaging resolution of Fig. 3a showed bulging towards the
step edge.

Figure 3: Surface structure of 13 u.c. LSMO thin films. (a) AFM imaging with a
$30\text{\,}\mathrm{nm}$ radius tip showing mostly contrast from the step
edges. The scale bar is equal to $100\text{\,}\mathrm{nm}$. (b) AFM image with
a tip radius of $5\text{\,}\mathrm{nm}$ indicating a granular surface
morphology. Scale bar equals $40\text{\,}\mathrm{nm}$. (c) High resolution STM
image ($800\text{\,}\mathrm{mV}$, $100\text{\,}\mathrm{pA}$) revealing a
staggered growth. Numbers indicate consecutive layers. The scale bar is equal
to $10\text{\,}\mathrm{nm}$. (d) Height distribution of c with the peak
spacing indicating the stacking of layers.

To further explore the nanoscale surface structures of the LSMO films, we used
scanning tunneling microscopy (STM) in a UHV-SPM setup. The tip had a radius
of less than $1\text{\,}\mathrm{nm}$ and the STM was mainly sensitive to
electronic or height variations on the atomic scale. The STM image in Fig. 3c
showed that the semispherical structures are actually small stacked islands.
These morphologies are known as staggered growth, or “wedding cake” islands
30. Similar features were observed for Ca-doped manganite 34 and are termed
mounds 29. Our tip-radius-dependent imaging highlights that, without using
ultra-sharp SPM tips, nanoscale features may be hidden and the surface
morphology inaccurately interpreted as atomically smooth on a scale of the
width of the STO plateaus.

In Fig. 3c, the most deeply buried layer, as indicated by the number $1$, is
coalesced with only a few voids left in the area. We observe up to
$3\text{\,}$ layers stacked on top of this coalesced layer in the form of
"wedding cake" islands or staggered growth, as shown numerically in the
figure. By analyzing the statistical distribution of the height data, Fig. 3d,
we observe a combination of $4\text{\,}$ height distributions. Each peak
corresponds to a higher positioned layer in Fig. 3c. We conducted a Gaussian
fit to identify the peaks, as indicated by the colored curves. These peaks are
not evenly spaced, as indicated by the numbers corresponding to the increase
in height ($\Delta h$) of each stacking layer. However, they are close to the
values for the thickness of 1 u.c. ($0.385\text{\,}\mathrm{nm}$) except for
the layer numbered $1$ to $2$. This could be due to the fact that the area of
the layer $1$ is almost completely coalesced, so the convolution of the tip in
the few small gaps has a negative effect on the accurate measurement of the
thickness of the film. In this work, we observe clear layer-by-layer growth
but no half-integer heights, which is different from the work of Ref. 29,
where a peak spacing around $200\text{\,}\mathrm{pm}$ was observed indicating
a mix of surface termination growth.

The variation in the peak-to-peak height of staggered growth can be as large
as $2.5\text{\,}\mathrm{nm}$, as can be seen in the maximum height value of
Fig. 3c. These large heights originate from individual dispersed spherical-
like features across the surface. The film has an average roughness of
$1.1\text{\,}\mathrm{nm}$ and an RMS roughness of $338\text{\,}\mathrm{pm}$.
With a film thickness of $13\text{\,}$ u.c., this average roughness accounts
for approximately $20\text{\,}\mathrm{\char 37\relax}$ of the total film
thickness. Based on the XRD results of Fig. 2, we observed that the thickness
of the surface layer with low crystallinity in the $13\text{\,}$ u.c. LSMO
film was approximately $0.9\text{\,}\mathrm{nm}$. This corresponds rather well
with the observed staggered growth average film roughness by the STM and the
off-stochiomtery of 2.5 u.c. ($\sim$$1\text{\,}\mathrm{nm}$) from RXR.

To relate the observed surface structures to bulk crystallinity, we propose a
two-step growth process. First, the deposited species during PLD nucleate and
grow, manifesting as spherical features as observed in the SPM images. These
spherical features move across the surface by diffusion and lateral
aggregation but are impeded near the edges of the vicinal step due to the
Erlich-Swoebel (ES) barrier 30. This barrier near the step edge impedes
downward diffusion of surface features, leading to a densely packed morphology
of islands toward the step edge region. Consequently, just below the step
edge, there is a material deficiency, and few staggered structures can form
there. We argue that the spherical features themselves are not necessarily an
integer number of atomic u.c. stackings of the ABO3 perovskite. However, after
embedding the features into a staggered growth layer, the thickness converges
to integer u.c. heights (Fig. 3d), likely driven by a crystal restructuring.
The disorder in the morphology of the staggered growth breaks long-range
crystallinity and explains why a thinner fully crystalline bulk thickness is
obtained in comparison to the total film thickness by XRD fitting. Possibly,
Sr segregation or oxygen vacancies 42 break the stochiometery of the islands,
as previously observed with RXR in Fig. 2e. The spherical nature of the
islands is further identified by FFT filtering of the STM data in S4 Fig. 9.
When features associated with the staggered growth are removed, the spherical
features remain and can be clearly observed. During PLD, these spherical
features are formed on top of the islands and experience another ES barrier at
the periphery of the island below. The coalescence and growth of approximately
circular islands are naturally accompanied by gaps across layers. As the
islands expand in width, the gaps gradually coalesce. Only when these islands
have reached a sufficient size can second layer nucleation occur 49.
Consequently, we did not observe nucleation of smaller islands or spherical
features on nearly all of the highest islands. Rather rare in presence, we
sometimes observed small individual spherical features scattered throughout
the surface, as seen in S5 Fig. 10. This observation is consistent with the
work of Tershoff et al. 50. In their work, it was shown that nucleation of a
second layer occurs only when an island has reached a critical diameter in
combination with a sufficiently strong ES barrier of the island boundaries.
This seems to imply that before a critical size is reached, surface species
can diffuse through the ES barrier and contribute to an increase in the
lateral dimension of the islands. Only when a critical size has been reached,
the ES barrier is significant in magnitude to block species that diffuse
downward and form a new layer of staggered growth. The second step of the film
growth process occurs after the coalescing of a fully closed layer. We argue
that reconstruction can occur in these closed layers, forming a highly ordered
crystalline bulk, as seen in the XRD data. However, local defects can still
persist 29, leading to a chemical dopant-induced disorder 20.

###  2.2 Variation in surface electronic structure

Figure 4: Scanning tunneling spectroscopy on 13 u.c. LSMO thin films
demonstrating electronic phase separation. (a) Topography STM image
($800\text{\,}\mathrm{mV}$, $100\text{\,}\mathrm{pA}$) with numbers indicating
staggered growth islands. The purple dashed line indicates the step edge. The
numbers $1^{\prime}-4^{\prime}$ are layers located on an STO plateau below the
step edge. The scale bar is equal to $10\text{\,}\mathrm{nm}$. (b)
Corresponding dI/dV map at $800\text{\,}\mathrm{mV}$. The scale bar is equal
to $10\text{\,}\mathrm{nm}$. (c) Small-scale topography
($800\text{\,}\mathrm{mV}$, $100\text{\,}\mathrm{pA}$) and (d) corresponding
dI/dV map. The scale bar is equal to $2\text{\,}\mathrm{nm}$. The black circle
highlights the increased conductivity of a circular feature observed in the
topography. (e) Statistical dI/dV distribution of d fitted with a Gaussian and
showing two distributions of LDOS.

The local electronic structure of the complex morphology of the LSMO surface
was studied in more depth. The presence of electronic disorder or EPS was
investigated with scanning tunneling microscopy (STM) and tunneling
spectroscopy (STS), respectively. Figs. 4a and 4b show the topography and
simultaneously mapped dI/dV or local density of states (LDOS) measured at
$800\text{\,}\mathrm{mV}$ at $300\text{\,}\mathrm{K}$, respectively. For the
dI/dV map, the areas in red have a larger LDOS than the areas in blue.
Staggered growth island contours are highlighted with black lines and numbered
at each consecutive stacked layer. From the dI/dV map, we observe that the
layer, indicated with $1\text{\,}$ in Fig. 4b is rather homogeneous in LDOS
with only some smaller local LDOS spots. This layer is almost structurally
coalesced with few remaining voids. However, the staggered growth layers
$2\text{\,}$ \- $4\text{\,}$ show a greater variation in the intensity of
LDOS. The dI/dV signal can be correlated with the presence of surface
spherical features, as highlighted by the black circles in Figs. 4c, d. These
spherical features possess a larger LDOS, and hence it is likely that they
possess a non-stochiometric chemical composition, with an excess of Sr making
the spherical features more conductive. These spherical features are rather
dispersed throughout the staggered growth.

The EPS persists down to the nanometer scale. In Figs. 4c and 4d a
$10\text{\,}$$\times$$10\text{\,}\mathrm{nm}$ image reveals a clear
inhomogeneity of dI/dV with some electronic domains less than
$1\text{\,}\mathrm{nm}$ in diameter. During growth, Sr adatoms may diffuse
through the staggered growth before stochiometry is achieved in the bulk. The
surface roughness accompanying EPS has previously been reported for Ca-doped
LMO 34. Below the step edge, as indicated by the purple dotted line in width
Figs. 4a, b, a small region numbered $2^{\prime}$, of less than
$10\text{\,}\mathrm{nm}$ in width is observed. It has little staggered growth
and rather uniform LDOS, showing that surface morphology and the EPS are
correlated for the LSMO thin films.

Figure 5: Resistance vs temperature curves. (a) Resistance vs temperature of
$13\text{\,}$ u.c. LSMO thin film with resistance hysteresis observed between
the upward (up) and downward (down) temperature sweeps. (b) Fitting a
percolation model of resistivity of phase-separated ferromagnetic and
paramagnetic areas (orange curve) 35 and a purely ferromagnetic model (red
curve) 35.

The distribution of the dI/dV map of Fig. 4d indicates a bimodal distribution,
shown in Fig. 4e. A broader distribution of the blue areas is observed
compared to the red ones. This distribution is an indication of chemical
disorder caused by the local excess Sr and oxygen vacancies discussed
previously, which thus still persist in the relatively flat surface regions. A
possible effect of this disorder are magnetic inhomogeneities due to the local
variation in chemical composition. Such an magnetic inhomogeneity is important
to consider because of the spin-ordering effect in the OER 9.

To further verify this, we attempt to correlate a possible microscale magnetic
inhomogeneity with bulk transport measurements, which has been done before for
manganites 20, 22 and relates percolation theory 51 to the emergence of a
nonlinear response such as magnetoresistance. In our work, we observed two
nanoscale LDOS phases. In the percolation theory of phase separation, this
electronic disorder introduces hysteresis or memory effects in R(T) and M(T)
measurements. We performed the former, and the results in Fig. 5a indicate
such a hysteresis between the cooling-down and warming-up cycles. The
temperature dependence of the resistance can be regarded as a system in
competition between ferromagnetic (FM) metallic areas, with a certain spin
correlation length, and paramagnetic (PM) areas with a semiconducting nature.
Both regions can be identified by a varying spin-correlation length. For Ca-
doped manganites, a phenomenological model was developed in Ref. 35 that
describes the temperature dependence of the resistance as a percolation
between a volume fraction of the PM and FM areas. Here, we fit the same model
for LSMO, similar to the work in Ref. 52. In this model, the metallic
conductivity is described with electron-electron interaction by a $T^{2}$ term
and electron-spin fluctuation scattering by $T^{4.5}$ 35, 52. Therefore, the
FM resistance can be given as $\rho_{FM}(T)=\rho_{0}+AT^{2}+BT^{4.5}$. For
high temperatures, above $T_{c}$, the resistance can be described using a
magnetic polaron picture 35 represented as
$\rho_{PM}(T)=CT\text{exp}(\frac{E_{g}}{k_{b}T}$), with $E_{g}$ the activation
energy and $k_{b}$ the Boltzmann constant. We fitted this phenomenological
model to the R(T) data and a reasonable agreement is displayed in Fig. 5b. For
intermediate- and high-temperature regions, the fit agrees with the data, and
for the low-temperature regime, a deviation is observed. We fitted the low-
temperature regime with only the FM model, and a good agreement was reached,
as indicated by the red line in Fig. 5b. On the basis of these results, we
conjecture that the observed structural and/or electronic phase separation of
LSMO epitaxial thin films is likely correlated to the macroscopic film
resistive behavior, and complex FM and PM phase separation occurs, especially
at elevated temperatures.

##  3 Conclusion

The combination of XRD and RXR characterization and SPM imaging revealed a
complex surface and electronic morphology of LSMO thin films grown on STO by
PLD. Using element-specific RXR, we revealed a gradient in the chemical
composition of the film that showed a stochiometric bulk but a non-
stochiometric top layer of around $1\text{\,}\mathrm{nm}$ with an excess of
Sr, oxygen vacancies, formation of Mn2+ valence states, and a La deficiency.
This chemical disorder was further studied on the nanoscale with SPM where we
revealed a staggered growth on the surface of the films consisting of
nanoscale stacked islands and composed of spherical features. The formation of
staggered growth was described by a two-step growth model driven by an
Ehrlich-Schwoebel barrier on the surface and a reconstruction of the bulk
yielding high crystallinity. We correlated the staggered growth with the
formation of surface EPS driven by non-stochiometry and macroscale magnetic
inhomogeneities. For applications such as electrochemistry correlated to the
fundamental theoretical understanding of the complex oxides, careful
consideration of the surface morphology and electronic structure is important.
Therefore, the results presented in this work of the disentanglement between
the bulk and surface properties of LSMO make it a tantalizing model system for
evaluating the relationship between the surface and bulk spin ordering in
electrocatalysis 10, 53, 54, 9.

##  4 Methods

###  4.1 PLD and RHEED

Thin films of La0.67Sr0.33MnO3 were fabricated using PLD and a stochiometric
target from SurfaceNet. The vacuum system had a base pressure of
$5\text{\times}{10}^{-8}\text{\,}\mathrm{mbar}$ and was equipped with an in
situ RHEED and a KrF excimer laser of $248\text{\,}\mathrm{nm}$. B-site
terminated and step-terraced SrTiO3 (001) substrates were purchased from
CrysTec GmbH or Shinkosha Co. Ltd. The laser fluence was set to
$2.0\text{\,}\mathrm{J}\text{\,}{\mathrm{cm}}^{-2}$ and the frequency used was
$1\text{\,}\mathrm{Hz}$. The oxygen pressure was $0.266\text{\,}\mathrm{mbar}$
and the substrate temperature was $750\text{\,}\mathrm{\SIUnitSymbolCelsius}$.
The distance between the sample and the target was $5\text{\,}\mathrm{cm}$ and
a rectangular mask was used to obtain a laser spot size of
$2.24\text{\,}{\mathrm{mm}}^{2}$. The targets were pre-ablated at
$10\text{\,}\mathrm{Hz}$ to remove any possible surface contamination. After
deposition, the samples were slowly cooled with a rate of
$25\text{\,}\mathrm{\SIUnitSymbolCelsius}$/min inside the PLD at
$100\text{\,}\mathrm{mbar}$ oxygen pressure.

###  4.2 XRD

X-ray diffraction and reflectivity measurements were performed using a Bruker
D8 Discover diffractometer with Cu-K$\alpha$ radiation and an Eiger 2 R 500K
area detector. A Ge (022) monochromator was used for diffractograms and a
collimator of $1\text{\,}\mathrm{mm}$ diameter was used to obtain
Cu-K$\alpha$1 radiation and shape the incident beam. The detector was operated
in 0D mode with a small region of interest
($975\text{\,}\mathrm{\SIUnitSymbolMicro m}$ x
$4575\text{\,}\mathrm{\SIUnitSymbolMicro m}$) during the $2\Theta$-$\omega$
scans. To obtain the RSM, the detector was kept stationary while operating in
1D mode while an omega rocking curve was performed. Reflectivity measurements
were made using a $0.1\text{\,}\mathrm{mm}$ slit and a $1\text{\,}\mathrm{mm}$
collimator to shape the incident beam. The detector was operated in 0D mode
with a small region of interest ($975\text{\,}\mathrm{\SIUnitSymbolMicro m}$ x
$4575\text{\,}\mathrm{\SIUnitSymbolMicro m}$). Comparison of the measured and
expected diffractograms of the LSMO film peaks around the STO 002 peak in a
$2\Theta$-$\omega$ scan of the XRD spectra was performed using
InteractiveXRDFit software 55. More in-depth simulations of the XRD
diffractogram in this study were carried out using the GID SL program at
Sergei Stepanov’s X-Ray Server 38. The parameters used to vary the
crystallinity of the layer are the Debye-Waller like factors ($w_{0}$ and
$w_{h}$), which form a correction to scattering and absorption factors based
on the crystallinity of the material 38. Its value ranges from 0 to 1, where 0
is an amorphous layer and 1 corresponds to a perfect crystal without defects.
For the more crystalline layer, the values of $w_{0}$ and $w_{h}$ were taken
as $0.85\text{\,}$ to include a small number of defects and amount of
disorder, typically present in PLD grown films. For the lower crystalline
layer, a value of $0.75\text{\,}$ generated the best fit. A value of 1 was
used for $w_{0}$ and $w_{h}$ for the crystalline substrate. The roughness
values used were equal to $2\text{\,}\mathrm{\text{\r{A}}}$ for the substrate
and $4.5\text{\,}\mathrm{\text{\r{A}}}$ for the film, in accordance with the
roughness measured using AFM.

###  4.3 RXR

####  4.3.1 Data Acquisition

Data acquisition of the Resonant X-ray reflectometry (RXR) was done at the
Resonant Elastic and Inelastic X-ray Scattering (REIXS) beamline of the
Canadian Light Source (CLS) in Saskatoon, Canada 39. A flux of
$5\text{\times}{10}^{12}\text{\,}$ photons per second with a photon energy
resolution $\Delta$E/E of $\sim$10-4 was used for the measurements. To give
the desired linear and circular polarizations, an elliptical polarizing
undulator was used. Measurements were done at a temperature of
$300\text{\,}\mathrm{K}$ under a base pressure of
$1\text{\times}{10}^{-9}\text{\,}\mathrm{mbar}$. The reflection geometry scans
were made possible by an in-vacuum 4-circle diffractometer after the samples
were aligned with their surface normal in the scattering plane. Measurements
were done at several resonant photon energies at different resonances: Ti L2,3
($\sim$450-470 eV), Mn L2,3 ($\sim$635-660 eV), and La M4,5 ($\sim$830-860
eV), along with multiple nonresonant photon energies. The measurements were
done in specular reflection geometry. Circular-dichroic magnetic measurements
were performed under a homogeneous $0.6\text{\,}\mathrm{T}$ field that was
produced by inserting a permanent magnet array into the sample environment.
This aligned the magnetization in both the $xy$-plane of the film and the
scattering plane of the measurement. Dichroic measurements were done only at
the resonant photon energies of the Mn L2,3 resonance. A photodiode was used
to detect the reflected beam intensity, with the response function of the
photodiode determined by directly measuring the synchrotron beam. All measured
data were normalized by the incident beam flux and the response function to
obtain quantitative reflectivity spectra. The full RXR dataset is presented in
Ref. [9].

####  4.3.2 Modelling

Global Optimization of Resonant X-ray Reflectometry (GO-RXR), a software
package recently developed by the QMaX group at the University of Saskatchewan
was used for modelling of the RXR data. We used tabulated atomic form factors
for nonresonant energies 56 and measured x-ray absorption for the construction
of resonant scattering tensors for elements Ti, Mn, and La. For Mn we
differentiated the resonant scattering tensors for Mn2+ and for Mn in
stochiometric LSMO (implemented as a weighted linear combination of Mn3+ and
Mn4+ scattering tensors corresponding to Mn3.3+). A slab model was used to
determine the optical and magneto-optical profiles. Such a model is made up of
parametrized layers with defined elements, oxidation states, roughnesses,
thicknesses, and densities. By modeling the film at the u.c. level using the
model parameters we construct an element-specific discrete depth-dependent
density profile. This so-called u.c. model fixed the thickness of each layer
with a value that corresponds to the in-plane lattice constant of a
theoretical LSMO/STO heterostructure and models the roughness as a step
function. A density profile of the 13 u.c. LSMO samples is determined by
optimizing the density of the elements and oxidation states present in each
u.c. layer. First, an approximation of the density profile, along with the
form factors, is used to determine the expected energy- and depth-dependent
optical profile. The optical profile is then used to simulate the reflectivity
for a given energy, reflection angle, and polarization. Finally, we optimized
the density by fitting the simulations to the experimental data. The
concentration of Sr and La is fixed to 3:7 throughout the bulk of the film
(from the target stoichiometry) to reduce the parameter set; however it is
allowed to vary at the interface and surface of the film. The nonmagnetic
elemental density profile of Mn was first determined by optimizing the
parameters against an extended sigma-polarized experimental dataset.

###  4.4 Scanning probe microscopy

####  4.4.1 UHV atomic force microscopy

Atomic force microscopy was performed with a Scienta Omicron GmBH UHV VT-SPM
operating in ultra-high vacuum with a base pressure of
${10}^{-10}\text{\,}\mathrm{mbar}$. A self-sensing quartz tuning fork (AB38T)
was customized as a two-prong force oscillating sensor 43 with a Q-factor of
$15\,000\text{\,}$. Using a cleaved silicon wafer, a pyramidal-shaped tip is
formed with a radius of about $30\text{\,}\mathrm{nm}$. Frequency-Modulation
AFM using constant frequency shift feedback was used using a Phase-Lock Loop
(PLL). The tip oscillated with an amplitude of $1\text{\,}\mathrm{nm}$. The
ultrasharp-tip AFM sensor was fabricated by using a diamond tip (AdamaProbe).
This tip was glued to one prong of the self-sensing tuning fork using EPOTEK
E2101 UHV compatible two-component epoxy. AFM was operated in FM-AFM mode. The
Q-factor of the sensor was around $40\,000\text{\,}$. The tip oscillated with
an amplitude of $1.5\text{\,}\mathrm{nm}$. Post-processing of the AFM data was
performed with Gwyddion software 57. Images were line aligned using median of
differences and plane-leveled using mean subtraction.

####  4.4.2 Ambient atomic force microscopy

Contact-AFM was performed using a Nanosensor NCH-PPP Si cantilever on a Veeco
Dimension III microscope in ambient. Imaging was carried out with a normal
force of $5\text{\,}\mathrm{nN}$ with a scan speed of
$1\text{\,}\mathrm{l}$ines per second (512 lines/512 pixels). The friction
image was obtained by registering the lateral movement of the cantilever
detected by the 4-quadrant photodetector.

####  4.4.3 UHV scanning tunneling microscopy and scanning tunneling
spectroscopy

Scanning Tunneling Microscopy was performed with a Scienta Omicron GmbH UHV
VT-SPM operating in ultra-high vacuum with a base pressure of
${10}^{-10}\text{\,}\mathrm{mbar}$. STM tips were mechanically cut from PtIr
wire. The bias was applied to the tip and the sample was grounded. The imaging
was performed in constant current mode. For dI/dV mapping and spectroscopy, an
Oxford Instruments lock-in amplifier with an alternating voltage of
$100\text{\,}\mathrm{mV}$ applied at $4.1\text{\,}\mathrm{kHz}$ was used.
Post-processing of the STM data was performed with Gwyddion software 57. The
images were line-aligned using median of differences and plane-leveled using
mean subtraction.

Financial support from the Eindhoven University of Technology is acknowledged.
Support from the University of Twente in the framework of the tenure track
start-up package, the Natural Sciences and Engineering Research Council of
Canada (NSERC) Discovery Grant program, and the NSERC CREATE to INSPIRE
program are gratefully acknowledged.

## Author contributions

M.V., E.M, C.B., K.F. conceived and designed the experiments. E.M. synthesized
the samples and performed RHEED during synthesis. E.M. and E.K. performed and
analyzed x-ray diffraction measurements. L.K., R.S. and R.G. performed,
modeled, and analyzed resonant X-ray reflectivity measurements. M.V. performed
and analyzed scanning tunneling microscopy and spectroscopy mapping. M.V. and
E.M. performed and analyzed physical property measurements. G.K. advised on
the surface analysis and growth mode. C.B. and K.F. supervised the research.
M.V. and E.M. wrote the manuscript with contributions from all authors. All
authors have given their approval to the final version of the manuscript.

##  5 Supplementary

###  5.1 S1

Figure 6: Comparison of the measured and expected diffractogram of the LSMO
film peaks around the STO 002 peak in a $2\Theta$-$\omega$ scan. The expected
diffractrogram is obtained with the Interactive XRDFit software 55 using a
thickness of 13 u.c. and c = 3.845 $\mathrm{\text{\r{A}}}$.

###  5.2 S2

Figure 7: Contact-mode AFM. (a) Topographic map with contact-mode AFM showing
contrast predominantly from the step edges. (b) Friction map shows stronger
signal (blue) indicated along the step edges (indicated by the black arrow),
while it shows little on the plateaus.

###  5.3 S3

Figure 8: Line cross-section of Fig. 3a on the main text. Green arrows
indicate upward bulging of the topography near the step edge. The plateau’s
are indicated with the green lines.

###  5.4 S4

Figure 9: FFT filtering showing spherical features across the staggered growth
of the LSMO surface. (a) STM image of $13\text{\,}\mathrm{u}$.c. LSMO thin
film. (b) Fast Fourier filtering of the main staggered growth. (c) After the
removal of the staggered growth by FFT filtering leaves only the spherical
features of the film surface highlighted.

###  5.5 S5

Figure 10: STM observation of individual spherical features. STM image of
$13\text{\,}$ u.c. LSMO/STO film with staggered growth showing individual
spherical features distributed ontop of the upper layers, as indicated with
the arrows.

## References

  * Yin et al. 2020 Yin, Y.; Jiang, X.; Koten, M. A.; Shield, J. E.; Chen, X.; Yun, Y.; N’Diaye, A. T.; Hong, X.; Xu, X. Spin Rectification and Electrically Controlled Spin Transport in Molecular-Ferroelectrics-Based Spin Valves. _Physical Review Applied_ 2020, _13_ , 064011
  * Ogimoto et al. 2003 Ogimoto, Y.; Izumi, M.; Sawa, A.; Manako, T.; Sato, H.; Akoh, H.; Kawasaki, M.; Tokura, Y. Tunneling Magnetoresistance above Room Temperature in La${}_{\textrm{0.7}}$Sr${}_{\textrm{0.3}}$MnO${}_{\textrm{3}}$/SrTiO${}_{\textrm{3}}$/La${}_{\textrm{0.7}}$ Sr${}_{\textrm{0.3}}$MnO${}_{\textrm{3}}$ Junctions. _Japanese Journal of Applied Physics_ 2003, _42_ , L369–L372
  * Bruno et al. 2015 Bruno, F. Y. et al. Insight into spin transport in oxide heterostructures from interface-resolved magnetic mapping. _Nature Communications_ 2015, _6_ , 6306
  * Park et al. 1998 Park, J.-H.; Vescovo, E.; Kim, H.-J.; Kwon, C.; Ramesh, R.; Venkatesan, T. Direct evidence for a half-metallic ferromagnet. _Nature_ 1998, _392_ , 794–796
  * Bowen et al. 2003 Bowen, M.; Bibes, M.; Barthélémy, A.; Contour, J.-P.; Anane, A.; Lemaitre, Y.; Fert, A. Nearly total spin polarization in La${}_{\textrm{2/3}}$Sr${}_{\textrm{1/3}}$MnO${}_{\textrm{3}}$ from tunneling experiments. _Applied Physics Letters_ 2003, _82_ , 233–235
  * Dagotto et al. 2001 Dagotto, E.; Hotta, T.; Moreo, A. Colossal magnetoresistant materials: the key role of phase separation. _Physics Reports_ 2001, _344_ , 1–153
  * Chmaissem et al. 2003 Chmaissem, O.; Dabrowski, B.; Kolesnik, S.; Mais, J.; Jorgensen, J. D.; Short, S. Structural and magnetic phase diagrams of La${}_{\textrm{(1-x)}}$Sr${}_{\textrm{x}}$MnO${}_{\textrm{3}}$ and Pr${}_{\textrm{(1-y)}}$Sr${}_{\textrm{y}}$MnO${}_{\textrm{3}}$. _Physical Review B_ 2003, _67_ , 094431
  * Urushibara et al. 1995 Urushibara, A.; Moritomo, Y.; Arima, T.; Asamitsu, A.; Kido, G.; Tokura, Y. Insulator-metal transition and giant magnetoresistance in La${}_{\textrm{(1-x)}}$Sr${}_{\textrm{x}}$MnO${}_{\textrm{3}}$. _Physical Review B_ 1995, _51_ , 14103–14109
  * van der Minne et al. 2023 van der Minne, E. et al. Beyond domain alignment: Revealing the effect of intrinsic magnetic order on electrochemical water splitting. 2023; Publisher: arXiv Version Number: 1
  * Xu et al. 2023 Xu, H.; Qi, J.; Zhang, Y.; Liu, H.; Hu, L.; Feng, M.; Lü, W. Magnetic Field-Enhanced Oxygen Evolution Reaction via the Tuneability of Spin Polarization in a Half-Metal Catalyst. _ACS Applied Materials & Interfaces_ 2023, _15_ , 32320–32328
  * Scholz et al. 2016 Scholz, J.; Risch, M.; Stoerzinger, K. A.; Wartner, G.; Shao-Horn, Y.; Jooss, C. Rotating Ring–Disk Electrode Study of Oxygen Evolution at a Perovskite Surface: Correlating Activity to Manganese Concentration. _The Journal of Physical Chemistry C_ 2016, _120_ , 27746–27756
  * Lee et al. 2021 Lee, J.; Adiga, P.; Lee, S. A.; Nam, S. H.; Ju, H.; Jung, M.; Jeong, H. Y.; Kim, Y.; Wong, C.; Elzein, R.; Addou, R.; Stoerzinger, K. A.; Choi, W. S. Contribution of the Sub-Surface to Electrocatalytic Activity in Atomically Precise La${}_{\textrm{0.7}}$Sr${}_{\textrm{0.3}}$MnO${}_{\textrm{3}}$ Heterostructures. _Small_ 2021, _17_ , 2103632
  * Gonell et al. 2020 Gonell, F.; Sánchez-Sánchez, C. M.; Vivier, V.; Méthivier, C.; Laberty-Robert, C.; Portehault, D. Structure–Activity Relationship in Manganese Perovskite Oxide Nanocrystals from Molten Salts for Efficient Oxygen Reduction Reaction Electrocatalysis. _Chemistry of Materials_ 2020, _32_ , 4241–4247
  * Hale and Choudhury 2022 Hale, W.; Choudhury, P. Thermodynamic Stability and Intrinsic Activity of La${}_{\textrm{(1-x)}}$Sr${}_{\textrm{x}}$MnO${}_{\textrm{3}}$ (LSMO) as an Efficient Bifunctional OER/ORR Electrocatalysts: A Theoretical Study. _Catalysts_ 2022, _12_ , 260
  * Qi et al. 2022 Qi, J.; Zhang, Y.; Liu, H.; Xu, H.; Wang, C.; Hu, L.; Feng, M.; Lü, W. Strain Modified Oxygen Evolution Reaction Performance in Epitaxial, Freestanding, and Van Der Waals Manganite Thin Films. _Nano Letters_ 2022, _22_ , 7066–7072
  * Gracia 2017 Gracia, J. Spin dependent interactions catalyse the oxygen electrochemistry. _Physical Chemistry Chemical Physics_ 2017, _19_ , 20451–20456
  * Moreo et al. 2000 Moreo, A.; Mayr, M.; Feiguin, A.; Yunoki, S.; Dagotto, E. Giant Cluster Coexistence in Doped Manganites and Other Compounds. _Physical Review Letters_ 2000, _84_ , 5568–5571
  * Burgy et al. 2001 Burgy, J.; Mayr, M.; Martin-Mayor, V.; Moreo, A.; Dagotto, E. Colossal Effects in Transition Metal Oxides Caused by Intrinsic Inhomogeneities. _Physical Review Letters_ 2001, _87_ , 277202
  * Debnath and Lin 2003 Debnath, A. K.; Lin, J. G. Current-induced giant electroresistance in La${}_{\textrm{0.7}}$Sr${}_{\textrm{0.3}}$MnO${}_{\textrm{3}}$ thin films. _Physical Review B_ 2003, _67_ , 064412
  * Miao et al. 2020 Miao, T. et al. Direct experimental evidence of physical origin of electronic phase separation in manganites. _Proceedings of the National Academy of Sciences_ 2020, _117_ , 7090–7094
  * Zhu et al. 2016 Zhu, Y. et al. Chemical ordering suppresses large-scale electronic phase separation in doped manganites. _Nature Communications_ 2016, _7_ , 11260
  * Baldini et al. 2015 Baldini, M.; Muramatsu, T.; Sherafati, M.; Mao, H.-k.; Malavasi, L.; Postorino, P.; Satpathy, S.; Struzhkin, V. V. Origin of colossal magnetoresistance in LaMnO${}_{\textrm{3}}$ manganite. _Proceedings of the National Academy of Sciences_ 2015, _112_ , 10869–10872
  * López-Mir et al. 2016 López-Mir, L.; Balcells, L.; Valencia, S.; Kronast, F.; Martínez, B.; De Miguel, J. J.; Ocal, C. Growth Instabilities as a Source of Surface Chemical Structuration in Functional Perovskite Thin Films. _Crystal Growth & Design_ 2016, _16_ , 5479–5486
  * Zhang et al. 2021 Zhang, C.; Ding, S.; Qiao, K.; Li, J.; Li, Z.; Yin, Z.; Sun, J.; Wang, J.; Zhao, T.; Hu, F.; Shen, B. Large Low-Field Magnetoresistance (LFMR) Effect in Free-Standing La${}_{\textrm{0.7}}$Sr${}_{\textrm{0.3}}$MnO${}_{\textrm{3}}$ Films. _ACS Applied Materials & Interfaces_ 2021, _13_ , 28442–28450
  * Yin et al. 2018 Yin, L.; Wang, C.; Li, L.; Shen, Q.; Zhang, L. Large room temperature magnetoresistance in La${}_{\textrm{0.9}}$Sr${}_{\textrm{0.1}}$MnO${}_{\textrm{3}}$ thin films. _Journal of Alloys and Compounds_ 2018, _730_ , 327–332
  * Liu et al. 2016 Liu, G. Z.; Yang, Y. Y.; Qiu, J.; Chen, X. X.; Jiang, Y. C.; Yao, J. L.; Zhao, M.; Zhao, R.; Gao, J. Substrate-related structural, electrical, magnetic and optical properties of La${}_{\textrm{0.7}}$Sr${}_{\textrm{0.3}}$MnO${}_{\textrm{3}}$ films. _Journal of Physics D: Applied Physics_ 2016, _49_ , 075304
  * Carreira et al. 2019 Carreira, S. J.; Aguirre, M. H.; Briatico, J.; Steren, L. B. Nanoscale magnetic and charge anisotropies at manganite interfaces. _RSC Advances_ 2019, _9_ , 38604–38611
  * Gambardella et al. 2014 Gambardella, A.; Graziosi, P.; Bergenti, I.; Prezioso, M.; Pullini, D.; Milita, S.; Biscarini, F.; Dediu, V. A. Surface Nanostructures in Manganite Films. _Scientific Reports_ 2014, _4_ , 5353
  * Tselev et al. 2015 Tselev, A.; Vasudevan, R. K.; Gianfrancesco, A. G.; Qiao, L.; Ganesh, P.; Meyer, T. L.; Lee, H. N.; Biegalski, M. D.; Baddorf, A. P.; Kalinin, S. V. Surface Control of Epitaxial Manganite Films via Oxygen Pressure. _ACS Nano_ 2015, _9_ , 4316–4327
  * Pandya et al. 2016 Pandya, S.; Damodaran, A. R.; Xu, R.; Hsu, S.-L.; Agar, J. C.; Martin, L. W. Strain-induced growth instability and nanoscale surface patterning in perovskite thin films. _Scientific Reports_ 2016, _6_ , 26075
  * Kelley et al. 2021 Kelley, K. P.; Sharma, V.; Zhang, W.; Baddorf, A. P.; Nascimento, V. B.; Vasudevan, R. K. Exotic Long-Range Surface Reconstruction on La${}_{\textrm{0.7}}$Sr${}_{\textrm{0.3}}$MnO${}_{\textrm{3}}$ Thin Films. _ACS Applied Materials & Interfaces_ 2021, _13_ , 9166–9173
  * De Jong et al. 2005 De Jong, M. P.; Bergenti, I.; Dediu, V. A.; Fahlman, M.; Marsi, M.; Taliani, C. Evidence for Mn 2+ ions at surfaces of La${}_{\textrm{0.7}}$Sr${}_{\textrm{0.3}}$MnO${}_{\textrm{3}}$ thin films. _Physical Review B_ 2005, _71_ , 014434
  * Dulli et al. 2000 Dulli, H.; Dowben, P. A.; Liou, S.-H.; Plummer, E. W. Surface segregation and restructuring of colossal-magnetoresistant manganese perovskites La${}_{\textrm{0.65}}$Sr${}_{\textrm{0.35}}$MnO${}_{\textrm{3}}$. _Physical Review B_ 2000, _62_ , R14629–R14632
  * Kelly et al. 2009 Kelly, S.; Galli, F.; Komissarov, I.; Aarts, J. Correlations between the morphology and the electronic structure at the surface of thin film manganites, investigated with STM. 2009; arXiv:0902.0626 [cond-mat]
  * Li et al. 2002 Li, G.; Zhou, H.-D.; Feng, S. J.; Fan, X.-J.; Li, X.-G.; Wang, Z. D. Competition between ferromagnetic metallic and paramagnetic insulating phases in manganites. _Journal of Applied Physics_ 2002, _92_ , 1406–1410
  * Koster 1999 Koster, G. _Artificially layered oxides by pulsed laser deposition_ ; University of Twente: Enschede, 1999; OCLC: 67824844
  * Yang et al. 2010 Yang, F.; Kemik, N.; Biegalski, M. D.; Christen, H. M.; Arenholz, E.; Takamura, Y. Strain engineering to control the magnetic and magnetotransport properties of La${}_{\textrm{0.67}}$Sr${}_{\textrm{0.33}}$MnO${}_{\textrm{3}}$ thin films. _Applied Physics Letters_ 2010, _97_ , 092503
  * Stepanov 2004 Stepanov, S. A. X-ray server: an online resource for simulations of x-ray diffraction and scattering. Denver, CO, 2004; pp 16–26
  * Green et al. 2020 Green, R. J.; Sutarto, R.; He, F.; Hepting, M.; Hawthorn, D. G.; Sawatzky, G. A. Resonant Soft X-ray Reflectometry and Diffraction Studies of Emergent Phenomena in Oxide Heterostructures. _Synchrotron Radiation News_ 2020, _33_ , 20–24
  * Hamann-Borrero et al. 2016 Hamann-Borrero, J. E.; Macke, S.; Choi, W. S.; Sutarto, R.; He, F.; Radi, A.; Elfimov, I.; Green, R. J.; Haverkort, M. W.; Zabolotnyy, V. B.; Lee, H. N.; Sawatzky, G. A.; Hinkov, V. Valence-state reflectometry of complex oxide heterointerfaces. _npj Quantum Materials_ 2016, _1_ , 16013
  * Porter et al. 2017 Porter, S. B.; Venkatesan, M.; Dunne, P.; Doudin, B.; Rode, K.; Coey, J. M. D. Magnetic Dead Layers in La${}_{\textrm{0.7}}$Sr${}_{\textrm{0.3}}$MnO${}_{\textrm{3}}$ Revisited. _IEEE Transactions on Magnetics_ 2017, _53_ , 1–4
  * Li et al. 2012 Li, Z.; Bosman, M.; Yang, Z.; Ren, P.; Wang, L.; Cao, L.; Yu, X.; Ke, C.; Breese, M. B. H.; Rusydi, A.; Zhu, W.; Dong, Z.; Foo, Y. L. Interface and Surface Cation Stoichiometry Modified by Oxygen Vacancies in Epitaxial Manganite Films. _Advanced Functional Materials_ 2012, _22_ , 4312–4321
  * Verhage et al. 2023 Verhage, M.; Çiftçi, T. H.; Reul, M.; Cromwijk, T.; van Stralen, T. J. N.; Koopmans, B.; Kurnosikov, O.; Flipse, K. Switchable-magnetisation planar probe MFM sensor. 2023; arXiv:2302.11387 [physics]
  * Giessibl 2019 Giessibl, F. J. The qPlus sensor, a powerful core for the atomic force microscope. _Review of Scientific Instruments_ 2019, _90_ , 011101
  * Heath et al. 2021 Heath, G. R.; Kots, E.; Robertson, J. L.; Lansky, S.; Khelashvili, G.; Weinstein, H.; Scheuring, S. Localization atomic force microscopy. _Nature_ 2021, _594_ , 385–390
  * Boschker et al. 2011 Boschker, H.; Huijben, M.; Vailionis, A.; Verbeeck, J.; Van Aert, S.; Luysberg, M.; Bals, S.; Van Tendeloo, G.; Houwman, E. P.; Koster, G.; Blank, D. H. A.; Rijnders, G. Optimized fabrication of high-quality La${}_{\textrm{0.67}}$Sr${}_{\textrm{0.33}}$MnO${}_{\textrm{3}}$ thin films considering all essential characteristics. _Journal of Physics D: Applied Physics_ 2011, _44_ , 205001
  * Molinari et al. 2022 Molinari, A.; Gorji, S.; Michalička, J.; Kübel, C.; Hahn, H.; Kruk, R. Tailoring interface epitaxy and magnetism in La${}_{\textrm{(1-x)}}$Sr${}_{\textrm{x}}$MnO${}_{\textrm{3}}$/SrTiO${}_{\textrm{3}}$ heterostructures via temperature-driven defect engineering. _Journal of Applied Physics_ 2022, _132_ , 105304
  * Chen et al. 2019 Chen, Z.; Khajeh, A.; Martini, A.; Kim, S. H. Chemical and physical origins of friction on surfaces with atomic steps. _Science Advances_ 2019, _5_ , eaaw0513
  * Beausoleil et al. 2014 Beausoleil, A.; Desjardins, P.; Rochefort, A. Impact of nucleation on step-meandering instabilities during step-flow growth on vicinal surfaces. _Physical Review E_ 2014, _89_ , 032406
  * Tersoff et al. 1994 Tersoff, J.; Denier Van Der Gon, A. W.; Tromp, R. M. Critical island size for layer-by-layer growth. _Physical Review Letters_ 1994, _72_ , 266–269
  * Nan 1993 Nan, C.-W. _Physics of inhomogeneous inorganic materials_ ; Pergamon Press Ltd, 1993; Vol. 37
  * Yin et al. 2016 Yin, L.; Wang, C.; Shen, Q.; Zhang, L. Strain-induced Curie temperature variation in La${}_{\textrm{0.9}}$Sr${}_{\textrm{0.1}}$MnO${}_{\textrm{3}}$ thin films. _RSC Advances_ 2016, _6_ , 96093–96102
  * Ren et al. 2021 Ren, X.; Wu, T.; Sun, Y.; Li, Y.; Xian, G.; Liu, X.; Shen, C.; Gracia, J.; Gao, H.-J.; Yang, H.; Xu, Z. J. Spin-polarized oxygen evolution reaction under magnetic field. _Nature Communications_ 2021, _12_ , 2608
  * Li et al. 2019 Li, X.; Sun, Y.; Wu, Q.; Liu, H.; Gu, W.; Wang, X.; Cheng, Z.; Fu, Z.; Lu, Y. Optimized Electronic Configuration to Improve the Surface Absorption and Bulk Conductivity for Enhanced Oxygen Evolution Reaction. _Journal of the American Chemical Society_ 2019, _141_ , 3121–3128
  * Lichtensteiger 2018 Lichtensteiger, C. InteractiveXRDFit : a new tool to simulate and fit X-ray diffractograms of oxide thin films and heterostructures. _Journal of Applied Crystallography_ 2018, _51_ , 1745–1751
  * Henke et al. 1993 Henke, B.; Gullikson, E.; Davis, J. X-Ray Interactions: Photoabsorption, Scattering, Transmission, and Reflection at E = 50-30,000 eV, Z = 1-92. _Atomic Data and Nuclear Data Tables_ 1993, _54_ , 181–342
  * Nečas and Klapetek 2012 Nečas, D.; Klapetek, P. Gwyddion: an open-source software for SPM data analysis. _Open Physics_ 2012, _10_


Send correspondence to Jonathan S. Kent
E-mail<EMAIL_ADDRESS>
# Unsupervised Learning for Target Tracking and Background Subtraction in
Satellite Imagery

Jonathan S. Kent  Ball Aerospace & Technologies Corp., 2875 Presidential
Drive, Fairborn, OH, USA 45324  Charles C. Wamsley  Ball Aerospace &
Technologies Corp., 2875 Presidential Drive, Fairborn, OH, USA 45324  Davin
Flateau  Ball Aerospace & Technologies Corp., 2875 Presidential Drive,
Fairborn, OH, USA 45324  Amber Ferguson  Ball Aerospace & Technologies Corp.,
2875 Presidential Drive, Fairborn, OH, USA 45324

###### Abstract

This paper describes an unsupervised machine learning methodology capable of
target tracking and background suppression via a novel dual-model approach.
“Jekyll” produces a video bit-mask describing an estimate of the locations of
moving objects, and “Hyde” outputs a pseudo-background frame to subtract from
the original input image sequence. These models were trained with a custom-
modified version of Cross Entropy Loss.

Simulated data were used to compare the performance of Jekyll and Hyde against
a more traditional supervised Machine Learning approach. The results from
these comparisons show that the unsupervised methods developed are competitive
in output quality with supervised techniques, without the associated cost of
acquiring labeled training data.

###### keywords:

Unsupervised Learning, Satellite Imagery, Target Tracking, Machine Learning

##  1 INTRODUCTION

Target tracking and background subtraction are central tasks in the field of
satellite imagery analysis [1, 2, 3], made more difficult by the existence of
background motion, weather patterns, and both static and transient sensor
artifacts. However, manual segmentation of available data remains costly in
time, effort, and money [4], and transfers that cost to supervised learning-
based projects, which require a set of manually created targets to train on.
Therefore, despite the increased theoretical complexity required, it is
fruitful to develop unsupervised algorithms and model architectures capable of
operating in this area, which do not require explicit targets and therefore
are not burdened by the cost of creating them [5]. The solution proposed here
involves developing two networks, called “Jekyll” and “Hyde,”111In the book
“The Strange Case of Dr. Jekyll and Mr. Hyde,” Dr. Jekyll and Mr. Hyde are two
halves of the same person, and a recurring theme involves Dr. Jekyll using his
wealth and influence to mask over the crimes committed by Mr. Hyde, while they
are investigated by Mr. Utterson. Similarly, Jekyll produces a bit-mask at
cost to itself that covers the errors generated by Hyde, and they are compared
to a supervised model named Utterson. [6] which respectively produce a target
tracking bit-mask over a series of frames, and an approximation of a
background image. These two outputs are compared to the input used to generate
them, with a new loss function defined to express the difference between the
Hyde-generated pseudo-background and every frame of the input, but masked out
by the presence of an object detection by Jekyll.

The data domain in use for these experiments consists of Earth-imagery frames
that change over time, i.e. “video” from an infrared “camera”, containing
targets of interest which are spatially unresolved. These targets are observed
by the sensors with variable Signal-to-Noise ratios.

The data used for the models and experiments in this work were simulated using
the Air Force Institute of Technology (AFIT) Sensor and Scene Emulation Tool,
or ASSET [7, 8]. However, the work presented in this paper focuses solely on
developing an initial approach to an unsupervised target tracking step, rather
than providing a complete solution for target tracking and background
subtraction, robust to background motion, weather, and sensor artifacts. To
facilitate this, those confounding effects were left out of the data generated
using ASSET. Further work will be required to address them for real world
applications.

The actual data-set generated consists of two $500\times 500\times 500$111500
frames, 500 pixels of width and height. data tensors: an input data tensor
containing Gaussian scaled222Here, Gaussian scaling of a data-set
$\mathcal{D}_{u}$ into $\mathcal{D}_{s}$ consists of
$\mathcal{D}_{s}\leftarrow\frac{\mathcal{D}_{u}-\mu(\mathcal{D}_{u})}{\sigma(\mathcal{D}_{u})}$,
which gives $\mathcal{D}_{s}$ a mean of $0$ and a standard deviation of $1$.
Where $\mathcal{D}_{u}$ may have values in the hundreds of thousands, which
would break the numerics of neural network processing, $\mathcal{D}_{s}$ will
be constrained relatively close to $0$.[9] infrared intensity “image” or
“video” data including backgrounds and target objects, and a synthetic target
data tensor consisting of a corresponding pixel-by-pixel binary target mask
[10] denoting the locations of target objects. The synthetic target data is
used for reporting numerical and graphical results, as well as to train a
supervised model that can be used as a benchmark for the unsupervised models.
These tensors were carved into samples consisting of $16$ frames of $64\times
64$ video, which were separated into Training, Validation, and Testing subsets
to avoid over-fitting during experimentation [11] with 50%, 20%, and 30% of
the samples respectively.

##  2 RELATED WORK

Modern developments in unsupervised learning were kick-started by the
development of the Generative Adversarial Networks (GAN) framework [12], which
was one of the first to introduce a methodology involving multiple networks
“playing against” one another, hence adversarial networks. Typically, GANs are
employed in the task of generating new samples from a high-dimensional
distribution, by feeding a generator network a random vector and having a
discriminator network attempt to determine which of a collection of samples
have been generated artificially. Somewhat similar in nature to the GAN is the
Variational Auto-Encoder (VAE) [13], which attaches the output of an encoding
network directly to the input of a decoding network, and trains both according
to reconstruction loss. In the case of VAEs, the two networks share a loss
function and are encouraged to work together to minimize it. The use of multi-
network systems for unsupervised learning inspired the approach used in this
paper.

Previous work involving VAEs have shown that it is possible to perform image
segmentation in an unsupervised setting [14]. This was achieved by minimizing
both the reconstruction error of the VAE, and the normalized cut [15] of the
encoded state. Although image segmentation and target tracking possess
theoretical similarities, in an unsupervised setting the tendency of
segmentation algorithms is towards segmenting the background into, for
example, mountains and valleys, rather than background versus target objects.
[16] introduced an extremely clever unsupervised target tracking scheme that
achieved high accuracy. It produces a single target track forwards and
backwards through time, and takes the differences between the two tracks as
its loss, assuming the only consistent solution is to track a moving target,
using regularization to rule out staying put. Unfortunately, it is unsuitable
for analyzing satellite imagery, as it can neither track multiple objects
simultaneously, nor render the size of detected objects.

Both [17] and [18] accomplished unsupervised object detection using classical
methods. [17] leverages algorithmic clustering of similar arrangements of
pixels together, and [18] uses a modified form of the Winnow algorithm [19].
Both of these predate current Computer Vision research, and as a result do not
leverage the capabilities of neural Machine Learning, instead producing much
less powerful linear classification models akin to a single convolutional
layer. Both algorithms are limited by their problem formulation, in that they
are intended to be used with a single background, e.g. a separate model would
be developed for each camera in a security system because they look at
different scenes around a house, despite them all being used for the same
task. This leaves them both incapable of generalization, and both assume that
unusual deviation from the mean value of a pixel implies an object of
interest. However, by abstracting much of the algorithmic work of [17] or [18]
to the Machine Learning model, their shared assumption that unusual deviation
implies interest can be reused in designing a loss function.

##  3 Model Formulation and Loss Function

###  3.1 Formulation

Both Jekyll and Hyde are formulated as feed-forward, three dimensional
convolutional neural networks (CNN) [20, 21, 22], implemented with PyTorch
[23, 24]. Due to the nature of the satellite sensor, the imagery that is used
represents an approximate intensity mapping, scaled according to mean and
standard deviation, rather than color photography, and therefore is to be
treated as single-channel rather than multi-channel imagery. Both Jekyll and
Hyde accept a $K\times 1\times N\times W\times H$ tensor as their input, where
$K$ is the number of samples in the batch, $N$ is the number of frames in a
sample, and $W\times H$ describes the width and height of the input image.

Both models employ a number of skip/highway connections [25] in their
architecture, concatenating the hidden states from the $n^{th}$ layer in the
first half of the network with the inputs to the $n^{th}$-from-the-last layer
in the second half of the network. This helps to combat gradient degradation
in the early layers, as well as giving access to more primitive data to the
later layers. The structure of these networks could be analogized to an
hourglass, skip connections notwithstanding. The first half of the network
increases the number of filters and decreases the width and height of the
hidden state, with the second half of the network reversing the process by
decreasing the number of filters and increasing the width and height. The
first half consists of convolution and max-pool operations, and the latter
half of transpose convolutions and max-unpool operations. ReLU activations
[26] are used between convolution-type operations. This is somewhat similar to
the model architectures described in [27, 28, 29, 30], but simplified and
using 3D convolutions.

The only architectural difference between Jekyll and Hyde is an output
function, which is executed immediately after their last layer. Jekyll
possesses a final Sigmoid activation function [11], turning a hidden state
into a tensor of probabilities denoting the estimated likelihood that a given
pixel at a given time contains a target object. Hyde’s final output is taken
as the frame-wise mean over its last layer, producing a single frame to be
used as an approximated background.

For direct comparison between supervised and unsupervised methods in this
domain, a competing supervised model, named “Utterson” [6], is trained against
synthetic target data. Utterson possesses an architecture identical to that of
Jekyll, and is tested on the same task as Jekyll; producing a bit-mask
denoting the positions of targets of interest. For a loss function, Utterson
uses vanilla Binary Cross Entropy loss, as is natively supported in PyTorch
[11, 23]. A supervised model against which to compare the unsupervised models
will allow for direct analysis of the costs and benefits associated with using
unsupervised instead of supervised learning on this task.

###  3.2 Loss Function

The goal of the loss function is to impel Hyde to produce an approximation of
the background of the input image sequence sans moving target objects, and get
Jekyll to label the locations of those moving objects without also including
portions of the background. This will be accomplished by using a modified form
of Cross Entropy Loss [11] along with a cost associated with masking. While
Cross Entropy Loss is typically used for classification problems, by
formulating target tracking as “classifying pixels between is-a-target and
just-the-background,” it becomes applicable.

The typical formulation of Binary Cross Entropy Loss is given as:

$-\sum_{x\in\mathcal{X}}Y(x)\textbf{ln}\big{(}\hat{Y}(x)\big{)}\ +\
\big{(}1-Y(x)\big{)}\textbf{ln}\big{(}1-\hat{Y}(x)\big{)}$

where $Y(x)$ is the true label of $x$, and takes the value $1$ where $x$ is a
target, and $0$ where $x$ is the background. $\hat{Y}(x)$ is the probability
estimated by the model that $x$ is a target. By using the squared difference
between the input and Hyde’s pseudo-background as an analogue for $Y(x)$, and
replacing the $\big{(}1-Y(x)\big{)}\textbf{ln}\big{(}1-\hat{Y}(x)\big{)}$
term111This term breaks down for $Y(x)>1$, and so requires modification. The
original term’s use was that it would increase loss where the model
incorrectly output high values, encouraging it not to simply declare
everything to be a target. But it will be demonstrated that, in this case,
this functionality can be replicated by assigning a constant cost to
outputting any value at all, encouraging the model only to “spend” where it
expects to lower the other term by a greater amount. with a simpler cost term,
an alternate version of entropic loss can be used in the unsupervised case. In
addition, Hyde contributes to decreasing loss by producing a background that
minimizes the difference between it and the input, except where Jekyll is
masking it out. As a result, the same loss can be used to optimize both
models. This loss function will now be constructed from its elements.

For the purposes of simplifying notation, the loss function is described as it
applies to a single $N\times W\times H$ sample, rather than to the full
$K\times 1\times N\times W\times H$ batch input.

The tensor describing the frame-wise linear differential between Hyde’s output
$H$ and the input $i$, the frames of which are $i_{1},i_{2}\ ...\ i_{N}$:

$\hat{\Delta}(\theta_{h},i)=\begin{bmatrix}i_{1}-H(\theta_{h},i),&i_{2}-H(\theta_{h},i),&\dots&i_{N}-H(\theta_{h},i)\end{bmatrix}$

The element-wise square of the linear differential:

$\hat{\Delta}^{2}(\theta_{h},i)=\hat{\Delta}(\theta_{h},i)\odot\hat{\Delta}(\theta_{h},i)$

The tensor containing the squared differentials, masked out by the negative
natural logarithm of Jekyll’s bit-mask:

$\hat{\mathcal{L}}_{h}(\theta_{h},\theta_{j},i)=-\textbf{ln}\big{(}J(\theta_{j},i)+\epsilon\big{)}\odot\hat{\Delta}^{2}(\theta_{h},i)$111Consider
that where $J(\theta_{j},i)\approx 1$,
$-\textbf{ln}\big{(}J(\theta_{j},i)\big{)}\approx 0$, so where
$J(\theta_{j},i)$ is large, the difference $\hat{\Delta}^{2}(\theta_{h},i)$
does not contribute significantly to loss. Alternatively,
where$J(\theta_{j},i)\approx 0$, $-\textbf{ln}\big{(}J(\theta_{j},i)\big{)}\gg
0$, meaning that any substantial value in $\hat{\Delta}^{2}(\theta_{h},i)$
will contribute heavily. A small additive $\epsilon=0.001$ is used to prevent
numerical errors.

The loss term contributed by masked squared “pseudo-background” error loss as
the element-wise mean of the masked squared differentials:

$\mathcal{L}_{h}(\theta_{h},\theta_{j},i)=\textbf{mean}\big{(}\hat{\mathcal{L}}_{h}(\theta_{h},\theta_{j},i)\big{)}$

Including the mean quantity masked with a multiplicative hyper-parameter:

$\mathcal{L}(\theta_{h},\theta_{j},i)=\mathcal{L}_{h}(\theta_{h},\theta_{j},i)+\alpha\cdot\textbf{mean}\big{(}J(\theta_{j},i)\big{)}$222This
“pixel-wise cost” addition to the entropic loss essentially forces the model
to make a decision; increase loss via an expenditure associated with masking
out more pixels, or increase loss by choosing not to mask out error between
the background and the input at a particular location. Thus, the model will be
incentivized to mask out error where and only where entropic loss would be
expected to exceed $\alpha$, which, with an accurate background estimation,
would only be at the location of a moving object.

Which serves as the loss function for both Jekyll and Hyde. This unified loss
function allows for the computer to execute a single back-propagation
operation per training batch, which speeds up computation appreciably.

##  4 HYPER-PARAMETERS USED AND TRAINING REGIME

Initially, the simulated data is sliced into samples with $N=16,\ W=64,\
H=64$, and separated into Training, Validation, and Testing subsets with 50%,
20%, and 30% of the samples in them respectively, with a total of
approximately 1,000 samples over all three subsets. It was found
experimentally that $\alpha=1$ produced high quality results, where $\alpha$
is the multiplicative hyper-parameter associated with assigning pixel-wise
cost to masking.

Jekyll, Hyde, and Utterson were each given 100 epochs with PyTorch’s native
implementation of Adam optimizer [31, 23]. All three models had a weight decay
of 0.01, Hyde and Utterson had a learning rate of $5.0*10^{-4}$, and Jekyll
had a learning rate of $5.0*10^{-5}$. All other hyper-parameters were left as
default. Jekyll’s learning rate was made lower than Hyde’s as it was found
that, learning at the same rate, Jekyll would begin masking out large errors
that Hyde produced early in training, with Hyde never learning to correct the
errors, and Jekyll never being able to stop masking them out. Jekyll and Hyde
were trained using the loss function described earlier in this paper, while
Utterson was trained with vanilla Binary Cross-Entropy Loss.

##  5 RESULTS

Numerical results describing the accuracy of the target tracking models, the
unsupervised Jekyll and the supervised Utterson, on the testing data-set are
reported in Figures 1 and 2.111Due to the nature of the differences in
approach to evaluation by the supervised and unsupervised models, Utterson was
incapable of producing an output exceeding $\sim 0.45$. To account for this in
numerical and graphical comparisons, Utterson’s outputs were linearly re-
scaled to $(0,1)$, similar to those of Jekyll. These were obtained for a
variety of “threshold” values, by, for a given threshold $t$, considering any
model output greater than or equal to $t$ to be a positive prediction (i.e.
predicting the presence of a target), and any output less than $t$ to be a
negative prediction (i.e. predicting the absence of a target). As the
threshold increases, a model is less likely to be read as predicting the
presence of a target as the model must be more “confident” for this to be the
case, but it is also less likely to raise a false alarm. The numerical results
include: Positive Predictive Value, the likelihood of a model’s positive
predictions to be correct; Negative Predictive Value, the likelihood of a
model’s negative predictions to be correct; Sensitivity, the likelihood for a
model to pick up a particular target pixel; and Specificity, the likelihood
for a model to reject a particular non-target pixel.222These numerical results
are meant as a rough comparison between models, and as support for the claims
made. They do not precisely represent the quality or value of the outputs of
the models, e.g. both models tend to over-represent the size of the target
objects, as can be seen in the included imagery, lowering the PPV
significantly, while still accurately representing the locations and motions
of targets. Of these results, Sensitivity is the most meaningful, as it
provides a measurement of the ability of the models to pick up targets.

Figure 1: The Positive Predictive Values (PPV) and Negative Predictive Values
(NPV) by threshold on the left and right, respectively. PPV is a measure of
the likelihood that, if the model has labeled a pixel as containing a target,
it does actually contain a target. NPV measures the likelihood that if the
model says there is no target there, that it does not contain a target. The
threshold (t) determines at what point the numerical output by the model is
considered to be a positive prediction, e.g. for $t=0.3$, if the model outputs
a 0.4, then it is considered to be predicting the presence of a target, and if
it outputs 0.2, it is read as predicting the absence of a target.

Figure 2: The Sensitivity and Specificity by threshold on the left and right,
respectively. Sensitivity measures, of the pixels that do actually contain a
target object, the proportion of which that were actually picked up by the
model. Specificity measures, of the pixels that do not contain a target
object, the proportion that was appropriately ignored by the model.

In addition, imagery is provided in Figures 3, 4, 5, 6, and 7 demonstrating
the results. Of the column titles: “Input” refers to the input passed to
Jekyll, Hyde, and Utterson; “Hyde”, “Jekyll”, and “Utterson” each show the
respective outputs of those models; “Subtr” shows the result of subtracting
Hyde’s pseudo-background from the input; and “Label” shows the ground truths,
the locations of targets of interest that were synthetically generated
alongside the data. The samples shown come from the test subset of the data,
and show frames 1, 5, 9, and 13 of a 16 frame sequence, with frame 1 of each
sequence on the top of the figure, with the time increasing in each subsequent
row.

It can clearly be seen in the graphical results that the unsupervised approach
demonstrated by Jekyll and Hyde is competitive with the supervised approach
represented by Utterson, within this particular problem domain on this data-
set. This is supported by the similarity of the Sensitivity values between
Jekyll and Utterson for thresholds below $0.4$.

Figure 3: Below is a figure showing each element of the Jekyll and Hyde
formulation: an input series or “video” of “images” containing small moving
objects, a static pseudo-background generated by one unsupervised model
“Hyde”, a sequence containing the estimated objects detected by another
unsupervised model “Jekyll,” Hyde’s pseudo-background subtracted from the
respective input frame, the results of a supervised model “Utterson” for
comparison, and the true labels/locations of the objects of interest. The
first, second, third, and fourth rows relate to the $1^{st},5^{th},9^{th}$,
and $13^{th}$ frames of a 16-frame input sequence respectively. In this
instance, there are stationary features in this input sequence that closely
resemble the target objects. The target tracking models demonstrate both their
accuracy and value by ignoring these. Figure 4: Here, the target tracking
models manage to pick up a small, dark, very slowly moving object on a dark
background near other dark objects. By looking closely at the locations marked
out by the models, one can just barely make out the target, which would be
otherwise unnoticed. Figure 5: This example shows the models’ ability to track
a large number of objects at the same time, while also being able to ignore
parts of the background that superficially resemble the objects being tracked.
Figure 6: The models are capable not only of accurately tracking moving
objects, but also of accurately producing a negative result. Figure 7: An
example of the limitations of the models being used; when two tracked objects
come close to one other, the model will track them both as a single misshapen
object or “blob”, rather than as discrete entities. This is the result of both
Jekyll and Utterson giving “slack” to target objects, and masking them as
larger than they are, as a precaution against costly failure to mask
something.

##  6 CONCLUSIONS AND FUTURE WORK

While this formulation of target tracking as an unsupervised learning problem
is potentially extremely powerful, this particular approach is limited in
scope. On its own, it is not robust to transient sensor artifacts, to
background motion, or to weather. It also does not perform any sort of
discrimination between targets worth tracking and those not worth tracking,
classification of different kinds of targets, or prediction of trajectories.

However, this work does present an important step towards the development of a
completely unsupervised pipeline, by allowing for the transformation of
imagery into target locations. Artifact removal [32, 33, 34, 35], image
stabilization [36, 37, 38], and saliency estimation [39, 40, 41, 42, 43] are
all active research areas making great strides, and the coalescing of these
under an unsupervised or semi-supervised Machine Learning paradigm would allow
for an extremely robust, accurate, scalable, and inexpensive solution for a
variety of challenges in satellite imagery analysis.

“A little song, a little dance, a little seltzer down your pants.” -Tim

## References

  * [1] Blackman, S. S., [Multiple-target tracking with radar applications ] (1986).
  * [2] Fox, P. J., Liu, J., and Weiner, N., “Integrating out astrophysical uncertainties,” Phys. Rev. D 83, 103514 (May 2011).
  * [3] Kopsiaftis, G. and Karantzalos, K., “Vehicle detection and traffic density monitoring from very high resolution satellite video data,” in [2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) ], 1881–1884 (2015).
  * [4] Rashtchian, C., Young, P., Hodosh, M., and Hockenmaier, J., “Collecting image annotations using amazon’s mechanical turk,” in [Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk ], 139–147 (2010).
  * [5] Hastie, T., Tibshirani, R., and Friedman, J., “Unsupervised learning,” in [The elements of statistical learning ], 485–585, Springer (2009).
  * [6] Stevenson, R. L., Grennell, A., Somerset, R., Young, D., O’Connell, P., and Casson, G., [The Strange Case of Dr. Jekyll and Mr. Hyde ], Didier (1947).
  * [7] Young, S. R., Steward, B. J., and Gross, K. C., “Development and validation of the AFIT scene and sensor emulator for testing (ASSET),” 101780A (May 2017).
  * [8] AFIT, “AFIT Sensor And Scene Emulation Tool,” (2020).  publisher: Air Force Institute of Technology.
  * [9] Zill, D., Wright, W. S., and Cullen, M. R., [Advanced engineering mathematics ], Jones & Bartlett Learning (2011).
  * [10] Iliadis, M., Spinoulas, L., and Katsaggelos, A. K., “Deepbinarymask: Learning a binary mask for video compressive sensing,” arXiv preprint arXiv:1607.03343 (2016).
  * [11] Goodfellow, I., Bengio, Y., and Courville, A., [Deep Learning ], MIT Press (2016).
  * [12] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y., “Generative Adversarial Networks,” arXiv:1406.2661 [cs, stat] (June 2014).  arXiv: 1406.2661.
  * [13] Kingma, D. P. and Welling, M., “Auto-Encoding Variational Bayes,” arXiv:1312.6114 [cs, stat] (May 2014).  arXiv: 1312.6114.
  * [14] Xia, X. and Kulis, B., “W-Net: A Deep Model for Fully Unsupervised Image Segmentation,” (Nov. 2017).
  * [15] Shi, J. and Malik, J., “Normalized cuts and image segmentation,” IEEE Transactions on pattern analysis and machine intelligence 22(8), 888–905 (2000).
  * [16] Wang, N., Song, Y., Ma, C., Zhou, W., Liu, W., and Li, H., “Unsupervised Deep Tracking,” (Apr. 2019).
  * [17] Ghasemi, A. and Safabakhsh, R., “Unsupervised foreground-background segmentation using growing self organizing map in noisy backgrounds,” in [2011 3rd International Conference on Computer Research and Development ], 1, 334–338 (Mar. 2011).
  * [18] Nair, V. and Clark, J., “An unsupervised, online learning framework for moving object detection,” in [Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004\. CVPR 2004. ], 2, 317–324, IEEE, Washington, DC, USA (2004).
  * [19] Littlestone, N., “Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm,” Machine learning 2(4), 285–318 (1988).
  * [20] Zhang, W., Itoh, K., Tanida, J., and Ichioka, Y., “Parallel distributed processing model with local space-invariant interconnections and its optical architecture,” Applied optics 29(32), 4790–4797 (1990).
  * [21] LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D., “Backpropagation applied to handwritten zip code recognition,” Neural computation 1(4), 541–551 (1989).
  * [22] Ji, S., Xu, W., Yang, M., and Yu, K., “3d convolutional neural networks for human action recognition,” IEEE transactions on pattern analysis and machine intelligence 35(1), 221–231 (2012).
  * [23] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al., “Pytorch: An imperative style, high-performance deep learning library,” in [Advances in neural information processing systems ], 8026–8037 (2019).
  * [24] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A., “Automatic differentiation in pytorch,” (2017).
  * [25] Srivastava, R. K., Greff, K., and Schmidhuber, J., “Highway networks,” CoRR abs/1505.00387 (2015).
  * [26] Glorot, X., Bordes, A., and Bengio, Y., “Deep sparse rectifier neural networks,” in [Proceedings of the fourteenth international conference on artificial intelligence and statistics ], 315–323 (2011).
  * [27] Ronneberger, O., Fischer, P., and Brox, T., “U-net: Convolutional networks for biomedical image segmentation,” in [International Conference on Medical image computing and computer-assisted intervention ], 234–241, Springer (2015).
  * [28] Long, J., Shelhamer, E., and Darrell, T., “Fully convolutional networks for semantic segmentation,” in [2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) ], 3431–3440, IEEE, Boston, MA, USA (June 2015).
  * [29] Mao, X.-J., Shen, C., and Yang, Y.-B., “Image restoration using convolutional auto-encoders with symmetric skip connections,” arXiv preprint arXiv:1606.08921 (2016).
  * [30] Quan, T. M., Hildebrand, D. G., and Jeong, W.-K., “Fusionnet: A deep fully residual convolutional neural network for image segmentation in connectomics,” arXiv preprint arXiv:1612.05360 (2016).
  * [31] Kingma, D. P. and Ba, J., “Adam: A method for stochastic optimization,” (2014).  cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
  * [32] Winkler, I., Haufe, S., and Tangermann, M., “Automatic classification of artifactual ica-components for artifact removal in eeg signals,” Behavioral and brain functions 7(1), 30 (2011).
  * [33] Allman, D., Reiter, A., and Bell, M. A. L., “A machine learning method to identify and remove reflection artifacts in photoacoustic channel data,” in [2017 IEEE International Ultrasonics Symposium (IUS) ], 1–4, IEEE (2017).
  * [34] Biswas, R., Blackburn, L., Cao, J., Essick, R., Hodge, K. A., Katsavounidis, E., Kim, K., Kim, Y.-M., Le Bigot, E.-O., Lee, C.-H., et al., “Application of machine learning algorithms to the study of noise artifacts in gravitational-wave data,” Physical Review D 88(6), 062003 (2013).
  * [35] Zhang, Y., Haghdan, M., and Xu, K. S., “Unsupervised motion artifact detection in wrist-measured electrodermal activity data,” in [Proceedings of the 2017 ACM International Symposium on Wearable Computers ], 54–57 (2017).
  * [36] Walha, A., Wali, A., and Alimi, A. M., “Video stabilization for aerial video surveillance,” AASRI Procedia 4, 72–77 (2013).
  * [37] Saitwal, K. A., Cobb, W. K., and Yang, T., “Image stabilization techniques for video surveillance systems,” (Jan. 5 2016).  US Patent 9,232,140.
  * [38] Deng, H.-B., Jia, Y.-D., Xu, Y.-H., and Liang, W., “An airborne image stabilization method based on projection and the gaussian mixture model,” in [2007 International Conference on Machine Learning and Cybernetics ], 1, 345–349, IEEE (2007).
  * [39] Li, J., Tian, Y., Huang, T., and Gao, W., “Probabilistic multi-task learning for visual saliency estimation in video,” International journal of computer vision 90(2), 150–165 (2010).
  * [40] Tang, Y. and Wu, X., “Saliency detection via combining region-level and pixel-level predictions with cnns,” in [European Conference on Computer Vision ], 809–825, Springer (2016).
  * [41] Xia, C., Qi, F., and Shi, G., “Bottom–up visual saliency estimation with deep autoencoder-based sparse reconstruction,” IEEE transactions on neural networks and learning systems 27(6), 1227–1240 (2016).
  * [42] Hu, Y.-T., Huang, J.-B., and Schwing, A. G., “Unsupervised video object segmentation using motion saliency-guided spatio-temporal propagation,” in [Proceedings of the European conference on computer vision (ECCV) ], 786–802 (2018).
  * [43] Aytekin, C., Iosifidis, A., and Gabbouj, M., “Probabilistic saliency estimation,” Pattern Recognition 74, 359–372 (2018).