SED-ML Validator: tool for debugging simulation experiments

Bilal Shaikh  $+$ 1,∗, Andrew Philip Freiburger  $+$ 2,∗, Matthias König  $+$
3, Frank T. Bergmann  $+$ 4,5,
David P. Nickerson  $+$ 6, Herbert M. Sauro  $+$ 7, Michael L. Blinov  $+$ 8,
Lucian P. Smith  $+$ 7,
Ion I. Moraru  $+$ 8 and Jonathan R. Karr  $+$ 1,†

1Department of Genetics and Genomic Sciences, Icahn School of Medicine at
Mount Sinai, New York, NY 10029, USA, 2Department of Civil Engineering,
University of Victoria, Victoria, BC V8P 5C2, Canada, 3Department of
Theoretical Biology, Humboldt University, 10115 Berlin,
Germany, 4BioQUANT/COS, Heidelberg University, 69120 Heidelberg, Germany,
5Department of
Computing and Mathematical Sciences, California Institute of Technology,
Pasadena 91125, CA,
USA, 6Auckland Bioengineering Institute, University of Auckland, Auckland
1010, New Zealand,
7Department ofBioengineering, University of Washington, Seattle, WA 98105, USA
and 8Center
for Cell Analysis & Modeling, University of Connecticut School of Medicine,
Farmington, CT
06030, USA

∗These authors contributed equally to this work.
†To whom correspondence should be addressed<EMAIL_ADDRESS>
## Abstract

Summary: More sophisticated models are needed to address problems in
bioscience, synthetic biology, and precision medicine. To help facilitate the
collaboration needed for such models, the community developed the Simulation
Experiment Description Markup Language (SED-ML), a common format for
describing simulations. However, the utility of SED-ML has been hampered by
limited support for SED-ML among modeling software tools and by different
interpretations of SED-ML among the tools that support the format. To help
modelers debug their simulations and to push the community to use SED-ML
consistently, we developed a tool for validating SED-ML files. We have used
the validator to correct the official SED-ML example files. We plan to use the
validator to correct the files in the BioModels database so that they can be
simulated. We anticipate that the validator will be a valuable tool for
developing more predictive simulations and that the validator will help
increase the adoption and interoperability of SED-ML.

Availability: The validator is freely available as a webform, HTTP API,
command-line program, and Python package at
https://run.biosimulations.org/utils/validate and
https://pypi.org/project/biosimulators-utils. The validator is also embedded
into interfaces to 11 simulation tools. The source code is openly available as
described in the Supplementary data.

Contact<EMAIL_ADDRESS>
##  1 Introduction

Expanded capabilities to predict biological behavior are needed to help
engineers design synthetic biological systems and help physicians precisely
diagnose and treat disease (Carrera and Covert,, 2015; Marucci et al.,, 2020).
Achieving more predictive models will likely require deep collaboration among
large teams of modelers, experimentalists, and clinicians (Szigeti et al.,,
2018; Singla and White,, 2021; Waltemath et al.,, 2011).

To facilitate collaboration, the Computational Modeling in Biology Network
(COMBINE; Hucka et al.,, 2015) developed the Simulation Experiment Description
Markup Language (SED-ML; Waltemath et al.,, 2011), a common format for
describing simulations. At its core, SED-ML describes individual simulations
of individual models. Initially, SED-ML focused on continuous kinetic models
described with CellML (Cuellar et al.,, 2003) and the Systems Biology Markup
Language (SBML; Keating et al.,, 2020). Recently, we have expanded SED-ML to a
broader range of models including spatial, flux balance, qualitative, and
rule-based models; a broader range of simulation algorithms such as flux
balance analysis (FBA) and asynchronous logical simulation; and additional
model languages such as the BioNetGen Language (Faeder et al.,, 2009), the
SBML Flux Balance Constraints (Olivier and Bergmann,, 2015) and Qualitative
Models (Chaouiya et al.,, 2013) packages, and Smoldyn (Shaikh et al.,, 2021).

On top of this core functionality, SED-ML can describe sets of simulations of
variants of models, such as an ensemble of stochastic simulations or a
parameter scan of a model. In addition, SED-ML can describe how to create
tables and figures of simulation results.

Several tools can create SED-ML files, including web applications such as JWS
Online (Peters et al.,, 2017), RunBioSimulations, and SED-ML Web Tools
(Bergmann et al., 2017b, ). SED-ML files can be executed with several tools
such as COPASI (Bergmann et al., 2017a, ), iBioSim (Watanabe et al.,, 2018),
OpenCOR (Garny and Hunter,, 2015), Tellurium (Choi et al.,, 2018), and Virtual
Cell (Moraru et al.,, 2008). Furthermore, SED-ML files can be published with
repositories such as BioModels (Malik-Sheriff et al.,, 2020), JWS Online
(Peters et al.,, 2017), and Physiome (Sarwar et al.,, 2019). More information
about these and other SED-ML tools is available at https://sed-ml.org.

However, the utility of SED-ML has been hampered by limited support for SED-ML
among modeling software tools and by different interpretations of SED-ML among
these tools. For example, we have found that Tellurium can only execute a few
of the simulations in BioModels.

To help modelers debug their simulations and to push the community to use SED-
ML consistently, we developed a tool that thoroughly validates SED-ML files.
The tool is available as a webform, HTTP API, command-line program, and Python
API. Here, we articulate how modelers can use the validator, summarize the
validation rules the validator checks, describe how the validator communicates
issues about SED-ML files, and highlight how we have already used the
validator to correct the official SED-ML examples, identify and fill gaps in
the SED-ML specifications, and identify bugs in the implementation of SED-ML
by multiple software tools. We also outline how we plan to use the validator
to correct the SED-ML files in BioModels. The Supplementary data provides more
information about the validator and how we are using it to drive convergence
around SED-ML.

##  2 Methods

Because SED-ML files typically describe simulations of external model files
encoded in languages such as CellML, NeuroML (Cannon et al.,, 2014), and SBML,
we designed the validator to validate COMBINE archives (Bergmann et al.,,
2014). COMBINE archives are zip archives that contain one or more SED-ML
files, other files that the SED-ML files reference, an OMEX manifest file that
summarizes the contents of the archive, and optionally OMEX metadata files
(Neal et al.,, 2020) that capture metadata about the archive and its contents.
COMBINE archives can be created with several tools such as CombineArchiveWeb
and RunBioSimulations. More information about these tools is available at
https://sed-ml.org.

Figure 1: The SED-ML validator helps investigators quickly detect errors and
other potential problems in SED-ML and model files organized into COMBINE
archives.

After creating a COMBINE archive with a SED-ML file, modelers can use the
validator via its webform, HTTP API, command-line program, or Python API.
Supplementary data S3 and S4 outline how to install and use the validator. To
make the validator easy to access, we have also embedded it into standardized
interfaces to 11 popular simulation tools (Supplementary data S5.1).

The validator thoroughly checks that COMBINE archives and their contents are
consistent with the COMBINE archive, OMEX manifest, OMEX metadata, and SED-ML
formats, as well as with the languages of the models in the archive. For
example, the validator checks that each reference to a SED-ML element matches
the id of an element, that the network of model sources is acyclic, and that
each XPath target for each variable of an XML-encoded model matches a single
model element. The validator uses LibCellML (https://libcellml.org),
LibNeuroML (Vella et al.,, 2014), and LibSBML (Bornstein et al.,, 2008) to
check that CellML, NeuroML, and SBML models involved in SED-ML files are
valid. Supplementary data S2 outlines all of the validation rules that the
validator evaluates.

When COMBINE archives are invalid, the validator reports as many errors as can
be identified simultaneously, each with contextual information about the
element responsible for the error. For example, when the target of a variable
of a data generator in a SED-ML file does not match an element of the
associated model, the validator provides information about the invalid target
and the model that it should match. Supplementary data S4.7 illustrates
several example error messages.

The validator also reports warnings about potential mistakes, such as the use
of experimental features of SED-ML that few simulation tools support. We
implemented warnings for simulations based on common mistakes that we have
observed in SED-ML files. We implemented warnings for models using libraries
for model languages such as LibSBML.

##  3 Real-world examples

As a first real-world test, we used the validator to identify and fix issues
with the official SED-ML examples (Supplementary data S5.3). The validator
alerted us to two common problems with each example, as well as less common
issues with several files. The validator also identified files that use a
combination of SED-ML elements that the SED-ML specifications do not
officially support. This finding prompted us to add a warning for this
combination of elements and clarify the description of these examples on the
SED-ML website. To ensure these examples remain valid, we also set up an
automated action that uses our validator to check these files each time they
are changed.

Encouraged by this success, we plan to use our validator to identify and fix
issues with the SED-ML files in BioModels (Supplementary data S5.4). We
anticipate these corrections will enable these files to be simulated with
multiple tools, which will increase the utility of the models in BioModels.

To avoid similar errors in the future, we have also submitted several
proposals to clarify the specifications of SED-ML (Supplementary data S5.5)
and filed numerous bug reports for several software tools that support SED-ML
(Supplementary data S5.6). In addition, we aim to help the BioModels Team
incorporate our validator into their curation workflow to ensure that
BioModels publishes valid SED-ML files going forward.

##  4 Discussion

We believe that our validator will be a key resource for debugging simulation
experiments and that this work will push the community to use SED-ML
consistently. Taken together, we believe these advancements will increase the
community’s ability to collaborate on simulation experiments, which we
anticipate will foster more sophisticated models.

Once the next version of SED-ML (L1V4) is approved, we plan to expand the
validator to SED-ML’s new features for additional types of observables and
plots, data reductions, and model calibration. We also aim to expand the
capabilities of the validator to validate additional types of files that could
be included in COMBINE archives, such as PETab and Vega files, two emerging
formats for model calibration and data visualization.

## Funding

This work was supported by the National Institutes of Health [grant number
P41EB023912], BMBF LiSyM [grant number 031L0054], and DFG QuaLiPerF [grant
number 436883643].

Conflict of Interest: none declared.

## References

  * Bergmann et al., (2014) Bergmann,F.T. et al. (2014) COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project.  BMC Bioinformatics,  15 (1), 1–9.
  * (2) Bergmann,F.T. et al. (2017a) COPASI and its applications in biotechnology.  J. Biotechnol.,  261, 215–220.
  * (3) Bergmann,F.T. et al. (2017b) SED-ML Web Tools: generate, modify and export standard-compliant simulation studies.  Bioinformatics,  33 (8), 1253–1254.
  * Bornstein et al., (2008) Bornstein,B.J. et al. (2008) LibSBML: an API library for SBML.  Bioinformatics,  24 (6), 880–881.
  * Cannon et al., (2014) Cannon,R.C. et al. (2014) LEMS: a language for expressing complex biological models in concise and hierarchical form and its use in underpinning NeuroML 2.  Front. Neuroinform.,  8, 79.
  * Carrera and Covert, (2015) Carrera,J. and Covert,M.W. (2015) Why build whole-cell models?  Trends Cell. Biol.,  25 (12), 719–722.
  * Chaouiya et al., (2013) Chaouiya,C. et al. (2013) SBML qualitative models: a model representation format and infrastructure to foster interactions between qualitative modelling formalisms and tools.  BMC Syst. Biol.,  7 (1), 1–15.
  * Choi et al., (2018) Choi,K. et al. (2018) Tellurium: an extensible Python-based modeling environment for systems and synthetic biology.  Biosystems,  171, 74–79.
  * Cuellar et al., (2003) Cuellar,A.A. et al. (2003) An overview of CellML 1.1, a biological model description language.  Simulation,  79 (12), 740–747.
  * Faeder et al., (2009) Faeder,J.R., Blinov,M.L. and Hlavacek,W.S. (2009) Rule-based modeling of biochemical systems with BioNetGen.  Methods Mol Biol,  500, 113–167.
  * Garny and Hunter, (2015) Garny,A. and Hunter,P.J. (2015) OpenCOR: a modular and interoperable approach to computational biology.  Front. Physiol.,  6, 26.
  * Hucka et al., (2015) Hucka,M. et al. (2015) Promoting coordinated development of community-based information standards for modeling in biology: the COMBINE initiative.  Front. Bioeng. Biotechnol.,  3, 19.
  * Keating et al., (2020) Keating,S.M. et al. (2020) SBML Level 3: an extensible format for the exchange and reuse of biological models.  Mol. Syst. Biol.,  16 (8), e9110.
  * Malik-Sheriff et al., (2020) Malik-Sheriff,R.S. et al. (2020) BioModels–15 years of sharing computational models in life science.  Nucleic Acids Res.,  48 (D1), D407–D415.
  * Marucci et al., (2020) Marucci,L., Barberis,M., Karr,J., Ray,O., Race,P.R., de Souza Andrade,M., Grierson,C., Hoffmann,S.A., Landon,S., Rech,E. et al. (2020) Computer-aided whole-cell design: taking a holistic approach by integrating synthetic with systems biology.  Front. Bioeng. Biotechnol.,  8, 942.
  * Moraru et al., (2008) Moraru,I.I. et al. (2008) Virtual Cell modelling and simulation software environment.  IET Syst. Biol.,  2 (5), 352–362.
  * Neal et al., (2020) Neal,M.L. et al. (2020) Open modeling and exchange (OMEX) metadata specification version 1.0.  J. Integr. Bioinform.,  17 (2-3).
  * Olivier and Bergmann, (2015) Olivier,B.G. and Bergmann,F.T. (2015) The Systems Biology Markup Language (SBML) Level 3 package: Flux balance Constraints.  J. Integr. Bioinform.,  12 (2), 660–690.
  * Peters et al., (2017) Peters,M. et al. (2017) The JWS Online simulation database.  Bioinformatics,  33 (10), 1589–1590.
  * Sarwar et al., (2019) Sarwar,D.M. et al. (2019) Model annotation and discovery with the Physiome Model Repository.  BMC Bioinformatics,  20 (1), 1–10.
  * Shaikh et al., (2021) Shaikh,B. et al. (2021) RunBioSimulations: an extensible web application that simulates a wide range of computational modeling frameworks, algorithms, and formats.  Nucleic Acids Res.,  49 (W1).
  * Singla and White, (2021) Singla,J. and White,K.L. (2021) A community approach to whole-cell modeling.  Curr. Opin. Syst. Biol.,  26, 33–38.
  * Szigeti et al., (2018) Szigeti,B., Roth,Y.D., Sekar,J.A., Goldberg,A.P., Pochiraju,S.C. and Karr,J.R. (2018) A blueprint for human whole-cell modeling.  Curr. OpiN. Syst. Biol.,  7, 8–15.
  * Vella et al., (2014) Vella,M. et al. (2014) libNeuroML and PyLEMS: using Python to combine procedural and declarative modeling approaches in computational neuroscience.  Front. Neuroinform.,  8, 38.
  * Waltemath et al., (2011) Waltemath,D. et al. (2011) Reproducible computational biology experiments with SED-ML-the Simulation Experiment Description Markup Language.  BMC Syst. Biol.,  5 (1), 1–10.
  * Watanabe et al., (2018) Watanabe,L. et al. (2018) iBioSim 3: a tool for model-based genetic circuit design.  ACS Syn. Biol.,  8 (7), 1560–1563.


Amirsaeed<EMAIL_ADDRESS>Xuelu<EMAIL_ADDRESS>Vishal
<EMAIL_ADDRESS>Department of Electrical Engineering
The Pennsylvania State University
University Park, USA AL with Hierarchically-Adaptive Sample Assessment

# Maturity-Aware Active Learning for Semantic Segmentation with
Hierarchically-Adaptive Sample Assessment

###### Abstract

Active Learning (AL) for semantic segmentation is challenging due to heavy
class imbalance and different ways of defining “sample” (pixels, areas, etc.),
leaving the interpretation of the data distribution ambiguous. We propose
“Maturity-Aware Distribution Breakdown-based Active Learning” (MADBAL), an AL
method that benefits from a hierarchical approach to define a multiview data
distribution, which takes into account the different "sample" definitions
jointly, hence able to select the most impactful segmentation pixels with
comprehensive understanding. MADBAL also features a novel uncertainty
formulation, where AL supporting modules are included to sense the features’
maturity whose weighted influence continuously contributes to the uncertainty
detection. In this way, MADBAL makes significant performance leaps even in the
early AL stage, hence reducing the training burden significantly. It
outperforms state-of-the-art methods on Cityscapes and PASCAL VOC datasets as
verified in our extensive experiments.

##  1 Introduction

Neural networks in the past decade have been dominant solutions for a wide
majority of computer vision problems [El Helou et al.(2021)El Helou, Zhou,
Susstrunk, and Timofte, Yazdani et al.(2021)Yazdani, Agrawal, Johnstonbaugh,
Kothapalli, and Monga, Lu and Weng(2007), Minaee et al.(2021)Minaee, Boykov,
Porikli, Plaza, Kehtarnavaz, and Terzopoulos, Li and Monga(2020)]; however,
these solutions often suffer from being data-eccentric, which means a burden
in both the data collection and annotation. While this burden exists for
almost every computer vision task, it becomes more costly and laborious for
tasks that need fine-grained annotations, such as image segmentation.

Active learning (AL) methods have been proposed to overcome this bottleneck by
incrementally selecting the samples for improving the performance of the
current model, which has been trained on a limited training set. AL offers
some criteria based on which the samples in an unlabeled pool are assessed,
ranked, selected, and then added to the current training pool. The majority of
existing AL methods [Ash et al.(2019)Ash, Zhang, Krishnamurthy, Langford, and
Agarwal, Aghdam et al.(2019)Aghdam, Gonzalez-Garcia, Weijer, and López, Gal et
al.(2017)Gal, Islam, and Ghahramani, Nguyen and Smeulders(2004)] rely on a
combination of the criteria such as model uncertainty and the diversity in the
labeled pool. Although these criteria seem intuitive and are well-defined for
tasks such as classification, for segmentation their definition becomes
ambiguous, hence challenging to quantify. This is due to the fact that in
segmentation, “sample” does not have a concrete definition based on which the
formulation can uniquely rank the samples like classification. In
segmentation, a “sample” can be perceived either as a pixel, a patch, or an
image. Moreover, the informativeness of the samples for a specific definition
is correlated to the information of surrounding samples. This leaves the
existing segmentation AL methods [Cai et al.(2021)Cai, Xu, Liew, and Foo,
Casanova et al.(2020)Casanova, Pinheiro, Rostamzadeh, and Pal, Shin et
al.(2021)Shin, Xie, and Albanie], focusing on only one interpretation of
samples, with a narrow insight into the data distribution.

In this work, we propose a systematic and inclusive AL strategy as a natural
evolution of existing works with different sample considerations, tied with
backbone-agnostic, AL supporting network components. Specifically, our key
contributions are:

  * •

Distribution Breakdown: We propose a hierarchical approach to estimate the
data distribution based on different definitions of “sample,” which allows for
a multilevel assessment of the data. We traverse this hierarchy level by
level, while at each level breaking down the distribution of the data
according to the corresponding “sample” definition (see Fig. 2). This means
that the representativeness of the selected data is checked across multiple
views making the training set as insightful as possible.

  * •

Maturity-Awareness: We propose a set of backbone-agnostic, AL supporting
modules associated with carefully devised uncertainty terms which together are
capable of detecting the most impactful samples for network performance
improvement. AL supporting modules help monitor the flow of information
through different layers with different features’ maturity level (see Fig. 1).
This flow is interpreted via our proposed uncertainty formulation which
evaluates the model maturity for different samples.

  * •

Integration of the aforementioned algorithmic pieces results in a model
referred to as “Maturity-Aware Distribution Breakdown-based Active Learning”
(MADBAL). We evaluate the performance of our model on Cityscapes [Cordts et
al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson, Franke, Roth, and
Schiele] and Pascal VOC 2012 [Everingham et al.(2015)Everingham, Eslami, Van
Gool, Williams, Winn, and Zisserman] datasets and prove that not only does
MADBAL outperform state of the art w.r.t different metrics, but also exhibits
immediate performance leaps unlike state of the art where the improvements are
more gradual. This makes MADBAL a preferred AL solution for reducing the
training burden overhead from two standpoints: 1) Lower number of AL steps
needed for achieving acceptable results. 2) No requirement for a rich,
carefully-selected initial labeled pool.

##  2 Related Works

Selection criteria in the AL literature consist mainly of two types of nature:
uncertainty and diversity. Uncertainty-based criteria [Gal et al.(2017)Gal,
Islam, and Ghahramani] focus on how certain the model is in its prediction for
a possible candidate and select ones with more uncertainty. They can be
mathematically formulated in a variety of ways such as posterior probability
of the predicted class [Lewis(1995)] or the margin between the posterior
probabilities of the predicted class and the the class that received the
second highest predicted probability [Joshi et al.(2009)Joshi, Porikli, and
Papanikolopoulos]. This complements diversity-based criteria [Nguyen and
Smeulders(2004)] with the main objective to help the training set maintain a
representation as close as possible to the whole distribution of the data.
This would lead to detection and addition of the samples so that the distance
between the training set and the unlabeled pool is minimized (Core-set AL)
[Sener and Savarese(2017)], or the most representative subset of the unlabeled
pool is constructed [Elhamifar et al.(2013)Elhamifar, Sapiro, Yang, and
Sasrty]. AL methods consider one or a hybrid combination of these two criteria
for their selection. For example, BALD [Houlsby et al.(2011)Houlsby, Huszár,
Ghahramani, and Lengyel] uses a Bayesian framework to select samples based on
the uncertainty of sampled networks. Later, BatchBALD [Kirsch et
al.(2019)Kirsch, van Amersfoort, and Gal] was proposed as a modification of
BALD to take the diversity into consideration. Besides these criteria,
expected model change is used in a few works [Settles et al.(2007)Settles,
Craven, and Ray, Roy and McCallum(2001), Konyushkova et al.(2017)Konyushkova,
Sznitman, and Fua, Freytag et al.(2014)Freytag, Rodner, and Denzler] as a
criterion to select samples that cause the greatest change in the current
model or its output. For example, Freytag _et al._ [Freytag et
al.(2014)Freytag, Rodner, and Denzler] use the current model to predict the
output changes, while Settles _et al_. [Settles et al.(2007)Settles, Craven,
and Ray] rely on the predicted gradient length to select samples.

AL methods for semantic segmentation are categorized into image-based and
region-based methods: Image-based methods [Yang et al.(2017)Yang, Zhang, Chen,
Zhang, and Chen, Sinha et al.(2019)Sinha, Ebrahimi, and Darrell, Xie et
al.(2020)Xie, Feng, Chen, Sun, Ma, and Song, Gorriz et al.(2017)Gorriz,
Carlier, Faure, and Giro-i Nieto] are often faster with lower computational
complexity owing to their definition of “sample,” which gives them smaller
search space at the expense of adding redundant classes at every AL step. This
in turn leads to less budget-efficiency of these approaches. As an image-based
method for medical image segmentation, Yang _et al_. [Yang et al.(2017)Yang,
Zhang, Chen, Zhang, and Chen] propose a CNN architecture and a heuristic
method to find the most representative samples among top k with highest
uncertainty. Within the same domain, [Gorriz et al.(2017)Gorriz, Carlier,
Faure, and Giro-i Nieto] leverages drop-out to represent the Monte Carlo
sampling at test time for melanoma segmentation. [Sinha et al.(2019)Sinha,
Ebrahimi, and Darrell] leverages the min-max game between the adversarial
network and the variational autoencoder (VAE) to discriminate between
challenging and easy samples based on the features in the latent space.
Inspired by the work of Yoo _et al_. [Yoo and Kweon(2019)] for dedicating
network components for loss prediction, Xie _et al_. [Xie et al.(2020)Xie,
Feng, Chen, Sun, Ma, and Song] develop a difficulty-aware network to generate
difficulty heatmaps using the missclassified/correctly classified pixels in
the labeled pool. Region-based methods [Shin et al.(2021)Shin, Xie, and
Albanie, Cai et al.(2021)Cai, Xu, Liew, and Foo, Siddiqui et
al.(2020)Siddiqui, Valentin, and Nießner, Qiao et al.(2022)Qiao, Zhu, Long,
Zhang, Wang, Du, and Yang, Mackowiak et al.(2018)Mackowiak, Lenz, Ghori,
Diego, Lange, and Rother], unlike image-based methods, show higher performance
with significantly lower budget as they are able to select only the regions
with the most helpful classes for annotation, hence no need for annotating
useless regions. This has led to emergence of more region-based methods
recently. CEREALS [Mackowiak et al.(2018)Mackowiak, Lenz, Ghori, Diego, Lange,
and Rother] estimates the cost of annotating regions and finds a trade-off
between the informativeness and annotation cost of the candidates. Golestaneh
_et al_. [Golestaneh and Kitani(2020)] utilize the fact that the most
uncertain regions show high uncertainty under equivariant transformations.
Recently, Cai _et al_. [Cai et al.(2021)Cai, Xu, Liew, and Foo] was one of the
pioneers in estimating the data distribution by using the trained model at the
current step to find the dominant labels across superpixels and select the
most uncertain superpixels whose dominant labels belong to less frequent
classes, which inspires our uncertainty formulation at the superpixel level;
nevertheless we extend [Cai et al.(2021)Cai, Xu, Liew, and Foo] by introducing
AL supporting modules at other levels. Focusing on pixels, PixelPick [Shin et
al.(2021)Shin, Xie, and Albanie] in each round of AL selects an equal number
of pixels with highest uncertainty from each image. Recently, [Qiao et
al.(2022)Qiao, Zhu, Long, Zhang, Wang, Du, and Yang] deploys a regional
Gaussian attention module to select regions and leverages contextual guidance
to extend the regional annotations to unlabeled regions, while borrowing the
idea of the loss prediction module from [Xie et al.(2020)Xie, Feng, Chen, Sun,
Ma, and Song]. The proved benefits of loss prediction module in [Yoo and
Kweon(2019), Xie et al.(2020)Xie, Feng, Chen, Sun, Ma, and Song, Qiao et
al.(2022)Qiao, Zhu, Long, Zhang, Wang, Du, and Yang] motivates us in including
it in our AL supporting modules; however, as it will be elaborated, ours
benefits from a more effective training protocol (separate training phases),
allocating boundary-aware output channels, and more effective ground truth
formulation.

Figure 1: The proposed backbone-agnostic architecture. We use pixels m and n
as examples of easy and challenging samples to understand sample assessment
mechanism. For the easy sample, the predicted probability distributions
($\hat{Y}_{m}^{i}$) show high similarity to the final distribution
($\hat{Y}_{m}$) (measured by Jenson-Shannon divergence). For the challenging
sample, the predicted probabilities ($\hat{Y}_{n}^{i}$) show a confusing trend
at shallow stages, taking longer to show a consistent trend and high
similarity to the final distribution ($\hat{Y}_{n}$). For the details of the
modules and their training schemes refer to 3.1.

##  3 Methods

Our method includes two main components: 1) AL supporting modules, components
of which reflect the information flow needed for maturity-awareness, and 2)
selection strategy, which reflects our hierarchical distribution breakdown
scheme integrated with our custom uncertainty formulation.

###  3.1 AL Supporting Modules

Our network consists of conventional modules essential for carrying out the
main segmentation task (Mainstream Segmentation modules – see Fig. 1) which
makes it backbone-agnostic. These modules are trained through a preliminary
phase of training (training phase I) with cross-entropy (CE) loss. Once these
modules are trained, the training of AL supporting modules, whose purpose is
critical at the time of sample selection, starts based on their designated
goal.
Varied-Maturity Heads besides the main segmentation head (Fully-mature head –
see Fig. 1), include three heads with access to different depths of the
backbone layers. Indeed, as the depth increases, the maturity of features
provided to these heads increases. Starting from the shallowest, we denote
them with Shallow-mature Head, Semi-mature Head, and Almost-mature Head. Each
head in training phase II is trained for the segmentation task and assigned a
loss term
($\underset{k\in\\{1,2,3\\}}{\operatorname{\mathcal{L}^{k}_{\textit{seg}}}}$)
which is CE loss defined on segmentation outputs.
Loss Prediction Module is in charge of predicting the probability of each
pixel’s error-proneness for the segmentation task. This module makes use of a
different version of the feature maps provided to the varied-maturity heads
and the endpoint features of the backbone (see Fig. 1). These features are
weighted by the weight maps provided by the Weight Prediction Block to help
the model reweight the features based on their importance. We define the
ground truth for the loss prediction task by considering the class-specific
average loss across the labeled pool as a threshold for determining the loss
labels:

$Y^{i}_{L}=\begin{cases}1,&\text{if }L_{\textit{I}}^{i}\geq\tau_{Y_{i}}\\\
0,&\text{otherwise}\end{cases}$ (1)

Where $Y^{i}_{loss}$, $\tau_{c}$, and $L_{\textit{phase I}}^{i}$ are the loss
label for pixel $i$, the phase I mean loss (CE) across all the pixels
belonging to class $c$ in the labeled pool, and phase I loss for pixel $i$,
respectively. Since each pixel is labeled based on how it compares to other
members of its class, the model acquires a more insightful loss prediction
capability specialized for each class. Moreover, as the model is already
trained for the main task, the ground truth does not change during the
training of this module unlike existing works [Xie et al.(2020)Xie, Feng,
Chen, Sun, Ma, and Song, Qiao et al.(2022)Qiao, Zhu, Long, Zhang, Wang, Du,
and Yang]. Next, we follow the training for the loss prediction task
separately for the pixels lying on the boundary and center regions via
assigning two output channels. This aids the module for a better focus on
different levels of error as it is known that the segmentation error is
generally higher on the boundary of the objects [Liu et al.(2020)Liu, Lu, and
Xu, Zou et al.(2019)Zou, Xiang, Chen, Chen, and Qiao]. Thus, the loss for loss
prediction module would be:

$\mathcal{L}_{\textit{L}}^{m\in\\{\textit{c, b}\\}}(\theta_{\textit{ seg
heads}},\theta_{\textit{L}},\theta_{\textit{W}})=\\\ \frac{1}{|\\{x:x\in
m\\}|}\sum_{\\{i:x_{i}\in
m\\}}\Bigl{(}-Y_{i}^{L}log(\sigma(\hat{Y}^{i}_{L-m}))-(1-Y_{i}^{L})log(1-\sigma(\hat{Y}^{i}_{L-m}))\Bigr{)}$
(2)

Where $\theta_{\textit{seg heads}}$, $\theta_{\textit{L}}$, and
$\theta_{\textit{W}}$ denote the parameters of varied-maturity segmentation
heads, loss prediction module, and weight prediction block, respectively.
Additionally, $|.|$, $\sigma(.)$, and $\hat{Y}^{i}_{L-m}$ are the cardinality
operator, the Sigmoid function, and the output of the channel $m$ (boundary or
center) of the loss prediction module, respectively.
Now, we have everything for the loss of training phase II in place:

$\mathcal{L}_{\textit{II}}=\lambda_{0}\mathcal{L}^{c}_{\textit{L}}+\lambda_{1}\mathcal{L}^{b}_{\textit{L}}+\sum_{k=1}^{3}\lambda_{k+1}\mathcal{L}^{k}_{\textit{seg}}$
(3)

Where $\lambda_{k}$’s are the regularization constants chosen by cross-
validation and parameter search.

Figure 2: Our hierarchical distribution breakdown approach.

###  3.2 Selection Strategy

We follow a hierarchical approach to breakdown the distribution of the data
through which we measure the uncertainty of samples by starting to look from a
low field of view (pixel level: top level of the hierarchy) going
incrementally to the highest field of view (Image level). At each field of
view, we assess the uncertainty level within different scopes to find regions
needing attention for sampling.
Pixel Level-Getting Aware of the Maturity: We start the uncertainty assessment
by analyzing the pixels individually. At this level, we feed all the samples
to the trained model and for the pixels in the unlabeled pool, measure the
uncertainty based on:

$u(x)=\Bigl{(}H(\hat{Y}_{x})+W^{1}_{x}JS(\hat{Y}_{x},\hat{Y}^{1}_{x})+W^{2}_{x}JS(\hat{Y}_{x},\hat{Y}^{2}_{x})+W^{3}_{x}JS(\hat{Y}_{x},\hat{Y}^{3}_{x})\Bigr{)}\\\
\Bigl{[}\Bigl{(}1-\delta(x)\Bigr{)}e^{\sigma(\hat{Y}_{L-b}^{x})}+\delta(x)e^{\sigma(\hat{Y}_{L-c}^{x})}\Bigr{]}$
(4) $\delta(x)=\begin{cases}1,&\text{if }x\in center\\\
0,&\text{otherwise}\end{cases}$

Here $H(P)$, $JS(P_{1},P_{2})$, $W^{k}_{x}$, and $\hat{Y}^{k}_{x}$ denote
entropy of probability distribution $P$, Jenson-Shannon divergence between
distributions $P_{1}$ and $P_{2}$, weight map predicted by the weight
prediction block for the $k$-th head, and the output distribution by the
$k$-th head for pixel $x$, respectively. Via Eq. 4, we measure the uncertainty
of the model for pixel $x$ by checking: i) the entropy of the final output
distribution (reflected by the first term), ii) the similarity of the final
output distribution to those of the varied-maturity heads (reflected by the
second-fourth terms). The intuition is that the easier a pixel is to classify,
the less depth is needed to produce an output similar to the final one. The
importance of each term is determined by the weight map corresponding to the
pixel and segmentation head ($W^{k}_{x}$). Lastly, iii) error-proneness of the
pixel (reflected by the exponential terms), determined based on the score
given for the pixel by its corresponding channel (center or boundary) of the
loss prediction module.
Superpixel Level: Next, we zoom out and look through the superpixel level. By
definition, a superpixel is a group of perceptually similar pixels. First, we
assign each superpixel to its dominant label $Do(s)$ (the predicted class for
the majority of the pixels within that superpixel [Cai et al.(2021)Cai, Xu,
Liew, and Foo]); however, unlike [Cai et al.(2021)Cai, Xu, Liew, and Foo], we
estimate the probability of the class $C_{i}$ within cluster $k$ ($cl_{k}$) by
counting the superpixels with the dominant label of $C_{i}$:

$P_{cl_{k}}(C_{i})=\frac{|\\{s:Do_{s}=C_{i}\&s\in cl_{k}\\}|}{|\\{s:s\in
cl_{k}\\}|}$ (5)

Now, we assess the uncertainty of each superpixel by:

$u(s)=\frac{\underset{x\in s}{\operatorname{\sum}}u(x)}{|\\{x|x\in
s\\}|}e^{-P_{cl_{k}}(Do(s))}\textit{ s.t. }s\in cl_{k}$ (6)

Based on Eq. 6 the uncertainty of a superpixel in a cluster is proportional to
the average uncertainty of its pixels and inversely proportional to the
abundance of its dominant label.
Cluster Level: Having the uncertainty of the superpixels in each cluster, we
now assess the uncertainty of each cluster:

$u(cl_{k})=\frac{\sum_{s\in cl_{k}}u(s)}{|\\{s:s\in cl_{k}\\}|}$ (7)

The uncertainty of each cluster determines the budget it is assigned in the
sample selection step. The more uncertain a cluster is, the larger budget it
is assigned to:

$B_{cl_{k}}=\left\lceil\frac{u(cl_{k})}{\sum_{j=1}^{N_{clusters}}u(cl_{j})}B_{t}\right\rceil$
(8)

Where $B_{cl_{k}}$, $B_{t}$, and $\left\lceil.\right\rceil$ are the budget
assigned to cluster $k$, the total budget, and the ceil function,
respectively.
Image Level: Once the uncertainty scores of all the lower fields of view are
figured out, we query pixels for each image based on: 1) considering budget
dedicated to each cluster in the image, 2) finding superpixels with highest
uncertainty within that cluster, and 3) selecting pixels with highest
uncertainty within these superpixels. It is worth mentioning that traversing
from the top to the bottom of the hierarchy helps us achieve a global insight
of the uncertainty across different regions of the image, while the trip back
to the top aids with finely detecting and selecting a small, yet impactful
number of samples for annotation.

Figure 3: Comparison with SOTA on Cityscapes with two of the most popular
backbones in AL methods (ResNet50 and MobileNetv2).

##  4 Experiments 111Find the numerical data for the plots and codes
here:github/MADBAL

Implementation Details: We evaluate MADBAL on Cityscapes [Cordts et
al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson, Franke, Roth, and
Schiele] and Pascal VOC 2012 [Everingham et al.(2015)Everingham, Eslami, Van
Gool, Williams, Winn, and Zisserman] datasets by training on samples from the
training set and testing on the validation set. Our initial labeled pools have
10 and 20 randomly selected pixels per image for VOC and Cityscapes,
respectively. The AL budget in our experiments is 10 and 20 pixels per image
for VOC and Cityscapes, respectively. We use SEEDS algorithm [Bergh et
al.(2012)Bergh, Boix, Roig, Capitani, and Gool] for superpixel extraction and
set the number of superpixels per images for both datasets equal to the number
of squares when the image is divided to the squares of size $16\times 16$. we
cluster these superpixels; however, superpixels are of irregular (not
necessarily rectangular or vector) shapes, which is not acceptable by K-means.
To address this, we first fit the superpixel at the center of a rectangular
patch with minimum size and then resize that patch to a certain size
($16\times 16$). Consequently, we feed the resized patch to the backbone of a
pretrained VGG 16 [Simonyan and Zisserman(2014)] and apply K-means to the
extracted feature vectors. This way the clustering would be done based on the
perceptual properties of the superpixels. For each dataset, we conduct our
experiments three times with its most prevalent backbones in the literature:
ResNet50 [He et al.(2016)He, Zhang, Ren, and Sun], MobileNetv2 [Sandler et
al.(2018)Sandler, Howard, Zhu, Zhmoginov, and Chen], and MobileNetv3 [Howard
et al.(2019)Howard, Sandler, Chu, Chen, Chen, Tan, Wang, Zhu, Pang, Vasudevan,
et al.] for Cityscapes and ResNet50 and MobileNetv3 for VOC.

During training, for Cityscapes, we acquire random crops of size $768\times
768$ from the samples and for VOC random crops of size $256\times 256$. Our
models are deployed using Pytorch and we use stochastic gradient descent
optimizer with an initial learning rate of 0.01, momentum of 0.9 and a poly
learning rate scheduler decaying the learning rate from the initial value to
zero linearly through 150 epochs (for phase I) and 30 epochs (for phase II).
The hyperparameres in Eq. 3 are 1, 1, 0.05, 0.1, and 0.15 respectively,
selected via cross-validation on initial labeled pools of Cityscapes.

Comparison with State of the Art: Figs. 3 , 4, and 5a report the average _mean
intersection over union_ (mIoU) of three repetitions of our experiments for
each dataset and backbone w.r.t. different budget measures. We can observe
that our annotation cost is two orders of magnitude lower than the majority of
the SOTA (regardless of the backbone) w.r.t. the percentage of annotated
pixels, while outperforming SOTA with a significant margin w.r.t. number of
clicks. Moreover, MADBAL starts with a lower performance than Shin _et al._ ’s
[Shin et al.(2021)Shin, Xie, and Albanie], which is mostly depending on the
richness of the initial labeled pool, and makes considerably large leaps and
outperforms their method quickly. This implies MADBAL’s effectiveness in
selecting the most important samples early on. To get a qualitative sense of
these leaps, Fig. 5b visualizes the performance of MADBAL through the first
two AL steps on a validation sample from each dataset. Finally, Tab. 1
compares various weakly-supervised and interactive weak supervision methods on
VOC, confirming the benefits of MADBAL trained with only 20 pixels per image.

Figure 4: Comparison with SOTA on VOC. w.r.t. number of clicks (a) and percentage of annotated pixels (b).  Table 1: Comparison with Weakly-supervised methods and PixelPick [Shin et al.(2021)Shin, Xie, and Albanie] on VOC. Method | Backbone | Train set (anno. type) | mIoU
---|---|---|---
Weakly-supervised methods
GAIN [Li et al.(2019)Li, Wu, Peng, Ernst, and Fu] | VGG16 | 10.k imgs (classes) | 55.3
MDC [Wei et al.(2018)Wei, Xiao, Shi, Jie, Feng, and Huang] | VGG16 | 10.k imgs (classes) | 60.4
DSRG [Huang et al.(2018)Huang, Wang, Wang, Liu, and Wang] | ResNet101 | 10.5k imgs (classes) | 61.4
FickleNet [Lee et al.(2019)Lee, Kim, Lee, Lee, and Yoon] | ResNet101 | 10.5k imgs (classes) | 64.9
BoxSup [Dai et al.(2015)Dai, He, and Sun] | VGG16 | 10.5k imgs (boxes) | 62.0
ScribbleSup [Lin et al.(2016)Lin, Dai, Jia, He, and Sun] | VGG16 | 10.5k imgs (scribbles) | 63.1
Interactive weak supervision
PixelPick [Shin et al.(2021)Shin, Xie, and Albanie] | ResNet50 | 1.5k imgs (20 pixels per image) | 65.6
MADBAL | ResNet50 | 1.5k imgs (20 pixels per image) | 72.4

Ablation Study: We validate our design of MADBAL by conducting experiments
devised to show how presence and absence of various components affect the
performance. For these experiments, we incorporate MobileNetV3 [Howard et
al.(2019)Howard, Sandler, Chu, Chen, Chen, Tan, Wang, Zhu, Pang, Vasudevan, et
al.] backbone and Cityscapes [Cordts et al.(2016)Cordts, Omran, Ramos,
Rehfeld, Enzweiler, Benenson, Franke, Roth, and Schiele] dataset.

Figure 5: (a): Performance results on Cityscapes based on the number of clicks
(for each method the exact values of the last AL step are shown with its
matching color). (b): Visualization results on both datasets. First round of
AL is completed with 20 and 10 pixels per image, and second round with 40 and
20 for Cityscapes and VOC, respectively.

i) Effect of maturity-awareness and loss prediction in this set of experiments
is focused through four different AL scenarios: 1) AL with MADBAL. 2) AL with
a modified MADBAL in which weight prediction block is dropped and averaging is
used instead (i.e. $\frac{1}{3}$ is used instead of weight maps). This is to
show the essence of giving different importance to different intermediate
features and its benefits for recognition of impactful pixels (denoted with
“Averaging”). 3) AL with a modified MADBAL in which the loss prediction module
only accesses the backbone features (i.e. no inputs from the varied-maturity
heads) and the uncertainty score calculation (Eq. 4) does not have Jenson-
Shannon divergence terms. This helps observe the effect of maturity-awareness
directly by removing the corresponding terms in uncertainty score formulation
(denoted with “No maturity-awareness”). 4) AL with vanilla backbone (no loss
prediction module, weight prediction block, and varied-maturity heads) to
analyze the performance of MADBAL solely relying on distribution breakdown
(denoted with “Vanilla”).

ii) Effect of distribution breakdown, as another important piece of novelty
during the sample selection stage, is studied via 3 AL scenarios: 1) AL with
MADBAL to show the benefits of the distribution breakdown, 2) AL with modified
MADBAL in which superpixels are assigned to random clusters while keeping the
number of clusters the same. This is to observe how clustering superpixels
based on their perception plays a role in detecting the most uncertain samples
while keeping the diversity (denoted with “Random dist-breakdown”). 3) AL with
modified MADBAL which does not benefit from distribution breakdown at all. In
this scenario N pixels with highest uncertainty scores in each image are
queried for annotation (denoted with “No dist-breakdown”).

We continue each AL progress until 90$\%$ performance of fully-supervised
model ($0.9\times 68.5=65.1\%$) is achieved. Fig. 6 depicts the results. As
expected, MADBAL, with its fully extended features, achieves $90\%$
performance with only 50 pixels per image owing to all the devised components.
For the first ablation study, the second-best performance belongs to
“Averaging,” which matches the intuition as the algorithm benefits from the
varied-maturity heads both for loss prediction and uncertainty score
calculation; however, removing learnable weight maps adversely affects its
performance compared to MADBAL. “Vanilla” and “No maturity-awareness” show the
worst performances due to missing the critical components. Between the two,
“Vanilla” is inferior as it does not benefit from the maturity-awareness nor
from the loss prediction module. “No maturity-awareness,” on the other hand,
shows better performance owing to loss prediction module helping with better
assessment of samples’ uncertainty. For the second ablation, it is worth
noticing that “Random dist-breakdown” is still showing a better performance
compared to “No dist-breakdown.” This can be attributed to the inevitable
diversity the clustering (whether it be a perception-based clustering
algorithm or random clustering) brings to the pulled samples in each round of
sample selection. In other words, by grouping the superpixels, each of which
corresponds roughly to an object class, we prevent “over-selection” of pixels
belonging to the same object category in each step. Hence, despite its lower
performance, “Random dist-breakdown” is still able to achieve higher
performance than “No dist-breakdown.”

Figure 6: Ablation study on the effect of proposed components. (a), (b): the
effect of maturity-awareness and loss prediction. (c), (d): the effect of
distribution breakdown. When all the components are put into work the highest
annotation efficiency is achieved. The more components are dropped, the more
degradation on the efficiency is resulted.

##  5 Conclusion

In this work we proposed an active learning framework for semantic
segmentation by integrating maturity-awareness and distribution breakdown.
Maturity-awareness helps develop an effective understanding and recognition of
the most critical pixels for performance improvement, while distribution
breakdown provides a hierarchical approach to have an inclusive insight of the
data distribution across different fields of view. Combined with a novel
uncertainty formulation, the proposed MADBAL is shown to outperform many state
of the art methods with significant margin. MADABAL can significantly reduce
training burdens and also be impactful for tasks where annotation is expensive
and not readily available.

## References

  * [Aghdam et al.(2019)Aghdam, Gonzalez-Garcia, Weijer, and López] Hamed H Aghdam, Abel Gonzalez-Garcia, Joost van de Weijer, and Antonio M López.  Active learning for deep detection neural networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3672–3680, 2019.
  * [Ash et al.(2019)Ash, Zhang, Krishnamurthy, Langford, and Agarwal] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal.  Deep batch active learning by diverse, uncertain gradient lower bounds.  _arXiv preprint arXiv:1906.03671_ , 2019.
  * [Bergh et al.(2012)Bergh, Boix, Roig, Capitani, and Gool] Michael Van den Bergh, Xavier Boix, Gemma Roig, Benjamin de Capitani, and Luc Van Gool.  Seeds: Superpixels extracted via energy-driven sampling.  In _European conference on computer vision_ , pages 13–26. Springer, 2012.
  * [Cai et al.(2021)Cai, Xu, Liew, and Foo] Lile Cai, Xun Xu, Jun Hao Liew, and Chuan Sheng Foo.  Revisiting superpixels for active learning in semantic segmentation with realistic annotation costs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10988–10997, 2021.
  * [Casanova et al.(2020)Casanova, Pinheiro, Rostamzadeh, and Pal] Arantxa Casanova, Pedro O. Pinheiro, Negar Rostamzadeh, and Christopher J. Pal.  Reinforced active learning for image segmentation.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=SkgC6TNFvr.
  * [Cordts et al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson, Franke, Roth, and Schiele] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3213–3223, 2016.
  * [Dai et al.(2015)Dai, He, and Sun] Jifeng Dai, Kaiming He, and Jian Sun.  Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1635–1643, 2015.
  * [El Helou et al.(2021)El Helou, Zhou, Susstrunk, and Timofte] Majed El Helou, Ruofan Zhou, Sabine Susstrunk, and Radu Timofte.  Ntire 2021 depth guided image relighting challenge.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 566–577, June 2021.
  * [Elhamifar et al.(2013)Elhamifar, Sapiro, Yang, and Sasrty] Ehsan Elhamifar, Guillermo Sapiro, Allen Yang, and S Shankar Sasrty.  A convex optimization framework for active learning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 209–216, 2013.
  * [Everingham et al.(2015)Everingham, Eslami, Van Gool, Williams, Winn, and Zisserman] Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  _International journal of computer vision_ , 111(1):98–136, 2015.
  * [Freytag et al.(2014)Freytag, Rodner, and Denzler] Alexander Freytag, Erik Rodner, and Joachim Denzler.  Selecting influential examples: Active learning with expected model output changes.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, _Computer Vision – ECCV 2014_ , pages 562–577, Cham, 2014. Springer International Publishing.  ISBN 978-3-319-10593-2.
  * [Gal et al.(2017)Gal, Islam, and Ghahramani] Yarin Gal, Riashat Islam, and Zoubin Ghahramani.  Deep bayesian active learning with image data.  In _International Conference on Machine Learning_ , pages 1183–1192. PMLR, 2017.
  * [Golestaneh and Kitani(2020)] S Alireza Golestaneh and Kris M Kitani.  Importance of self-consistency in active learning for semantic segmentation.  _arXiv preprint arXiv:2008.01860_ , 2020.
  * [Gorriz et al.(2017)Gorriz, Carlier, Faure, and Giro-i Nieto] Marc Gorriz, Axel Carlier, Emmanuel Faure, and Xavier Giro-i Nieto.  Cost-effective active learning for melanoma segmentation.  _arXiv preprint arXiv:1711.09168_ , 2017.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * [Houlsby et al.(2011)Houlsby, Huszár, Ghahramani, and Lengyel] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel.  Bayesian active learning for classification and preference learning.  _arXiv preprint arXiv:1112.5745_ , 2011.
  * [Howard et al.(2019)Howard, Sandler, Chu, Chen, Chen, Tan, Wang, Zhu, Pang, Vasudevan, et al.] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1314–1324, 2019.
  * [Huang et al.(2018)Huang, Wang, Wang, Liu, and Wang] Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, and Jingdong Wang.  Weakly-supervised semantic segmentation network with deep seeded region growing.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7014–7023, 2018.
  * [Joshi et al.(2009)Joshi, Porikli, and Papanikolopoulos] Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos.  Multi-class active learning for image classification.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2372–2379, 2009.  10.1109/CVPR.2009.5206627.
  * [Kirsch et al.(2019)Kirsch, van Amersfoort, and Gal] Andreas Kirsch, Joost van Amersfoort, and Yarin Gal.  Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/95323660ed2124450caaac2c46b5ed90-Paper.pdf.
  * [Konyushkova et al.(2017)Konyushkova, Sznitman, and Fua] Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua.  Learning active learning from data.  _Advances in neural information processing systems_ , 30, 2017.
  * [Lee et al.(2019)Lee, Kim, Lee, Lee, and Yoon] Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon.  Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5267–5276, 2019.
  * [Lewis(1995)] David D Lewis.  A sequential algorithm for training text classifiers: Corrigendum and additional data.  In _Acm Sigir Forum_ , volume 29, pages 13–19. ACM New York, NY, USA, 1995.
  * [Li et al.(2019)Li, Wu, Peng, Ernst, and Fu] Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Fu.  Guided attention inference network.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(12):2996–3010, 2019.
  * [Li and Monga(2020)] Xuelu Li and Vishal Monga.  Group based deep shared feature learning for fine-grained image classification.  _arXiv preprint arXiv:2004.01817_ , 2020.
  * [Lin et al.(2016)Lin, Dai, Jia, He, and Sun] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun.  Scribblesup: Scribble-supervised convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3159–3167, 2016.
  * [Liu et al.(2020)Liu, Lu, and Xu] Wenrui Liu, Zongqing Lu, and He Xu.  Auxiliary edge detection for semantic image segmentation.  In _Proceedings of the 2020 6th International Conference on Computing and Artificial Intelligence_ , ICCAI ’20, 2020.
  * [Lu and Weng(2007)] Dengsheng Lu and Qihao Weng.  A survey of image classification methods and techniques for improving classification performance.  _International journal of Remote sensing_ , 28(5):823–870, 2007.
  * [Mackowiak et al.(2018)Mackowiak, Lenz, Ghori, Diego, Lange, and Rother] Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, and Carsten Rother.  Cereals-cost-effective region-based active learning for semantic segmentation.  _arXiv preprint arXiv:1810.09726_ , 2018.
  * [Minaee et al.(2021)Minaee, Boykov, Porikli, Plaza, Kehtarnavaz, and Terzopoulos] Shervin Minaee, Yuri Y Boykov, Fatih Porikli, Antonio J Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos.  Image segmentation using deep learning: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 2021.
  * [Nguyen and Smeulders(2004)] Hieu T Nguyen and Arnold Smeulders.  Active learning using pre-clustering.  In _Proceedings of the twenty-first international conference on Machine learning_ , page 79, 2004.
  * [Ouali et al.(2020)Ouali, Hudelot, and Tami] Yassine Ouali, Céline Hudelot, and Myriam Tami.  Semi-supervised semantic segmentation with cross-consistency training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12674–12684, 2020.
  * [Qiao et al.(2022)Qiao, Zhu, Long, Zhang, Wang, Du, and Yang] Yu Qiao, Jincheng Zhu, Chengjiang Long, Zeyao Zhang, Yuxin Wang, Zhenjun Du, and Xin Yang.  Cpral: Collaborative panoptic-regional active learning for semantic segmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 2108–2116, 2022.
  * [Roy and McCallum(2001)] Nicholas Roy and Andrew McCallum.  Toward optimal active learning through monte carlo estimation of error reduction.  _ICML, Williamstown_ , 2:441–448, 2001.
  * [Sandler et al.(2018)Sandler, Howard, Zhu, Zhmoginov, and Chen] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [Sener and Savarese(2017)] Ozan Sener and Silvio Savarese.  Active learning for convolutional neural networks: A core-set approach.  _arXiv preprint arXiv:1708.00489_ , 2017.
  * [Settles et al.(2007)Settles, Craven, and Ray] Burr Settles, Mark Craven, and Soumya Ray.  Multiple-instance active learning.  _Advances in neural information processing systems_ , 20, 2007.
  * [Shin et al.(2021)Shin, Xie, and Albanie] Gyungin Shin, Weidi Xie, and Samuel Albanie.  All you need are a few pixels: semantic segmentation with pixelpick.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1687–1697, 2021.
  * [Siddiqui et al.(2020)Siddiqui, Valentin, and Nießner] Yawar Siddiqui, Julien Valentin, and Matthias Nießner.  Viewal: Active learning with viewpoint entropy for semantic segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9433–9443, 2020.
  * [Simonyan and Zisserman(2014)] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * [Sinha et al.(2019)Sinha, Ebrahimi, and Darrell] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell.  Variational adversarial active learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5972–5981, 2019.
  * [Wei et al.(2018)Wei, Xiao, Shi, Jie, Feng, and Huang] Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie, Jiashi Feng, and Thomas S Huang.  Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7268–7277, 2018.
  * [Xie et al.(2020)Xie, Feng, Chen, Sun, Ma, and Song] Shuai Xie, Zunlei Feng, Ying Chen, Songtao Sun, Chao Ma, and Mingli Song.  Deal: Difficulty-aware active learning for semantic segmentation.  In _Proceedings of the Asian Conference on Computer Vision_ , 2020\.
  * [Yang et al.(2017)Yang, Zhang, Chen, Zhang, and Chen] Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z Chen.  Suggestive annotation: A deep active learning framework for biomedical image segmentation.  In _International conference on medical image computing and computer-assisted intervention_ , pages 399–407. Springer, 2017.
  * [Yazdani et al.(2021)Yazdani, Agrawal, Johnstonbaugh, Kothapalli, and Monga] Amirsaeed Yazdani, Sumit Agrawal, Kerrick Johnstonbaugh, Sri-Rajasekhar Kothapalli, and Vishal Monga.  Simultaneous denoising and localization network for photoacoustic target localization.  _IEEE Transactions on Medical Imaging_ , 40(9):2367–2379, 2021.  10.1109/TMI.2021.3077187.
  * [Yoo and Kweon(2019)] Donggeun Yoo and In So Kweon.  Learning loss for active learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 93–102, 2019.
  * [Zou et al.(2019)Zou, Xiang, Chen, Chen, and Qiao] Nan Zou, Zhiyu Xiang, Yiman Chen, Shuya Chen, and Chengyu Qiao.  Boundary-aware cnn for semantic segmentation.  _IEEE Access_ , 7:114520–114528, 2019.  10.1109/ACCESS.2019.2935816.


# Self-induced inverse spin Hall effect in La0.67Sr0.33MnO3 films

Pushpendra Gupta  Laboratory for Nanomagnetism and Magnetic Materials, School
of Physical Sciences, National Institute of Science Education and Research
(NISER), An OCC of Homi Bhabha National Institute (HBNI), Jatni 752050, India.
In Jun Park  Department of Physics, Indiana University – Purdue University
Indianapolis, Indianapolis, Indiana 46202, USA  Anupama Swain  Laboratory for
Nanomagnetism and Magnetic Materials, School of Physical Sciences, National
Institute of Science Education and Research (NISER), An OCC of Homi Bhabha
National Institute (HBNI), Jatni 752050, India.  Abhisek Mishra  Laboratory
for Nanomagnetism and Magnetic Materials, School of Physical Sciences,
National Institute of Science Education and Research (NISER), An OCC of Homi
Bhabha National Institute (HBNI), Jatni 752050, India.  Vivek P. Amin
<EMAIL_ADDRESS>Department of Physics, Indiana University – Purdue University
Indianapolis, Indianapolis, Indiana 46202, USA  Subhankar Bedanta
<EMAIL_ADDRESS>Laboratory for Nanomagnetism and Magnetic Materials,
School of Physical Sciences, National Institute of Science Education and
Research (NISER), An OCC of Homi Bhabha National Institute (HBNI), Jatni
752050, India.  Center for Interdisciplinary Sciences (CIS), National
Institute of Science Education and Research (NISER), An OCC of Homi Bhabha
National Institute (HBNI), Jatni 752050, India.

###### Abstract

The efficient generation of spin currents and spin torques via spin-orbit
coupling is an important goal of spintronics research. One crucial metric for
spin current generation is the spin Hall angle, which is the ratio of the spin
Hall current to the transversely flowing charge current. A typical approach to
measure the spin Hall angle in nonmagnetic materials is to generate spin
currents via spin pumping in an adjacent ferromagnetic layer and measure the
transverse voltage from the inverse spin Hall effect in the nonmagnetic layer.
However, given that the spin Hall effect also occurs in ferromagnets, single
ferromagnetic layers could generate a self-induced transverse voltage during
spin pumping as well. Here we show that manganite based La0.67Sr0.33MnO3
(LSMO) films deposited by pulsed laser deposition exhibit a significant self-
induced inverse spin Hall voltage while undergoing spin pumping. A spin
pumping voltage of 1.86 $\mu$V is observed in the LSMO (12 nm) film. Using
density functional theory and the Kubo formalism, we calculate the intrinsic
spin current conductivities of these films and show that they are in
reasonable agreement with the experimental measurements.

inverse spin Hall effect, ferromagnetic resonance, manganite, first principle
calculation

##  I Introduction

Many spintronic applications depend on all-electrical control of magnetization
dynamics. Such control can be achieved by electrically generating a spin
current in a nonmagnetic layer that flows into a neighboring ferromagnetic
layer and exerts a spin-orbit torque (SOT) Liu  _et al._ (2012); Miron  _et
al._ (2010); Manchon  _et al._ (2019); Amin  _et al._ (2020); Shao  _et al._
(2021). The spin-transfer torque, or transfer of spin angular momentum to the
ferromagnetic layer’s magnetization, can result in magnetization switching in
magnetic memories and self-sustained oscillations in spin torque oscillators.
Thus, an important step in achieving energy-efficient magnetization control is
to optimize the efficiency of spin current generation.

It is well known that for nonmagnetic materials with appreciable spin-orbit
coupling (such as heavy metals like Pt or Ta), an applied electric field
generates a spin current where the electric field, spin flow, and spin
polarization are mutually orthogonal. This phenomenon, known as the spin Hall
effect Sinova  _et al._ (2015), is typically used as a spin current source in
spintronic devices. Likewise, if a spin current with orthogonal spin flow and
spin polarization flows into a heavy metal, a potential difference forms that
is oriented perpendicularly to the spin flow and spin polarization. This
process is known as the inverse spin Hall effect (ISHE). One prominent method
to measure the strength of the inverse spin Hall effect in a nonmagnetic
material is to generate a spin current via spin pumping Mosendz  _et al._
(2010); Hahn  _et al._ (2013); Czeschka  _et al._ (2011); Wegrowe and Drouhin
(2011) in a neighboring ferromagnetic material and measure the resulting
inverse spin Hall voltage.

Both experimental Miao  _et al._ (2013); Tsukahara  _et al._ (2014); Humphries
_et al._ (2017); Kanagawa  _et al._ (2018); Wang  _et al._ (2019) and
theoretical Amin  _et al._ (2018); Go  _et al._ (2020); Mook  _et al._ (2020);
Yahagi  _et al._ (2021); Salemi and Oppeneer (2022) studies have shown strong
evidence that single ferromagnetic layers also generate transversely-flowing
spin currents under an applied electric field. However, due to the lower
symmetry of ferromagnets compared to nonmagnets, the electric field, spin
flow, and spin polarization need not be mutually orthogonal. Spin current
generation in single ferromagnet layers can be attributed to the spin Hall
effect Wang  _et al._ (2019); Amin  _et al._ (2019); Omori  _et al._ (2019),
spin anomalous Hall effect Taniguchi  _et al._ (2015); Das  _et al._ (2017);
Amin  _et al._ (2019), spin planar Hall effect Taniguchi  _et al._ (2015);
Amin  _et al._ (2018); Safranski  _et al._ (2018, 2020), and the magnetic spin
Hall effect Davidson  _et al._ (2020); Mook  _et al._ (2020); Salemi and
Oppeneer (2022). It is possible that some of the pumped spin current can flow
into the ferromagnetic layer and be converted via the ISHE into a transverse
voltage. To confirm this would require measuring an appreciable inverse spin
Hall voltage while performing spin pumping in a single ferromagnet layer.

A recent report by Miao et al. shows that the injection of a pure spin current
from yttrium iron-garnet (YIG) to NiFe (Py) resulted in an ISHE in Py Miao
_et al._ (2013). This report suggests the presence of appreciable spin-orbit
coupling in the Py film, which is endorsed by the observation of an
anisotropic magnetoresistance. Further, Tsukahara et al. Tsukahara  _et al._
(2014) have measured a self-induced ISHE in a single layer Py film, i.e.
without any adjacent high spin-orbit coupling material. These results indicate
that the spin current generated within Py could be converted via the ISHE into
a tranversely-oriented voltage. In addition, there are other studies of the
intrinsic ISHE in Co and Fe films Kanagawa  _et al._ (2018).

On a similar note, manganites such as La0.67Sr0.33MnO3 (LSMO), have attracted
attention in recent years in spin-to-charge conversion studies due to its low
damping, high Curie temperature ($T_{c}$ $\sim$ 350 K), and high spin
polarization Luo  _et al._ (2017); Qin  _et al._ (2017). LSMO is an oxide-
based ferromagnet that has been explored only as a spin current source in spin
pumping studies Gupta  _et al._ (2021); Luo  _et al._ (2017, 2014); Lee  _et
al._ (2016); Gupta  _et al._ (2023). In particular, in LSMO/Pt bilayers,
antidamping like spin torques have been observed Gupta  _et al._ (2021), where
the spin torques were primarily attributed to the spin Hall effect in Pt.
However, to date, a self-induced inverse spin Hall effect has not been
reported in manganites like LSMO.

In this work, we report a self-induced ISHE in epitaxial LSMO films under
ferromagnetic resonance. Further, we extract the intrinsic spin Hall
conductivity of our LSMO films and compare them to first-principles
calculations of the bulk spin Hall conductivity. The spin Hall conductivities
extracted from both the experimental results and theoretical calculations are
on the order of $10~{}\ohm^{-1}\text{cm}^{-1}$ and thus show reasonable
agreement.

##  II Experimental details

Samples STO(100)/LSMO(t nm) have been deposited via pulsed laser deposition
technique having a laser of wavelength 248 nm and a vacuum chamber with base
pressure 3 $\times$ 10-7 mbar. The samples are called SL1, SL2, and SL3 with
thicknesses (t) of 12 nm, 35 nm, and 55 nm respectively. For the growth of
LSMO films, the substrate temperature was kept at 740 $\degree$C during the
deposition. Laser fluence and frequency are kept at 1.4 J/cm2 and 2 Hz,
respectively. The oxygen partial pressure during the growth was maintained at
0.47 mbar. Post-deposition annealing of the sample was performed at the same
temperature (740 $\degree$C) for 30 mins at 250 mbar oxygen pressure. X-ray
reflectivity (XRR) and x-ray diffraction (XRD) have been performed with a
diffractometer (model- Rigaku smartLab) with wavelength $\lambda$= 0.154 nm.
Cross-sectional high-resolution transmission electron microscopy (HR-TEM) has
been performed to check the epitaxy of deposited films by TEM (JEOL, F-200).
The dc magnetometry was performed by a superconducting quantum interference
device (SQUID) based magnetometer manufactured by Quantum Design (MPMS3).

Co-planar wave-guide (CPW) based ferromagnetic resonance (FMR) spectroscopy
(manufactured by NanOsc, Sweden) was carried out to study magnetization
dynamics. The sample was kept on top of the CPW in a flip-chip manner Gupta
_et al._ (2021); Singh  _et al._ (2019). To prevent shunting, a 25 $\mu$m
polymer tape was placed between the sample and the CPW. A dc magnetic field
($H$) was applied perpendicular to the radio frequency field ($h_{rf}$). ISHE
measurement was performed at 7 GHz frequency and 25 mW microwave power. The
voltage obtained due to ISHE was measured by a Nanovoltmeter (Keithley 2182A).

##  III Results and discussion

Figure 1: (a) Schematic of sample geometry and ISHE measurement. (b) XRD
pattern for sample SL2, in which LSMO film diffraction peaks are observed
corresponding to STO (100) planes. The zoomed view of (200) peak is shown in
the inset of (b). (c) HR-TEM image for SL2 sample. The insets are the zoomed
part of LSMO films and STO substrate for measuring the lattice constants of
LSMO and STO, respectively. (d) Thermoremanent magnetization (TRM) data in
which the ferromagnetic to paramagnetic transition is observed at 320 K.

The schematic of samples studied in this work and ISHE measurement are shown
in Fig. 1(a). The XRD data shown in Fig. 1(b) convey that LSMO films had been
grown in (100) orientations which is also the orientation of the STO
substrate. This confirms that the LSMO films are highly epitaxial in nature.
Film thickness was estimated using X-ray reflectivity (data not shown). Fig.
1(c) depicts a cross-sectional HRTEM image of sample SL2, which indicates
clear and sharp interfaces of STO/LSMO. The zoomed-in image confirms the
epitaxial nature of LSMO.

Figure 2: FMR signal (solid circles) for different frequencies for samples
SL1, SL2, and SL3. Solid lines are the fits using the Lorentzian equation.

Fig. 1(d) shows the magnetization (M) vs. temperature (T) plot for the LSMO
(35 nm) sample. For this measurement, sample was cooled from 400 K to 5 K in
the presence of a 100 Oe magnetic field. After cooling to 5 K, the magnetic
field was switched off and magnetization data were recorded with temperature.
These protocols are called thermoremanent magnetization (TRM). From the
figure, it can be clearly seen that the Curie temperature ($T_{c}$) of the
deposited film is 320 K, which confirms the ferromagnetic phase of the sample
at room temperature (300 K).

Figure 3: Frequency ($f$) vs. resonance field ($H_{res}$) plot and (b)
Linewidth ($\Delta H$) vs. $f$ plot for STO/LSMO samples extracted from Fig.
2. Solid lines are the best fits for equations (1) and (2).

To study the magnetization dynamics, we have performed FMR from 3.5-10 GHz.
The FMR spectra for all the samples have been shown in Fig. 2. To calculate
the Gilbert damping constant ($\alpha$), each FMR spectrum was fitted with the
derivative of the standard Lorentzian equation. From the fitting of FMR data,
resonance field ($H_{res}$) and linewidth ($\Delta H$) have been extracted. f
vs. $H_{res}$ plots are shown in Fig. 3(a) for all the samples. The data are
fitted using the Kittel equation

$f=\frac{\gamma}{2\pi}\sqrt{(H_{res}+H_{K})(H_{res}+4\pi M_{eff}+H_{K})}$ (1)

where $\gamma$(=$\frac{g\mu_{B}}{\hbar}$), $g$, $\mu_{B}$, $H_{K}$ and
$M_{eff}$ are gyromagnetic ratio, Lande g-factor, Bohr magneton, in-plane
anisotropic fields and effective demagnetization, respectively.

Further, f vs. $\Delta H$ data shown in Fig. 3(b) are fitted using the
following linear equation

$\Delta H=\Delta H_{0}+\dfrac{4\pi\alpha f}{\gamma}$ (2)

where $\Delta H_{0}$ is the inhomogeneous linewidth. The values of Gilbert
damping constant ($\alpha$) for samples SL1, SL2, and SL3 are evaluated to be
0.0086±0.0002, 0.0077±0.0008 and 0.0097±0.0009, respectively. The values of
$\alpha$ match well with the previously reported values Luo  _et al._ (2013).

Figure 4: Measured dc voltage ($V_{meas}$) vs. field (H) measured at f=7 GHz
for SL1. $V_{sym}$ and $V_{asym}$ components from $V_{meas}$ have been
separated using equation (3).

Further, Gupta et al. has shown the presence of a significant anisotropic
magnetoresistance in the LSMO/Pt system, which may arise due to the spin-orbit
coupling of the LSMO film (though the adjacent Pt film could contribute via
the proximity effect)Gupta  _et al._ (2021). The presence of spin-orbit
coupling suggests there could be appreciable spin to charge conversion in the
LSMO films. As LSMO shows metallic ferromagnetic behavior at room temperature,
there is a possibility of an intrinsic ISHE in LSMO films similar to that
found in the transition metal ferromagnets. The intrinsic ISHE causes a
voltage across the sample transversely oriented to the spin flow, which is
measured via making contacts with Cu wires from the sample to a nanovoltmeter.
Fig. 4 shows the measured voltage ($V_{meas}$) data at 0 and 180$\degree$.
$V_{meas}$ fitted with the equation (3) to separate out the symmetric
($V_{sym}$) and asymmetric ($V_{asym}$) contributions to voltage Iguchi and
Saitoh (2017).

$\begin{split}V_{meas}=V_{sym}\frac{(\Delta H)^{2}}{(H-H_{res})^{2}+(\Delta
H)^{2}}+\\\ V_{asym}\frac{(2\Delta H)(H-H_{res})}{(H-H_{res})^{2}+(\Delta
H)^{2}}\end{split}$ (3)

From Fig. 4, it can be clearly seen that the $V_{sym}$ is more dominating than
the $V_{asym}$ component. Also, it can be noticed that $V_{sym}$ changed its
polarity when the sample was rotated to 180$\degree$. This is a characteristic
property that confirms the ISHE in LSMO films.

It is well known that the $V_{sym}$ mainly comes due to spin pumping while the
spin rectification effects integrated to AMR, AHE, etc., contribute to
$V_{asym}$ voltage Iguchi and Saitoh (2017). To calculate the spin pumping
voltage from the measured voltage, a complete angle-dependent ISHE has been
performed from 0 to 360$\degree$ in a step of 5$\degree$. Each measured
voltage data is fitted to get $V_{sym}$ and $V_{asym}$ values using equation
(3). Further, $V_{sym}$ and $V_{asym}$ are plotted with their corresponding
angles as shown in Fig. 5(a) and (b). The $V_{sym}$ data are fitted using the
following equation Conca  _et al._ (2017)

$V_{sym}=V_{sp}cos^{3}(\phi+\phi_{0})+V_{AHE}cos(\phi+\phi_{0})cos\theta+\\\
V_{sym}^{AMR\perp}cos(2(\phi+\phi_{0}))cos(\phi+\phi_{0})+\\\
V_{sym}^{AMR\parallel}sin(2(\phi+\phi_{0}))cos(\phi+\phi_{0})$ (4)

while the $V_{asym}$ data are fitted using the following equation Conca  _et
al._ (2017)

$V_{asym}=V_{AHE}cos(\phi+\phi_{0})sin\theta+\\\
V_{asym}^{AMR\perp}cos(2(\phi+\phi_{0}))cos(\phi+\phi_{0})\\\
+V_{asym}^{AMR\parallel}sin(2(\phi+\phi_{0}))cos(\phi+\phi_{0})$ (5)

where $\theta$ is the angle between the electric and magnetic fields of the
applied microwave, which is 90$\degree$. Angle $\phi$ is defined as the angle
between the voltage measurement direction and perpendicular to the magnetic
field direction, while $\phi_{0}$ is the extra factor to take care of the
misalignment in the sample positioning. $V_{AHE}$ is anomalous Hall voltage,
which arises due to the FM nature of the sample.
$V_{sym}^{AMR\perp,\parallel}$ and $V_{asym}^{AMR\perp,\parallel}$ are
perpendicular(parallel) components of symmetric and asymmetric contributions
to $V_{AMR}$ and can be calculated using the following equation Conca  _et
al._ (2017)

$V_{AMR}^{{\perp,\parallel}}=\sqrt{(V_{asym}^{AMR\perp,\parallel})^{2}+(V_{sym}^{AMR\perp,\parallel})^{2}}$
(6)

From the fitting, a significant spin pumping voltage has been obtained for
samples, which are listed in Table I. The value of $V_{sp}$ is dominating over
any other spin rectification effects. As there is no SOC layer adjacent to the
LSMO film, $V_{sp}$ can be attributed to the intrinsic ISHE of LSMO films.

Figure 5: (a) $\phi$ dependent $V_{sym}$ and (b) $V_{asym}$ for LSMO films.
Red and black solid lines are the best fits for $V_{sym}$ and $V_{asym}$,
using equations (4) and (5), respectively.

Further, $I_{c}$ vs $t_{LSMO}$ data have been plotted as shown in Fig. 6. As
$I_{c}$ has a contribution from magnonic charge pumping ($I_{MCP}$) and
inverse spin Hall effect ($I_{ISHE}$) hence the data has been fitted using the
equation $I_{c}=I_{MCP}+I_{ISHE}$ where $I_{MCP}$ and $I_{ISHE}$ can be
written in terms of following expressions Huang  _et al._ (2020); Azevedo  _et
al._ (2015)

$I_{MCP}=(\sigma_{LSMO}t_{LSMO}+\sigma_{STO}t_{STO})w\\\
\times(\frac{\Delta\rho}{2M_{s}^{2}})(\frac{M_{eff}\gamma}{4\pi\alpha^{2}})A^{(d)}h_{rf}^{2}$
(7)

$I_{SHE}=\theta_{SHA}w(\frac{2e}{\hbar})\lambda_{LSMO}\tanh(\frac{t_{LSO}}{2\lambda_{LSMO}})j_{s}^{LSMO}$
(8)

where $\theta_{SHA}$, $\Delta\rho$, $A^{(d)}$, $h_{rf}$ and $\lambda_{LSMO}$
are the spin Hall angle, anisotropic magneto-resistivity, spin orbit coupling
parameter, RF magnetic field of the microwave, and spin diffusion length,
respectively. $\sigma_{LSMO}$, $t_{LSMO}$, $\sigma_{STO}$, and $t_{STO}$ are
the conductivity and thickness of the LSMO and STO layers respectively. The
spin current density at the LSMO/STO interface, $j_{s}^{LSMO}$, is given by
Tserkovnyak  _et al._ (2005); Huang  _et al._ (2020)

$j_{s}^{LSMO}=\frac{(g_{r}^{\uparrow\downarrow})\gamma^{2}h_{rf}^{2}\hbar[4\pi
M_{s}\gamma+\sqrt{(4\pi M_{s}\gamma)^{2}+4\omega^{2}}]}{8\pi\alpha^{2}[(4\pi
M_{s}\gamma)^{2}+4\omega^{2}]}$ (9)

From the above equation we have calculated the $g_{r}^{\uparrow\downarrow}$
value to 1.2$\times$1016 m-2and $j_{s}^{LSMO}$ has been considered as a
fitting parameter. From the fitting of the $I_{c}$ vs $t_{LSMO}$ data shown in
Fig. 6, the $\theta_{SHA}$ value is extracted to 0.33.

Figure 6: Charge current ($I_{c}$) vs thickness ($t_{LSMO}$) of LSMO. The
solid line represents the fitting to $I_{C}=I_{MCP}+I_{ISHE}$ where the
$I_{MCP}$ and $I_{SHE}$ are represented by equations (7) and (8),
respectively.

Finally, we have obtained the spin Hall conductivity using
$\theta_{SHA}$=$\sigma_{SH}$/$\sigma_{C}$ spin Hall conductivity and charge
conductivity, respectively. The calculated values of $\sigma_{SH}$ for SL1,
SL2, and SL3 are 76.4, 58.9, and 50.0 $\ohm^{-1}cm^{-1}$, respectively.

In an earlier report spin current has been generated in the Py layer itself
via FMR due to the high resistivity of the adjacent SiO2 layer which prevents
the spins to diffuse through it (SiO2 layer) Tsukahara  _et al._ (2014). A
similar self-induced ISHE observation has also been made in Co and Fe films
Kanagawa  _et al._ (2018). In our case, the resistivity of STO is
comparatively less than SiO2 Saadatkia  _et al._ (2018). When FMR occurs in
the LSMO layer, the spin current is generated, which tends to move towards the
LSMO/STO interface. At the interface, spins get scattered due to the charged
impurities which might be causing a spin gradient in the LSMO layer. This spin
gradient creates spin current which gets converted into charge current via
ISHE, which leads to the generation of potential difference.

Table 1: $V_{sp}$, and $V_{AHE}$ values for all samples from the fitting of
the data shown in Fig. 5(a) and (b) using equations (4) and (5). $V_{AMR}$
values were calculated using equation (6).

Sample | $V_{sp}$ | $V_{AHE}$ | $V_{AMR}^{\perp}$ | $V_{AMR}^{\parallel}$
---|---|---|---|---
| ($\mu V$) | ($\mu V$) | ($\mu V$) | ($\mu V$)
S1 | 1.86 | 0.12 | 1.43 | 0.39
S2 | 1.83 | 0.31 | 1.47 | 0.13
S3 | 1.39 | 0.12 | 1.31 | 0.15

##  IV Computation Details

Figure 7: Bulk crystal structure of perovskite LaMnO3 or SrMnO3. The green,
red, and purple spheres represent La (or Sr), O, and Mn atoms, respectively. A
simpler model was chosen to reduce computational costs.

In order to evaluate the appreciable ISHE in LSMO from the experimental work,
we performed the density functional theory (DFT) calculations as implemented
in the Quantum Espresso packageGiannozzi  _et al._ (2017, 2009, 2020). To
reduce computational costs, we limited our theoretical calculation to LaMnO3
(LMO) and SrMnO3 (SMO), instead of LSMO since LSMO requires a large unit cell
in order to meet its chemical composition with symmetry and redundant
calculations to average the disorder. The representative unit cell of LMO or
SMO is shown in Fig. 7, where green, red, and purple spheres are La (or Sr),
O, and Mn atoms, respectively. Cut-off energies for plane wave and charge
density of 90 Ry and 900 Ry and 9$\times$9$\times$9 k-points grids were used
to ensure the total energy converged within 7$\times$10-9 Ry. Once the ground
states were obtained, we interpolated them into an orbital basis using the
Wannier90 package Pizzi  _et al._ (2020). With interpolated basis, the spin
Hall conductivity tensor was calculated using Kubo’s formula as implemented in
Qiao  _et al._ (2018); Tsirkin (2021)

$\sigma_{\alpha\beta\gamma}=\frac{-e\hbar}{N_{k}V_{c}}\sum_{k}\sum_{n,m}(f_{nk}-f_{mk})\times\\\
\frac{Im[\langle\psi_{nk}|\frac{1}{2}\\{S^{\gamma},\nu_{\alpha}\\}|\psi_{mk}\rangle\langle\psi_{mk}|\nu_{\beta}|\psi_{nk}]}{(\epsilon_{nm}-\epsilon_{mk})^{2}-(\hbar\omega)+i\eta)^{2}}$
(10)

where $\alpha$, $\beta$, and $\gamma$ are the direction of spin current,
electric field, and spin polarization, respectively. 400$\times$400$\times$400
k-grids were used to calculate the spin Hall conductivity tensors of LMO and
SMO.

For non-magnetic materials, it is known that the non-zero elements of spin
Hall conductivity tensor can occur when the three components (i.e., $\alpha$,
$\beta$, and $\gamma$) are mutually orthogonal, and they have the same
magnitude. However, due to the lower symmetry from magnetization,
ferromagnetic materials have different non-zero spin Hall conductivity
components depending on the magnetization direction Davidson  _et al._ (2020).
For instance, when the magnetization is parallel to $x$ direction,
$\sigma_{xyz}=-\sigma_{xzy}$, $\sigma_{yzx}=-\sigma_{zyx}$, and
$\sigma_{zxy}=-\sigma_{yxz}$ while it is parallel to $z$ direction,
$\sigma_{xyz}=-\sigma_{yxz}$, $\sigma_{yzx}=-\sigma_{xzy}$, and
$\sigma_{zxy}=-\sigma_{zyx}$. Therefore, we showed our calculation results for
only distinctive non-zero components (i.e., $\sigma_{xyz}$, $\sigma_{yzx}$,
and $\sigma_{zxy}$), as shown in Table II. Clearly, there are non-zero values
for spin Hall conductivity tensor on the order of 10 $\ohm^{-1}cm^{-1}$, which
are on the same order from experimentally obtained values, although they are
smaller than the ones from other perovskite structures ($\sim$ 100
$\ohm^{-1}cm^{-1}$ ), such as SrIrO3 Nan  _et al._ (2019), SrRuO3 Ou  _et al._
(2019), BaOsO3, and SrOsO3 Jadaun  _et al._ (2020) or Pt ($\sim$ 2000
$\ohm^{-1}cm^{-1}$) Qiao  _et al._ (2018). We note that extrinsic mechanisms
of the ISHE could also contribute to the experimental measurements but are not
included in the theoretical calculations.

Table 2: Calculated non-zero spin Hall conductivity tensor elements of LMO and
SMO

Spin Hall conductivity | $\sigma_{xyz}$ | $\sigma_{yzx}$ | $\sigma_{zxy}$
---|---|---|---
| ($\ohm^{-1}cm^{-1}$) | ($\ohm^{-1}cm^{-1}$) | ($\ohm^{-1}cm^{-1}$)
$LaMnO_{3}(M\parallel x)$ | 17.457 | -18.671 | 14.547
$LaMnO_{3}(M\parallel z)$ | -18.677 | 14.558 | 14.471
$SrMnO_{3}(M\parallel x)$ | 1.066 | 16.657 | -3.595
$SrMnO_{3}(M\parallel z)$ | 17.050 | -3.638 | -1.019

##  V Conclusion

In summary, we measured a self-induced ISHE in highly epitaxial LSMO films and
performed first-principles calculations of the spin Hall conductivities of
bulk LSMO. The LSMO films were prepared using pulsed layer deposition and
characterized by different techniques to confirm their epitaxial nature. The
spin pumping voltage ($V_{sp}$) dominates over any other spin rectification
effects in our samples, indicating the presence of appreciable spin-to-charge
conversion in LSMO films. The spin Hall conductivity was calculated from first
principles using density functional theory and the Kubo formalism, and the
results qualitatively match our experimental findings. This study will help
shed light on spin current generation in manganite-based films.

##  VI Acknowledgments

SB, PG, AM and AS thank the Department of Atomic Energy (DAE) and SERB
(Science and Engineering Research Board), Govt. of India, for providing
financial support. PG acknowledges UGC for SRF fellowship. VA and IJP
acknowledge support from the National Science Foundation (NSF DMR-2105219).

## References

  * Liu  _et al._ (2012) L. Liu, C.-F. Pai, Y. Li, H. W. Tseng, D. C. Ralph, and R. A. Buhrman, Science 336, 555 (2012).
  * Miron  _et al._ (2010) I. M. Miron, G. Gaudin, S. Auffret, B. Rodmacq, A. Schuhl, S. Pizzini, J. Vogel, and P. Gambardella, Nature Materials 9, 230 (2010).
  * Manchon  _et al._ (2019) A. Manchon, J. Železný, I. M. Miron, T. Jungwirth, J. Sinova, A. Thiaville, K. Garello, and P. Gambardella, Rev. Mod. Phys. 91, 035004 (2019).
  * Amin  _et al._ (2020) V. P. Amin, P. M. Haney, and M. D. Stiles, J. Appl. Phys. 128, 151101 (2020).
  * Shao  _et al._ (2021) Q. Shao, P. Li, L. Liu, H. Yang, S. Fukami, A. Razavi, H. Wu, K. Wang, F. Freimuth, Y. Mokrousov, M. D. Stiles, S. Emori, A. Hoffmann, J. Åkerman, K. Roy, J.-P. Wang, S.-H. Yang, K. Garello, and W. Zhang, IEEE Transactions on Magnetics 57, 1 (2021).
  * Sinova  _et al._ (2015) J. Sinova, S. O. Valenzuela, J. Wunderlich, C. H. Back, and T. Jungwirth, Rev. Mod. Phys. 87, 1213 (2015).
  * Mosendz  _et al._ (2010) O. Mosendz, J. Pearson, F. Fradin, G. Bauer, S. Bader, and A. Hoffmann, Physical review letters 104, 046601 (2010).
  * Hahn  _et al._ (2013) C. Hahn, G. de Loubens, M. Viret, O. Klein, V. Naletov, and J. B. Youssef, Physical review letters 111, 217204 (2013).
  * Czeschka  _et al._ (2011) F. D. Czeschka, L. Dreher, M. S. Brandt, M. Weiler, M. Althammer, I.-M. Imort, G. Reiss, A. Thomas, W. Schoch, W. Limmer, _et al._ , Physical review letters 107, 046601 (2011).
  * Wegrowe and Drouhin (2011) J.-E. Wegrowe and H.-J. Drouhin, Entropy 13, 316 (2011).
  * Miao  _et al._ (2013) B. Miao, S. Huang, D. Qu, and C. Chien, Physical review letters 111, 066602 (2013).
  * Tsukahara  _et al._ (2014) A. Tsukahara, Y. Ando, Y. Kitamura, H. Emoto, E. Shikoh, M. P. Delmo, T. Shinjo, and M. Shiraishi, Physical Review B 89, 235317 (2014).
  * Humphries  _et al._ (2017) A. M. Humphries, T. Wang, E. R. J. Edwards, S. R. Allen, J. M. Shaw, H. T. Nembach, J. Q. Xiao, T. J. Silva, and X. Fan, Nat. Commun. 8 (2017), 10.1038/s41467-017-00967-w.
  * Kanagawa  _et al._ (2018) K. Kanagawa, Y. Teki, and E. Shikoh, AIP Advances 8, 055910 (2018).
  * Wang  _et al._ (2019) W. Wang, T. Wang, V. P. Amin, Y. Wang, A. Radhakrishnan, A. Davidson, S. R. Allen, T. J. Silva, H. Ohldag, D. Balzar, B. L. Zink, P. M. Haney, J. Q. Xiao, D. G. Cahill, V. O. Lorenz, and X. Fan, Nature Nanotechnology 14, 819–824 (2019).
  * Amin  _et al._ (2018) V. P. Amin, J. Zemen, and M. D. Stiles, Phys. Rev. Lett. 121, 136805 (2018).
  * Go  _et al._ (2020) D. Go, F. Freimuth, J.-P. Hanke, F. Xue, O. Gomonay, K.-J. Lee, S. Blügel, P. M. Haney, H.-W. Lee, and Y. Mokrousov, Phys. Rev. Research 2, 033401 (2020).
  * Mook  _et al._ (2020) A. Mook, R. R. Neumann, A. Johansson, J. Henk, and I. Mertig, Physical Review Research 2, 023065 (2020).
  * Yahagi  _et al._ (2021) Y. Yahagi, D. Miura, A. Sakuma, and J. Železnỳ, Physical Review B 104, 094417 (2021).
  * Salemi and Oppeneer (2022) L. Salemi and P. M. Oppeneer, Phys. Rev. B 106, 024410 (2022).
  * Amin  _et al._ (2019) V. P. Amin, J. Li, M. D. Stiles, and P. M. Haney, Physical Review B 99, 220405 (2019).
  * Omori  _et al._ (2019) Y. Omori, E. Sagasta, Y. Niimi, M. Gradhand, L. E. Hueso, F. Casanova, and Y. Otani, Physical Review B 99, 014403 (2019).
  * Taniguchi  _et al._ (2015) T. Taniguchi, J. Grollier, and M. D. Stiles, Phys. Rev. Appl. 3, 044001 (2015).
  * Das  _et al._ (2017) K. S. Das, W. Y. Schoemaker, B. Van Wees, and I. J. Vera-Marun, Physical Review B 96, 220408 (2017).
  * Safranski  _et al._ (2018) C. Safranski, E. A. Montoya, and I. N. Krivorotov, Nature Nanotechnology 14, 27 (2018).
  * Safranski  _et al._ (2020) C. Safranski, J. Z. Sun, J.-W. Xu, and A. D. Kent, Phys. Rev. Lett. 124, 197204 (2020).
  * Davidson  _et al._ (2020) A. Davidson, V. P. Amin, W. S. Aljuaid, P. M. Haney, and X. Fan, Physics Letters A 384, 126228 (2020).
  * Luo  _et al._ (2017) G. Luo, J. Lin, W.-C. Chiang, and C.-R. Chang, Scientific Reports 7, 6612 (2017).
  * Qin  _et al._ (2017) Q. Qin, S. He, W. Song, P. Yang, Q. Wu, Y. P. Feng, and J. Chen, Applied Physics Letters 110, 112401 (2017).
  * Gupta  _et al._ (2021) P. Gupta, B. B. Singh, K. Roy, A. Sarkar, M. Waschk, T. Brueckel, and S. Bedanta, Nanoscale 13, 2714 (2021).
  * Luo  _et al._ (2014) G. Luo, C. Chang, and J. Lin, Journal of Applied Physics 115, 17C508 (2014).
  * Lee  _et al._ (2016) H. K. Lee, I. Barsukov, A. Swartz, B. Kim, L. Yang, H. Hwang, and I. Krivorotov, AIP Advances 6, 055212 (2016).
  * Gupta  _et al._ (2023) P. Gupta, B. B. Singh, A. Mishra, A. Kumar, A. Sarkar, M. Waschk, and S. Bedanta, in  _SPIN_ (World Scientific, 2023).
  * Singh  _et al._ (2019) B. B. Singh, S. K. Jena, M. Samanta, K. Biswas, B. Satpati, and S. Bedanta, physica status solidi (RRL)–Rapid Research Letters 13, 1800492 (2019).
  * Luo  _et al._ (2013) G. Luo, C. Chang, and J. Lin, IEEE transactions on magnetics 49, 4371 (2013).
  * Iguchi and Saitoh (2017) R. Iguchi and E. Saitoh, journal of the physical society of japan 86, 011003 (2017).
  * Conca  _et al._ (2017) A. Conca, B. Heinz, M. Schweizer, S. Keller, E. T. Papaioannou, and B. Hillebrands, Physical Review B 95, 174426 (2017).
  * Huang  _et al._ (2020) Y.-H. Huang, Y.-C. Weng, C.-T. Liang, and J. Lin, AIP Advances 10, 015041 (2020).
  * Azevedo  _et al._ (2015) A. Azevedo, R. Cunha, F. Estrada, O. A. Santos, J. Mendes, L. Vilela-Leão, R. Rodríguez-Suárez, and S. Rezende, Physical Review B 92, 024402 (2015).
  * Tserkovnyak  _et al._ (2005) Y. Tserkovnyak, A. Brataas, G. E. Bauer, and B. I. Halperin, Reviews of Modern Physics 77, 1375 (2005).
  * Saadatkia  _et al._ (2018) P. Saadatkia, P. Stepanov, and F. Selim, Materials Research Express 5, 016202 (2018).
  * Giannozzi  _et al._ (2017) P. Giannozzi, O. Andreussi, T. Brumme, O. Bunau, M. B. Nardelli, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, M. Cococcioni, _et al._ , Journal of physics: Condensed matter 29, 465901 (2017).
  * Giannozzi  _et al._ (2009) P. Giannozzi, S. Baroni, N. Bonini, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, G. L. Chiarotti, M. Cococcioni, I. Dabo, _et al._ , Journal of physics: Condensed matter 21, 395502 (2009).
  * Giannozzi  _et al._ (2020) P. Giannozzi, O. Baseggio, P. Bonfà, D. Brunato, R. Car, I. Carnimeo, C. Cavazzoni, S. De Gironcoli, P. Delugas, F. Ferrari Ruffino, _et al._ , The Journal of chemical physics 152, 154105 (2020).
  * Pizzi  _et al._ (2020) G. Pizzi, V. Vitale, R. Arita, S. Blügel, F. Freimuth, G. Géranton, M. Gibertini, D. Gresch, C. Johnson, T. Koretsune, _et al._ , Journal of Physics: Condensed Matter 32, 165902 (2020).
  * Qiao  _et al._ (2018) J. Qiao, J. Zhou, Z. Yuan, and W. Zhao, Physical Review B 98, 214402 (2018).
  * Tsirkin (2021) S. S. Tsirkin, npj Computational Materials 7, 33 (2021).
  * Nan  _et al._ (2019) T. Nan, T. Anderson, J. Gibbons, K. Hwang, N. Campbell, H. Zhou, Y. Dong, G. Kim, D. Shao, T. Paudel, _et al._ , Proceedings of the National Academy of Sciences 116, 16186 (2019).
  * Ou  _et al._ (2019) Y. Ou, Z. Wang, C. S. Chang, H. P. Nair, H. Paik, N. Reynolds, D. C. Ralph, D. A. Muller, D. G. Schlom, and R. A. Buhrman, Nano Letters 19, 3663 (2019).
  * Jadaun  _et al._ (2020) P. Jadaun, L. F. Register, and S. K. Banerjee, Proceedings of the National Academy of Sciences 117, 11878 (2020).