smalltableaux,centertableaux,nobaseline

# Qudit circuits with $\bm{{\operatorname{SU}(d)}}$ symmetry II

Austin Hulse<EMAIL_ADDRESS>Duke Quantum Center, Duke University,
Durham, NC 27708, USA  Department of Physics, Duke University, Durham, NC
27708, USA  Hanqing Liu  0000-0003-3544-6048<EMAIL_ADDRESS>Theoretical
Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA
Iman Marvian<EMAIL_ADDRESS>Duke Quantum Center, Duke University,
Durham, NC 27708, USA  Department of Physics, Duke University, Durham, NC
27708, USA  Department of Electrical and Computer Engineering, Duke
University, Durham, NC 27708, USA

# A framework for semi-universality: Semi-universality of $3$-qudit
$\operatorname{SU}(d)$-invariant gates

Austin Hulse<EMAIL_ADDRESS>Duke Quantum Center, Duke University,
Durham, NC 27708, USA  Department of Physics, Duke University, Durham, NC
27708, USA  Hanqing Liu  0000-0003-3544-6048<EMAIL_ADDRESS>Theoretical
Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA
Iman Marvian<EMAIL_ADDRESS>Duke Quantum Center, Duke University,
Durham, NC 27708, USA  Department of Physics, Duke University, Durham, NC
27708, USA  Department of Electrical and Computer Engineering, Duke
University, Durham, NC 27708, USA

###### Abstract

Quantum circuits with symmetry-respecting gates have attracted broad interest
in quantum information science. While recent work has developed a theory for
circuits with Abelian symmetries, revealing important distinctions between
Abelian and non-Abelian cases, a comprehensive framework for non-Abelian
symmetries has been lacking. In this work, we develop novel techniques and a
powerful framework that is particularly useful for understanding circuits with
non-Abelian symmetries. Using this framework we settle an open question on
quantum circuits with $\operatorname{SU}(d)$ symmetry. We show that 3-qudit
$\operatorname{SU}(d)$-invariant gates are semi-universal, i.e., generate all
$\operatorname{SU}(d)$-invariant unitaries, up to certain constraints on the
relative phases between sectors with inequivalent representation of symmetry.
Furthermore, we prove that these gates achieve full universality when
supplemented with 3 ancilla qudits. Interestingly, we find that studying
circuits with 3-qudit gates is also useful for a better understanding of
circuits with 2-qudit gates. In particular, we establish that even though
2-qudit $\operatorname{SU}(d)$-invariant gates are not themselves semi-
universal, they become universal with at most 11 ancilla qudits. Additionally,
we investigate the statistical properties of circuits composed of random
$\operatorname{SU}(d)$-invariant gates. Our findings reveal that while
circuits with 2-qudit gates do not form a 2-design for the Haar measure over
$\operatorname{SU}(d)$-invariant unitaries, circuits with 3-qudit gates
generate a $t$-design, with $t$ that is quadratic in the number of qudits.

##  I Introduction

The universality of 2-qudit quantum gates is a celebrated result in the fields
of quantum computing and control theory [divincenzo1995two, lloyd1995almost,
deutsch1995universality, brylinski2002universal]. According to this result,
any unitary transformation on any finite number of qudits can be realized with
a finite sequence of two qudit gates. However, in the presence of a global
symmetry, this universality fails: generic unitaries that respect a global
continuous symmetry cannot be realized, even approximately, using $k$-qudit
gates that respect the same symmetry, with any fixed $k$ [Marvian2022Restrict,
marvian2022quditcircuit, Marvian2024Rotationally, marvian2024theoryabelian].
In general, the locality of gates imposes various types of restrictions on the
set of realizable unitaries. For instance, it restricts the possible relative
phases between sectors with inequivalent irreducible representations (charges)
of the symmetry. To distinguish these more common types of restrictions from
other types, Ref. [marvian2024theoryabelian] proposes the notion of semi-
universality, a weakening of the notion of universality. This concept, which
is the main focus of the present paper, is defined below (see
LABEL:sec:semiundef for the formal definition).

Recall that under the action of a symmetry group $G$, the total Hilbert space
$\mathcal{H}$ decomposes into subspaces (charge sectors),
$\mathcal{H}=\bigoplus_{\lambda}\mathcal{H}_{\lambda}$, corresponding to
inequivalent irreducible representations of $G$. A set of gates respecting
this symmetry is called _semi-universal_ , if for any unitary $V$ that
respects the symmetry, there exists a set of phases $\\{\theta_{\lambda}\\}$
such that
$V\sum_{\lambda}\operatorname{e}^{\mathrm{i}\theta_{\lambda}}\Pi_{\lambda}$
can be realized as a sequence of gates in that set, where $\Pi_{\lambda}$ is
the orthogonal projector to the charge sector $\mathcal{H}_{\lambda}$.

One may hope that even though 2-qudit gates are not generally universal, at
least they might be semi-universal. Previous works have shown that this is
indeed the case for qubits with $\operatorname{U}(1)$ and
$\operatorname{SU}(2)$ symmetry [Marvian2024Rotationally,
marvian2024theoryabelian]. In particular, in the case of SU(2) symmetry,
2-qubit gates realized by the Heisenberg exchange interaction, which is
$\operatorname{SU}(2)$-invariant, are semi-universal. On the other hand,
surprisingly, it turns out that for $d\geq 3$, 2-qudit
$\operatorname{SU}(d)$-invariant gates on qudit systems are not semi-
universal. Indeed, Ref. [marvian2022quditcircuit] identifies new conservation
laws that restrict the time evolution of the system,111In particular, in
[marvian2022quditcircuit], it is shown that in certain subspaces, the dynamics
of qudits under 2-qudit $\operatorname{SU}(d)$-invariant unitaries can be
mapped to the dynamics of a fermionic system evolving under a free (non-
interacting) Hamiltonian. even when the state is restricted to one
$\operatorname{SU}(d)$ charge sector. Since 2-qudit
$\operatorname{SU}(d)$-invariant gates are not semi-universal in general, it
is natural to ask whether $k$-local gates are; and, in particular, what is the
minimum locality that achieves semi-universality.

Ancillae | Local dimension $d$ | Semi-universality | Universality
---|---|---|---
✗ | 2 | $k=2$ | $k=2\lfloor\frac{n}{2}\rfloor$
| $d\geq 3$ | $k=3$ | $k=n$
✓ | 2 |  $k=2$ with no ancillae |  $k=2$ and $2$ ancillae
| $d\geq 3$ |  $k=2$ and $\leq 8$ ancillae |  $k=2$ and $\leq 11$ ancillae, or $k=3$ and $\leq 3$ ancillae
Table 1: Locality of gates needed to achieve (semi)-universality. This table
lists the minimum $k$, such that $k$-qudit $\operatorname{SU}(d)$-invariant
unitaries achieve semi-universality and universality on $n\geq 3$ qudits, with
or without ancillae. The results for $\operatorname{SU}(2)$ symmetry were
shown in [Marvian2024Rotationally], and the results on $\operatorname{SU}(d)$
symmetry with $d\geq 3$ are established in this work. Note that for $d=2$, 2
ancilla qudits are needed to achieve universality, even if one is allowed to
use $k$-qubit gates, with $k<2\lfloor\frac{n}{2}\rfloor$.

###  I.1 Summary of results

In this paper, we develop new powerful tools and a framework for understanding
semi-universality in quantum circuits with arbitrary symmetries. While recent
work [marvian2024theoryabelian] has found a simple characterization of
circuits with Abelian symmetries, it is known that circuits with non-Abelian
symmetries can show significantly more complicated behaviors. For instance, in
the presence of non-Abelian symmetries, the unitaries realized in one charge
sector $\mathcal{H}_{\lambda}$, may dictate the unitaries in other (possibly
multiple) sectors, whereas this cannot happen in the case of Abelian
symmetries [marvian2024theoryabelian, marvian2022quditcircuit].

Applying these tools to the important example of $\operatorname{SU}(d)$
symmetry, we settle an open question about circuits with this symmetry. It was
recently shown that 2-qudit $\operatorname{SU}(d)$-invariant gates are not
semi-universal when $d>2$, while they are for $d=2$ [marvian2022quditcircuit,
Marvian2024Rotationally]. Furthermore, using properties of the Young-Jucys-
Murphy elements and Okounkov-Vershik’s approach to the representation theory
of the symmetric group [okounkov2005], Ref. [Zheng_2023] argues that 4-local
$\operatorname{SU}(d)$-invariant unitaries are semi-universal. However, prior
to the present work, it was not known if semi-universality can be achieved
with 3-qudit unitaries or not.

Here, we settle this open question and prove that 3-qudit unitaries that
respect a global $\operatorname{SU}(d)$ symmetry on $d$-dimensional qudits are
indeed semi-universal. It is worth emphasizing that our proof of the semi-
universality of 3-qudit gates is elementary and based on the tools developed
in this paper, which are applicable to other symmetries (see
LABEL:sec:semiuni, and in particular, LABEL:MainLemma and
LABEL:thm:mulblocks).

In our construction, the generating gate set includes all 2-qudit
$\operatorname{SU}(d)$-invariant gates

$\displaystyle\exp({\mathrm{i}\theta\mathbf{P}_{ij}})\
:\quad\theta\in[0,2\pi)\ ,$ (1)

where $\mathbf{P}_{ij}$ is the SWAP operator on qudits222We label the qudits
as if they were in a chain for convenience: this geometry plays no role in the
proof of semi-universality. $i$ and $j$, and a single 3-qudit gate, e.g., one
of the unitaries

$\displaystyle
R_{+}=\exp({\mathrm{i}\pi\Pi_{\vbox{\hbox{\scalebox{0.5}{\scriptsize\ydiagram{3}}}}}})\
,\ \ \text{ or }\ \ \
R_{-}=\exp({\mathrm{i}\pi\Pi_{\vbox{\hbox{\scalebox{0.5}{\scriptsize\ydiagram{1,1,1}}}}}})\
$

Here, $\Pi_{\vbox{\hbox{\scalebox{0.5}{\scriptsize\ydiagram{3}}}}}$ and
$\Pi_{\vbox{\hbox{\scalebox{0.5}{\scriptsize\ydiagram{1,1,1}}}}}$ are,
respectively, the Hermitian projectors to the symmetric and anti-symmetric
subspaces of the three-qudit Hilbert space $(\mathbb{C}^{d})^{\otimes 3}$,
which means
$R_{+}=\mathbb{I}-2\Pi_{\vbox{\hbox{\scalebox{0.5}{\scriptsize\ydiagram{1,1,1}}}}}$
and
$R_{-}=\mathbb{I}-2\Pi_{\vbox{\hbox{\scalebox{0.5}{\scriptsize\ydiagram{1,1,1}}}}}$
are reflection unitaries, and are also permutationally invariant gates (see
Fig. 1). Here we emphasize that $R_{\pm}$ are not special: almost any single
3-qudit $\operatorname{SU}(d)$-invariant unitary together with 2-qudit
unitaries in Eq. 1 are semi-universal. LABEL:prop:2local-criteria provides a
simple characterization of 3-qudit gates that are capable of achieving semi-
universality. In particular, we find that any 3-qudit gate that cannot be
realized with 2-qudit gates is sufficient to make them universal.

It is worth noting that when restricted to a 3-qudit system, 2-qudit
$\operatorname{SU}(d)$-invariant gates already achieve semi-universality, and
the gate $R_{+}$ is just a relative phase between the $\operatorname{SU}(d)$
charge sectors. However, interestingly, when acting on 3 qudits in a system
with $n>3$ qudits, such gates can drastically change the set of realizable
unitaries and make them semi-universal.

Figure 1: Semi-universality with 3-qudit gates. Any set of
$\operatorname{SU}(d)$-invariant gates is called _semi-universal_ , if they
generate all $\operatorname{SU}(d)$-invariant unitaries, up to possible
constraints on the relative phases between sectors with inequivalent irreps of
$\operatorname{SU}(d)$. While 2-qudit $\operatorname{SU}(d)$-invariant gates
are not semi-universal for $d\geq 3$, we show that amending them with any
single generic 3-qudit unitary makes them semi-universal. In this schematic
circuit, 2-qudit gates are in the form of Eq. 1 for arbitrary $\theta$, and
$R_{+}$ is the 3-qudit reflection unitary defined in Sec. I.1.

Table 1 summarizes the results on (semi-)universality from the current paper
and [Marvian2024Rotationally, marvian2022quditcircuit]. As listed in this
table, in addition to the semi-universality of 3-qudit gates, which is shown
in LABEL:sec:4ex and LABEL:sec:over, in this work, we also prove that

  1. 1.

Without ancilla qudits, for $d>2$, $(n-1)$-qudit
$\operatorname{SU}(d)$-invariant gates are still not sufficient to achieve
universality on $n$ qudits (see LABEL:sec:imposs).

  2. 2.

With at most 8 ancilla qudits, semi-universality can be achieved with 2-qudit
$\operatorname{SU}(d)$-invariant gates when $d>2$ (see LABEL:lem:ancillauni).

  3. 3.

With at most 11 ancilla qudits, universality can be achieved with 2-qudit
$\operatorname{SU}(d)$-invariant gates when $d>2$ (see LABEL:cor:anc).

Therefore, a remarkable (and perhaps, unexpected) corollary of our study of
3-qudit $\operatorname{SU}(d)$-invariant gates is a significantly simpler
understanding of the computational universality of 2-qudit gates using ancilla
qudits! This has been recently established in [vanmeter2021universality] by
applying an advanced result in the mathematical Lie-algebraic literature by
Marin [marin2007algebre]. (This work characterizes the Lie algebra generated
by transpositions as a subalgebra of the group algebra of permutations.)

The semi-universality of 3-qudit gates allows us to characterize the group
generated by $k$-qudit $\operatorname{SU}(d)$-invariant unitaries on $n$
qudits, denoted by $\mathcal{V}_{k}^{(n)}$ for $k\geq 3$ (Note that we have
suppressed the $d$ dependence to simplify the notation). Recall that according
to the general results of [Marvian2022Restrict], $\mathcal{V}_{k}^{(n)}$ is a
compact connected Lie group. Then, as we discuss in LABEL:sec:imposs, in the
regime $n\geq k\gg d\geq 2$, the difference between the dimensions of this
group and the subgroup generated by 3-qudit $\operatorname{SU}(d)$-invariant
gates is approximately

$\displaystyle\dim\mathcal{V}_{k}^{(n)}-\dim\mathcal{V}^{(n)}_{3}\approx\frac{k^{d-1}}{d!(d-1)!}+\mathcal{O}(k^{d-2})\
.$ (2)

In Fig. 2 we plot

$\displaystyle\rho_{k,d}^{(n)}:=\frac{\dim\mathcal{V}_{k}^{(n)}-\dim\mathcal{V}_{3}^{(n)}}{\dim\mathcal{V}^{(n)}-\dim\mathcal{V}_{3}^{(n)}}\approx\Big{(}\frac{k}{n}\Big{)}^{d-1}\
,$ (3)

for various $d$ together with their asymptotic behavior, where
$\mathcal{V}^{(n)}=\mathcal{V}_{n}^{(n)}$ denotes the group of all
$\operatorname{SU}(d)$-invariant unitaries on $n$ qudits. This ratio
determines how the dimension of the Lie group of realizable unitaries grows
with $k$. In particular, $\rho_{k,d}^{(n)}=1$ means that universality is
achieved. Interestingly, when $d\geq 3$ this happens only if $k=n$, i.e.,
gates act on all qudits in the system.333On the other hand, in the case of
$d=2$, i.e., qubits with SU(2) symmetry, when $n$ is odd, the universality can
be achieved with gates acting on $n-1$ qubits. Note that the right-hand side
of Eq. 2 is independent of $n$ the number of qudits (indeed, according to the
general results of [Marvian2022Restrict], this is a consequence of the fact
that the symmetry group $G=\operatorname{SU}(d)$ is connected).

Figure 2: Convergence to universality as a function of the locality of gates.
According to the general no-go theorem of [Marvian2022Restrict], in symmetric
quantum circuits with continuous symmetries, without ancilla qudits,
universality cannot be achieved with $k$-qudit gates with a fixed $k$. On the
other hand, we show that in the case of $\operatorname{SU}(d)$ symmetry, semi-
universality is achieved with 3-qudit gates, which means for $k\geq 3$, the
only constraints on the realizable unitaries are on the relative phases
between sectors with inequivalent irreducible representations of
$\operatorname{SU}(d)$. Roughly speaking, the ratio $\rho_{k,d}^{(n)}$ defined
in Eq. 3, describes the fraction of these constraints that vanish with
$k$-qudit gates with $k\geq 3$. Universality is achieved when
$\rho_{k,d}^{(n)}=1$. Here, we plot this ratio for a system with $n=10^{4}$
qudits for different values of $d$, as specified in the plot. The dots are
$\rho_{k,d}^{(n)}$ and the lines are its asymptotic expression $(k/n)^{d-1}$.
The denominators of the ratio in Eq. 3 are $4998$, $8.3\times 10^{6}$,
$7.0\times 10^{9}$, $3.5\times 10^{12}$ and $7.9\times 10^{23}$ for
$d=2,3,4,5,10$, respectively.

In LABEL:Sec:design we also discuss the implications of this result on the
statistical properties of random quantum circuits with
$\operatorname{SU}(d)$-invariant gates. As was noted in
[marvian2022quditcircuit], the additional conservation laws, that restrict
unitaries realized by 2-qudit circuits, imply that the distribution of
unitaries generated by such random circuits is not a 2-design for the Haar
distribution over the group of $\operatorname{SU}(d)$-invariant unitaries
$\mathcal{V}^{(n)}$ (see LABEL:Sec:design for the definition of $t$-designs).
As we show in LABEL:Sec:design, a corollary of the semi-universality of
3-qudit gates is that, assuming the number of qudits is $n>d$, the
distribution of unitaries generated by random circuits formed from such gates
is a $t$-design up to $t\approx n^{2}/2$.

Finally, we note that while the main emphasis of this paper is on semi-
universality in the context of symmetric quantum circuits, many of the ideas
and techniques developed here are broadly useful in the context of quantum
computing and control theory. In particular, LABEL:MainLemma, which provides
the necessary and sufficient conditions for semi-universality, and
LABEL:thm:mulblocks, which is used for extending controllability from a
subspace to the full space, are of independent interest.

###  I.2 Outline

In LABEL:sec:semiuni we formally define semi-universality for an arbitrary
unitary symmetry group, and present a number of generally applicable tools. In
particular, we provide a necessary and sufficient condition for semi-
universality to hold in LABEL:MainLemma, and we describe how using ancillae,
semi-universality can be promoted to universality.

In LABEL:sec:semiSU we specify to the case of $\operatorname{SU}(d)$ symmetry
on $d$-dimensional qudits and state LABEL:thm:semi-universality, the semi-
universality of 3-qudit $\operatorname{SU}(d)$-invariant unitaries. We also
describe one of the main tools used in this paper, namely Schur-Weyl duality.

To explain the applications of the tools developed in LABEL:sec:semiuni, in
LABEL:sec:4ex we present a detailed discussion of the examples of $n=3$ and
$n=4$ qudits. In particular, we describe in detail how semi-universality fails
for 2-qudit gates and prove that it holds for 3-qudit gates using
LABEL:MainLemma. This serves as the base case of the induction argument for
LABEL:thm:semi-universality, which is proven in LABEL:sec:over.

In LABEL:sec:imposs, we show that, when $d\geq 3$ and without using ancilla,
universality on $n$ qudits cannot be achieved without full nonlocal control,
i.e. $n$-local gates are required. We consider the use of ancilla qudits for
achieving (semi-)universality in LABEL:sec:ancilla. The statistical properties
of circuits generated from $\operatorname{SU}(d)$-invariant $3$-local gates
are studied in LABEL:Sec:design.

Finally, in LABEL:sec:tools, we prove LABEL:MainLemma. We also describe more
general scenarios in which semi-universality does not hold, even when there is
subsystem universality on all charge sectors. In particular, in
LABEL:Mainlemma_gen we present a characterization of $G$-invariant groups that
are subsystem universal.

Table 2: Table of notations.


11institutetext: University of Chicago, Chicago IL, USA 22institutetext:
Argonne National Lab, Lemont IL, USA

# Ultrafast Focus Detection for Automated Microscopy

Maksim Levental 11 Ryan Chard 22 Kyle Chard 1122 Ian Foster Gregg Wildenberg
112222

###### Abstract

Technological advancements in modern scientific instruments, such as scanning
electron microscopes (SEMs), have significantly increased data acquisition
rates and image resolutions enabling new questions to be explored; however,
the resulting data volumes and velocities, combined with automated
experiments, are quickly overwhelming scientists as there remain crucial steps
that require human intervention, for example reviewing image focus. We present
a fast out-of-focus detection algorithm for electron microscopy images
collected serially and demonstrate that it can be used to provide near-real-
time quality control for neuroscience workflows. Our technique, Multi-scale
Histologic Feature Detection, adapts classical computer vision techniques and
is based on detecting various fine-grained histologic features. We exploit the
inherent parallelism in the technique to employ GPU primitives in order to
accelerate characterization. We show that our method can detect of out-of-
focus conditions within just 20ms. To make these capabilities generally
available, we deploy our feature detector as an on-demand service and show
that it can be used to determine the degree of focus in approximately 230ms,
enabling near-real-time use.

##  1 Introduction

A fundamental goal of neuroscience is to map the anatomical relationships of
the brain, an approach broadly called connectomics. Electron microscopy, an
imaging method traditionally limited to small single 2D images, provides
sufficient resolution to directly visualize the connections, or synapses,
between neurons. Recently, automated serial electron microscopy (SEM)
techniques have been developed where thousands, if not tens of thousands, of
individual images are automatically acquired in series and then registered
(i.e., aligned) to produce a volumetric dataset. Such datasets allow
neuroscientists to follow the tortuous path neurons take through the brain to
connect with each other (hence the name connectomics). However, many of the
steps that comprise the collection of such datasets for connectomics require
manual inspection, causing significant slowdowns in the rate at which datasets
can be acquired. Such bottlenecks significantly impact the size of the
datasets that can be reasonably acquired and studied. Furthermore, advances in
electron microscopes have increased the rate that datasets can be acquired;
for example, $\sim$ 10 Tbs/24hr [6], which, when used to map an entire, mouse
brain will result in approximately 1 exabyte of data.

Auto-focus technology is a critical component of many imaging systems; from
consumer cameras (for purposes of convenience) to industrial inspection tools
to scientific instrumentation [27]. Such technology is typically either active
or passive; active methods exploit some auxiliary device or mechanism to
measure the distance of the optics from the scene, while passive methods
analyze the definition or sharpness of an image by virtue of a proxy measure
called a criterion function. Many electron microscopes incorporate auto-focus
systems that attempt to focus the microscope before image acquisition. Despite
such functionality, out-of-focus (OOF) images still occur at high rates
(between 1% and 10%), depending on the quality of the tissue sections being
imaged. For instance, it is common to experience occasional staining
artifacts, and tears or compression artifacts (i.e., section wrinkles) during
ultra-thin serial sectioning. These imperfections can cause auto-focus systems
to fail if the microscope centers on them. This results in the system failing
to find the correct focal plane, thus necessitating post-acquisition
evaluation. These OOF error modes prevent effective automation, since a
prerequisite of many downstream transformations is that the images collected
all have high degree-of-focus (DOF). Without properly focused images, all
downstream computational steps (e.g., 2D tile montaging, 3D alignment,
automatic segmentation) will fail.

The DOF of images acquired by an electron microscope is also of critical
importance with respect to automation. While seemingly a small step in a
potential automation pipeline, focus detection is nevertheless an extremely
critical step. In general, imaging tissue sections requires loading and
unloading sets of $\sim$ 100-200 sections at a time. Failure to detect a
single OOF image in situ causes significant delays because the affected sample
sets need to be reloaded, desired field of view must be reconfigured, and
reacquired images need to be realigned into the image stack. All such
remediation steps are time and labor intensive, and effectively stops any
downstream automation until the problem is remedied. Under ideal conditions,
it is estimated that fixing a single image would take several hours of manual
intervention, which increases if multiple images in distinct parts of the
series have to be manually reacquired and aligned.

In this work we focus on ensuring images acquired by the electron microscope
have high DOF, in order to further progress towards to goal of end-to-end
automation. To this end, we propose a new technique, Multi-scale Histologic
Feature Detection (MHFD), that involves a second pass over the collected
image, after it has been acquired, using a computer vision system to detect a
failure to successfully achieve high DOF. Our technique relies on employing
feature detection [15] as a criterion function, in accordance with the
hypothesis that the quantity of features detected is positively correlated
with DOF. Using this insight, we develop a feature detector based on scale-
space representations of images (see Section 2.2) but optimized for latency.
The design and implementation of our feature detector prioritizes
parallelization, specifically in order to target GPU deployments.

Our solution achieves low latency detection of the OOF condition with high
accuracy (see Section 5). To provide access to these capabilities, we have
deployed them as a service that can be consumed on-demand and integrated in
automated workflows. The service leverages Argonne National Laboratory’s
Leadership Computing Facility to provide access to A100 GPUs to rapidly
analyze images as they are captured. This allows users to detect low quality
images and correct their collection while the sample is still in the
microscope, effectively eliminating costly delays in reloading, aligning, and
imaging the sample. An important caveat in our work: we explicitly aim to
augment existing microscopy equipment without the need for costly and complex
retrofitting. This precludes mere improvements to existing auto-focus systems
as they are, in essence, proprietary black boxes from the perspective of the
end user of an electron microscope.

The rest of this article is organized as follows: Section 2 reviews background
information on connectomics and scale-space feature detectors. Section 3
describes our focus detection method, in particular optimizations made in
order to achieve near-real-time performance. Section 4 describes how we
deliver MHFD as a service. Section 5 presents evaluation results. Section 6
discusses related work. Finally, we conclude in Section 7.

##  2 Background

We briefly review a common connectomics workflow and then describe scale-space
representations.

###  2.1 Connectomics

Connectomics is defined as the study of comprehensive maps of connections
within an organism’s nervous system (called _connectomes_). The data
acquisition pipeline for connectomics consists of the following steps:

  1. 1.

A piece of nervous system (e.g., brain), ranging from $\sim$ 1mm3 to 1 cm3 is
stained with heavy metals (e.g., osmium tetroxide, uranyl acetate, lead) in
order to provide contrast in resulting images [11];

  2. 2.

After staining, the section is dehydrated and embedded in a plastic resin to
stabilize the tissue for serial sectioning, which is performed with an
Automated Serial Sections to Tape (ATUM) device [12] (where ultrathin sections
are automatically sectioned and collected on polyimide tape);

  3. 3.

The sections are mounted to a silicon wafer, with each wafer containing
200-300 sections;

  4. 4.

The wafer is loaded into a SEM, where the user marks a region of interest
(ROI) within the sections for the microscope to image;

  5. 5.

The SEM initiates a protocol to automatically image the ROI over all the
sections at a desired resolution;

  6. 6.

For each section, the SEM attempts to auto-focus before imaging by sampling
different focal planes over a set range of focal depths

The series of collected images are then algorithmically aligned to each other
to produce a 3D volumetric image stack where biological features are segmented
either manually or by automatic segmentation techniques.

Since the imaging and post-acquisition process (e.g., retakes of blurry
images, 3D alignment, segmentation) is slow, connectomics is practically
constrained to small volumes ($\sim$ 100 µm3), but technologies are rapidly
advancing, with near future goals of mapping an entire mouse brain [1]. Even
with 100 µm3 volumes, the scope of the biological problem is large. For
instance, a single mouse neuron is estimated to receive $\sim$ 5000-7000
connections [25] and the cell density of the mouse cortex is $\sim 1.5\times
10^{5}$ cells/mm3 [10]. A 100 µm3 volume will therefore contain $\sim$ 150
neurons receiving 7.55 synapses, all of which neuroscientists seek to
automatically segment and study. For an entire mouse brain, there are $\sim
7\times 10^{6}$ neurons and $3.5\times 10^{1}0$ connections. Ensuring that
automatic segmentation algorithms accurately segment neurons depends on having
the highest possible quality images and any error in image quality is very
likely to produce segmentation errors that propagate in a non-linear fashion.
For instance, if the connection between two neurons is improperly assigned,
the other neurons that those pair of neurons connect to will also be
improperly connected, and so on. Not only is the biological scope of the
problem large, but datasets are also large, both in terms of the number of
images and data size. Again, using the range of 100 µm3 to 1 cm3 datasets,
these volumes will equate to 2,500 to 250,000 sections and $\sim$ 0.7
terabytes or 1 exabyte of data, respectively. Thus, the scope of the data both
in terms of the biological goal and data management demands automation in the
connectomic pipeline, in order to minimize errors and the need for manual OOF
detection and correction.

###  2.2 Scale-space representations

We base our multi-scale histologic feature detection technique on classical
scale-space representations of signals and images. We give a brief overview
(see [15] for a more comprehensive review). The fundamental principle of
scale-space feature detection is that natural images possess structural
features at multiple scales and features at a particular scale are isolated
from features at other scales. Thus, any image $I\left(x,y\right)$ can be
transformed into a scale-space representation $L\left(x,y,t\right)$, where
$L\left(x^{\prime},y^{\prime},t^{\prime}\right)$ represents the pixel
intensity at pixel coordinates $\left(x^{\prime},y^{\prime}\right)$ and
_scale_ $t^{\prime}$. How to construct the representation of the image at each
scale is discussed below. More importantly, such a representation lends itself
readily to scale sensitive feature detection, owing to the fact that features
at a particular scale are decoupled from features at other scales, thereby
eliminating confounding detections. Examples of structural features that can
be detected and characterized using scale-space representations include edges,
corners, ridges, and so called blobs (roughly circular regions of uniform
intensity).

A scale-space representation at a particular scale is constructed by
convolution of the image with a filter that satisfies the following
constraints: non-enhancement of local extrema, scale invariance, and
rotational invariance. Other relevant constraints are discussed in [9]. One
such filter is the symmetric, mean zero, two dimensional, Gaussian filter
[13]:

$G\left(x,y,\sigma\right)\coloneqq\frac{1}{2\pi\sigma^{2}}e^{-\frac{x^{2}+y^{2}}{2\sigma^{2}}}$

The scale-space representation $L(x,y,t)$ of an image $I(x,y)$ is defined to
be the convolution of that image with a mean zero Gaussian filter:

$L\left(x,y,t\right)\coloneqq G\left(x,y,t\right)*I\left(x,y\right)$

where $t$ determines the scale. $L(x,y,t)$ has the interpretation that image
structures of scale smaller than $\sqrt{t^{2}}=t$ have been removed due to
blurring. This is due to the fact that the variance of the Gaussian filter is
$t^{2}$ and features of this scale are therefore “beneath the noise floor” of
the filter or, in effect, suppressed by filtering procedure. A corollary is
that features with approximate length scale $t$ will have maximal response
upon being filtered by $G(x,y,t)$. That is to say, for a $t$ scale feature at
pixel coordinates $\left(x,y\right)$ and for scales
$t^{\prime}<t<t^{\prime\prime}$ we have

$L\left(x,y,t^{\prime}\right)<L\left(x,y,t\right)<L\left(x,y,t^{\prime\prime}\right)$

This is due to the fact that for scales $t^{\prime}<t$, small scale features
will dominate the response and for $t<t^{\prime\prime}$, as already mentioned,
the feature will have been suppressed.

Note that the aforementioned presumes having identified the pixel coordinates
$\left(x,y\right)$ as the locus of the feature. Hence, in order to detect
features across both scale and space dimensions, maximal responses in spatial
dimensions $\left(x,y\right)$ need to also be characterized. For such
characterization one generally employs standard calculus, in order to identify
critical points of the second derivatives of $L\left(x,y,t\right)$. Hence, we
can construct scale-sensitive feature detectors by considering critical points
of linear and non-linear combinations of spatial derivatives
$\partial_{x},\partial_{y}$ and derivatives in scale $\partial_{t}$. For
example the scale derivative of the Laplacian

$\partial_{t}\nabla^{2}L\coloneqq\partial_{t}\left(\partial_{x}^{2}+\partial_{y}^{2}\right)L$
(1)

effectively detects regions of uniform pixel intensity (i.e., blobs).

Equation (1) permits a discretization called Difference of Gaussians (DoG)
[18]:

$t^{2}\nabla^{2}L\approx t\times\left(L\left(x,y,t+\delta
t\right)-L\left(x,y,t\right)\right)$

Therefore, we define the following parameters: $n$, which determines the
granularity of the scales detected; $\min_{t}$, the minimum scale detected;
$\max_{t}$, the maximum scale detected; $\delta
t\coloneqq\left(\max_{t}-\min_{t}\right)/n$;
$t_{i}\coloneqq\min_{t}+\left(i-1\right)\times\delta t$, the discrete scales
detected. We then define the discretized DoG filter:

$\operatorname{DoG}\left(x,y,i\right)\coloneqq
t_{i}\times\left(L\left(x,y,t_{i+1}\right)-L\left(x,y,t_{i}\right)\right)$ (2)

This produces a sequence $\left\\{\operatorname{DoG}\left(x,y,i\right)\mid
i=1,\dots,n\right\\}$ of filtered and scaled images (called a Gaussian pyramid
[8]). Note that there are alternative conventions for how each difference in
the definition of $\operatorname{DoG}\left(x,y,i\right)$ should be scaled
(including partitioning into so called octaves [5]); we observe that linear
scaling is sufficient, in terms of accuracy and complexity, for the purposes
of detecting OOF conditions.

##  3 Multi-scale Histologic Feature Detection

We propose to use histologic feature detection at multiple scales as a
criterion function, reasoning that the absolute quantity of features detected
at multiple scales is positively correlated with DOF (see Figure 1). For our
particular use case, this is tantamount to detecting histologic structures
ranging from cell walls to whole organelles. The key insight is that the
ability to resolve structure across the range of feature scales is highly
correlated with a high-definition image. To this end, we develop a feature
detector based on Equation (1) but optimized for latency (rather than
accuracy).

(a) Histologic features of an in-focus section.

(b) Histologic features of an out-of-focus section.

Figure 1: Comparison of sections with histologic feature recognition as a
function of focal depth.

In order to verify our hypothesis, that detecting features across a range of
scales is correlated with DOF, we compare the number of histologic features
detected as a function of absolute deviation from in-focus ($\lvert
f-f^{\prime}\rvert$ where $f^{\prime}$ is the correct focal depth) for a
series of sections with known focal depth (see Figure 2(a)). We observe a
strong log-linear relationship (see Figure 2(b)). Fitting a log-linear
relationship produces a line with $r=-0.9754$, confirming our hypothesis that
quantity of histologic features detected is a good proxy measure for DOF. Note
that the log-linear relationship corresponds to a roughly quadratic decrease
in the number of histologic features detected. This is to be expected since,
intuitively, a twice improved DOF of a two dimensional image yields improved
detection along both spatial dimensions and thus a four times increased
quantity of histologic features detected.

(a) Number of histologic features as a function of absolute deviation from
focused ($\lvert f-f^{\prime}\rvert$ where $f^{\prime}$ is the correct focal
depth).

(b) Log plot and line fit with $r=-0.9754$.

Figure 2: Comparison of histologic feature recognition as a function of focal
depth.

Recall, we aim to achieve near-real-time quality control of SEM images to
facilitate error detection and correction while data are being collected. We
therefore require low-latency multi-scale histologic feature detection. Here
we present the design and implementation of our system that leverages GPUs to
rapidly classify images by determining their degree of focus. Our starting
point is Equation (2) for possible optimizations. Computing maxima of
$\operatorname{DoG}\left(x,y,i\right)$ in the scale dimension (equivalently
critical points of Equation (1)) necessarily entails computing maxima in a
small pixel neighborhood at every scale. We first make the heuristic
assumption that, in each pixel neighborhood that corresponds to a feature,
there is a single unique and maximal response at some scale $t$. This response
corresponds to the scale at which the variance of the Gaussian filter $G$ most
closely corresponds to the scale of the feature (see Section 2.2). We
therefore search for _local maxima_ in spatial dimensions $x,y$ but _global
maxima_ in the scale dimension:

$C:=\left\\{\left(\hat{x}_{j},\hat{y}_{j},\hat{i}_{j}\right)\right\\}\coloneqq\operatorname*{\mathtt{argmaxlocal}}_{x,y}\operatorname*{\mathtt{argmax}}_{i}\operatorname{DoG}\left(x,y,i\right)$
(3)

where the subscript $j$ indexes over the features detected. Once all such
maxima are identified it suffices to compute and report the cardinality,
$\left|C\right|$, as the criterion function value.

It is readily apparent that our histologic feature detector is parallelizable:
for each scale $t_{i}$ we can compute $L\left(x,y,t_{i}\right)$ independently
of all other $L\left(x,y,t_{j}\right)$ (for $j\neq i$). A further
parallelization is possible for the $\operatorname*{\mathtt{argmax}}$
operation, since the maxima are computed independently across distinct
neighborhoods of pixels. In order to maximally exploit this, we first perform
the inner $\operatorname*{\mathtt{argmax}}$ in Equation (3) on a block of
columns of $\left\\{\operatorname{DoG}\left(x,y,i\right)\right\\}$ in
parallel, thereby effectively reducing the Gaussian pyramid to a single image.
Note that when GPU memory is sufficient we can compute the
$\operatorname*{\mathtt{argmax}}$ across all columns simultaneously (and
otherwise within a constant number of steps). We then perform the outer
$\operatorname*{\mathtt{argmaxlocal}}_{x,y}$ on disjoint pixel neighborhoods
of the flattened image in parallel as well.

Note that the implementation of the inner $\operatorname*{\mathtt{argmax}}$ is
“free”, since the $\operatorname*{\mathtt{argmax}}$ primitive is implemented
in exactly this way in most GPGPU libraries [19], and thus our substitution of
$\operatorname*{\mathtt{argmax}}_{i}$ for
$\operatorname*{\mathtt{argmaxlocal}}_{i}$ yields a moderate latency
improvement. The outer $\operatorname*{\mathtt{argmaxlocal}}$ is implemented
using a comparison against $\operatorname{\mathtt{maxpool\\_2d}}(n,n)$ (with
$n=3$) (see [9307788] for details on this technique). Employing
$\operatorname{\mathtt{maxpool\\_2d}}$ in this way has the added benefit of
effectively performing non-maximum suppression [20], since it rejects spurious
candidate maxima within a $3\times 3$ neighborhood of a true maximum.

Typically one would compute $L(x,y,t_{i})$ in the conventional way (by
linearly convolving $G$ and $I$) but prior work has shown that performing the
convolution in the Fourier domain is much more efficient [9307788]; namely

$L\left(x,y,t_{i}\right)=\mathcal{F}^{-1}\big{\\{}\mathcal{F}\left\\{G\left(x,y,t_{i}\right)\right\\}\cdot\mathcal{F}\left\\{I\left(x,y\right)\right\\}\big{\\}}$

where
$\mathcal{F}\left\\{\cdot\right\\},\mathcal{F}^{-1}\left\\{\cdot\right\\}$ are
the Fourier transform and inverse Fourier transform, respectively. This
approach has the additional advantage that we can make use of highly optimized
Fast Fourier Transform (FFT) routines made available by GPGPU libraries.

One remaining detail is histogram stretching of the images. Due to the dynamic
range (i.e., variable bit depth) of the microscope, we need to normalize the
histogram of pixel values. We implement this normalization by saturating
$.175\%$ of the darkest pixels, saturating $.175\%$ of the lightest pixels,
and mapping the entire range to $[0,1]$. We find this gives us consistently
robust results with respect to noise and anomalous features. This histogram
normalization is also parallelized using GPU primitives. We present our
technique in Algorithm (1).

Algorithm 1 Multi-scale Histologic Feature Detection

1:$I\left(x,y\right),n,\min_{t},\max_{t},M$

2:$I^{\prime}\left(x,y\right)\coloneqq\texttt{HistorgramStretch}(I\left(x,y\right))$

3:Broadcast$\left(I^{\prime}\left(x,y\right),M\right)$

4:parfor $m\coloneqq 1,\dots,M$ do

5: parfor $i\in I_{m}$ do

6:
$L\left(x,y,t_{i}\right)\coloneqq\mathcal{F}^{-1}\big{\\{}\mathcal{F}\left\\{G\left(x,y,t_{i}\right)\right\\}\cdot\mathcal{F}\left\\{I^{\prime}\left(x,y\right)\right\\}\big{\\}}$

7: end

8:end

9:Gather$\left(L\left(x,y,t_{i}\right),M\right)$

10:parfor $i\coloneqq 1,\dots,n+1$ do

11: $\operatorname{DoG}\left(x,y,i\right)\coloneqq
t_{i}\times\left(L\left(x,y,t_{i+1}\right)-L\left(x,y,t_{i}\right)\right)$

12:end

13:$\left\\{\left(\hat{x}_{j},\hat{y}_{j},\hat{i}_{j}\right)\right\\}\coloneqq\operatorname*{\mathtt{argmaxlocal}}_{x,y}\operatorname*{\mathtt{argmax}}_{i}\operatorname{DoG}\left(x,y,i\right)$

14:DOF
$\coloneqq\left|\left\\{\left(\hat{x}_{j},\hat{y}_{j},\hat{i}_{j}\right)\right\\}\right|$

##  4 Histologic Feature Detection as a Service

A key challenge to using our histologic feature detector is that it requires
powerful GPUs with large quantities of RAM, something that many commodity GPUs
and edge devices lack. To make our detector generally accessible we have
deployed it as an on-demand service using the funcX platform [7]. funcX is a
high performance function-as-a-service platform designed to provide secure,
fire-and-forget remote execution. funcX federates access to remote research
cyberinfrastructure via a single, multi-tenant cloud service. Users submit a
function invocation request to funcX which then routes the request to the
desired _endpoint_ for execution. Endpoints may be deployed by users on remote
computing resources, including clouds, clusters, and edge devices.

We registered our MHFD tool as a funcX function, configuring it such that it
requires as input arguments only the location of the input image. The function
executes the MHFD tool on an accessible GPU and the resulting feature count
and DOF is returned asynchronously via the funcX service. Registration as a
funcX function allows others to execute the tool on their own funcX endpoints.

We enable automated invocation of the MHFD via Globus Flows [2]—a research
automation platform. funcX is accessible as a Flows Action Provider, enabling
users to deploy a flow that detects data creation, transfers data from
instrument to analysis cluster, executes the MHFD, and returns results to
users.

##  5 Evaluation

We evaluate our optimized histologic feature detector in terms of runtime
performance (in order to assess its fitness a realtime OOF detector). Data
used herein were collected using brains prepared in the same manner and as
previously described [11]. Using a commercial ultramicrotome (Powertome, RMC),
the cured block was trimmed to a ~1.0mm x 1.5 mm rectangle and ~2,000, 40nm
thick sections were collected on polyimide tape (Kapton) using an automated
tape collecting device (ATUM, RMC) and assembled on silicon wafers as
previously described [12]. Images at different focal distances were acquired
using backscattered electron detection with a Gemini 300 scanning electron
microscope (Carl Zeiss), equipped with ATLAS software for automated imaging.
Dwell times for all datasets were 1.0 microsecond.

Table 1: Test platform (ALCF ThetaGPU) CPU |  Dual AMD Rome 7742 @ 2.25GHz
---|---
GPU |  8x NVIDIA A100-40GB
HD |  4x 3.84 U.2 NVMe SSD
RAM |  1TB
Software |  CuPy-8.3.0, CUDA-11.0, NVIDIA-450.51.05

We perform runtime experiments across a range of parameters of interest
(section resolution, number of feature scales). Our test platform is a NVIDIA
DGX A100 (see Table 1). Experiments consist of computing the DOF of a sample
section for a given configuration. All experiments are repeated $k$ times
(with $k=21$) and all metrics reported are median statistics, where we discard
the first execution as it is an outlier due to various initializations (e.g,.
pinning CUDA memory).

For a section resolution of $1024\times 1024$ pixels we achieve approximately
a 50Hz runtime in the single GPU configuration; this is near-real-time. We
observe that, as expected, runtime grows linearly with the number of feature
scales and quadratically with the resolution of the section; naturally, this
is owing to the parallel architecture of the GPU. The principle defect of our
technique is that it is highly dependent on the available RAM of the GPU on
which it is deployed. In practice, most GPUs available at the edge, i.e.,
proximal to microscopy instruments, will have insufficient RAM to accommodate
large section resolutions and wide feature scale ranges. In fact, even the
40GB of the DGX’s A100 is exhausted at resolutions above $4096\times 4096$ for
more than approximately 20 feature scales.

Therefore, we further investigate parallelizing MHFD across multiple GPUs. Our
implementation parallelizes MHFD in a straightforward fashion: we partition
the set of filters across the GPUs, perform the “lighter” FFT-IFFT pair on
each constituent GPU, and then gather the results to the root GPU (arbitrarily
chosen). That is to say we actually carry out

$\left\\{L\left(x,y,t_{i}\right)\mid i\in
I_{m}\right\\}=\big{\\{}\mathcal{F}^{-1}\big{\\{}\mathcal{F}\left\\{G\left(x,y,t_{i}\right)\right\\}\cdot\mathcal{F}\left\\{I\left(x,y\right)\right\\}\big{\\}}\mid
i\in I_{m}\big{\\}}$

where for $m=1,\dots,M$ the set $I_{m}$ indexes the scales allocated to a node
$m$. By partitioning the set of Gaussian filters
$\left\\{G\left(x,y,t_{i}\right)\right\\}$ across $M$ nodes, we effectively
perform distributed filtering. We use CUDA-aware OpenMPI to implement the
distribution. Note that for such multi-GPU configurations the range of feature
scales was chosen to be a multiple of the number of GPUs (hence the
proportionally increasing sparsity of data in Figure 3(a)).

(a) Median runtime as a function of number of feature scales at resolution
$=1024\times 1024$.

(b) Median runtime as a function of section resolution with 16 feature scales.

Figure 3: Scaling experiments for runtime with respect to number of GPUs,
resolution, and number of feature scales.

We observe that, as one would expect, runtime is inversely proportional to
number of GPUs (see Figure 3(b)) but for instances where a single GPU
configuration is sufficient it is also optimal. More precise timing reveals
that parallelization across multiple GPUs incurs high network copy costs
during the gather phase (see Figure 4). Note that this latency persists even
after taking advantage of CUDA IPC [22]. In effect, this is a fairly obvious
demonstration of Amdahl’s law. Therefore, parallelization across multiple GPUs
should be considered in instances where full resolution section images are
necessary (e.g., when feature scale ranges are very wide, with detection at
the lower end of the scale being critical). In all other cases, preprocessing
by downsampling, by bilinear interpolation, in order to satisfy GPU RAM
constraints yields a more than reasonable tradeoff between accuracy and
latency.

Figure 4: Breakdown of runtime into the four major phases for two GPUs across
feature scales at resolution $=1024\times 1024$.

To evaluate the on-demand use of MHFD we deployed a funcX endpoint on the
ThetaGPU cluster (Table 1). We registered our MHFD tool as a funcX function
and passed the path to the test dataset as input. It is important to note that
the SEM imagery is not passed through the funcX platform in these tests but
instead is assumed data to be resident on the shared file system. We performed
over 1000 invocations of the function after first executing a task to
initialize the environment. The tests use a single GPU and a dataset with
section resolution of $1024\times 1024$ pixels to be comparable to earlier
results. Our findings show that the mean time to perform the feature detection
was $25$ms, with a standard deviation of $6$ms, which is similar to the
previous result for the same dataset when not using funcX. The mean time taken
to submit the request to the funcX service and retrieve the result was
$233$ms, with a standard deviation of $44$ms, meaning the funcX platform
introduces an overhead of approximately $200$ms. While this overhead is
roughly $8\times$ greater than the MHFD analysis itself, the total time
required to determine the focus of a dataset is suitable to classify images
and report errors as data is collected. In practice, data must also be moved
to the ThetaGPU cluster’s filesystem before it can be analyzed. When
evaluating the time to transfer the single 23MB image from Argonne’s
Structural Biology Center (where the SEM resides) to the ThetaGPU cluster, we
found it was moved at over 200MB/s and took roughly $100$ms. This is due to
the two 40Gb/s connections between the SBC and computing facility.

##  6 Related work

Automating the control and optimization of scientific instruments is a common
area of research that spans a diverse collection of devices and applies an
equally diverse set of techniques, including HPC analysis, ML-in-the-loop
[21], and edge-accelerated processing [16]. Laszewski et al. [14] and Bicer et
al. [4] demonstrate two approaches to perform real-time processing of
synchrotron light source data in order to steer experiments. Both of these
cases employ HPC systems to rapidly analyze and reconstruct data to guide
instruments toward areas of interest. Others have also used FPGAs to act on
streams of instrument imagery [24].

There is also much work in developing and improving auto-focus algorithms and
their applications to microscopy. Yeo et al. [26] was one of the first
investigations of applying auto-focus to microscopy. They compare several
criterion functions and conclude that the so-called Tenengrad criterion
function is most accurate and most robust to noise. The crucial difference
between their evaluation criteria and ours is they select for criterion
functions that are suited for optical microscopy, i.e., criterion functions
that are robust to staining/coloring (whereas all of our samples are
grayscale). Redondo et al. [23] reviews sixteen criterion functions and their
computational cost in the context of automated microscopy. Bian et al. [3]
address the same issues that motivate us in that they aim to support automated
processes in the face of topographic variance in the samples (which leads to
comparing OOF rates). Their solutions distinguish themselves in that they
employ active devices (such as low-coherence interferometry). Interestingly,
seemingly contemporaneously with our project Luo et al. [17] proposed a deep
learning architecture that auto-focuses in a “single-shot” manner. Such a
solution is appealing given the affinity with our own application of GPGPU to
the problem and we intend to experiment with applying it to our data.

##  7 Conclusions

We presented an OOF detection technique designed to augment existing
microscopy instrumentation. Rather than focusing the microscope, as auto-
focusing algorithms would, our algorithm operates downstream of image
acquisition to report out-of-focus events to the user. This enables the user
to intervene and initiate reacquisition protocols (on the microscope) before
unknowingly proceeding with collecting the next series of images or proceeding
with downstream image processing and analysis. Our technique is effective and
operates at near-real-time latencies. Thus, this human-in-the-loop remediation
protocol already saves the user much wasted collection time triaging defective
collection runs.

## Acknowledgements

This work was supported by the U.S. Department of Energy, Office of Science,
under contract DE-AC02-06CH11357.

## References

  * [1] Abbott, L.F., Bock, D.D., Callaway, E.M., Denk, W., Dulac, C., Fairhall, A.L., Fiete, I., et al.: The mind of a mouse. Cell 182(6), 1372–1376 (2020)
  * [2] Ananthakrishnan, R., Blaiszik, B., Chard, K., Chard, R., McCollam, B., Pruyne, J., Rosen, S., Tuecke, S., Foster, I.: Globus platform services for data publication. In: Practice and Experience on Advanced Research Computing. pp. 14:1–14:7 (2018)
  * [3] Bian, Z., Guo, C., Jiang, S., Zhu, J., Wang, R., Song, P., Zhang, Z., Hoshino, K., Zheng, G.: Autofocusing technologies for whole slide imaging and automated microscopy. Journal of Biophotonics 13(12), e202000227 (2020)
  * [4] Bicer, T., Gursoy, D., Kettimuthu, R., Foster, I.T., Ren, B., De Andrede, V., De Carlo, F.: Real-time data analysis and autonomous steering of synchrotron light source experiments. In: 13th International Conference on e-Science (e-Science). pp. 59–68. IEEE (2017)
  * [5] Burt, P., Adelson, E.: The laplacian pyramid as a compact image code. IEEE Transactions on Communications 31(4), 532–540 (1983)
  * [6] Carl Zeiss AG: ZEISS MultiSEM Research Partner Program (10 2018), the World’s Fastest Scanning Electron Microscopes
  * [7] Chard, R., Babuji, Y., Li, Z., Skluzacek, T., Woodard, A., Blaiszik, B., Foster, I., Chard, K.: FuncX: A federated function serving fabric for science. In: Proceedings of the 29th International symposium on high-performance parallel and distributed computing. pp. 65–76 (2020)
  * [8] Derpanis, K.G.: The gaussian pyramid (2005)
  * [9] Duits, R., Florack, L., De Graaf, J., ter Haar Romeny, B.: On the axioms of scale space theory. Journal of Mathematical Imaging and Vision 20(3), 267–298 (2004)
  * [10] Herculano-Houzel Suzana, Watson Charles, P.G.: Distribution of neurons in functional areas of the mouse cerebral cortex reveals quantitatively different cortical zones. Frontiers in Neuroanatomy 7 (2013)
  * [11] Hua, Y., Laserstein, P., Helmstaedter, M.: Large-volume en-bloc staining for electron microscopy-based connectomics. Nature communications 6(1), 1–7 (2015)
  * [12] Kasthuri, N., Hayworth, K.J., Berger, D.R., Schalek, R.L., Conchello, J.A., Knowles-Barley, S., Lee, D., Vázquez-Reina, A., Kaynig, V., Jones, T.R., et al.: Saturated reconstruction of a volume of neocortex. Cell 162(3), 648–661 (2015)
  * [13] Koenderink, J.J.: The structure of images. Biological cybernetics 50(5), 363–370 (1984)
  * [14] von Laszeski, G., Insley, J.A., Foster, I., Bresnahan, J., Kesselman, C., Su, M., Thiebaux, M., et al.: Real-time analysis, visualization, and steering of microtomography experiments at photon sources. Tech. rep., Argonne National Lab. (2000)
  * [15] Lindeberg, T.: Feature detection with automatic scale selection. International Journal of Computer Vision 30, 79–116 (2004)
  * [16] Liu, Z., Ali, A., Kenesei, P., Miceli, A., Sharma, H., Schwarz, N., Trujillo, D., Yoo, H., Coffee, R., Layad, N., et al.: Bridging data center ai systems with edge computing for actionable information retrieval. In: 3rd Annual Workshop on Extreme-scale Experiment-in-the-Loop Computing (XLOOP). pp. 15–23. IEEE (2021)
  * [17] Luo, Y., Huang, L., Rivenson, Y., Ozcan, A.: Single-shot autofocusing of microscopy images using deep learning. ACS Photonics 8(2), 625–638 (2021)
  * [18] Marr, D., Hildreth, E.: Theory of edge detection. Proceedings of the Royal Society of London. Series B. Biological Sciences 207(1167), 187–217 (1980)
  * [19] Merrill, D.: Cuda unbound (cub). https://github.com/NVIDIA/cub (2021)
  * [20] Neubeck, A., Van Gool, L.: Efficient non-maximum suppression. In: 18th International Conference on Pattern Recognition. vol. 3, pp. 850–855. IEEE (2006)
  * [21] Pan, J., Libera, J.A., Paulson, N.H., Stan, M.: Flame stability analysis of flame spray pyrolysis by artificial intelligence. The International Journal of Advanced Manufacturing Technology 114(7), 2215–2228 (2021)
  * [22] Potluri, S., Wang, H., Bureddy, D., Singh, A.K., Rosales, C., Panda, D.K.: Optimizing mpi communication on multi-gpu systems using cuda inter-process communication. In: 26th International Parallel and Distributed Processing Symposium Workshops PhD Forum. pp. 1848–1857 (2012)
  * [23] Redondo, R., Cristóbal, G., Garcia, G.B., Deniz, O., Salido, J., del Milagro Fernandez, M., Vidal, J., Valdiviezo, J.C., Nava, R., Escalante-Ramírez, B., Garcia-Rojo, M.: Autofocus evaluation for brightfield microscopy pathology. Journal of Biomedical Optics 17(3), 1 – 9 (2012)
  * [24] Stevanovic, U., Caselle, M., Cecilia, A., Chilingaryan, S., Farago, T., Gasilov, S., Herth, A., Kopmann, A., Vogelgesang, M., Balzer, M., et al.: A control system and streaming daq platform with image-based trigger for x-ray imaging. IEEE Transactions on Nuclear Science 62(3), 911–918 (2015)
  * [25] Wildenberg, G.A., Rosen, M.R., Lundell, J., Paukner, D., Freedman, D.J., Kasthuri, N.: Primate neuronal connections are sparse in cortex as compared to mouse. Cell Reports 36(11), 109709 (2021)
  * [26] Yeo, T., Ong, S., Jayasooriah, Sinniah, R.: Autofocusing for tissue microscopy. Image and Vision Computing 11(10), 629–639 (1993)
  * [27] Yu Sun, Duthaler, S., Nelson, B.J.: Autofocusing algorithm selection in computer microscopy. In: International Conference on Intelligent Robots and Systems. pp. 70–76 (2005)


# GigaPevt: Multimodal Medical Assistant

Pavel Blinov1 Konstantin Egorov1 Ivan Sviridov1 Nikolay Ivanov1 Stepan Botman1
Evgeniy Tagin1 Stepan Kudin1 Galina Zubkova1&Andrey Savchenko1
1Sber AI Lab
{Blinov.P.D, Egorov.K.Ser, IAnatoSviridov, NikIvanov, SABotman, EyTagin,
SSKudin, GVZubkova<EMAIL_ADDRESS>
###### Abstract

Building an intelligent and efficient medical assistant is still a challenging
AI problem. The major limitation comes from the data modality scarceness,
which reduces comprehensive patient perception. This demo paper presents the
GigaPevt, the first multimodal medical assistant that combines the dialog
capabilities of large language models with specialized medical models. Such an
approach shows immediate advantages in dialog quality and metric performance,
with a 1.18% accuracy improvement in the question-answering task.

##  1 Introduction

The availability of high-quality healthcare remains a significant social
problem for many countries due to high-cost issues, shortage of medical staff,
or inaccessibility in far-end regions. From a technological point of view, one
possible solution lies in developing intelligent medical assistants. There are
many attempts to build such an assistant in the form of a symptom checker
application (Ada Health, K Health, Symptomate, Babylon Health) Wallace et al.
(2022). The major limitation of such systems is using the only text modality,
where patients describe the symptoms and get answers in the chat. The system
misses a significant part of the non-verbal doctor-patient communication
context in that case. Another flaw of such AI assistants is the design in a
survey systems paradigm with predefined questions that miss the human touch in
the communication process.

However, the rapid development of large language models (LLMs) opens new ways
to create next-generation assistants in healthcare. In this research, we
introduce an innovative multimodal medical assistant called GigaPevt (Figure
1).111The GigaPevt name is a wordplay in many facets. First, the GP stands for
General Practitioner. Second, in Russian, the word consonant to a therapist
with the smaller metric prefix Giga- instead of Tera-. Third, it references
the core Russian LLM GigaChat. The GigaPevt combines specialized medical
models with the rich dialog capabilities of LLM, which account for broad
context in visual, audio, and text modalities. That allows us to mitigate the
problems mentioned above, significantly improves patient interaction
experience, and boosts the quality of dialog.

Figure 1: GigaPevt UI

The GigaPevt demonstration video is available at https://youtu.be/yETMJqIThPE.

##  2 GigaPevt architecture

Given the assortment of potential deployment scenarios, we implemented the
GigaPevt as a client-server application, as shown in Figure 2. We use a rich
client written in Python to run lightweight models that require low latency
for a comfortable user experience. These are the face detector, Text-to-Speech
(TTS), and Automatic Speech Recognition (ASR) API
services.222https://developers.sber.ru/docs/ru/salutespeech/category-overview
Client logic handles data flow, client state management, and network
interactions with the rest of the system.

We use Flask as a back-end framework that handles the server side. The server
has a model manager, specialized models, and the GP Dialog Logic component
responsible for the patient-assistant interaction.

Figure 2: GigaPevt architecture

##  3 Specialized models

###  3.1 Video-based Facial Analytics

We analyze facial videos in a frame-wise way to identify a user and predict
the attributes list. The facial region is detected using the
MediaPipe333https://developers.google.com/mediapipe library, and the obtained
facial image is fed into the corresponding neural network.

####  3.1.1 User Identification

We apply open-source face recognition model to identify, authorize, and manage
the user session. The facial image is resized to 224x224 and fed into the fast
OFAMobileNetV3 model Savchenko et al. (2023) to extract facial descriptors.
The latter model was pre-trained on the part of the VGGFace2 dataset Cao et
al. (2018). The training dataset contains feature vectors of known subjects
obtained by the same procedure given their facial photos. Extracted facial
embeddings are normalized using $L_{2}$ norm and matched with a training
dataset by the k-NN with the Euclidean distance. In the experimental study, we
gathered a testing dataset with 426 photos of 104 persons. As a result, the
recognition accuracy equals 98.19%.

Question |  GigaChat answer |  LLaMA answer |  GigaPevt answer
---|---|---|---
Is my heart rate comfortable for exercise? |  To determine a comfortable heart rate when exercising, you need to consider your age and level of physical fitness. Generally, for people aged 20 to 59 years, exercising at a heart rate between 50% and 85% of your maximum heart rate is recommended. |  The normal heart rate for an adult is between 60 and 100 beats per minute. If your heart rate is within this range, has a regular rhythm, and you do not feel discomfort during exercise, it may be comfortable for you. However, consult your doctor for testing and advice if you have any doubts. |  No, your heart rate is currently not comfortable for exercise. Your heart rate is 115 beats per minute, which is higher than normal. For comfortable physical exercise, it is recommended to have a heart rate ranging from 60 to 100 beats per minute. I suggest you consult a doctor to determine the optimal level of physical activity for you.
What do you think? Am I obese? |  To determine if you are obese, you need to know your weight and height. Obesity is defined as excess accumulation of fat, which can lead to various diseases and poor quality of life. |  As an artificial intelligence model, I cannot determine your weight or estimate your body. However, if you suspect you are obese, it is best to see your doctor to determine your body mass index (BMI) and get advice. The most important thing is not to put off visiting your doctor if you have concerns about your health. |  Based on your body mass index, which is within the normal range, you are not obese. However, if you have other signs or symptoms, such as fainting, dizziness, or shortness of breath, it is recommended that you consult your doctor.
Table 1: Examples of answers to health-related questions (translated from
Russian)

####  3.1.2 Socio-Demographic Model

We used a high-performance MobileNet-V1 model to predict age, gender, and
ethnicity Savchenko (2021). It was pre-trained to recognize faces from the
VGGFace2 dataset Cao et al. (2018). Unfortunately, the age groups in this
dataset are very imbalanced, so the trained models work incorrectly for the
faces of very young or old people. Hence, we decided to add all (15K) images
from the Adience dataset Eidinger et al. (2014). As the latter contains only
age intervals, e.g., (60-100), we put all images from this interval to the
average age, i.e., 80. The ethnicity classifier was trained on the subset of
the UTKFace dataset Zhang et al. (2017) with different class weights to
achieve better performance for imbalanced classes. As a result, we obtained
gender classification accuracy (93.79%), which is 2.7% greater than the best-
known accuracy of a much larger VGG-16 neural network in the DEX method Rothe
et al. (2015). The age prediction MAE (Mean Absolute Error) is equal to 5.74
years, which is 0.7 years better when compared to the DEX. Finally, the
accuracy of ethnicity prediction is equal to 87.6%, which is on par with the
best-known models Savchenko (2021).

####  3.1.3 Facial Expression Recognition

The EfficientNet-B0 model was used to predict facial emotions Savchenko
(2022). In particular, we pre-trained it to identify faces on the VGGFace2
dataset as in the previous subsection. Next, we add three heads for predicting
eight basic expressions (Anger, Contempt, Disgust, Fear, Happiness, Neutral,
Sadness, and Surprise), valence, and arousal using the training part of the
AffectNet dataset Mollahosseini et al. (2017). In contrast to typical face
recognition training, we use highly cropped faces in both training stages to
remove hair, background, etc., and concentrate on facial expressions only.
Thus, we obtained competitive results on the validation part of the AffectNet
dataset. In particular, our accuracy for eight classes equals 61.93%, 0.5-2%
greater than that of models of similar size and efficiency Pourmirzaei et al.
(2021); Savchenko (2021).

####  3.1.4 Body Mass Index Model

The Body Mass Index (BMI) estimation is addressed as an image regression
problem. The dataset for this task includes samples (face-BMI value) from
Reddit,444https://www.reddit.com/r/progresspics a public social web platform
enriched with the FIW-BMI dataset Jiang et al. (2019). The detected faces are
resized to $256\times 256$ and fed into the ResNet34 model He et al. (2015) as
a backbone with MSE as a metric and loss function. On the test set, we get the
MSE equal to 4 units.

####  3.1.5 rPPG Model

The remote photoplethysmography (rPPG) model allows extracting a
photoplethysmography signal using a video of a user’s face. Our system used
the Plane Orthogonal-to-Skin (POS) algorithm described in Wang et al. (2017).
This algorithm is the strong baseline in the field; it adopts signal
processing techniques on subtle changes of the RGB values in the time domain.
The algorithm runs based on a face landmark detection system, as it relies on
averaging pixel values change over face regions of interest (cheeks, nose, and
forehead), which are the most valuable for pulse wave detection Wang et al.
(2018). According to Liu et al. (2023), this method has high accuracy,
robustness, and speed. Calculating rPPG also allows us to infer several
derived parameters of the state of the cardiovascular system, such as heart
rate, RR intervals (intervals between consecutive heartbeats), stress index,
PNN50, and many others. All these health indicators allow GigaPevt to
determine the user’s condition more accurately and comprehensively, improving
the patient interaction experience.

###  3.2 GP Dialog Logic

To manage and control the dialog, we used the GigaChat Pro
model555https://developers.sber.ru/portal/products/gigachat. As the assistant
continuously receives a stream of additional parameters from sensors, we can
substantially enrich the prompts. Even the simple inclusion of such augmented
parameters in a text prompt is helpful. But our ultimate goal is to obtain
controllable, high-quality dialog focused on the anamnesis collection. To
achieve this goal, we resorted to the Retrieval-Augmented Generation (RAG)
technique Lewis et al. (2020) as it can help supplement the user’s request
with the found relevant information and the Chain of Thoughts (CoT) Wei et al.
(2023) as it was successfully used in challenging NLP scenarios Li et al.
(2023). We started to assemble a database of medical dialogues and questions.
These improvements substantially boost the quality of answers and performance
in downstream health-related tasks.

##  4 Experiments

For a diverse assessment of the GigaPevt assistant, we evaluate it from
quantitative and qualitative sides.

###  4.1 Performance Evaluation

We evaluate the proposed assistant on the two most relevant downstream tasks
from the RuMedBench Blinov et al. (2022).

The RuMedDaNet is a binary question-answering task with 512 test medical-
related questions in Russian. As shown in Table 2, the GigaPevt model with the
RAG technique substantially outperforms GigaChat and Human assessment by 1.18%
and 0.39%, respectively. It is the most indicative task for assessing the
increase in the quality of dialogue because it is close to our assistant’s
target scenario.

The RuMedNLI is a task of natural language inference with 1536 Russian text
pairs to decide whether the second text in a pair is entailment,
contradiction, or neutral to the first. For this task, the performance gain is
positive but relatively moderate (0.38%).

Model | RuMedDaNet | RuMedNLI
---|---|---
GigaPevt | 93.36 | 71.59
Human | 92.97 | 85.67
GigaChat | 92.18 | 71.21
ChatGPT-3.5 | 89.26 | 61.33
Table 2: LLMs performance metrics (Accuracy, %). The best results are shown in
bold.

###  4.2 Dialog Showcases

Table 1 lists several examples to assess the qualitative gap in the answers
between the GigaChat model, open LLaMA Touvron et al. (2023) and GigaPevt. The
GigaChat and LLaMA answers are the base model answers in a zero-shot mode
without additional information. GigaPevt answers is the model with visual
input, RAG, and CoT for enriching our prompts. Tuned prompts have the
following form: constitution (description of how the model should behave and
its role) + visual information about a person (pulse, body mass index, etc.) +
postfix (recommendations for model).

We can see that the model with prompt tuning, RAG, visual information from the
camera, etc., allows us to answer questions that require a visual inspection
immediately. Due to generally more awareness about the patient, our assistant
gives more specific and concrete answers.

##  5 Conclusion and Future Work

This demo aims to inspire multimodality research in the medical domain. We
demonstrate the prototype of medical assistant technology, which processes a
rich multimodal context. Several experiments confirm assistant effectiveness.

We plan to use more advanced knowledge management techniques for LLMs, such as
Advanced and Modular RAG Gao et al. (2024), to make our model more complete
and focused on the medical domain. Also, our plan includes closer integration
with a patient electronic health record for better context awareness and the
ability to target more specific questions.

## References

  * Blinov et al. [2022] Pavel Blinov, Arina Reshetnikova, Aleksandr Nesterov, Galina Zubkova, and Vladimir Kokh.  Rumedbench: a russian medical language understanding benchmark.  In International Conference on Artificial Intelligence in Medicine, pages 383–392. Springer, 2022.
  * Cao et al. [2018] Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman.  VGGFace2: A dataset for recognising faces across pose and age.  In International Conference on Automatic Face and Gesture Recognition, 2018.
  * Eidinger et al. [2014] Eran Eidinger, Roee Enbar, and Tal Hassner.  Age and gender estimation of unfiltered faces.  IEEE Transactions on Information Forensics and Security, 9(12):2170–2179, 2014.
  * Gao et al. [2024] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang.  Retrieval-augmented generation for large language models: A survey, 2024\.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * Jiang et al. [2019] Min Jiang, Yuanyuan Shang, and Guodong Guo.  On visual bmi analysis from facial images.  Image and Vision Computing, 89:183–196, 2019.
  * Lewis et al. [2020] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc., 2020.
  * Li et al. [2023] Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter.  Chain of code: Reasoning with a language model-augmented code emulator, 2023.
  * Liu et al. [2023] Xin Liu, Girish Narayanswamy, Akshay Paruchuri, Xiaoyu Zhang, Jiankai Tang, Yuzhe Zhang, Soumyadip Sengupta, Shwetak Patel, Yuntao Wang, and Daniel McDuff.  rppg-toolbox: Deep remote ppg toolbox, 2023.
  * Mollahosseini et al. [2017] Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor.  AffectNet: A database for facial expression, valence, and arousal computing in the wild.  IEEE Transactions on Affective Computing, 10(1):18–31, 2017.
  * Pourmirzaei et al. [2021] Mahdi Pourmirzaei, Gholam Ali Montazer, and Farzaneh Esmaili.  Using self-supervised auxiliary tasks to improve fine-grained facial representation.  arXiv preprint arXiv:2105.06421, 2021.
  * Rothe et al. [2015] Rasmus Rothe, Radu Timofte, and Luc Van Gool.  DEX: Deep expectation of apparent age from a single image.  In Proceedings of the International Conference on Computer Vision (CVPR) Workshops, pages 10–15. IEEE, 2015.
  * Savchenko et al. [2023] Andrey V. Savchenko, Lyudmila V. Savchenko, and Ilya Makarov.  Fast search of face recognition model for a mobile device based on neural architecture comparator.  IEEE Access, 11:65977–65990, 2023.
  * Savchenko [2021] Andrey V Savchenko.  Facial expression and attributes recognition based on multi-task learning of lightweight neural networks.  In Proceedings of the 19th International Symposium on Intelligent Systems and Informatics (SISY), pages 119–124. IEEE, 2021.
  * Savchenko [2022] Andrey V Savchenko.  MT-EmotiEffNet for multi-task human affective behavior analysis and learning from synthetic data.  In Proceedings of European Conference on Computer Vision (ECCV) Workshops, pages 45–59. Springer, 2022.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * Wallace et al. [2022] William Wallace, Calvin Chan, Swathikan Chidambaram, Lydia Hanna, Fahad Mujtaba Iqbal, Amish Acharya, Pasha Normahani, Hutan Ashrafian, Sheraz R Markar, Viknesh Sounderajah, et al.  The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review.  NPJ Digital Medicine, 5(1):118, 2022.
  * Wang et al. [2017] Wenjin Wang, Albertus C. den Brinker, Sander Stuijk, and Gerard de Haan.  Algorithmic principles of remote ppg.  IEEE Transactions on Biomedical Engineering, 64(7):1479–1491, 2017\.
  * Wang et al. [2018] Chen Wang, Thierry Pun, and Guillaume Chanel.  A comparative survey of methods for remote heart rate detection from frontal face videos.  Frontiers in Bioengineering and Biotechnology, 6:33, 05 2018.
  * Wei et al. [2023] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models, 2023.
  * Zhang et al. [2017] Zhifei Zhang, Yang Song, and Hairong Qi.  Age progression/regression by conditional adversarial autoencoder.  In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pages 5810–5818. IEEE, 2017.