††thanks: These authors have contributed equally to this work.††thanks: These
authors have contributed equally to this work.

# Experimental Determination of Electronic States via Digitized Shortcut-to-
Adiabaticity

Ze Zhan  Zhejiang Province Key Laboratory of Quantum Technology and Device,
Department of Physics, Zhejiang University, Hangzhou, 310027, China  Chongxin
Run  Zhejiang Province Key Laboratory of Quantum Technology and Device,
Department of Physics, Zhejiang University, Hangzhou, 310027, China  Zhiwen
Zong  Zhejiang Province Key Laboratory of Quantum Technology and Device,
Department of Physics, Zhejiang University, Hangzhou, 310027, China  Liang
Xiang  Zhejiang Province Key Laboratory of Quantum Technology and Device,
Department of Physics, Zhejiang University, Hangzhou, 310027, China  Ying Fei
Zhejiang Province Key Laboratory of Quantum Technology and Device, Department
of Physics, Zhejiang University, Hangzhou, 310027, China  Zhenhai Sun
Zhejiang Province Key Laboratory of Quantum Technology and Device, Department
of Physics, Zhejiang University, Hangzhou, 310027, China  Yaozu Wu  Zhejiang
Province Key Laboratory of Quantum Technology and Device, Department of
Physics, Zhejiang University, Hangzhou, 310027, China  Zhilong Jia  Key
Laboratory of Quantum Information, University of Science and Technology of
China, Hefei, 230026, China  Peng Duan  Key Laboratory of Quantum Information,
University of Science and Technology of China, Hefei, 230026, China  Jianlan
Wu<EMAIL_ADDRESS>Zhejiang Province Key Laboratory of Quantum
Technology and Device, Department of Physics, Zhejiang University, Hangzhou,
310027, China  Yi Yin<EMAIL_ADDRESS>Zhejiang Province Key Laboratory of
Quantum Technology and Device, Department of Physics, Zhejiang University,
Hangzhou, 310027, China  Guoping Guo<EMAIL_ADDRESS>Key Laboratory of
Quantum Information, University of Science and Technology of China, Hefei,
230026, China  Origin Quantum Computing, Hefei, 230026, China

###### Abstract

A digitized shortcut-to-adiabaticity (STA) method is implemented in a
superconducting quantum device to determine electronic states in two example
systems, the H2 molecule and the topological Bernevig-Hughes-Zhang (BHZ)
model. For H2, a short internuclear distance is chosen as a reference point,
at which the ground and excited states are obtained within a few steps of the
digitized STA. The relation between the electronic states and the internuclear
distance is subsequently determined by treating the states at the reference
distance as the initial states in a one-step digitized STA process. The same
approach is applied to the BHZ model, and the valence and conduction bands are
excellently obtained based on two pre-determined reference points.

The rapid advancement in quantum computation has shed the light of rich
practical applications ChuangBook ; GoogleNat19 . In a conventional computer,
the computational burden of an electronic state increases exponentially with
the system size. The evolution of a quantum state in a quantum computing
device can circumvent the difficulty of searching a multi-dimensional Hilbert
space ChuangBook . The electronic ground state can be determined by the
criterion of the lowest energy, from which a variational quantum eigensolver
(VQE) protocol has been proposed for a quantum computing device PeruzzoNC14 .
The VQE has been experimentally implemented to calculate various systems
ranging from small molecules (e.g., H2 MalleyPRX16 , LiH and BeH2 KandalaNat17
) to an atom chain of H12 GoogleSci20 . Due to the intrinsic requirement of
energy minimization, it is more difficult to calculate the electronic excited
states with VQE. In a recent experiment, the excited eigenstates of H2 are
determined in a space composed of local states excited from the ground state
CollessPRX18 .

An important family of quantum computing methods is rooted in quantum
adiabaticity FarhiSci01 , in which a quantum system evolves along its
instantaneous eigenstate under a slowly varying Hamiltonian, and both dynamic
and geometric phases are accumulated over time. In additional to geometric
gates, adiabatic algorithms has been applied to the state preparation, quantum
annealing, etc ArnabRMP08 . To accelerate the adiabatic process and avoid non-
adiabatic errors, a shortcut-to-adiabaciticy (STA) has been proposed with the
assistance of a counter-diabatic Hamiltonian BerryJPhysA09 ; XChenPRL2010 .
Despite its theoretical feasibility, the STA implementation has been well
studied in a single-qubit system ZZXPRA17 ; SCPMA2018 ; WangNJP2018 ;
ZZXNJP2018 , but becomes difficult in multi-qubit systems due to the
experimental difficulty of creating a complicated STA Hamiltonian. On the
other hand, digital quantum computing paradigm based on the Trotter-Suzuki
decomposition offers a universal tool for various quantum problems BarendsNC15
; BarendsNat16 . With a continual improvement in equipments and algorithms,
the digital quantum computing has gradually become practically available,
e.g., in the VQE determination of the H12 chain GoogleSci20 . A digitized
algorithm was recently proposed to efficiently reproduce a multi-qubit STA
process, and was tested in the preparation of entangled two-qubit states
chenxiPRAPP20 . In this Letter, we apply the digitized STA method to
experimentally determine both ground and excited electronic states in two
example systems, the molecule H2 and the topological Bernevig-Hughes-Zhang
(BHZ) model ZhangSci06 . Especially, the experimentally determined states at
one or a few reference points serves a good choice of initial STA states to
subsequently determine electronic states in the whole parameter space.

Figure 1: A schematic diagram of digitized STA which arises from an adiabatic
process and its corresponding STA process.

A digitized STA is constructed from an adiabatic process (see Fig. 1). To
extract the eigenergies and eigenstates
$\\{\varepsilon_{n},|\varphi_{n}\rangle\\}$ of a target Hamiltonian $H$, we
can begin with a simple Hamiltonian $H_{0}$ and design a time-varying
Hamiltonian,

$\displaystyle H_{\mathrm{ad}}(t)=H_{0}+\lambda(t)(H-H_{0}),$ (1)

where the coefficient satisfies $\lambda(t\\!=\\!0)\\!=\\!0$ and
$\lim_{T\rightarrow\infty}\lambda(t\\!=\\!T)\\!=\\!1$. If the initial state is
an eigenstate, $|\Psi(0)\rangle=|\varphi_{n}(H_{0})\rangle$, the adiabatic
evolution spontaneously drags the quantum system to its final eigenstate,
$|\Psi(t\\!=\\!T)\rangle\\!\propto\\!|\varphi_{n}(H)\rangle$. For the STA
process within a finite operation time $T$, a counter-diabatic Hamiltonian is
introduced as

$\displaystyle
H_{\mathrm{cd}}(t)=i\sum_{n}\left[|\dot{\varphi}_{n}(t)\rangle\langle\varphi_{n}(t)|-\langle\varphi_{n}(t)|\dot{\varphi}_{n}(t)\rangle|\varphi_{n}(t)\rangle\langle\varphi_{n}(t)|\right]$
(2)

using the instantaneous eigenstates
$|\varphi_{n}(t)\rangle\equiv|\varphi_{n}(H_{\mathrm{ad}}(t))\rangle$. The
quantum system subject to the total Hamiltonian
$H_{\mathrm{tot}}(t)\\!=\\!H_{\mathrm{ad}}(t)+H_{\mathrm{cd}}(t)$ is dragged
to the exact eigenstate $|\Psi(T)\rangle\propto|\varphi_{n}(H)\rangle$ under
the conditions of $\dot{\lambda}(0)\\!=\\!0$ and $\dot{\lambda}(T)\\!=\\!0$.
The reduced Planck constant $\hbar$ is set to be unity throughout this paper.
The digitized STA is then designed based on the above total Hamiltonian. After
dividing the operation time into $M$ segments ($\Delta t\\!=\\!T/M$), we apply
the Trotter-Suzuki decomposition to the time evolution operator
$U\\!=\\!T_{+}\exp[-i\int_{0}^{T}H_{\mathrm{tot}}(t)dt]$, which leads to
$U\\!\approx\\!U_{M}\cdots U_{2}U_{1}$ with
$U_{m=1,\cdots,M}\\!=\\!\exp[-iH_{\mathrm{tot}}(m\Delta t)\Delta t]$. The
total Hamiltonian can be expanded into
$H_{\mathrm{tot}}(t)\\!=\\!\sum_{j}\omega_{j}(t)\Gamma_{j}/2$, where
$\Gamma_{j}$ is a quantum operator and $\omega_{j}(t)$ is its associated
coefficient. The Trotter-Suzuki decomposition allows a new factorization,
$U_{m}\\!\approx\\!\cdots
R_{\Gamma_{2}}(\theta_{m;2})R_{\Gamma_{1}}(\theta_{m;1})$, where
$R_{\Gamma_{j}}(\theta_{m;j})\\!=\\!\exp[-i\theta_{m;j}\Gamma_{j}/2]$ can be
viewed as a generalized rotation operator over an angle
$\theta_{m;j}\\!=\\!\omega_{j}(m\Delta t)\Delta t$. In practice,
$R_{\Gamma_{j}}(\theta_{m;j})$ can be realized by a combination of sequential
single- and two-qubit gates. In practice, the operation time $T$ is an
adjustable parameter to define the angle sequence $\\{\theta_{m;j}\\}$ while
the real experimental time is determined by the total time of combined gates.

The digitized STA is implemented in a quantum computing device composed of six
superconducting cross-shaped transmon qubits WangPRAPP2019 , from which two
qubits ($Q_{A}$ and $Q_{B}$) are selected. The device is mounted in an
aluminum sample box and in a dilution refrigerator with a base temperature
around 10 mK. The frequency-tunable qubits are initially biased at an
operation point through two $Z$-control lines. In our experiment, the two
qubits are set at $\omega_{A}/2\pi\\!=\\!6.21$ GHz and
$\omega_{B}/2\pi\\!=\\!5.70$ GHz. The two anharmonicity parameters are
$\Delta_{A}/2\pi\\!\approx\\!\Delta_{B}/2\pi\\!=\\!-250$ MHz, enabling a
selective microwave drive to control each qubit through its $XY$-control line.
At the operation point, the relaxation times are $T_{A;1}\\!=\\!5.8$ $\mu$s
and $T_{B;1}\\!=\\!6.9$ $\mu$s while the pure dephasing times are
$T_{A;\phi}\\!=\\!26$ $\mu$s and $T_{B;\phi}\\!=\\!28$ $\mu$s. The readout
fidelities of the ground and the excited states are $F_{A;0}\\!=\\!99\%$ and
$F_{A;1}\\!=\\!95\%$ for qubit $A$, and $F_{B;0}\\!=\\!97\%$ and
$F_{B;1}\\!=\\!93\%$ for qubit $B$.

Figure 2: Determination of the ground state of H2 via a digitized STA process
for the internuclear distance at $R=0.05$ Å. (a) A quantum circuit diagram of
the $M$-step digitized STA process. (b) The evolution of the state fidelities
in four digitized STA experiments ($1\leq M\leq 4$) in comparison with the
ideal result. (c) The evolution of the state energies in four digitized STA
experiments.

Example 1 - a hydrogen molecule. For two electrons of H2, the single-electron
basis functions are combined by two delocalized orbitals
$\\{|\phi_{g}\rangle,|\phi_{u}\rangle\\}$ and two spin states
$\\{|+\rangle,|-\rangle\\}$, i.e.,
$|\chi_{i=1,\cdots,4}(\vec{r},s)\rangle\\!=\\!|\phi_{g/u}(\vec{r})\rangle\otimes|s\\!=\\!+/-\rangle$
with $\vec{r}$ and $s$ the coordinate and spin variables PeruzzoNC14 ;
MalleyPRX16 ; CollessPRX18 . In the number representation with regard to
$\\{|\chi_{i}\rangle\\}$, the second quantization rewrites the Hamiltonian
into $H=\sum h_{ij}a^{+}_{i}a_{j}+\sum h_{ijkl}a^{+}_{i}a^{+}_{j}a_{k}a_{l}$,
where $a_{i}$ and $a^{+}_{i}$ are annihilation and creation operators
satisfying $[a_{i},a^{+}_{j}]_{+}=\delta_{i,j}$. The single- and two-electron
integrals ($h_{ij}$ and $h_{ijkl}$) depend on the internuclear distance $R$
PeruzzoNC14 ; MalleyPRX16 ; CollessPRX18 . The Bravyi-Kitaev transformation
SeeleyJCP01  is applied and the target Hamiltonian is approximated as

$\displaystyle H=g_{0}+gZ_{A}+gZ_{B}+g_{12}Y_{A}Y_{B},$ (3)

where $\\{X_{\alpha},Y_{\alpha},Z_{\alpha}\\}$ is the set of Pauli matrices
($\alpha=A,B$), and the parameters $\\{g_{0},g,g_{12}\\}$ are functions of $R$
PeruzzoNC14 ; MalleyPRX16 ; CollessPRX18 .

In the first step, we explore the ground state $|\varphi_{0}(H)\rangle$ at
$R=0.05$ Å, where the energy parameters are
$g_{0}\\!=\\!10.08~{}E_{\mathrm{h}}$, $g\\!=\\!-1.055~{}E_{\mathrm{h}}$ and
$g_{12}\\!=\\!0.1557~{}E_{\mathrm{h}}$ ($E_{\mathrm{h}}\\!\approx\\!27.2$ eV)
CollessPRX18 . The initial Hamiltonian is selected to be
$H_{0}=g(Y_{A}+Y_{B})$ while the time-adjusting coefficient in Eq. (1) is
$\lambda(0\leq t\leq T)=\sin^{2}(\pi t/2)$. This form of $\lambda(t)$ is
applied to all the experiments in this Letter. The initial ground state
$|\varphi_{0}(H_{0})\rangle=(1/2)(|0\rangle+i|1\rangle)_{A}\otimes(|0\rangle+i|1\rangle)_{B}$
is experimentally generated by applying the $R_{X_{A}}(-\pi/2)$ and
$R_{X_{B}}(-\pi/2)$ gates onto the two-qubit state $|00\rangle$. Following the
single-qubit approximation chenxiPRAPP20 ; WangPRAPP2019 , the total
Hamiltonian is formulated as
$H_{\mathrm{tot}}(t)\approx(1/2)[\omega_{1}(t)Y_{A}Y_{B}+\sum_{\alpha=A,B}\omega_{2}(t)Y_{\alpha}+\omega_{3}(t)X_{\alpha}+\omega_{4}(t)Z_{\alpha}]$
Suppl . The whole time evolution of the $M$-step digitized STA is given by
$U\\!=\\!\prod_{m=M}^{1}U_{m}$ and each $m$-th partial time evolution operator
is set to be

$\displaystyle U_{m}$ $\displaystyle=$ $\displaystyle
R_{Z_{A}}(\theta_{m;4})R_{Z_{B}}(\theta_{m;4})R_{X_{A}}(\theta_{m;3})R_{X_{B}}(\theta_{m;3})$
(4) $\displaystyle\times
R_{Y_{A}}(\theta_{m;2})R_{Y_{B}}(\theta_{m;2})R_{Y_{A}Y_{B}}(\theta_{m;1}),$

where the two-qubit rotation $R_{Y_{A}Y_{B}}(\theta)$ is experimentally
realized by BarendsNC15

$\displaystyle
R_{Y_{A}Y_{B}}(\theta)=R_{X_{A}}(\pi/2)R_{X_{B}}(\pi/2)U_{\mathrm{CN}}R_{Z_{B}}(\theta)U_{\mathrm{CN}}R_{X_{A}}(-\pi/2)R_{X_{B}}(-\pi/2).$
(5)

The rotation angles $\\{\theta_{m;j}\\}$ are theoretically determined by the
time-varying parameters $\\{\omega_{j}(t)\\}$ Suppl , and experimentally
adjusted to improve the accuracy of $U_{m}$. The corresponding pulse sequence
is shown in Fig. 2(a). In practice, we perform the digitized STA experiment up
to four steps ($1\\!\leq\\!M\\!\leq\\!4$) and measure the state evolution with
the quantum state tomography (QST) during each step. As compared to the
theoretical simulation, the experimental error of the state preparation is
$<5$% while a one-step evolution of $U_{m}$ causes an additional $>5\%$ error
Suppl . To reduce the error accumulation, we repeatedly prepare the
intermediate state $|\Psi(t=(m-1)\Delta t)\rangle$ using a universal quantum
circuit ShendePRA03  during each $m$-th step. Because the final quantum state
through a real evolution is a mixed state, we diagonalize the measured density
matrix and extract the eigenvector with largest population as our experimental
result Suppl . These treatments are also used in other experiments in this
paper, but may be unnecessary with the future improvement of sample quality
and control accuracy.

To quantify the state evolution in each $M$-step experiment, we introduce the
fidelity function
$\mathcal{F}_{m}\\!=\\!|\langle\varphi_{0}(H)|\Psi(t\\!=\\!m\Delta
t)\rangle|^{2}$ by comparing both intermediate and final states $|\Psi(m\Delta
t)\rangle$ with the ideal ground state $|\varphi_{0}(H)\rangle$. The
experimental results of $\mathcal{F}_{m}$ are presented in Fig. 3(b). For
$M\geq 2$, the digitized STA efficiently drags the initial state
($\mathcal{F}_{0}=23\%$) to the ideal state (the best fidelity
$\mathcal{F}_{M}\approx 99\%$). Although the Trotter-Suzuki decomposition
implicitly requires a large number of $M$, this fast convergence could be
attributed to a weak two-qubit interaction ($|g_{12}|\\!\ll\\!|g|$ at
$R\\!=\\!0.05$ Å). We also calculate the state energies
$E_{m}\\!=\\!\langle\Psi(m\Delta t)|H|\Psi(m\Delta t)\rangle$ through each
$M$-step process (see Fig. 3(c)). In comparison to the ideal value
$\varepsilon_{0}(R\\!=\\!0.05~{}\mathrm{\AA})\\!=\\!7.96$ $E_{\mathrm{h}}$,
the ground state energy of H2 is experimentally determined with an excellent
accuracy of $>99\%$.

Figure 3: Determination of the 1st excited state of H2 via a digitized STA
process for the internuclear distance at $R=0.05$ Å. (a) A quantum circuit
diagram of the $M$-step digitized STA process. (b) The evolution of the state
fidelities in four digitized STA experiments ($1\leq M\leq 4$) in comparison
with the ideal result. (c) The evolution of the state energies in four
digitized STA experiments.

Next we explore the first excited state $|\varphi_{n=1}(H)\rangle$ at
$R\\!=\\!0.05~{}\mathrm{\AA}$ while the other two excited states can be
investigated similarly. The initial Hamiltonian is changed to
$H_{0}\\!=\\!gZ_{A}$, and the initial state $|\Psi(0)\rangle\\!=\\!|01\rangle$
is obtained by applying the $R_{X_{B}}(\pi)$-gate onto the state $|00\rangle$.
Following the same procedure as above, each $m$-th partial time evolution
operator in the digitized STA is given by Suppl

$\displaystyle
U_{m}=R_{X_{A}}(\theta_{m;4})R_{Z_{A}}(\theta_{m;3})R_{Z_{B}}(\theta_{m;2})R_{Y_{A}Y_{B}}(\theta_{m;1}).$
(6)

The corresponding pulse sequence is plotted in Fig. 3(a). The experimental
results of the state fidelity
$\mathcal{F}_{m}\\!=\\!|\langle\varphi_{n=1}(H)|\Psi(m\Delta t)\rangle|^{2}$
and the state energy $E_{m}$ are presented in Figs. 3(b) and 3(c). Consistent
with the behavior in the ground state, the digitized STA efficiently drags the
initial state ($\mathcal{F}_{0}\\!=\\!50\%$) to the ideal first excited state
(the best fidelity $\mathcal{F}_{M}\\!\approx\\!98\%$) for $M\\!\geq\\!3$. The
experimental determination of the eigenenergy reaches the ideal value
$\varepsilon_{1}(R\\!=\\!0.05~{}\mathrm{\AA})\\!=\\!9.92$ $E_{\mathrm{h}}$
with the accuracy of $>99\%$.

Subsequently, we explore the electronic states of H2
($|\varphi_{n=0,1}(R)\rangle\\!\equiv\\!|\varphi_{n=0,1}(H(R))\rangle$) for
various internuclear distances. The above two digitized STA procedures are
applicable but the convergence slows down as $R$ increases, e.g.,
$M\\!\geq\\!40$ required for $R\\!=\\!1.55~{}\mathrm{\AA}$ from the
theoretical prediction. Since the operators involved in $H(R)$ are invariant
with the change of $R$, we utilize the above experimental result and treat
$R_{0}\\!=\\!0.05~{}\mathrm{\AA}$ as a reference point in the control
parameter ($R$) space. The initial Hamiltonian is set to
$H_{0}=[g_{12}(R)/g_{12}(R_{0})]H(R_{0})$. The two initial eigenstates,
$|\Psi(0)\rangle\\!=\\!|\varphi_{n=0,1}(R_{0})\rangle$, are directly prepared
with the universal quantum circuit ShendePRA03 . Interestingly, our
theoretical simulation predicts that a one-step STA process,
$U\\!=\\!R_{Z_{A}}(\theta_{2})R_{Z_{B}}(\theta_{2})R_{Y_{A}Y_{B}}(\theta_{1})$,
can produce the target eigenstates $|\varphi_{n=0,1}(R)\rangle$ with a high
fidelity Suppl . The pulse sequence is shown in the inset of Fig. 4(a). As
shown in Fig. 4(b), the experimental state fidelities are $\sim 97\%$ for the
ground state and $\sim 95\%$ for the first excited state in the range of large
internuclear distances
($1.05~{}\mathrm{\AA}\\!\leq\\!R\\!\leq\\!2.05~{}\mathrm{\AA}$). Figure 4(a)
presents the experimental energy landscape (the relation of
$\varepsilon_{n=0,1}$ vs. $R$) in an excellent agreement with the theoretical
prediction. An enlarged view of the energy landscape for
$1.05~{}\mathrm{\AA}\\!\leq\\!R\\!\leq\\!2.05~{}\mathrm{\AA}$ is shown in Fig.
4(c). We thus obtain an accurate description of the electronic structure of H2
using a delicate and efficient design of digitized STA.

Figure 4: Experimental determination of the energy landscapes for both the
ground and excited states of H2. (a) The two eigenergies
($\varepsilon_{n=0,1}$) as the functions of the internuclear distance $R$. The
quantum circuit diagram is shown in the inset. In the large-distance range
($1.05~{}\mathrm{\AA}\\!\leq\\!R\\!\leq\\!2.05~{}\mathrm{\AA}$), the
experimental results of the state fidelities and eigenenergies are enlarged in
(b) and (c).

Example 2 - the topological BHZ model. The BHZ model can be used to describe
the quantum spin Hall effect in a two-dimensional square lattice (the lattice
constant is $a$) ZhangSci06 . In the momentum space of
$\vec{k}=k_{x}\vec{e}_{x}+k_{y}\vec{e}_{y}$, an effective tight-binding
Hamiltonian is written as

$\displaystyle H$ $\displaystyle=$ $\displaystyle
g_{0}(\vec{k})+g(\vec{k})Z_{B}+Z_{A}\left[g_{12}(k_{x})X_{B}+g_{12}(k_{y})Y_{B}\right]$
(7)

with $g_{0}(\vec{k})\\!=\\!C_{1}-C_{2}f(\vec{k})$,
$g(\vec{k})\\!=\\!C_{3}f(\vec{k})$,
$g_{12}(k_{x/y})\\!=\\!C_{4}\sin(k_{x/y}a)$, and
$f(\vec{k})\\!=\\!8[\sin^{2}(k_{x}a/2)\\!+\\!\sin^{2}(k_{y}a/2)]$ Suppl . In
this Letter, the parameters of $\\{C_{i=1,\cdots,4}\\}$ are chosen from
HgTe/CdTe JiangPRB09 . The two-qubit interactions in Eq. (7) represent the
orbital-spin coupling. The symmetry of $H$ indicates two-fold degenerate
energy bands and a Dirac point at the $\Gamma$-point
($k_{x}\\!=\\!k_{y}\\!=\\!0$) of the first Brillouin zone (FBZ). The BHZ model
can thus be reduced to a single-qubit Hamiltonian which can be simulated by a
continuous STA process. To demonstrate the reliability of the digitized STA,
we however drive the two-qubit system to determine the valence and conduction
bands along the X-$\Gamma$-X linecut ($-\pi\\!\leq\\!k_{x}a\\!<\\!\pi$ and
$k_{y}\\!=\\!0$) in the FBZ.

Similar to the treatment in H2, we first explore the electronic states
$|\varphi_{n=0,1}(k_{x;0})\rangle\\!\equiv\\!|\varphi_{n=0,1}(H(k_{x}\\!=\\!k_{x;0},k_{y}\\!=\\!0))\rangle$
at two reference points, $k_{x;0}\\!=\\!\pm 0.1a^{-1}$. For both wavevectors,
the initial Hamiltonian is selected to be $H_{0}=g(k_{x;0})Z_{B}$. The initial
ground state of $H_{0}$ is $|\varphi_{0}(H_{0})\rangle\\!=\\!|01\rangle$
generated by applying the $R_{X_{B}}(\pi)$ gate to the two-qubit state
$|00\rangle$ while the initial excited state of $H_{0}$ is
$|\varphi_{1}(H_{0})\rangle\\!=\\!|00\rangle$. In the $M$-step digitized STA,
the partial time evolution operator at each step is given by Suppl

$\displaystyle U_{m}$ $\displaystyle=$ $\displaystyle
R_{Z_{B}}(\theta_{m;3})R_{Y_{B}}(\theta_{m;2})R_{Z_{A}X_{B}}(\theta_{m;1}).$
(8)

The corresponding pulse sequences for both the ground and excited states are
shown in Fig. 5(a) and Fig. 5(b), respectively. In practice, we perform the
two 4-step digitized STA experiments and determine
$|\varphi_{0}(k_{x;0})\rangle$ and $|\varphi_{1}(k_{x;0})\rangle$ with the
accuracies of 99% and 96%, respectively.

Figure 5: Digitized STA experiments for the BHZ model. (a) and (b) The quantum
circuit diagrams to determine the ground and excited eigenstate at the two
reference wavevectors, $k_{x;0}=\pm 0.1a^{-1}$. (c) The quantum circuit
diagram for wavevectors $k_{x}\neq k_{x;0}$. (d) The experimental
determination of the valence and conduction bands along the X-$\Gamma$-X
linecut. Inset is a enlarged view of (d) around the Dirac point.

Next we explore the electronic states $|\varphi_{n=0,1}(k_{x})\rangle$ at
different wavevectors along the X-$\Gamma$-X linecut. The initial Hamiltonian
is set to be $H_{0}=(g_{12}(k_{x})/g_{12}(k_{x;0}))H(k_{x;0})$ and the initial
states are changed to $|\Psi(0)\rangle\\!=\\!|\varphi_{0}(k_{x;0})\rangle$ and
$|\varphi_{1}(k_{x;0})\rangle$ accordingly. For positive ($k_{x}\\!>\\!0$) and
negative ($k_{x}\\!<\\!0$) wavevectors, the reference wavectors are chosen at
$k_{x;0}=0.1a^{-1}$ and $-0.1a^{-1}$, respectively. Instead of a multi-step
STA, a one-step process,
$U=R_{Z_{B}}(\theta_{3})R_{Z_{A}X_{B}}(\theta_{2})R_{Z_{B}}(\theta_{1})$, is
empirically applied (see Fig. 5(c)). The experimentally determined valence and
conduction bands,
$\varepsilon_{n=0,1}(k_{x})\\!=\\!\langle\varphi_{n}(k_{x})|H|\varphi_{n}(k_{x})\rangle$,
are shown in Fig. 5(d), which agrees excellently with the theoretical
prediction (the state fidelities $>96\%$). In particular, our experimental
results confirm the linear dispersion relation
$\varepsilon_{n}(k_{x})\\!\propto\\!k_{x}$ around the Dirac point
($k_{x}\\!=\\!0$) as shown in the inset of Fig. 5(d).

In conclusion, we applied the digitized STA to experimentally determine the
electronic states in two example systems, the hydrogen molecule and the BHZ
model. In both models, one or a few reference points $\Lambda_{0}$ are
selected in the space of the control parameters $\Lambda$ (e.g., the
internuclear distance $R$ and the wavector $\vec{k}$). With the initialization
at simple qubit states, the electronic states (ground and excited) at
$\Lambda_{0}$ are experimentally determined via multi-step digitized STA
processes with high fidelities ($96\%\sim 99\%$). For other control parameters
($\Lambda\neq\Lambda_{0}$), the pre-determined electronic states at
$\Lambda_{0}$ are used as the initial states and the eigenstates of the target
Hamiltonian is experimentally determined via a one-step STA process. An
excellent description of the electronic states with the change of the control
parameters are obtained accordingly. Our experimental study confirms the
reliability and efficiency of the digitized STA algorithm and our strategy of
$H(R_{0})\rightarrow H(R)$ show great potential of determining electronic
states in more complicated systems. Due to unavoidable errors, our experiments
require step-by-step corrections during the digitized STA process, but future
improvements can further sharpen the experimental accessibility.

The work reported here was supported by the National Key Research and
Development Program of China (Grant No. 2019YFA0308602, No. 2016YFA0301700),
the National Natural Science Foundation of China (Grants No. 12074336, No.
11934010, No. 11775129), the Fundamental Research Funds for the Central
Universities in China, and the Anhui Initiative in Quantum Information
Technologies (Grant No. AHY080000). Y.Y. acknowledge the funding support from
Tencent Corporation. This work was partially conducted at the University of
Science and Technology of China Center for Micro- and Nanoscale Research and
Fabrication.

## References

  * (1) M. A. Nielsen and I.L. Chuang, Quantum Computation and Quantum Information, Cambridge University Press, Cambridge, England (2010).
  * (2) Google AI Quantum, Nature 574, 505-510 (2019).
  * (3) A. Peruzzo, J. McClean, P. Shadbolt, M. H. Yung, X. Q. Zhou, P. J. Love, A. A. Guzik, and J. L. O’Brien, Nat. Commun. 5, 4213 (2014).
  * (4) P. J. J. O’Malley, R. Babbush, I. D. Kivlichan, J. Romero, J. R. McClean, R. Barends, _et al._ , Phys. Rev. X 6, 031007 (2016).
  * (5) A. Kandala, A. Mezzacapo, K. Temme, M. Takita, M. Brink, J. M. Chow, and J. M. Gambetta, Nature 549, 242-246 (2017).
  * (6) Google AI Quantum, Science 369, 1084-1089 (2020).
  * (7) E. Farhi, J. Goldstone, S. Gutmann, J. Lapan, A. Lundgren, and D. Preda, Science 292, 472-475 (2001).
  * (8) A. Das and B. K. Chakrabarti, Rev. Mod. Phys. 80, 1061 (2008).
  * (9) J. I. Colless, V. V. Ramasesh, D. Dahlen, M. S. Blok, M. E. Kimchi-Schwartz, J. R. McClean, J. Carter, W. A. de Jong, and I. Siddiqi, Phys. Rev. X. 8, 011021 (2018).
  * (10) M. V. Berry, J. Phys. A: Math. Theor. 42, 365303 (2009).
  * (11) X. Chen, I. Lizuain, A. Ruschhaupt, D. Guéry-Odelin, and J. G. Muga, Phys. Rev. Lett. 105, 123003 (2010).
  * (12) T. Wang, Z. Zhang, L. Xiang, Z. Jia, P. Duan, W. Cai, Z. Gong, Z. Zong, M. Wu, J. Wu, L. Sun, Y. Yin, and G. Guo, New J. Phys. 20, 065003 (2018).
  * (13) Z. Zhang, T. Wang, L. Xiang, Z. Jia, P. Duan, W. Cai, Z. Zhan, Z. Zong, J. Wu, L. Sun, Y. Yin, and G. Guo, New J. Phys. 20, 085001 (2018).
  * (14) Z. Zhang, T. Wang, L. Xiang, J. Yao, J. Wu, and Y. Yin, Phys. Rev. A 95, 042345 (2017).
  * (15) T. Wang, Z. Zhang, L. Xiang, Z. Gong, J. Wu, and Y. Yin, Sci. China Phys. Mech. Astron. 61, 047411 (2018).
  * (16) R. Barends, L. Lamata, J. Kelly, L. García-Álvarez, A. G. Fowler, A. Megrant, _et al._ , Nat. Commun. 6, 7654 (2015).
  * (17) R. Barends, A. Shabani, L. Lamata, J. Kelly, A. Mezzacapo, U. L. Heras, _et al._ , Nature 534, 222-226 (2016).
  * (18) N. N. Hegade, K. Paul, Y. Ding, M. Sanz, F. Albarran-Arriagada, E. Solano, and X. Chen, arXiv:2009.03539 (2020).
  * (19) B. A. Bernevig, T. L. Hughes, and S. C. Zhang, Science 314, 1757-1761 (2006).
  * (20) T. Wang, Z. Zhang, L. Xiang, Z. Jia, P. Duan, Z. Zong, Z. Sun, Z. Dong, J. Wu, Y. Yin, and G. Guo, Phys. Rev. Appl. 11, 034030 (2019).
  * (21) J. T. Seeley, M. J. Richard, and P. J. Love, J. Chem. Phys. 137, 224109 (2012).
  * (22) See Supplementary Material at [URL will be inserted by publisher] for a discussion of Shortcut-to-adiabaticity and its expression for a single qubit, Bravyi-Kitaev mapping of a hydrogen molecule, and the digitized STA for hydrogen molecule and BHZ model.
  * (23) V. V. Shende, I. L. Markov, and S. S. Bullock, Phys. Rev. A, 69, 062321 (2004).
  * (24) H. Jiang, L. Wang, Q. F. Sun, and X. C. Xie, Phys. Rev. B 80, 165316 (2009).


# A Method for Fast Autonomy Transfer in Reinforcement Learning

Dinuka Sahabandu1, Bhaskar Ramasubramanian2, Michail Alexiou3,
J. Sukarno Mertoguno4, Linda Bushnell1, Radha Poovendran1 1Network Security
Lab, Department of Electrical and Computer Engineering, University of
Washington, Seattle, WA 98195, USA.
{sdinuka, lb2<EMAIL_ADDRESS>and Computer Engineering, Western
Washington University, Bellingham, WA 98225, USA<EMAIL_ADDRESS>of Computer Science, Kennesaw State University, Marietta, GA 30060, USA.
<EMAIL_ADDRESS>School of Cybersecurity and Privacy, Georgia Institute
of Technology, Atlanta, GA 30332<EMAIL_ADDRESS>
###### Abstract

This paper introduces a novel reinforcement learning (RL) strategy designed to
facilitate rapid autonomy transfer by utilizing pre-trained critic value
functions from multiple environments. Unlike traditional methods that require
extensive retraining or fine-tuning, our approach integrates existing
knowledge, enabling an RL agent to adapt swiftly to new settings without
requiring extensive computational resources. Our contributions include
development of the Multi-critic Actor-Critic (MCAC) algorithm, establishing
its convergence, and empirical evidence demonstrating its efficacy. Our
experimental results show that MCAC significantly outperforms the baseline
Actor-Critic (AC) algorithm, achieving up to $22.76\times$ faster autonomy
transfer and higher reward accumulation. This advancement underscores the
potential of leveraging accumulated knowledge for efficient adaptation in RL
applications.

##  I Introduction

In the realms of artificial intelligence and data-driven control,
reinforcement learning (RL) stands out as a powerful paradigm for solving
complex decision-making problems in unknown or unseen environments [1]. RL
involves an agent learning to make decisions by interacting with an
environment, by aiming to maximize a cumulative reward signal provided by the
environment over time. This process of trial and error, coupled with the
feedback received through rewards, allows the agent to develop a strategy or
policy that guides its actions towards achieving its goals.

RL has been successfully applied in multiple domains, including robotics,
autonomous vehicles, games, and mobile networks [2, 3, 4, 5, 6, 7, 8]. Despite
these successes, RL faces a significant challenge when an agent encounters new
environments. In such a scenario, the agent must learn from the very beginning
in each new setting, a process that can be prohibitively time-consuming and
resource-intensive. The ability to transfer knowledge—utilizing insights
gained from previous experiences to expedite learning in new situations—has
therefore become a key area of focus. Knowledge transfer can dramatically
reduce the amount of interaction needed with a new environment to reach
optimal or near-optimal performance, thereby accelerating the learning process
and reducing computational costs.

Yet, effectively achieving this transfer of knowledge poses challenges. In
deep RL [9], where agents use deep neural networks to approximate policies or
value functions, techniques such as transfer learning and fine-tuning have
shown promise [10, 11]. While transfer learning and fine-tuning techniques
offer some solutions, they typically involve re-training parts of the network.
This process, albeit not as extensive as retraining the entire network, still
demands considerable computational effort and time. On the other hand,
traditional RL paradigms operating in discrete environments which use a
tabular approach to storing value functions (e.g., Q-learning [12], presents a
different challenge- _how can pre-learned knowledge be leveraged in discrete
environments without complex neural network architectures used in deep RL?_.

In this paper, we investigate the concept of rapid autonomy transfer within
the scope of classical reinforcement learning. Specifically, we examine this
problem through the lens of Actor-Critic (AC) algorithms under a discounted
reward framework. AC methods combine policy-based decision making (actor) with
value-based evaluation (critic), offering a balanced approach to learning by
directly integrating feedback on the quality of actions taken [13]. Actor-
Critic methods are distinguished by their rapid convergence properties and
minimal convergence errors, closely approaching optimal performance. The
discounted reward setting, central to our analysis, emphasizes the importance
of future rewards by applying a discount factor, which ensures that immediate
gains do not overshadow long-term benefits.

We propose a novel approach to enable rapid autonomy transfer in RL that uses
pre-trained critic value functions from multiple environments and combines
them to assist in training a new actor for a different environment. Our
approach bypasses the need for retraining or partially training critics by
focusing on the optimal integration of existing knowledge, effectively
leveraging the knowledge accumulated from multiple critics to navigate new
environments.

Specifically, our Multi-critic Actor-Critic (MCAC) learns a set of weights to
modulate influence of pre-trained value functions from diverse environments.
By leveraging pre-existing value functions without the need for retraining,
MCAC not only conserves computational resources but also accelerates
adaptation to new environments. This contribution marks a significant
advancement in the application of RL techniques, opening up new possibilities
for their use in diverse and dynamic settings. We make the following
contributions in this paper:

  * •

We introduce the MCAC algorithm to enable fast autonomy transfer in RL.

  * •

We establish convergence of weights and policies learned by our MCAC
algorithm.

  * •

Experimentally, we show that MCAC achieves autonomy transfer up to 22.76x
times faster than a baseline AC algorithm, while also achieving higher
rewards.

The rest of this paper is organized as: Sec. II describes related work and
Sec. III presents necessary preliminaries. Sec. IV introduces our MCAC
approach and presents our main results. Sec. V reports results of experimental
evaluations of MCAC and Sec. VI concludes the paper.

##  II Related Work

This section briefly describes related work in multi-critic learning, and
methods to aggregate information from critics.

Multi-Critic Actor Learning was proposed in [14] in the context of multi-task
learning as an alternative to maintaining separate critics for each task being
trained while training a single multi-task actor. Explicitly distinguishing
between tasks also eliminates a need for critics to learn to do so and
mitigates interference between task-value estimates. However, while [14] seeks
to ‘select’ a particular style of task satisfaction in a given environment,
our MCAC approach ‘combines’ multiple styles of task satisfaction to execute a
given task in a new environment. Recently, the authors of [15] introduced a
multi-actor mechanism that combined novel exploration strategies together with
a Q-value weighting technique to accelerate learning of optimal behaviors.
Different from [15], where experiments for any given environment provide
comparisons against benchmarks within the same environment, our goal in this
paper is to accomplish autonomy transfer across different environments.

Reward machines [16] provide a structured framework for defining, formalizing,
and managing rewards in an intuitive and human-understandable manner.
Structuring rewards in this way makes it possible to guide RL agents more
effectively. Reward machines were used to inform design of a Q-learning
algorithm in finite state-action environments in [17], and extended to
partially observable environments in [18]. However, the current state-of-the-
art in this domain does not provide a direct way to encode and aggregate
information from disparate sources and do not have a mechanism to encode
distracting rewards. In comparison, our MCAC technique aggregates information
from multiple scenarios and uses this information to achieve significant
speedup in terms of run time and number of learning episodes required.

##  III Preliminaries

This section introduces necessary preliminaries on Markov decision processes
(MDPs), reinforcement learning (RL), and stochastic approximation (SA).

###  III-A MDPs and RL

Let $(\Omega,\mathcal{F},\mathcal{P})$ denote a probability space, where
$\Omega$ is a sample space, $\mathcal{F}$ is a $\sigma-$algebra of subsets of
$\Omega$, and $\mathcal{P}$ is a probability measure on $\mathcal{F}$. A
random variable is a map $Y:\Omega\rightarrow\mathbb{R}$. We assume that the
environment of the RL agent is described by a Markov decision process (MDP)
[19].

######  Definition 1.

An MDP is a tuple $\mathcal{M}:=(S,A,P,r,\gamma)$, where $S$ is a finite set
of states, $A$ is a finite set of actions, and $P(s^{\prime}|s,a)$ is the
probability of transiting to state $s^{\prime}$ when action $a$ is taken in
state $s$. $r:S\times A\rightarrow\mathbb{R}$ is the reward obtained by the
agent when it takes action $a$ in state $s$. $\gamma\in(0,1]$ is a discounting
factor.

An RL agent typically does not have knowledge of $P$. Instead, it obtains a
(finite) reward $r$ for each action that it takes. Through repeated
interactions with the environment, the agent seeks to learn a policy $\pi$ in
order to maximize an objective
$\mathbb{E}_{\pi}[\sum_{t}\gamma^{t}r(s_{t},a_{t})]$ [1]. Let
$\boldsymbol{\pi}$ represent the set of stationary policies of the agent. A
policy $\pi\in\boldsymbol{\pi}$ is considered a deterministic stationary
policy if $\pi\in\\{0,1\\}^{|A|}$. It is said to be a stochastic stationary
policy if $\pi\in[0,1]^{|A|}$.

###  III-B Stochastic Approximation (SA) Algorithms

Let $h:\mathcal{R}^{m_{x}}\rightarrow\mathcal{R}^{m_{x}}$ be a continuous
function of parameters $x\in\mathcal{R}^{m_{x}}$. SA algorithms are designed
to solve equations of the form $h(x)=0$ based on noisy measurements of $h(x)$.
The classical SA algorithm is given by:

$x^{t+1}=x^{t}+\delta^{t}_{x}[h(x^{t})+n_{x}^{t}],\text{ for }t\geq 0.$ (1)

Here, $t$ denotes the iteration index, and $x^{t}$ represents the estimate of
$x$ at the $t^{\text{th}}$ iteration. The term $n_{x}^{t}$ represents the
zero-mean measurement noise associated with $x^{t}$, and $\delta^{t}_{x}$
denotes the learning rate. The stationary points of Eq. (1) coincide with the
solutions of $h(x)=0$ in the absence of noise ($n_{x}^{t}=0$). The convergence
of SA algorithms is typically analyzed through their associated Ordinary
Differential Equations (ODEs), represented as $\dot{x}=h(x)$.

The convergence of an SA algorithm requires specific assumptions on the
learning rate $\delta^{t}_{x}$:

######  Assumption 1.

$\sum_{t=0}^{\infty}\delta^{t}_{x}=\infty$ and
$\sum_{t=0}^{\infty}(\delta^{t}_{x})^{2}<\infty$.

Examples of $\delta^{t}_{x}$ satisfying Assumption 1 include
$\delta^{t}_{x}=1/t$ and $\delta^{t}_{x}=1/(t\log(t))$. A general convergence
result for SA algorithms is stated below.

######  Proposition 1 ([20, 21]).

Consider an SA algorithm in the following form defined over a set of
parameters $x\in\mathcal{R}^{m_{x}}$ and a continuous function
$h:\mathcal{R}^{m_{x}}\rightarrow\mathcal{R}^{m_{x}}$.

$x^{t+1}=\Theta({x^{t}+\delta^{t}_{x}[h(x^{t})+n_{x}^{t}+\kappa^{t}]}),~{}\text{for
$t\geq 0$},$ (2)

where $\Theta$ is a projection operator that projects each $x^{t}$ iterate
onto a compact and convex set $\Lambda\in\mathcal{R}^{m_{x}}$ and $\kappa^{t}$
is a bounded random sequence. Let the ODE associated with Eqn. (2) be

$\dot{x}=\bar{\Theta}(h(x)),$ (3)

where $\bar{\Theta}(h(x))=\lim\limits_{\eta\rightarrow 0}\frac{\Theta(x+\eta
h(x))-x}{\eta}$ and $\bar{\Theta}$ denotes a projection operator that
restricts evolution of Eqn. (3) to the set $\Lambda$. Let the nonempty compact
set $\boldmath{X}$ be a set of asymptotically stable equilibrium points of
Eqn. (3). Then $x^{t}$ converges almost surely to a point in $\boldmath{X}$ as
$n\rightarrow\infty$ if

  1. I)

$\delta^{t}_{x}$ satisfies the conditions in Assumption 1.

  2. II)

$\lim\limits_{t\rightarrow\infty}\left(\sup\limits_{\bar{t}>t}\left|\sum\limits_{l=t}^{\bar{t}}\delta^{t}_{x}n_{x}^{t}\right|\right)=0$
almost surely.

  3. III)

$\lim\limits_{t\rightarrow\infty}\kappa^{t}=0$ almost surely.

##  IV The Multi-Critic Actor Critic

In this section we introduce and describe the design of our multi-critic
actor-critic (MCAC) algorithm to enable fast autonomy transfer in RL using
pre-trained critics. To aid readability, we provide a description of notations
in Table I.

TABLE I: Descriptions of the notations Notation | Description
---|---
$S$ | State space of the current simulated environment
$A$ | Action space of the trained agent
$\hat{P}$ | Transition probability kernel of the current simulated env
$\hat{r}$ | Reward function of the trained agent
$\hat{\pi}$ | Set of stochastic stationary policies for the trained agent
$\hat{\pi}(s)$ | Stochastic stationary policy at state $s$
| for the trained agent
$\hat{P}(s^{\prime}|s,a)$ | Probability of transitioning from state $s$ to $s^{\prime}$
| under action $a$ in the current simulated environment
$\hat{r}(s,a,s^{\prime})$ | Reward received when transitioning from state $s$ to $s^{\prime}$
| under action $a$ for the trained actor
$\hat{V}$ | Modulated value vector of the trained actor
$\hat{V}(s)$ | Modulated value function of the trained actor at state $s$
$\bar{V}_{i}$ | Value vector of the $i$-th pre-trained critic
$\bar{V}_{i}(s)$ | Value function of the $i$-th pre-trained critic at state $s$
$W$ | Modulation weight vector
$\gamma$ | Discount factor
$\delta_{W}^{t}$ | Learning rate for weight updates at iteration $t$
$\Theta_{W}$ | Projection operator for weight updates
$\delta_{\hat{\pi}}^{t}$ | Learning rate for policy updates of the trained actor
| at iteration $t$
$\Theta_{\hat{\pi}}$ | Projection operator for policy updates of trained actor

Our intuition for the MCAC algorithm is grounded in the principles of ensemble
learning in machine learning [22] and model soups in large language models
[23]. These solutions utilize multiple trained models to achieve better
performance with significantly fewer resources and and in shorter time.
Following a similar line of thought, we hypothesize that using multiple pre-
trained critic values from training environments under different conditions
(e.g., different obstacle orientations) can enhance performance and achieve
faster convergence in actor-critic algorithms trained for a new environment
with varied conditions.

Specifically, given access to a set of pretrained critic value functions, the
MCAC approach aims to bypass the process of training a critic for the new
environment from scratch, opting instead to use a weighted sum of the existing
knowledge from the pretrained critic values for estimating the value function
in the current environment. A weighted average of pre-trained critic value
functions will provide a better initial condition for the value function
estimated for the new environment. This, in turn, will improve the agent’s
exploration and achieve better performance in terms of the accumulated reward
at convergence. Further, since MCAC will not require training new value
functions, but rather focuses on finding the best set of weights to compute
the weighted average of the pre-trained value functions of critics to estimate
the value function of the current environment, it reduces the number of
trainable parameters related to critics from $|S|$ to $N$, where $|S|$ is the
cardinality of the state space of the environment, and $N$ is the number of
pre-trained critic value functions available in the new environment.
Typically, $N\ll|S|$, which aids the agent in achieving significant speedup in
training. Additionally, starting from a better initial condition, as mentioned
previously, will also help the agent learn faster compared to learning from
scratch.

In order to setup the MCAC approach and algorithm, we first establish an
assumption regarding the availability of pre-trained value functions, derived
using the AC algorithm from various environment settings.

Let $\bar{V}_{i}(s)$ be the value function of the $i^{\text{th}}$ pre-trained
critic at state $s\in{S}$. Then define the vector
$\bar{V}_{i}:=[\bar{V}_{i}(s)]_{s\in{S}}$.

######  Assumption 2.

Converged value vectors $\bar{V}_{i}$ for $i=1,2,\ldots,N$ pre-trained critics
are readily available for the training the actor (i.e., agent) using the MCAC
algorithm.

Let $W=[w_{i}]_{i=1}^{N}$ be the weight vector where each weight $w_{i}$ is
associated with a pre-trained value function $\bar{V}_{i}$ of critic $i$.
Further, let $\sum_{i=1}^{N}w_{i}=1$ and $w_{i}\geq 0$ for all
$i\in\\{1,2,\ldots,N\\}$. Then, the value vector
$\hat{V}=[\hat{V}(s)]_{s\in{S}}$ of the actor is defined by

$\hat{V}=\Big{[}\sum_{i=1}^{N}w_{i}\bar{V}_{i}(s)\Big{]}_{s\in S}.$ (4)

The temporal difference (TD) error associated with the trained actor at state
$s\in{S}$ when taking action $a\in{A}$ is:

$\displaystyle\sum_{a\in{A}}\Big{[}\hat{V}(s)-\gamma\sum_{s^{\prime}\in{S}}\hat{P}(s,a,s^{\prime})[\hat{V}(s^{\prime})-\hat{r}(s,a,s^{\prime})]\Big{]}\hat{\pi}(s,a),$

where $0<\gamma<1$ denotes the discount factor. In actor-critic (AC) RL
algorithms [1], the TD error represents the difference between the predicted
reward and the actual reward obtained. The TD error is used to update the
value function and the policy in AC algorithms. A high TD error indicates that
the predictions are quite different from the actual outcomes, suggesting that
the actor’s value function and the policy need significant updating.
Conversely, a low TD error suggests that the actor’s predictions are accurate.

Minimizing the TD error for the trained actor can be expressed as the
following optimization problem:

$\displaystyle\min_{w,\hat{\pi}}\sum_{a\in{A}}\Big{[}\hat{V}(s)-\gamma\sum_{s^{\prime}\in{S}}\hat{P}(s,a,s^{\prime})[\hat{V}(s^{\prime})-\hat{r}(s,a,s^{\prime})]\Big{]}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\hat{\pi}(s,a)$
subject to:
$\displaystyle\hat{V}(s)-\gamma\sum_{s^{\prime}\in{S}}\hat{P}(s,a,s^{\prime})[\hat{V}(s^{\prime})-\hat{r}(s,a,s^{\prime})]\geq
0,$ (5) $\displaystyle\qquad\qquad\qquad\qquad\qquad\forall
s,s^{\prime}\in{S},{a\in{A}},$
$\displaystyle\hat{V}(s)=\sum_{i=1}^{N}w_{i}\bar{V}_{i}(s),\quad\forall
s\in{S}$ (6) $\displaystyle\sum_{a\in{A}}\hat{\pi}(s,a)=1,\quad\forall
s,s^{\prime}\in{S}$ (7) $\displaystyle\hat{\pi}(s,a)\geq 0,\quad\forall
s\in{S}{\text{ and }a\in{A}}$ (8)

######  Remark 1.

If the above optimization problem was framed within a typical actor-critic
(AC) framework, then the minimization would be performed over $\hat{V}$ and
$\hat{\pi}$, while omitting the equality constraint in Eqn. $\eqref{C2}$.
Then, using a stochastic approximation algorithm to solve the standard AC
algorithm would provide the following critic recursion.

$\hat{V}^{t+1}(s)=\hat{V}^{t}(s)+\delta^{t}_{V}[\gamma\hat{V}^{t}(s^{\prime})-\hat{V}^{t}(s)+\hat{r}(s,a,s^{\prime})].$

Here $\delta^{t}_{V}$ is the learning rate of the value updates.

In our MCAC algorithm, we replace the AC critic update by weight updates in
the following way:

$w_{i}^{t+1}=w_{i}^{t}+\delta^{t}_{W}\sqrt{w_{i}^{t}}[\gamma\bar{V}_{i}(s^{\prime})-\bar{V}_{i}(s)+\hat{r}(s,a,s^{\prime})].$

Here the term
$\gamma\bar{V}_{i}(s^{\prime})-\bar{V}_{i}(s)+\hat{r}(s,a,s^{\prime})$ ensures
weights are updated in a descent direction of the estimated TD error with
respect to the pre-trained critic value functions and $\delta^{t}_{W}$ is the
learning rate of the weight updates. We further limit evolution of the weight
vector $W^{t}$ to a probability simplex defined by $\sum_{i=1}^{N}w_{i}^{t}=1$
and $w_{i}^{t}\geq 0$ for all $i\in\\{1,2,\ldots,N\\}$. Therefore, we modify
above weight updates by introducing a projection map $\Theta_{W}$ as follows:

$w_{i}^{t+1}={\Theta}_{W}(w_{i}^{t}+\delta^{t}_{W}\sqrt{w_{i}^{t}}[\gamma\bar{V}_{i}(s^{\prime})-\bar{V}_{i}(s)+\hat{r}(s,a,s^{\prime})]).$
(9)

The rewards $\hat{r}_{i}(s,a,s^{\prime})$ associated with environments of the
$i^{\text{th}}$ pre-trained critic are not directly accessible during the
actor’s training phase during the MCAC algorithm. Hence, availability of these
rewards may be limited. We propose a possible methodology for approximating
values of $\hat{r}_{i}(s,a,s^{\prime})$ in instances where they are not
explicitly known below.

Following the standard definitions of the policy updates for actor-critic
reinforcement learning algorithm, we get:

$\hat{\pi}^{t+1}(s,a)={\Theta}_{\pi}(\hat{\pi}^{t}(s,a)-\delta_{\pi}^{t}\sqrt{\pi^{t}(s,a)}\hat{E}^{t}(s,a,s^{\prime})).$

Here
$\hat{E}^{t}(s,a,s^{\prime})=\sum_{i=1}^{m}w_{i}^{t}[\bar{V}_{i}^{t}(s)-\gamma\bar{V}_{i}^{t}(s^{\prime})]-\hat{r}(s,a,s^{\prime})$
ensures that policies are updated in a descent direction of the total
estimated TD error computed using the weighted pre-trained critic value
functions. The projection map $\Theta_{\pi}$ ensures that evolution of the
policy is such that $\sum_{a\in A(s)}\pi(s,a)=1$ and $\pi(s,a)\geq 0$.
$\delta_{\pi}^{t}$ is the learning rate of policy updates and is chosen such
that $\delta_{\pi}^{t}\ll\delta_{W}^{t}$ for all $t$. Thus, the policy is
updated in a slower time-scale compared to weights. Due to time scale
separation, iterations in faster time scales see iterations in slower times
scales as quasi-static, while the latter sees former as nearly equilibriated
(Chap. 6 of [24]).

The setup described in this section leads to design of our MCAC Algorithm,
shown in Algorithm IV.1. Next, we present results to establish convergence of
weight update iterations in the MCAC Algorithm to an equilibrium, and
convergence of policy update iterations to a stable equilibrium point. We will
leverage Proposition 1 to establish convergence of the updates. We defer
detailed proofs of these results to an extended version of the paper.

Algorithm IV.1 MCAC Algorithm

1:Input: State space (${{S}}$), Rewards (r), Pre-trained value vectors
$\bar{V}_{i}$ for $i=1,2,\ldots,m$, Rewards of pre-trained actors
$\bar{r}_{i}$ for $i=1,2,\ldots,N$, Max. iterations ($T>>0$)

2:Output: Policy, $\hat{\pi}^{\star}\leftarrow\hat{\pi}^{T}$

3:Initialization:  $t\leftarrow 0$, $W^{0}\leftarrow\frac{1}{N}\mathbf{1}$,
$\hat{\pi}^{0}~{}\leftarrow~{}\boldsymbol{\hat{\pi}}$, $s\leftarrow s_{0}$

4:while $t\leqslant T$ do

5: Draw $a$ from $\hat{\pi}(s)$

6: Reveal the next state $s^{\prime}$ according to ${P}$

7: Observe the reward $\hat{r}(s,a,s^{\prime})$

8:
$\hat{E}^{t}_{i}(s,a,s^{\prime})=\bar{V}_{i}(s)-\gamma\bar{V}_{i}(s^{\prime})+\hat{r}_{i}(s,a,s^{\prime})$

9:
$w_{i}^{t+1}={\Theta}_{W}(w_{i}^{t}+\delta^{t}_{W}\sqrt{w_{i}}\hat{E}^{t}_{i}(s,a,s^{\prime}))$

10:
$\hat{E}^{t}(s,a,s^{\prime})=\sum_{i=1}^{m}w_{i}^{t}[\bar{V}_{i}(s)-\gamma\bar{V}_{i}(s^{\prime})]+\hat{r}(s,a,s^{\prime})$

11:
$\hat{\pi}^{t+1}(s,a)={\Theta}_{\hat{\pi}}(\hat{\pi}^{t}(s,a)-\delta_{\pi}^{t}\sqrt{\hat{\pi}^{t}(s,a)}\hat{E}^{t}(s,a,s^{\prime}))$

12: Update the state: $s\leftarrow s^{\prime}$

13: $t\leftarrow t+1$

14:end while

(a) Pre-Trained Scenario 1

(b) Pre-Trained Scenario 2

(c) Pre-Trained Scenario 3

(d) Deployment Scenario 1

(e) Deployment Scenario 2

(f) Deployment Scenario 3

(g) Deployment Scenario 4

Figure 1: Case Study 1 Setup: This figure shows a $5\times 5$ grid. The blue
(green) color state is the start (goal) state and the grey shaded states
represent obstacles. The top row shows the configurations of obstacles that
are used to obtain the pretrained critics. The bottom row shows the obstacle
configurations on which our MCAC algorithm is evaluated. The four deployment
scenarios in the bottom row are obtained by combining the pretrained scenarios
in the top row in different ways.

######  Theorem 2.

The weight updates $w^{t}_{i}$ for all $i\in\\{1,2,\ldots,N\\}$ in Line 9 of
Algorithm IV.1 converge to a stable equilibrium point $W^{\star}\in\Omega$.

######  Theorem 3.

The policy updates $\pi^{t}(s,a)$ for all $a\in\pazocal{A}(s)$,
$s\in\mathbf{S}$ in Line 11 of Algorithm IV.1 converge to a stable equilibrium
point $\pi^{\star}\in\Pi$.

##  V Experiments

(a) Pre-Trained Scenario 1

(b) Pre-Trained Scenario 2

(c) Pre-Trained Scenario 3

(d) Pre-Trained Scenario 4

(e) Pre-Trained Scenario 5

(f) Deployment Scenario 1

(g) Deployment Scenario 2

(h) Deployment Scenario 3

(i) Deployment Scenario 4

Figure 2: Case Study 2 Setup: This figure shows a $16\times 16$ grid. The blue
(green) color state is the start (goal) state and the grey shaded states
represent obstacles. The top row shows the configurations of obstacles that
are used to obtain the pretrained critics. The bottom row shows the obstacle
configurations on which our MCAC algorithm is evaluated. The four deployment
scenarios in the bottom row are obtained by combining the pretrained scenarios
in the top row in different ways.

This section presents two case studies to compare our proposed multi-critic
actor-critic (MCAC) algorithm against a baseline actor-critic (Baseline AC).
We first present the details of the two case study scenarios developed for
comparing the two algorithms. Next we describe the hyper parameters used in
Baseline AC and MCAC algorithms and define metrics used to compare both
algorithms. We then present a comparison of results from the Baseline AC and
our MCAC algorithm on two case study scenarios.

###  V-A Case Study Environments

We demonstrate effectiveness of our MCAC algorithm through experiments
conducted in two distinct grid world environments of dimensions $5\times 5$
(Case Study 1) and $16\times 16$ (Case Study 2). Our experiments evaluate
capabilities of RL agents in scenarios of varying complexity. Each grid world
is populated with obstacles and potentially hazardous areas, challenging the
agent to learn to navigate effectively to a goal location, starting from an
initial position. In the $5\times 5$ grid, $1\times 1$ obstacles are placed at
predefined grid locations; the $16\times 16$ grid features a more complex
arrangement of obstacles, with $4\times 4$ obstacles placed at predefined
locations. Details about the grid environments are presented below.

State Space: The state space, ${S}$, of our grid-worlds is defined by the
discrete positions that an agent can occupy on a grid. For a grid of size
$L\times L$, the state space has $L^{2}$ possible positions, each uniquely
identified by its grid coordinates $(x,y)$. We examine $5\times 5$ and
$16\times 16$, grids leading to state spaces of $25$ and $256$ distinct
states, respectively.

Action Space: The action space, $A$, available to the agent consists of four
deterministic actions: move up ($\uparrow$), move down ($\downarrow$), move
left ($\leftarrow$), and move right ($\rightarrow$). When the agent takes an
action in a certain grid position (state), it moves one position in that
direction, unless there is something in the way (e.g., obstacles, edge of
grid). When the agent attempts to move beyond the grid’s edges, the agent’s
position remains unchanged.

Transition Probabilities:  The transition probabilities, $P(s^{\prime}|s,a)$,
describe the likelihood of moving from state $s$ to state $s^{\prime}$ given
action $a$. In our grid-worlds, if there is no obstacle, these probabilities
are either $0$ or $1$; a probability of $1$ indicates that action $a$ taken in
state $s$ will always result in transition to state $s^{\prime}$, and $0$
otherwise. When the agent is in an obstacle state, the agent will stay in the
obstacle state with probability $p_{\text{Stay}}=0.75$ and move to the state
suggested by the action with probability $1-p_{\text{Stay}}=0.25$.

Reward Structure The agent receives reward $r(s,a,s^{\prime})$ for each
transition from state $s$ to state $s^{\prime}$ via action $a$. Specifically,
$r(s,a,s^{\prime})=100$ when the agent takes an action that reaches the goal
state, and $r(s,a,s^{\prime})=-1$ for every other action taken by the agent.
This reward structure aims to balance exploration and exploitation by
incentivizing the discovery of the shortest path to the goal.

###  V-B Experiment Setup

TABLE II: This table presents a comparison of the actor-critic (AC) algorithm
and our multi-critic actor critic (MCAC) algorithm on Case Study 1 using four
metrics: average total reward at convergence ($\uparrow$), average number of
steps to reach goal state from start state ($\downarrow$), average runtime to
convergence ($\downarrow$), and number of episodes to convergence
($\downarrow$). All averages are calculated over 100 independent runs. Our
results show that across all deployment scenarios, MCAC consistently achieves
higher average reward, converges in fewer steps, takes less time to
convergence (with speedup (SU1) up to 2.31x), and fewer episodes to
convergence (with speedup (SU2) up to 10.44x). The total speedup $SU=SU1\times
SU2$ of our MCAC algorithm is up to 22.76x compared to the baseline AC
algorithm and 14.35x on average. Deployment
Scenario | Average Total Reward | Average Number of Steps | Average Runtime | Number of Episodes | Total SU
---|---|---|---|---|---
AC | MCAC | AC | MCAC | AC | MCAC | SU1 | AC | MCAC | SU2
|  |  |  |  |  |  |  |  |
1 | $85.98$ | $\mathbf{90.79}$ | 15.02 | $\mathbf{10.21}$ |  $0.18$ s |  $\mathbf{0.07}$ s | $2.41\times$ | 98 | 26 | $3.77\times$ | $9.09\times$
2 | $84.25$ | $\mathbf{89.37}$ | 16.75 | $\mathbf{11.63}$ | 0.17 s |  $\mathbf{0.08}$ s | $2.18\times$ | 94 | 9 | $10.44\times$ | $22.76\times$
3 | $82.44$ | $\mathbf{89.07}$ | 18.56 | $\mathbf{11.93}$ |  $0.17$ s |  $\mathbf{0.08}$ s | $2.17\times$ | 98 | 20 | $4.90\times$ | $10.63\times$
4 | 80.73 | $\mathbf{87.73}$ | 20.27 | $\mathbf{13.27}$ |  $0.20$ s |  $\mathbf{0.08}$ s | $2.31\times$ | 97 | 15 | $6.47\times$ | $14.95\times$
TABLE III: This table presents a comparison of the actor-critic (AC) algorithm
and our multi-critic actor critic (MCAC) algorithm on Case Study 2 using four
metrics: average total reward at convergence ($\uparrow$), average number of
steps to reach goal state from start state ($\downarrow$), average runtime to
convergence ($\downarrow$), and number of episodes to convergence
($\downarrow$). All averages are calculated over 100 independent runs. Our
results show that across all deployment scenarios, MCAC consistently achieves
higher average reward, converges in fewer steps, takes less time to
convergence (with speedup (SU1) up to 2.06x), and fewer episodes to
convergence (with speedup (SU2) up to 4.76x). The total speedup $SU=SU1\times
SU2$ of our MCAC algorithm is up to 9.37x compared to the baseline AC
algorithm, and 8.44x on average. Deployment
Scenario | Average Total Reward | Average Number of Steps | Average Runtime | Number of Episodes | Total SU
---|---|---|---|---|---
AC | MCAC | AC | MCAC | AC | MCAC | SU1 | AC | MCAC | SU2
|  |  |  |  |  |  |  |  |
1 | $-236.95$ | $\mathbf{-70.41}$ | 337.95 | $\mathbf{171.41}$ |  $2.66$ s |  $\mathbf{1.40}$ s | $1.89\times$ | 98 | 25 | $3.92\times$ | $7.41\times$
2 | $-223.89$ | $\mathbf{-82.40}$ | 324.89 | $\mathbf{183.40}$ | 2.99 s |  $\mathbf{1.45}$ s | $2.06\times$ | 100 | 22 | $4.55\times$ | $9.37\times$
3 | $-235.33$ | $\mathbf{-90.86}$ | 336.33 | $\mathbf{191.86}$ |  $2.85$ s |  $\mathbf{1.49}$ s | $1.91\times$ | 100 | 21 | $4.76\times$ | $9.09\times$
4 | -256.16 | $\mathbf{-109.88}$ | 357.16 | $\mathbf{210.88}$ |  $3.00$ s |  $\mathbf{1.55}$ s | $1.93\times$ | 90 | 22 | $4.09\times$ | $7.89\times$

For Case Study 1, we first train the AC algorithm on three pre-trained
scenarios given in the top row of Fig. 1 and save the converged value
functions at each state for each scenario. Then we run both AC and MCAC
algorithms on the four deployment scenarios shown in the bottom row of Fig. 1.
We use initial state $1$ and goal state $25$ for both pre-trained and
deployment scenarios corresponding to Case Study 1. NThe deployment scenarios
are obtained by the union of two or more of the obstacles presented in the
pre-trained scenarios.

For Case Study 2, we first train the AC algorithm on four pre-trained
scenarios given in the top row of Fig. 2 and save the converged value
functions at each state for each scenario. Then we run both AC and MCAC
algorithms on deployment scenarios shown in the bottom row of Fig. 2. Case
Study 2 presents more challenging tasks for the MCAC algorithm due to randomly
generated obstacles that increase in complexity across the deployment
scenarios.

For both environments, we run the AC and MCAC algorithms for 100 episodes, and
we conduct 100 independent experiments for each of the pre-trained and
deployment scenarios. An episode is defined as the agent starting from the
initial state and reaching the goal state.

###  V-C Metrics

We utilize the following metrics to compare performance of our MCAC algorithm
with the AC algorithm. All averages are over 100 independent experiments.

Average Total Reward: The total reward accumulated by the agent in the 100th
episode.

Average Number of Steps: The number of steps taken by the agent to reach the
goal in the 100th episode.

Average Runtime: The amount of time taken to run 100 episodes for each
algorithm.

Number of Episodes: The total number of episodes taken by the AC algorithm to
achieve the highest average total reward and the total number of episodes
taken by the MCAC algorithm to match or exceed the highest average total
reward achieved by the AC algorithm.

Speed Up - SU1, SU2, and Total SU: These metrics quantify efficiency gains of
the MCAC algorithm over the AC algorithm. SU1 is the ratio of the Average
Runtime of AC to Average Runtime of MCAC. SU2 is the ratio of the Number of
Episodes of AC to Number of Episodes of MCAC. The Total SU is the product of
SU1 and SU2, representing the overall speedup achieved by the MCAC algorithm
compared to the AC algorithm.

(a) Deployment Scenario 2 - Average Total Reward

(b) Deployment Scenario 2 - Average Number of Steps

Figure 3: This figure compares the baseline actor-critic (AC) algorithm and
our multi-critic actor critic (MCAC) algorithm in terms of average total
reward (left column) and average number of steps to reach the goal (right
column) in Deployment Scenario 2 for Case Study 1. Shaded regions indicate the
variance. MCAC consistently achieves higher average reward and does so in
significantly fewer episodes. Using MCAC also results in smaller variance
compared to the baseline AC algorithm. Our MCAC algorithm also achieves up to
10.44x speedup.

###  V-D Results

Table II compares performance of our MCAC algorithm against the AC algorithm
for Case Study 1. The results indicate that the MCAC algorithm consistently
achieves a higher average total reward (by $\approx+5$) and enables the agent
to reach the goal in fewer steps across all deployment scenarios. Our MCAC
algorithm also takes less time to convergence (with speedup (SU1) up to
2.31x), and fewer episodes to convergence (with speedup (SU2) up to 10.44x).
The total speedup $SU=SU1\times SU2$ of MCAC is up to 22.76x compared to the
baseline AC and 14.35x on average.

Table III compares performance of our MCAC algorithm against the AC algorithm
for Case Study 1. The results indicate that the MCAC algorithm consistently
achieves a higher average total reward (by $\approx+140$) and enables the
agent to reach the goal in fewer steps across all deployment scenarios. Our
MCAC algorithm also takes less time to convergence (with speedup (SU1) up to
2.06x), and fewer episodes to convergence (with speedup (SU2) up to 4.76x).
The total speedup $SU=SU1\times SU2$ of MCAC is up to 9.37x compared to the
baseline AC and 8.44x on average.

Fig. 3 compares the baseline actor-critic (AC) algorithm and our multi-critic
actor critic (MCAC) algorithm in terms of average total reward (left column)
and average number of steps to reach the goal (right column). We observe that
MCAC consistently achieves higher average reward and does so in significantly
fewer episodes. Using MCAC also results in smaller variance compared to the
baseline AC algorithm.

Our results show that by leveraging pre-existing knowledge of environments
through value functions, our MCAC algorithm achieves superior outcomes in
terms of optimality and convergence compared to the AC algorithm across
distinct environments. The initial knowledge equips MCAC with an informed
starting point, facilitating more effective exploration. Faster convergence of
MCAC can be attributed to this improved exploration capability, eliminating a
need to train critics from scratch. Instead, MCAC fine-tunes weights
corresponding to pre-trained critic value functions, efficiently estimating
the value of the current deployment environment to identify the optimal
solution.

##  VI Conclusion

This paper developed a novel algorithm to facilitate fast autonomy transfer in
reinforcement learning. Compared to methods that require retraining of
parameters when operating in new environments, our multi-critic actor-critic
(MCAC) algorithm used knowledge of pretrained value functions and integrated
these using a weighted average to expedite adaptation to new environments. We
demonstrated that MCAC achieved higher rewards and significantly faster (up to
22.76x) autonomy transfer compared to a baseline actor-critic algorithm across
multiple deployment scenarios in two distinct enrvironments. The success of
MCAC underscores significance of knowledge transfer in reinforcement learning,
offering a promising direction for future research in developing more
adaptable and efficient learning systems.

## References

  * [1] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. MIT Press, 2018.
  * [2] R. Hafner and M. Riedmiller, “Reinforcement learning in feedback control,” _Machine Learning_ , vol. 84, pp. 137–169, 2011.
  * [3] V. Mnih _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, 2015.
  * [4] D. Silver _et al._ , “Mastering the game of Go with deep neural networks and tree search,” _Nature_ , vol. 529, no. 7587, 2016.
  * [5] C. Zhang, P. Patras, and H. Haddadi, “Deep learning in mobile and wireless networking: A survey,” _IEEE Communications Surveys & Tutorials_, vol. 21, no. 3, pp. 2224–2287, 2019.
  * [6] D. Sadigh, S. Sastry, S. A. Seshia, and A. D. Dragan, “Planning for autonomous cars that leverage effects on human actions.” in _Robotics: Science and Systems_ , 2016.
  * [7] Z. Yan and Y. Xu, “Data-driven load frequency control for stochastic power systems: A deep reinforcement learning method with continuous action search,” _IEEE Trans. on Power Systems_ , vol. 34, no. 2, 2018.
  * [8] C. You, J. Lu, D. Filev, and P. Tsiotras, “Advanced planning for autonomous vehicles using reinforcement learning and deep inverse RL,” _Robotics and Autonomous Systems_ , vol. 114, pp. 1–18, 2019.
  * [9] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep reinforcement learning: A brief survey,” _IEEE Signal Processing Magazine_ , vol. 34, no. 6, pp. 26–38, 2017.
  * [10] Z. Zhu, K. Lin, A. K. Jain, and J. Zhou, “Transfer learning in deep reinforcement learning: A survey,” _IEEE Trans. on Pattern Analysis and Machine Intelligence_ , 2023.
  * [11] V. Campos, P. Sprechmann, S. S. Hansen, A. Barreto, S. Kapturowski, A. Vitvitskyi, A. P. Badia, and C. Blundell, “Beyond fine-tuning: Transferring behavior in reinforcement learning,” in _ICML 2021 Workshop on Unsupervised Reinforcement Learning_ , 2021.
  * [12] C. J. Watkins and P. Dayan, “Q-learning,” _Machine learning_ , vol. 8, pp. 279–292, 1992.
  * [13] V. Konda and J. Tsitsiklis, “Actor-critic algorithms,” _Advances in neural information processing systems_ , vol. 12, 1999.
  * [14] S. Mysore, G. Cheng, Y. Zhao, K. Saenko, and M. Wu, “Multi-critic actor learning: Teaching rl policies to act with style,” in _International Conference on Learning Representations_ , 2021.
  * [15] L. Li, Y. Li, W. Wei, Y. Zhang, and J. Liang, “Multi-actor mechanism for actor-critic reinforcement learning,” _Information Sciences_ , vol. 647, p. 119494, 2023.
  * [16] R. T. Icarte, T. Q. Klassen, R. Valenzano, and S. A. McIlraith, “Reward machines: Exploiting reward function structure in rl,” _Journal of Artificial Intelligence Research_ , vol. 73, pp. 173–208, 2022.
  * [17] R. T. Icarte, T. Klassen, R. Valenzano, and S. McIlraith, “Using reward machines for high-level task specification and decomposition in reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 2107–2116.
  * [18] R. T. Icarte, T. Q. Klassen, R. Valenzano, M. P. Castro, E. Waldie, and S. A. McIlraith, “Learning reward machines: A study in partially observable reinforcement learning,” _Artificial Intelligence_ , vol. 323, p. 103989, 2023.
  * [19] M. L. Puterman, _Markov decision processes: Discrete stochastic dynamic programming_. John Wiley & Sons, 2014.
  * [20] H. J. Kushner and D. S. Clark, _Stochastic approximation methods for constrained and unconstrained systems_. Springer Science & Business Media, 2012, vol. 26.
  * [21] M. Metivier and P. Priouret, “Applications of a Kushner and Clark lemma to general classes of stochastic algorithms,” _IEEE Trans. on Information Theory_ , vol. 30, no. 2, pp. 140–151, 1984.
  * [22] X. Dong, Z. Yu, W. Cao, Y. Shi, and Q. Ma, “A survey on ensemble learning,” _Frontiers of Computer Science_ , vol. 14, pp. 241–258, 2020.
  * [23] M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith _et al._ , “Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 23 965–23 998.
  * [24] V. S. Borkar, _Stochastic Approximation: A Dynamical Systems Viewpoint_. Springer, 2009, vol. 48.