††thanks: These two authors contributed equally††thanks: These two authors
contributed equally††thanks: Corresponding authors††thanks: Corresponding
authors††thanks: Corresponding authors

# Quantum Compiling with Reinforcement Learning on a Superconducting Processor

Z. T. Wang1,2,5 Qiuhao Chen3,4 Yuxuan Du6,4<EMAIL_ADDRESS>Z. H.
Yang2,5 Xiaoxia Cai1 Kaixuan Huang1 Jingning Zhang1 Kai Xu1,2,5,7 Jun Du1
Yinan Li3 Yuling Jiao3 Xingyao Wu4 Wu Liu4 Xiliang Lu3 Huikai Xu1 Yirong Jin1
Ruixia Wang1<EMAIL_ADDRESS>Haifeng Yu1,7<EMAIL_ADDRESS>S. P.
Zhao1,2,5,7 1Beijing Academy of Quantum Information Sciences, Beijing 100193,
China  2Beijing National Laboratory for Condensed Matter Physics, Institute of
Physics, Chinese Academy of Sciences, Beijing 100190, China  3School of
Mathematics and Statistics, Wuhan University, Wuhan, China  4JD Explore
Academy, Beijing, 101111, China  5School of Physical Sciences, University of
Chinese Academy of Sciences, Beijing 100190, China  6School of Computer
Science and Engineering, Nanyang Technological University, Singapore 639798,
Singapore  7Hefei National Laboratory, Hefei 230088, China

###### Abstract

To effectively implement quantum algorithms on noisy intermediate-scale
quantum (NISQ) processors is a central task in modern quantum technology. NISQ
processors feature tens to a few hundreds of noisy qubits with limited
coherence times and gate operations with errors, so NISQ algorithms naturally
require employing circuits of short lengths via quantum compilation. Here, we
develop a reinforcement learning (RL)-based quantum compiler for a
superconducting processor and demonstrate its capability of discovering novel
and hardware-amenable circuits with short lengths. We show that for the three-
qubit quantum Fourier transformation, a compiled circuit using only seven CZ
gates with unity circuit fidelity can be achieved. The compiler is also able
to find optimal circuits under device topological constraints, with lengths
considerably shorter than those by the conventional method. Our study
exemplifies the codesign of the software with hardware for efficient quantum
compilation, offering valuable insights for the advancement of RL-based
compilers.

##  I Introduction

Quantum computers are expected to outperform classical ones in various
applications [1, 2, 3, 4]. At present, the noisy intermediate-scale quantum
(NISQ) processors already exist and will play an important role in the coming
decades [5, 2] before scalable fault-tolerant quantum processors are developed
[6]. These NISQ processors consist of tens to a few hundred of noisy qubits
with relatively short coherence time and gate operations with errors.
Considerable effort has been devoted to maximizing their utility in different
ways such as quantum circuit compilation [7, 8, 9, 10, 11, 12, 13, 14] and
quantum error mitigation [7, 15, 16, 17, 18, 19]. In particular, quantum
compilation is the key step of quantum computation that translates quantum
algorithms into hardware instructions for a quantum processor. For the NISQ
processors, it is crucial to compile circuits with short length [7]
considering at the same time the qubit connectivity, gate expressivity, and
codesign of the software and hardware [12, 14] in order to achieve efficient
quantum compilation.

The performance of quantum compilers is in general characterized by three
metrics: fidelity, circuit length, and inference time, as illustrated in Fig.
1(a). Since identifying the optimal sequence of quantum gates is NP-hard [20],
various quantum compilers have been developed and each class of quantum
compilers holds its own strengths and weaknesses. For instance, Solovay-Kitaev
based compilers [21, 22, 23] excel in compiling accuracy with the ability to
achieve near-perfect fidelity, but at the cost of longer inference time and
circuit length, and with the limitation of compiling gates for at most two
qubits. To address these issues, heuristic compilers have been proposed [24,
25, 26, 27, 28, 29, 30, 31, 32], which use greedy search and variational
strategy to estimate the optimal gate sequence, with the central goal to
reduce the gate count and circuit length in quantum applications.

Figure 1: Architecture of the RL-based quantum compiler. (a) Three metrics to
measure the quality of a quantum compiler. (b) Precompilation. The action
space is first initialized based on the native gate set and circuit topology
of the superconducting processor. Then the agent starts the training of deep
Q-network (DQN). Namely, DQN constantly queries the database, highlighted by
two boxes ‘Percept’ and ‘Value’ in which small cuboids with different colors
and heights represent different unitaries and values, to generate training
data by pairing unitaries with the information of correct gate decompositions.
The data training is from easy to hard along the downward arrow. (c)
Inference. Given an unknown target unitary, the RL agent employs the AQ*
search guided by the trained DQN to predict the decomposed gate sequence,
which is finally implemented in the quantum processor. Here, compilation for
the two-qubit system is illustrated for simplicity.

Reinforcement learning (RL), as a general way of optimizing strategies with
complex decision-making problems, has recently emerged as a promising approach
to address limitations in quantum compilers. This approach includes a
precompilation procedure enabling manifest reduction of the overall execution
time [33]. Numerical studies have verified their abilities in generating near-
optimal gate sequences with high accuracy in single- and two-qubit systems
[34, 35]. In addition, a quantum compiler for an eight-qubit system is
proposed by training RL models to predict parameterized compiling circuit
structures with a variational quantum algorithm [36]. RL algorithms are also
tailored for qubit routing to minimize SWAP gate for real-world hardware
execution [37, 38]. Despite the progress, RL-based compilations on real
quantum processors and their performance remain largely unexplored.

In this work, we develop a quantum compiler based on deep Q-network (DQN) and
AQ* search strategy [35] for a superconducting multiqubit processor with a
given qubit connectivity and native gate set. Conceptually, the DQN alleviates
the challenge of exponentially increasing search space, and the AQ* search
provides an efficient inference. The synergy between DQN and AQ* search
guarantees efficiency in multiqubit compiling tasks. We conduct extensive
experiments on the processor to show that the compiler is able to discover
novel and hardware-amenable circuits with short lengths, showcasing remarkable
performance in two- and three-qubit systems. Particularly, we demonstrate that
for the three-qubit quantum Fourier transformation (QFT), the compiler can
find circuits having only seven CZ gates with unity circuit fidelity. In the
presence of hardware topological constraints, it is able to compile circuits
with lengths considerably shorter than those by conventional methods. Our work
offers an example of codesigning the software with hardware in quantum
compilation and allows for a practical assessment of quantum compilers for the
NISQ processors.

Figure 2: Compilation for two-qubit system. (a) SWAP gate. The blue-colored
circuit represents the standard decomposition, while the pink-colored circuit
shows the result by RL compilation with the inference time of $\sim 1.23$
seconds. The lower-left panel displays the statistical results of fidelity
$F_{2}$ measured on Q1-Q2 for each circuit with the same color, and histograms
drawn with dashed lines refer to the results of $F_{1}$. (b) Similar results
for $\text{R}_{\text{ZZ}}(\pi/{2})$ gate with pink and green colors
representing two gate sequences returned by the RL compiler. The inference
times are comparable to that for the SWAP gate.

##  II Results

###  II.1 RL-based compiler and experimental system

RL-based compiling starts with the considerations of the native gate set
$\mathcal{G}$ and the device topology of a quantum processor. In our
experiment, we use a $9$-qubit superconducting processor with tunable qubit
coupling and investigate the performance of the circuits from the RL-based
compiler (or RL compiler) for the two- and three-qubit chains formed in the
processor. A native gate set
$\mathcal{G}=\\{\pm\text{X}/2,\pm\text{Y}/2,\text{T},\text{T}^{\dagger},\text{CZ}\\}$
is used and the average single- and two-qubit gate fidelities are about
$99.9\%$ and $99.3\%$, respectively (see Supplementary Information A for
details). For a given target unitary $\hat{U}$, the compiler returns, within
the reference time, a gate sequence
$\\{\tilde{G}_{i}\\}_{i=1}^{N}\in\tilde{\mathcal{G}}^{N}$ approximating
$\hat{U}$, where $\tilde{\mathcal{G}}$ is the generalized gate set considering
device topology. For example, in the two-qubit case, the T gate in
$\mathcal{G}$ is extended to
$\text{T}\otimes\mathbb{I},\mathbb{I}\otimes\text{T}\in\tilde{\mathcal{G}}$.
We follow conventions to measure the performance of the RL compiler in terms
of the gate sequence length (or circuit length), the circuit fidelity, and the
inference time. For the circuit length, it amounts to the total gate number
$N=N_{1}+N_{2}$ in $\\{\tilde{G}_{i}\\}_{i=1}^{N}$, where $N_{1}$ and $N_{2}$
denote the number of single- and two-qubit gates, respectively. The circuit
fidelity is evaluated by
$F_{1}(\hat{U},\hat{V})=\text{Tr}(\hat{U}^{\dagger}\hat{V})/2^{M}$, where $M$
is the number of qubits and $\hat{V}=\prod_{i=1}^{N}\tilde{G}_{i}$ is the
circuit unitary [39]. The corresponding experimental fidelity is obtained by
quantum process tomography (QPT), i.e.,
$F_{2}(\hat{U},\hat{V})=\text{Tr}(\mathcal{\hat{E}}^{\dagger}\hat{\mathcal{F}})$,
where $\mathcal{\hat{E}}$ and $\hat{\mathcal{F}}$ are the calculated and
measured process matrices of $\hat{U}$ and $\hat{V}$, respectively.

The compiling process has two stages: precompilation and inference.
Precompilation aims to train a learning model that decomposes any unitary
$\hat{U}$ into an optimal sequence of gates in $\tilde{\mathcal{G}}$ [34] and
can be approached by deep Q-learning algorithm to capture the target learning
model, dubbed deep Q-Network (DQN) [40]. As shown in Fig. 1(b), the action
space $\mathcal{A}=\\{A_{i}\\}_{i=1}^{d}$, equivalent to
$\tilde{\mathcal{G}}$, contains elements representing legitimate operations on
the quantum processor. Once the action space is settled, the environment
$\mathsf{E}$ (a.k.a, database) and the agent $\mathsf{A}$ interact to learn
collaboratively with in total $L$ loops. Specifically, the first loop with
$l=1$ starts from the identity operation $\mathbb{I}$, as $\mathsf{E}$ applies
each action in $\mathcal{A}$ to $\mathbb{I}$ to synthesize $d$ percepts
(unitaries) $\mathcal{W}_{1}=\\{W_{k}\\}_{k=1}^{d}$ associated with $d$ values
$\mathcal{V}_{1}=\\{V_{k}\\}_{k=1}^{d}$. These values represent the minimum
number of gates required to revert the corresponding percept back to
$\mathbb{I}$ with a minus sign, i.e, $W_{k}=A_{k}$ and $V_{k}=-1$ for $\forall
k\in[d]$. Following the same routine in the second loop with $l=2$,
$\mathsf{E}$ applies each action in $\mathcal{A}$ to $\mathcal{W}_{1}$ to
synthesize $\mathcal{W}_{2}$ and calculates the corresponding values
$\mathcal{V}_{2}$. The cardinality of $\mathcal{W}_{2}$ is $d^{2}$ and there
are two possible outcomes of $0$ or $-2$. The prepared $\mathcal{W}_{1}$,
$\mathcal{A}$, and $\mathcal{V}_{2}$ are sent to $\mathsf{A}$ to conduct
learning, which is accomplished by the DQN. The Q-value function is utilized
to guide the optimization of DQN to minimize the distance between the
predictions and the ground truth [40]. After training $T$ epochs, $\mathsf{A}$
sends a terminal feedback to $\mathsf{E}$, which completes the current loop.
In the next loop $l=3$, $\mathsf{E}$ synthesizes more complex percepts
$\mathcal{W}_{3}$ and calculates the corresponding values $\mathcal{V}_{3}$,
and feeds $(\mathcal{W}_{2},\mathcal{A},\mathcal{V}_{3})$ to $\mathsf{A}$ to
train DQN. Progressively, the percepts $\mathcal{W}_{l}$ become increasingly
complicated and gradually cover the whole unitary space when the total loop
number $L\rightarrow\infty$, implying that DQN is forced to learn a general
rule for gate decomposition (see Methods).

The inference, as depicted in Fig. 1(c), leverages the trained DQN to
decompose any unitary $\hat{U}$ into a sequence of native gates. This process
utilizes the AQ* search algorithm [41], which iteratively expands the percept
with the highest value until the target identity is reached or a pre-defined
search depth is exceeded. Below we will see that circuits with the shortest
length to date can be obtained from the AQ* search, and the circuit fidelity
$F_{1}$ further increases after optimizing single-qubit gate parameters using
a post-processing toolkit based on the variational principle [42]. We will dub
our compiler with this post-processing as a variational RL-based (VRL)
compiler (see Methods for details). Note that for superconducting processors,
the parametrization of single-qubit gates does not increase any gate count and
gate operation time, and can be conveniently implemented [43].

Figure 3: Compilation for the three-qubit system. (a) QFT circuits by Qiskit
and RL compiler (inference time $\sim$ 800 seconds). Squares denote single-
qubit gates from $\mathcal{G}$. (b) Performance analysis of the QFT circuits
on Q2-Q3-Q6 qubits using the total variance (TV) metric. The QFT circuit is
evaluated on four varied input states $|000\rangle$, $|$+$00\rangle$,
$|$++$0\rangle$, $|$+++$\rangle$ and $\hat{U}^{\prime}|000\rangle$, where
$\hat{U}^{\prime}$ is randomly sampled from SU(8). Here,
$d_{\text{TV}}(P_{i},P_{u})$ quantifies the TV distance between the
distributions of the ideal output state $P_{i}$ and the uniform distribution
$P_{u}$ in the measurement of computational basis. For each panel in Subplot
(b), the notations $d_{\text{TV,Qiskit}}$, $d_{\text{TV,RL}}$, and
$d_{\text{TV,VRL}}$ represent $d_{\text{TV}}(P_{e},P_{u})$ in which $P_{e}$
refers to the distribution of the generated state for QFT circuits compiled by
Qiskit, RL, and VRL compilers, respectively. A lower difference with
$d_{\text{TV}}(P_{i},P_{u})$ suggests a better performance. (c) Statistical
performance comparison of Qiskit, RL, and VRL by preparing arbitrary input
states. The notaions in x-axis and y-axis follow those in Subplot (b). The
inset shows the fidelity (dashed line refers to $F_{1}$ and histogram refers
to $F_{2}$) of the QFT circuits compiled by three indicated compilers.
Subplots (d) and (e) analyze the statistical performance of Qiskit and RL
compilers for compiling $10$ two-qubit gates with topological constraints
under the metric of $F_{1}$ (dashed line), $F_{2}$ and $N_{2}$.

###  II.2 Compilation for two-qubit system

To compile the two-qubit gates, we build DQN through an adaptation of ResNet
architecture [44] with 60 million parameters, and the loop number is set as
$45$. The training dataset comprises 3.02 billion examples, as detailed in
Methods.

Figure 2 shows the compiling and experimental results of the widely used SWAP
and $\text{R}_{\text{ZZ}}$ gates. The experiment is performed on Q1-Q2 (see
chip schematic on the right side in Fig. 1(c)) and each decomposed gate
sequence is performed five times repeatedly to evaluate the statistical
performance. For the SWAP gate, the gate sequences by the standard
decomposition and RL compiler show optimal results, both with three CZ gates
($N_{2}=3$) and circuit fidelity $F_{1}$=1. However, the replacement of the H
gate by the $\pm$X/2 gates enables the RL compiler to achieve a higher
experimental fidelity $F_{2}=0.9859\pm 0.0035$ compared to $F_{2}=0.9849\pm
0.0010$ from the standard compilation. This improvement arises because
implementing an H gate typically requires multiple single-qubit gates from
$\mathcal{G}$. With $\text{R}_{\text{ZZ}}(\pi/2)$, our compiler returns two
solutions. The first solution has the same CZ gate number ($N_{2}=2$) as the
standard solution but differs in the adopted single-qubit gates. As with the
SWAP gate, this difference leads to a higher fidelity $F_{2}=0.9854\pm 0.0043$
compared to $F_{2}=0.9811\pm 0.0023$ in the standard case. With the second
solution, only one CZ gate ($N_{2}=1$) is used and a fidelity $F_{2}=0.9877\pm
0.0036$ is achieved. These results highlight the capability of the RL compiler
to discover novel circuits with a reduced number of two-qubit gates. Results
for more general two-qubit targets, including random gates from SU(4), are
discussed in Supplementary Information C.

###  II.3 Compilation for three-qubit system

For the compilation of three-qubit systems, the loop number in training DQN is
set to $L=44$, resulting in a training dataset of 27.4 billion examples. The
DQN architecture is identical to the two-qubit case, but with an increased
parameter space to enhance performance during inference. To demonstrate the
performance of the compiler, we first consider QFT, a key operation in many
quantum algorithms to gain speedups. The constraint of CZ gate applied only to
the nearest-neighbor qubits in the chain is considered. In Fig. 3(a), we find
that the RL compiler yields a circuit with CZ gate count of $N_{2}=7$, as
compared to the Qiskit compiler [45] with $N_{2}=15$ and compilers employing
heuristic search [24, 46, 47] with lowest $N_{2}=8$.

The distributions of the output states by QFT for the initial states
$|000\rangle$, $|$+$00\rangle$, $|$++$0\rangle$, $|$+++$\rangle$, and a random
state are visualized in Fig. 3(b), where the ideal results and the
experimental data measured on Q2-Q3-Q6 for the circuits by the Qiskit, RL, and
VRL compilers, are presented. We observe successive improvements in the
experimental results for Qiskit, RL, and VRL compilers. This can be seen more
clearly from the total variation (TV) distance (detailed in Supplementary
Information B) introduced to measure the distance between the experimental and
ideal output distributions [46]. We prepare stochastic initial states and the
experimental results of TV distance are shown in Fig. 3(c). In the inset of
Fig. 3(c), the experimental fidelities $F_{2}$ between the target unitary and
the circuits returned by the Qiskit, RL, and VRL compliers are presented,
showing the same trend of improvements. From the TV distance and the fidelity,
the experimental results from the RL-compiled circuit are still better than
those from the circuit by Qiskit, even though they have the fidelities $F_{1}$
of 0.94 and 1, respectively. These results unveil the interplay among circuit
fidelity, circuit length and structure, and gate errors and decoherence of the
quantum processor.

We next consider the compilation of two-qubit gates between $\text{Q}_{2}$ and
$\text{Q}_{6}$ in the Q2-Q3-Q6 chain, which may occur for devices with limited
qubit connectivity and can only be realized via CZ gates between
$\text{Q}_{2}$ and $\text{Q}_{3}$ or $\text{Q}_{3}$ and $\text{Q}_{6}$. In
Figs. 3(d) and (e), we show the results of compiling $10$ random two-qubit
gates. On average, the RL compiler uses $N_{2}=6.36$ CZ gates with fidelity
$F_{2}=0.9342\pm 0.0089$, whereas the Qiskit compiler uses $N_{2}=35.72$ CZ
gates with fidelity $F_{2}=0.6264\pm 0.0525$. These results show a clear
advantage of the RL compiler in the presence of device topological
constraints. The advantage can be attributed in part to the compiling approach
of starting from the matrix representation of the target operator and globally
seeking the gate sequence, whereas conventional compilers typically begin with
an initial circuit guess and locally optimize the structure or parameters.

###  II.4 Experimental error analysis

For NISQ processors, the experimental fidelity $F_{2}$ is always lower than
the circuit fidelity $F_{1}$ due to the imperfect gate operation, qubit
decoherence, and initial-state preparation and measurement errors. In Fig.
4(a), we present the results of $F_{2}$ as a function of circuit depth,
showing a clear decrease in fidelity with increasing depth. As shown in Fig.
4(b), here we take one block, each containing a CZ gate and 4 random
$\text{R}_{\phi}(\pi/2)$ gates, as the unit circuit depth metric and apply it
to the upper and lower two qubits alternately in the three-qubit chain.
Besides, for the zero circuit depth, it refers to two $\text{R}_{\phi}(\pi/2)$
gates applied to each qubit, where experimental results indicate its fidelity
ranging from 0.991 to 0.994. This means that the errors from initial-state
preparation and final-state readout are small compared to the gate errors from
the six single-qubit gates with an average fidelity of 0.999. Fig. 4(c) shows
$F_{2}$ versus CZ gate fidelity. Each CZ gate fidelity is an average over all
CZ gates in the QFT circuit for seven three-qubit sets with intentionally
reduced gate fidelities (see also Supplementary Information A). As indicated
by the dashed lines, extrapolating the CZ gate fidelity to unity would result
in a fidelity of about 0.97, where the discrepancy to the perfect fidelity is
caused primarily by the imperfect operations of 28 single-qubit gates in the
circuit. These results indicate that for shallow circuits (with short
operation time relative to qubit coherence time), the gate fidelity is the
dominant factor for the experimental performance.

Figure 4: Error analysis. (a) Experimental fidelity measured by QPT versus the
unit circuit depth with random circuits shown in (b) for Q2-Q3-Q6. At the
zeroth depth, only two $\text{R}_{\phi}(\pi/2)$ gates are applied to each
qubit, where
$\text{R}_{\phi}(\pi/2)=\exp({-i\frac{\pi}{4}(\cos{\phi}\text{X}+\sin{\phi}\text{Y})})$
and $\phi$ refers to a set of specific angles. The star indicates the circuit
in Fig. 3(a) by RL compiler. (c) Experimental fidelity of QFT circuit measured
by QPT versus averaged CZ gate fidelity for seven three-qubit sets of
Q2-Q1-Q4, Q2-Q3-Q6, Q3-Q6-Q5, Q4-Q5-Q2, Q4-Q5-Q6, Q5-Q2-Q3, and Q6-Q5-Q2. The
line is a linear fit. (d) Circuit depth dependences of fidelity measured using
XEB for two three-qubit sets. The lines fit as discussed in the text.

We use cross entropy benchmarking (XEB) [48, 49](see Supplementary Information
A) to measure fidelity as a function of circuit depth, which allows for the
measurement with much deeper depth. The results for two three-qubit sets are
shown in Fig. 4(d) with the depth extending to 200. The data align well with
the heuristic fitting function [50, 14]: $\alpha(d)$ = $K^{d}\cdot F_{\rm
CZU}^{m}\cdot F_{\rm CZL}^{d-m}\cdot F_{\text{R}_{\phi}(\pi/2)}^{4d}$. Here
$F_{\rm CZU}$ and $F_{\rm CZL}$ denote the CZ fidelities for the upper and
lower qubit pairs, $F_{\text{R}_{\phi}(\pi/2)}$ is the average single-qubit
gate fidelity, $d$ is the circuit depth, and $m$ is the number of blocks
applied on the upper two qubits. A fitting parameter $K\approx$ 0.992 is
obtained, which accounts for errors uncovered by gate infidelities such as
decoherence, non-Markovian noise, and crosstalk, etc. A $K$ value comparable
to the CZ gate fidelity implies a comparable role played by factors other than
gate fidelity for deep circuits.

##  III Discussion and outlook

We have developed an RL-based compiler for the superconducting processor with
given qubit connectivity, native gate set, gate fidelity, and coherence
properties. The compiler shows excellent performance in finding short circuits
with high fidelity and in experimental implementations. The influence of
decoherence as well as gate and measurement errors is experimentally
investigated, which shows the requirement of better device properties for
further improvements and may help predict the performance of RL-based
compilers on a specific superconducting processor. Compared to the previous
RL-based compilers [34, 33], our DQN model leverages more information about
percept performance, leading to efficient training and inference, thereby
facilitating effective compilation for multiqubit targets. A future direction
will be to design more informative RL models [51, 52] for training,
particularly for scenarios of increasing qubit number. Incorporating
supervised learning techniques during training can penalize the RL model for
generating redundant gate sequences [53, 54], thus promoting efficient
inference. Furthermore, exploring the potential of RL-based compilers for
simultaneous circuit optimization [55, 56, 57] and gate decomposition [58, 59,
60, 61, 50] holds promise for significantly improving compilation efficiency.

##  IV Methods

Here we introduce the technical details in the precompilation stage, including
training data generation and the optimization of DQN. We also describe the
implementation of AQ* search, exploited in the inference stage.

###  IV.1 Training data generation

For training data generation, we adopt two techniques that are widely used in
reinforcement learning to improve the performance and robustness of a
compiler.

The first technique is adding a bounded perturbation to the initial operation
$\mathbb{I}$ [34]. The perturbed operation is denoted by $\mathbb{I}^{\prime}$
satisfying $F_{1}(\mathbb{I},\mathbb{I}^{\prime})<10^{-4}$. The employment of
a set of $\\{\mathbb{I}^{\prime}\\}$ effectively mitigates the limitation of
merely using $\mathbb{I}$ in the sense that all expansions stem from a single
percept.

The second technique is injecting randomness in preparing $\mathcal{W}_{l}$
and $\mathcal{V}_{l}$ when $l$ becomes large. Note that the construction rule
of $\mathcal{W}_{l}$ and $\mathcal{V}_{l}$ presented in the main text implies
that their cardinalities exponentially scale with $l$. Such computational
overhead and the memory cost become prohibitive for large $l$. For this
reason, when $l$ exceeds a threshold, we only collect a fraction of all
possible synthesized unitaries to form $\mathcal{W}_{l}$. The quantity of this
threshold depends on the problem in hand and the accessible memories of
classical hardware, as detailed in Supplementary Information B.

###  IV.2 Optimization of DQN

As discussed in the main text, when $l=2$, the prepared
$(\mathcal{W}_{1},\mathcal{A},\mathcal{V}_{2})$ is sent to the agent
$\mathsf{A}$ to conduct learning. This process is completed by DQN guided by
the Q-value function. Here we first expand this process and then extend the
optimization of DQN to the general setting.

When $l=2$, the percepts $\mathcal{W}_{2}$ received by $\mathsf{A}$ take the
form
$\mathcal{W}_{2}=\\{A_{1}A_{1},\cdots,A_{k}A_{k^{\prime}},\cdots,A_{d}A_{d}\\}$
and the values in $\mathcal{V}_{2}$ contain two outcomes, i.e.,
$V_{kk^{\prime}}\in\\{0,-2\\}$ for $\forall k,k^{\prime}\in[d]$. As shown in
Fig. 1(b), given these training data, DQN $Q_{\bm{\theta}}$ takes
$W_{i}\in\mathcal{W}_{1}$ as input and predicts the values of
$[A_{1}W_{i},A_{2}W_{i},...,A_{d}W_{i}]$ as
$\hat{\bm{y}}=[Q_{\bm{\theta}}(W_{i},A_{1}),Q_{\bm{\theta}}(W_{i},A_{2}),...,Q_{\bm{\theta}}(W_{i},A_{d})]\in\mathbb{R}^{d}$,
where the trainable parameters are denoted by $\bm{\theta}$. The optimization
of $\bm{\theta}$, which aims to minimize the distance between the predictions
$\hat{\bm{y}}$ and the ground truth
$\bm{y}=[V_{1i},V_{2i},\cdots,V_{di}]\in\\{-2,0\\}^{d}$, is guided by the
Q-value function in the mathematical form

$\mathcal{L}_{l}(\bm{\theta})=\frac{1}{d\cdot|\mathcal{W}_{l}|}\sum_{W_{i}\in\mathcal{W}_{l}}\sum_{j=1}^{d}\left(Q_{\bm{\theta}}(W_{i},A_{j})-V_{ji}\right)^{2},$
(1)

with setting $l=2$. The gradient descent optimizer is employed to update
$\bm{\theta}$ to minimize $\mathcal{L}_{l}(\bm{\theta})$ with $T$ epochs. This
completes the optimization with $l=2$.

Following the same routine, at the $l$-th loop, we feed
$(\mathcal{W}_{l-1},\mathcal{A},\mathcal{V}_{l})$ to $\mathsf{A}$ to conduct
learning. More specifically, the gradient descent optimizer is employed to
update $\bm{\theta}$ to minimize $\mathcal{L}_{l}(\bm{\theta})$. The
optimization of the $l$-th loop is completed after $T$ epochs.

We apply a slight modification to the loss $\mathcal{L}_{l}(\bm{\theta})$ when
$l$ exceeds a threshold value. Such a modification originates from the fact
that for large $l$, the acquisition of the values $\mathcal{V}_{l}$ is
computationally expensive, which prohibits the optimization of
$\mathcal{L}_{l}(\bm{\theta})$. To this end, we replace the values
$\mathcal{V}_{l}$ by its estimation $\hat{\mathcal{V}}_{l}$, i.e.,

$\displaystyle\hat{V}_{ji}=\max_{A\in\mathcal{A}}\left(Q_{\bm{\theta}}(A_{j}W_{i},A)-1\right),~{}\forall
j.$

Accordingly, the loss function is rewritten as

$\mathcal{L}_{l}(\bm{\theta})=\frac{1}{d\cdot|\mathcal{W}_{l}|}\sum_{W_{i}\in\mathcal{W}_{l}}\sum_{j=1}^{d}\left(Q_{\bm{\theta}}(W_{i},A_{j})-\hat{V}_{ji}\right)^{2}.$
(2)

Supported by the theoretical analysis in [62, 40], the optimization of the
above two equations is equivalent when $L\rightarrow\infty$. The algorithmic
details of DQN are deferred to Supplementary Information B.

###  IV.3 AQ* Search

AQ* search is a heuristic-based search algorithm designed to find a path
between a starting node $p_{s}$ and a goal node $p_{t}$ on a graph structure
[41]. In quantum compiling, each node corresponds to a percept representing a
synthesized unitary, the starting node and the goal node correspond to the
target unitary $\hat{U}$ and the initial percept $\mathbb{I}$ (or
$\mathbb{I}^{\prime}$), respectively. As indicated by [63, 41], A* search [64]
or AQ* search can solve Rubik’s cube, or equivalently quantum compiling, using
shorter solutions in less time than Monte Carlo tree search.

We recap the workflow of AQ* search for self-consistency [41]. Concisely, we
denote a node by $p$, which stands for a percept (or equivalently a
synthesized unitary) in $\\{\mathcal{W}_{l}\\}_{l=1}^{L}$. AQ* search
leverages an evaluation function

$\displaystyle f(p,A;\hat{U})=g(p;\hat{U})+Q_{\bm{\theta}}(p,A)$

to find the path between $p_{s}$ to $p_{t}$, where $A\in\mathcal{A}$ refers to
an action, $\hat{U}$ represents the target unitary, $g(p;\hat{U})$ counts the
number of the exploited actions from $\hat{U}$ to $p$ during the search with a
minus sign, and $Q_{\bm{\theta}}(p,a)$ is the output of the trained DQN after
optimizing Eq. (1). Supported by the evaluation function $f(\cdot,\cdot)$, AQ*
search seeks the solution in an iterative manner. It maintains a set of
intermediate percepts $\\{\hat{p}\\}$, which initially only contains the
starting node $p_{i}$. In each iteration, for all
$p^{\prime}\in\\{\hat{p}\\}$, the algorithm expands $p^{\prime}$ with a
successor node $p^{\star}=A^{\star}p^{\prime}$ by leveraging the evaluation
function with

$\displaystyle A^{\star}=\arg\max_{A\in\mathcal{A}}f(p^{\prime},A;\hat{U}).$

All the successor nodes are added into $\\{\hat{p}\\}$ if they have not been
encountered before. Note that this process is efficient since the expansion
for different $p^{\prime}$ can be executed in parallel. The iteration process
is terminated depending on two conditions. First, a node
$p^{\prime}\in\\{\hat{p}\\}$ has approached the goal node $p_{t}$ within a
small error, i.e., $1-F_{1}(p^{\prime},p_{t})<10^{-4}$. Second, the runtime of
searching exceeds a predefined threshold. In this case, AQ* search outputs a
node in $\\{\hat{p}\\}$ closest to $p_{t}$.

###  IV.4 Variational RL-based Compiler

The variational RL-based compiler adheres to the variational principle by
optimizing parameterized quantum gates while simulating the quantum operations
classically. Similar ideals have also been broadly used in tensor networks
[65]. In particular, given a target operation $\hat{U}$, the nodes (percepts)
set $\\{\hat{p}\\}$ generated by the AQ* Search can be transformed into a
sequence of quantum gates $\\{\tilde{G}_{i}\in\tilde{\mathcal{G}}\\}$
satisfying $\hat{V}=\prod\tilde{G}_{i}\approx\hat{U}$. Notably, the gate set
$\tilde{\mathcal{G}}$ is pre-fixed during compiling and remains fixed
throughout the process, limiting potential optimization opportunities. The
proposed variational RL-based compiler operates by taking $\hat{U}$ and
$\hat{V}$ as inputs. It then parameterizes all single-qubit gates within
$\\{\tilde{G}_{i}\\}$ and optimizes the resulting variational ansatz
$\hat{V}_{\bm{\gamma}}=\prod\tilde{G}_{i}(\gamma_{i})$ towards $\hat{U}$ on a
classical computer. The loss function is defined as

$\mathcal{L}(\hat{V}_{\bm{\gamma}},\hat{U})=\text{Tr}\left(\hat{U}^{\dagger}\cdot\hat{V}_{\bm{\gamma}}\right).$

Subsequently, a gradient descent algorithm is used to adjust the parameter
vector $\bm{\gamma}$ and minimize this loss function.

## Acknowledgements

We thank H. L. Sheng for help in the preparation of figures. Numerical
calculations were performed on the supercomputing system of the Supercomputing
Center at Wuhan University. This work was supported by the National Natural
Science Foundation of China (Grants Nos. 92365206, 21973003, 22288201,
22303005, 12104056, and T2322030), the Fundamental Research Funds for the
Central Universities and Innovation Program for Quantum Science and Technology
(Grant No. 2021ZD0301800), the National Nature Science Foundation of Hubei
(Grant No. 2024AFA045) and the Beijing Nova Program (Grant No. 2022000216).

## References

  * [1] Peter W Shor.  Polynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer.  SIAM Review, 41(2):303–332, 1999.
  * [2] Andrew J Daley, Immanuel Bloch, Christian Kokail, Stuart Flannigan, Natalie Pearson, Matthias Troyer, and Peter Zoller.  Practical quantum advantage in quantum simulation.  Nature, 607(7920):667–676, 2022.
  * [3] Hsin-Yuan Huang, Michael Broughton, Jordan Cotler, Sitan Chen, Jerry Li, Masoud Mohseni, Hartmut Neven, Ryan Babbush, Richard Kueng, John Preskill, et al.  Quantum advantage in learning from experiments.  Science, 376(6598):1182–1186, 2022.
  * [4] Hsin-Yuan Huang, Richard Kueng, Giacomo Torlai, Victor V Albert, and John Preskill.  Provably efficient machine learning for quantum many-body problems.  Science, 377(6613):eabk3333, 2022.
  * [5] John Preskill.  Quantum computing in the nisq era and beyond.  Quantum, 2:79, 2018.
  * [6] John Preskill.  Fault-tolerant quantum computation.  In Introduction to quantum computation and information, pages 213–269. World Scientific, 1998.
  * [7] Kishor Bharti, Alba Cervera-Lierta, Thi Ha Kyaw, Tobias Haug, Sumner Alperin-Lea, Abhinav Anand, Matthias Degroote, Hermanni Heimonen, Jakob S Kottmann, Tim Menke, et al.  Noisy intermediate-scale quantum algorithms.  Rev. Mod. Phys, 94(1):015004, 2022.
  * [8] Thaddeus D Ladd, Fedor Jelezko, Raymond Laflamme, Yasunobu Nakamura, Christopher Monroe, and Jeremy Lloyd O’Brien.  Quantum computers.  Nature, 464(7285):45–53, 2010.
  * [9] Ali JavadiAbhari, Shruti Patil, Daniel Kudrow, Jeff Heckey, Alexey Lvov, Frederic T Chong, and Margaret Martonosi.  Scaffcc: A framework for compilation and analysis of quantum computing programs.  In Proceedings of the 11th ACM Conference on Computing Frontiers, pages 1–10, 2014.
  * [10] Frederic T Chong, Diana Franklin, and Margaret Martonosi.  Programming languages and compiler design for realistic quantum hardware.  Nature, 549(7671):180–187, 2017.
  * [11] Alexandru Paler, Ilia Polian, Kae Nemoto, and Simon J Devitt.  Fault-tolerant, high-level quantum circuits: form, compilation and description.  Quantum Sci. Technol, 2(2):025003, 2017.
  * [12] Norbert M Linke, Dmitri Maslov, Martin Roetteler, Shantanu Debnath, Caroline Figgatt, Kevin A Landsman, Kenneth Wright, and Christopher Monroe.  Experimental comparison of two quantum computing architectures.  Proc. Natl. Acad. Sci., 114(13):3305–3310, 2017.
  * [13] Bettina Heim, Mathias Soeken, Sarah Marshall, Chris Granade, Martin Roetteler, Alan Geller, Matthias Troyer, and Krysta Svore.  Quantum programming languages.  Nat. Rev. Phys, 2(12):709–722, 2020.
  * [14] Petar Jurcevic, Ali Javadi-Abhari, Lev S Bishop, Isaac Lauer, Daniela F Bogorin, Markus Brink, Lauren Capelluto, Oktay Günlük, Toshinari Itoko, Naoki Kanazawa, Abhinav Kandala, George A Keefe, Kevin Krsulich, William Landers, Eric P Lewandowski, Douglas T McClure, Giacomo Nannicini, Adinath Narasgond, Hasan M Nayfeh, Emily Pritchett, Mary Beth Rothwell, Srikanth Srinivasan, Neereja Sundaresan, Cindy Wang, Ken X Wei, Christopher J Wood, Jeng-Bang Yau, Eric J Zhang, Oliver E Dial, Jerry M Chow, and Jay M Gambetta.  Demonstration of quantum volume 64 on a superconducting quantum computing system.  Quantum Sci. Technol, 6(2):025020, mar 2021.
  * [15] Suguru Endo, Simon C Benjamin, and Ying Li.  Practical quantum error mitigation for near-future applications.  Phys. Rev. X, 8(3):031027, 2018.
  * [16] Antonio D Córcoles, Abhinav Kandala, Ali Javadi-Abhari, Douglas T McClure, Andrew W Cross, Kristan Temme, Paul D Nation, Matthias Steffen, and Jay M Gambetta.  Challenges and opportunities of near-term quantum computing systems.  Proceedings of the IEEE, 108(8):1338–1352, 2019.
  * [17] Janusz Kusyk, Samah M Saeed, and Muharrem Umit Uyar.  Survey on quantum circuit compilation for noisy intermediate-scale quantum computers: Artificial intelligence to heuristics.  IEEE Transactions on Quantum Engineering, 2:1–16, 2021.
  * [18] Zhenyu Cai, Ryan Babbush, Simon C. Benjamin, Suguru Endo, William J. Huggins, Ying Li, Jarrod R. McClean, and Thomas E. O’Brien.  Quantum error mitigation.  Rev. Mod. Phys., 95:045005, Dec 2023.
  * [19] He-Liang Huang, Xiao-Yue Xu, Chu Guo, Guojing Tian, Shi-Jie Wei, Xiaoming Sun, Wan-Su Bao, and Gui-Lu Long.  Near-term quantum computing techniques: Variational quantum algorithms, error mitigation, circuit compilation, benchmarking and classical simulation.  Sci. China Phys. Mech. Astron, 66(5):250302, 2023.
  * [20] Adi Botea, Akihiro Kishimoto, and Radu Marinescu.  On the complexity of quantum circuit compilation.  In Eleventh annual symposium on combinatorial search, 2018.
  * [21] A Yu Kitaev.  Quantum computations: algorithms and error correction.  Russian Mathematical Surveys, 52(6):1191, 1997.
  * [22] Christopher M. Dawson and Michael A. Nielsen.  The solovay-kitaev algorithm.  Quantum Inf. Comput., 6:81–95, 2005.
  * [23] Alexei Yu Kitaev, Alexander Shen, Mikhail N Vyalyi, and Mikhail N Vyalyi.  Classical and quantum computation.  Number 47. American Mathematical Soc., 2002.
  * [24] Marc Grau Davis, Ethan Smith, Ana Tudor, Koushik Sen, Irfan Siddiqi, and Costin Iancu.  Heuristics for quantum compiling with a continuous gate set.  arXiv:1912.02727, 2019.
  * [25] Ed Younis, Koushik Sen, Katherine Yelick, and Costin Iancu.  Qfast: Quantum synthesis using a hierarchical continuous circuit space.  arXiv:2003.04462, 2020.
  * [26] Sumeet Khatri, Ryan LaRose, Alexander Poremba, Lukasz Cincio, Andrew T Sornborger, and Patrick J Coles.  Quantum-assisted quantum compiling.  Quantum, 3:140, 2019.
  * [27] Kunal Sharma, Sumeet Khatri, Marco Cerezo, and Patrick J Coles.  Noise resilience of variational quantum compiling.  New J. Phys, 22(4):043006, 2020.
  * [28] Xiaosi Xu, Simon C Benjamin, and Xiao Yuan.  Variational circuit compiler for quantum error correction.  Phys. Rev. Applied, 15(3):034068, 2021.
  * [29] Liam Madden and Andrea Simonetto.  Best approximate quantum compiling problems.  ACM Transactions on Quantum Computing, 3(2):1–29, 2022.
  * [30] Péter Rakyta and Zoltán Zimborás.  Approaching the theoretical limit in quantum gate decomposition.  Quantum, 6:710, 2022.
  * [31] Tyson Jones and Simon C Benjamin.  Robust quantum compilation and circuit optimisation via energy minimisation.  Quantum, 6:628, 2022.
  * [32] David A Herrera-Martí.  Policy gradient approach to compilation of variational quantum circuits.  Quantum, 6:797, 2022.
  * [33] Lorenzo Moro, Matteo GA Paris, Marcello Restelli, and Enrico Prati.  Quantum compiling by deep reinforcement learning.  Commun. Phys, 4(1):1–8, 2021.
  * [34] Yuan-Hang Zhang, Pei-Lin Zheng, Yi Zhang, and Dong-Ling Deng.  Topological quantum compiling with reinforcement learning.  Phys. Rev. Lett., 125(17):170501, 2020.
  * [35] Qiuhao Chen, Yuxuan Du, Qi Zhao, Yuling Jiao, Xiliang Lu, and Xingyao Wu.  Efficient and practical quantum compiler towards multi-qubit systems with deep reinforcement learning.  arXiv:2204.06904, 2022.
  * [36] Zhimin He, Lvzhou Li, Shenggen Zheng, Yongyao Li, and Haozhen Situ.  Variational quantum compiling with double q-learning.  New J. Phys, 23(3):033002, 2021.
  * [37] Steven Herbert and Akash Sengupta.  Using reinforcement learning to find efficient qubit routing policies for deployment in near-term quantum computers.  arXiv:1812.11619, 2019.
  * [38] Akash Sengupta Matteo G. Pozzi, Steven J. Herbert and Robert D. Mullins.  Using reinforcement learning to perform qubit routing in quantum compilers.  arXiv:2007.15957, 2020.
  * [39] Feng Bao, Hao Deng, Dawei Ding, Ran Gao, Xun Gao, Cupjin Huang, Xun Jiang, Hsiang-Sheng Ku, Zhisheng Li, Xizheng Ma, Xiaotong Ni, Jin Qin, Zhijun Song, Hantao Sun, Chengchun Tang, Tenghui Wang, Feng Wu, Tian Xia, Wenlong Yu, Fang Zhang, Gengyan Zhang, Xiaohang Zhang, Jingwei Zhou, Xing Zhu, Yaoyun Shi, Jianxin Chen, Hui-Hai Zhao, and Chunqing Deng.  Fluxonium: An alternative qubit platform for high-fidelity operations.  Phys. Rev. Lett., 129:010502, Jun 2022.
  * [40] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [41] Forest Agostinelli, Alexander Shmakov, Stephen McAleer, Roy Fox, and Pierre Baldi.  A* search without expansions: Learning heuristic functions with deep q-networks.  arXiv:2102.04518, 2021.
  * [42] Andrew D McLachlan.  A variational solution of the time-dependent schrodinger equation.  Molecular Physics, 8(1):39–44, 1964.
  * [43] David C. McKay, Christopher J. Wood, Sarah Sheldon, Jerry M. Chow, and Jay M. Gambetta.  Efficient $z$ gates for quantum computing.  Phys. Rev. A, 96:022330, Aug 2017.
  * [44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [45] Gadi Aleksandrowicz, Thomas Alexander, Panagiotis Barkoutsos, Luciano Bello, Yael Ben-Haim, David Bucher, F Jose Cabrera-Hernández, Jorge Carballo-Franquis, Adrian Chen, Chun-Fu Chen, et al.  Qiskit: An open-source framework for quantum computing.  2022\.
  * [46] Akel Hashim, Ravi K. Naik, Alexis Morvan, Jean-Loup Ville, Bradley Mitchell, John Mark Kreikebaum, Marc Davis, Ethan Smith, Costin Iancu, Kevin P. O’Brien, Ian Hincks, Joel J. Wallman, Joseph Emerson, and Irfan Siddiqi.  Randomized compiling for scalable quantum computing on a noisy superconducting quantum processor.  Phys. Rev. X, 11:041039, Nov 2021.
  * [47] Ying Lu, Peng-Fei Zhou, Shao-Ming Fei, and Shi-Ju Ran.  Quantum compiling with a variational instruction set for accurate and fast quantum computing.  Phys. Rev. Res., 5:023096, May 2023.
  * [48] Sergio Boixo, Sergei Isakov, Vadim Smelyanskiy, Ryan Babbush, Nan Ding, Zhang Jiang, John Martinis, and Hartmut Neven.  Characterizing quantum supremacy in near-term devices.  Nat. Phys, 14, 06 2018.
  * [49] C. Neill, P. Roushan, K. Kechedzhi, S. Boixo, S. V. Isakov, V. Smelyanskiy, A. Megrant, B. Chiaro, A. Dunsworth, K. Arya, R. Barends, B. Burkett, Y. Chen, Z. Chen, A. Fowler, B. Foxen, M. Giustina, R. Graff, E. Jeffrey, T. Huang, J. Kelly, P. Klimov, E. Lucero, J. Mutus, M. Neeley, C. Quintana, D. Sank, A. Vainsencher, J. Wenner, T. C. White, H. Neven, and J. M. Martinis.  A blueprint for demonstrating quantum supremacy with superconducting qubits.  Science, 360(6385):195–199, 2018.
  * [50] Y. Kharkov, A. Ivanova, E. Mikhantiev, and A. Kotelnikov.  Arline benchmarks: Automated benchmarking platform for quantum compilers.  arXiv:2202.14025, 2022.
  * [51] Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J. R. Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, David Silver, Demis Hassabis, and Pushmeet Kohli.  Discovering faster matrix multiplication algorithms with reinforcement learning.  Nature, 610(7930):47–53, 2022.
  * [52] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis.  Highly accurate protein structure prediction with alphafold.  Nature, 596(7873):583–589, 2021.
  * [53] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [54] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  arXiv:2203.02155, 2022.
  * [55] Thomas Fösel, Murphy Yuezhen Niu, Florian Marquardt, and Li Li.  Quantum circuit optimization with deep reinforcement learning.  arXiv:2103.07585, 2021.
  * [56] Mateusz Ostaszewski, Lea M Trenkwalder, Wojciech Masarczyk, Eleanor Scerri, and Vedran Dunjko.  Reinforcement learning for optimization of variational quantum circuit architectures.  Advances in Neural Information Processing Systems, 34:18182–18194, 2021.
  * [57] Gerhard W Dueck, Anirban Pathak, Md Mazder Rahman, Abhishek Shukla, and Anindita Banerjee.  Optimization of circuits for ibm’s five-qubit quantum computers.  In 2018 21st Euromicro Conference on Digital System Design (DSD), pages 680–684. IEEE, 2018.
  * [58] Juha J Vartiainen, Mikko Möttönen, and Martti M Salomaa.  Efficient decomposition of quantum gates.  Phys. Rev. Lett., 92(17):177902, 2004.
  * [59] Christopher C Paige and Musheng Wei.  History and generality of the cs decomposition.  Linear Algebra and its Applications, 208:303–326, 1994.
  * [60] A.M. Krol, A. Sarkar, I. Ashraf, Z. Al-Ars, and K Bertels.  Efficient decomposition of unitary matrices in quantum circuit compilers.  Appl. Sci, 12(2):759, 2022.
  * [61] Mikko Möttönen, Juha J Vartiainen, Ville Bergholm, and Martti M Salomaa.  Quantum circuits for general multiqubit gates.  Phys. Rev. Lett., 93(13):130502, 2004.
  * [62] Dimitri Bertsekas.  Reinforcement learning and optimal control.  Athena Scientific, 2019.
  * [63] Forest Agostinelli, Stephen McAleer, Alexander Shmakov, and Pierre Baldi.  Solving the rubik’s cube with deep reinforcement learning and search.  Nat. Mach. Intell, 1(8):356–363, 2019.
  * [64] Wei Zeng and Richard L Church.  Finding shortest paths on real road networks: the case for a.  International journal of geographical information science, 23(4):531–543, 2009.
  * [65] J. Ignacio Cirac, David Pérez-García, Norbert Schuch, and Frank Verstraete.  Matrix product states and projected entangled pair states: Concepts, symmetries, theorems.  Rev. Mod. Phys., 93:045003, Dec 2021.
  * [66] Xuegang Li, Huikai Xu, Junhua Wang, Ling-Zhi Tang, Dan-Wei Zhang, Chuhong Yang, Tang Su, Chenlu Wang, Zhenyu Mi, Weijie Sun, Xuehui Liang, Mo Chen, Chengyao Li, Yingshan Zhang, Kehuan Linghu, Jiaxiu Han, Weiyang Liu, Yulong Feng, Pei Liu, Guangming Xue, Jingning Zhang, S. P. Zhao, Yirong Jin, Shi-Liang Zhu, Haifeng Yu, and Qi-Kun Xue.  Mapping topology-disorder phase diagram with a quantum simulator.  arXiv:2301.12138, 2024.
  * [67] Xuegang Li, Yingshan Zhang, Chuhong Yang, Zhiyuan Li, Junhua Wang, Tang Su, Mo Chen, Yongchao Li, Chengyao Li, Zhenyu Mi, Xuehui Liang, Chenlu Wang, Zhen Yang, Yulong Feng, Kehuan Linghu, Huikai Xu, Jiaxiu Han, Weiyang Liu, Peng Zhao, Teng Ma, Ruixia Wang, Jingning Zhang, Yu Song, Pei Liu, Ziting Wang, Zhaohua Yang, Guangming Xue, Yirong Jin, and Haifeng Yu.  Vacuum-gap transmon qubits realized using flip-chip technology.  Appl. Phys. Lett, 119(18), 11 2021.  184003\.
  * [68] Eric Bogatin.  Signal and Power Integrity: Simpified.  Pearson; 3rd edition, 2018.
  * [69] Jonas Butscher.  Shaping of fast flux pulses for two-qubit gates, 2018.
  * [70] Shoukuan Zhao, Diandong Tang, Xiaoxiao Xiao, Ruixia Wang, Qiming Sun, Zhen Chen, Xiaoxia Cai, Zhendong Li, Haifeng Yu, and Wei-Hai Fang.  Quantum computation of conical intersections on a programmable superconducting quantum processor.  arXiv:2402.12708, 2024.
  * [71] J. Kelly, R. Barends, B. Campbell, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, A. G. Fowler, I.-C. Hoi, E. Jeffrey, A. Megrant, J. Mutus, C. Neill, P. J. J. O’Malley, C. Quintana, P. Roushan, D. Sank, A. Vainsencher, J. Wenner, T. C. White, A. N. Cleland, and John M. Martinis.  Optimal quantum control using randomized benchmarking.  Phys. Rev. Lett., 112:240504, Jun 2014.
  * [72] Frank Arute, Kunal Arya, Ryan Babbush, Dave Bacon, Joseph C Bardin, Rami Barends, Rupak Biswas, Sergio Boixo, Fernando GSL Brandao, David A Buell, et al.  Quantum supremacy using a programmable superconducting processor.  Nature, 574(7779):505–510, 2019.
  * [73] Andrew M. Childs, Isaac L. Chuang, and Debbie W. Leung.  Realization of quantum process tomography in nmr.  Phys. Rev. A, 64:012314, Jun 2001.
  * [74] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533, 2015.
  * [75] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv:1412.6980, 2014.
  * [76] David C McKay, Thomas Alexander, Luciano Bello, Michael J Biercuk, Lev Bishop, Jiayin Chen, Jerry M Chow, Antonio D Córcoles, Daniel Egger, Stefan Filipp, et al.  Qiskit backend specifications for openqasm and openpulse experiments.  2018\.
  * [77] Aram W Harrow, Benjamin Recht, and Isaac L Chuang.  Efficient discrete approximations of quantum gates.  J. Math. Phys, 43(9):4445–4451, 2002.
  * [78] Marco Cerezo, Andrew Arrasmith, Ryan Babbush, Simon C Benjamin, Suguru Endo, Keisuke Fujii, Jarrod R McClean, Kosuke Mitarai, Xiao Yuan, Lukasz Cincio, et al.  Variational quantum algorithms.  Nat. Rev. Phys, 3(9):625–644, 2021.
  * [79] Amira Abbas, David Sutter, Christa Zoufal, Aurélien Lucchi, Alessio Figalli, and Stefan Woerner.  The power of quantum neural networks.  Nat Comput Sci, 1(6):403–409, 2021.
  * [80] Hsin-Yuan Huang, Michael Broughton, Masoud Mohseni, Ryan Babbush, Sergio Boixo, Hartmut Neven, and Jarrod R McClean.  Power of data in quantum machine learning.  Nat. Commun, 12(1):1–9, 2021.
  * [81] Yuxuan Du, Yibo Yang, Dacheng Tao, and Min-Hsiu Hsieh.  Problem-dependent power of quantum neural networks on multiclass classification.  Phys. Rev. Lett., 131:140601, Oct 2023.
  * [82] M Cerezo, Guillaume Verdon, Hsin-Yuan Huang, Lukasz Cincio, and Patrick J Coles.  Challenges and opportunities in quantum machine learning.  Nat Comput Sci, 2(9):567–576, 2022.
  * [83] Jinkai Tian, Xiaoyu Sun, Yuxuan Du, Shanshan Zhao, Qing Liu, Kaining Zhang, Wei Yi, Wanrong Huang, Chaoyue Wang, Xingyao Wu, et al.  Recent advances for quantum neural networks in generative learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [84] Chu Guo, Youwei Zhao, and He-Liang Huang.  Verifying random quantum circuits with arbitrary geometry using tensor network states algorithm.  Phys. Rev. Lett., 126:070502, Feb 2021.
  * [85] Feng Pan, Pengfei Zhou, Sujie Li, and Pan Zhang.  Contracting arbitrary tensor networks: General approximate algorithm and applications in graphical models and quantum circuit simulations.  Phys. Rev. Lett., 125:060503, Aug 2020.
  * [86] Yong (Alexander) Liu, Xin (Lucy) Liu, Fang (Nancy) Li, Haohuan Fu, Yuling Yang, Jiawei Song, Pengpeng Zhao, Zhen Wang, Dajia Peng, Huarong Chen, Chu Guo, Heliang Huang, Wenzhao Wu, and Dexun Chen.  Closing the ”quantum supremacy” gap: achieving real-time simulation of a random quantum circuit using a new sunway supercomputer.  In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’21, New York, NY, USA, 2021. Association for Computing Machinery.
  * [87] Thorsten B. Wahl and Sergii Strelchuk.  Simulating quantum circuits using efficient tensor network contraction algorithms with subexponential upper bound.  Phys. Rev. Lett., 131:180601, Oct 2023.
  * [88] Feng Pan, Keyang Chen, and Pan Zhang.  Solving the sampling problem of the sycamore quantum circuits.  Phys. Rev. Lett., 129:090502, Aug 2022.

Supplementary Information of “Quantum Compiling with Reinforcement Learning on
a Superconducting Processor”

###### Contents

  1. I Introduction
  2. II Results
    1. II.1 RL-based compiler and experimental system
    2. II.2 Compilation for two-qubit system
    3. II.3 Compilation for three-qubit system
    4. II.4 Experimental error analysis
  3. III Discussion and outlook
  4. IV Methods
    1. IV.1 Training data generation
    2. IV.2 Optimization of DQN
    3. IV.3 AQ* Search
    4. IV.4 Variational RL-based Compiler
  5. SI A EXPERIMENTAL DETAILS
  6. SI B RL-BASED QUANTUM COMPILER
  7. SI C EXTENDED DISCUSSION AND RESULTS FOR TWO-QUBIT SYSTEM
    1. SI C.1 Construction of training data and test data
    2. SI C.2 Implementation of RL-based compiler
    3. SI C.3 More numerical and experimental results
    4. SI C.4 Summary of results for two-qubit system
  8. SI D EXTENDED DISCUSSION AND RESULTS FOR THREE-QUBIT SYSTEM
    1. SI D.1 Construction of training data and test data
    2. SI D.2 Implementation of RL-based compiler
    3. SI D.3 Summary of results for three-qubit system
  9. SI E VARIATIONAL RL-BASED QUANTUM COMPILER

##  Appendix SI A EXPERIMENTAL DETAILS

Device information. The superconducting processor used in this work contains 9
transmon qubits and 12 couplers fabricated using the flip-chip technology with
a tantalum base layer [66, 67]. The qubits and couplers are arranged in a
3$\times$3 square lattice, as is shown schematically at the bottom in Fig. S1.
The qubits have an anharmonicity around $-$200 MHz, and the couplers, which
are also transmon qubits, have an anharmonicity around $-$130 MHz. Each qubit
has a single control line for XY and Z signals, while each coupler has a flux
line to adjust the effective coupling strength between the neighboring qubits.

Figure S1: Experimental setup. Electronics and wirings for synthesizing and transmitting the control/readout signals. Each qubit has three control channels: XY (microwave pulse), fast Z (Z pulse), and slow Z (direct current). Each coupler has two control channels: fast Z and slow Z. Readout pulses are generated in a similar way to the XY signals and are passed through the processor via the readout transmission lines. All the control and readout lines are well attenuated and filtered for noise shielding and precise control. The superconducting processor shown at the bottom has 9 qubits (crosses) and 12 couplers (rectangles) arranged in a 3$\times$3 lattice configuration. The qubit index is indicated and the coupler between Qi and Qj will be denoted as Cij. Table SI:  Basic parameters of six qubits Q1-Q6 used in the present experiment. $\omega_{r}$, $\omega_{\rm{max}}$, and $\omega_{\rm{idle}}$ are the readout resonator frequency, the qubit maximum frequency, and the qubit idle frequency, respectively. $\alpha$ is the qubit anharmonicity. $T_{1}$ and $T_{2}^{\ast}$ are the energy relaxation time and dephasing time of the qubit at idle point. $F_{g}$ and $F_{e}$ are the readout fidelities for the ground and first-excited states. The infidelities of the single-qubit gate and two-qubit CZ gate (between Q1 and Q2, Q2 and Q3, Q4 and Q5, Q5 and Q6, Q1 and Q4, Q2 and Q5, Q3 and Q6) are calibrated by RB and/or XEB. The data in parenthesis for the two-qubit CZ gate are values intentionally increased for the study described in the main text. In the single-qubit case, the measurements are performed simultaneously for all qubits. Qubit | $\rm Q_{1}$ | $\rm Q_{2}$ | $\rm Q_{3}$ | $\rm Q_{4}$ | $\rm Q_{5}$ | $\rm Q_{6}$ |
---|---|---|---|---|---|---|---
$\omega_{r}/2\pi(\mathrm{GHz})$ | 7.3146 | 7.3000 | 7.2433 | 7.3530 | 7.3438 | 7.2810 |
$\omega_{\rm{max}}/2\pi(\mathrm{GHz})$ | 5.142 | 5.061 | 5.051 | 5.097 | 4.990 | 4.973 |
$\omega_{\rm{idle}}/2\pi(\mathrm{GHz})$ | 5.142 | 4.907 | 5.046 | 4.967 | 4.653 | 4.792 |
$\alpha/2\pi(\mathrm{MHz})$ | -198 | -196 | -201 | -196 | -224 | -193 |
$T_{1}(\mu{\rm s})$ | 20.4 | 46.4 | 37.8 | 25.1 | 52.28 | 25.9 |
$T_{2}^{\ast}(\mu{\rm s})$ | 5.40 | 2.03 | 22.9 | 2.73 | 1.60 | 2.03 |
$F_{g}(\%)$ | 96.6 | 94.7 | 95.7 | 94.4 | 94.1 | 93.1 |
$F_{e}(\%)$ | 93.8 | 91.4 | 94.0 | 90.2 | 90.1 | 92.0 |
1Q RB $(\%)$ | 0.049 | 0.038 | 0.045 | 0.048 | 0.120 | 0.100 |
1Q XEB $(\%)$ | 0.0593 | 0.048 | 0.043 | 0.0735 | 0.2021 | 0.1075 |
Qubits |  $\rm Q_{1}$-$\rm Q_{2}$ |  $\rm Q_{2}$-$\rm Q_{3}$ |  $\rm Q_{4}$-$\rm Q_{5}$ |  $\rm Q_{5}$-$\rm Q_{6}$ |  $\rm Q_{1}$-$\rm Q_{4}$ |  $\rm Q_{2}$-$\rm Q_{5}$ |  $\rm Q_{3}$-$\rm Q_{6}$
2Q RB$(\%)$ | 0.51 | 0.61 | (3.99) | (0.92) | 0.67 | (1.1) | 0.80 (1.47)

Six qubits from Q1 to Q6 are used in the present experiment (see Fig. S1) and
basic device parameters are listed in Tab. SI. We note that the frequency of
the readout resonator $\omega_{r}$, ranging from 7.300 to 7.353 GHz, is above
the maximum qubit frequency $\omega_{\rm{max}}$ that varies between 4.973 and
5.142 GHz. The qubit idle frequency $\omega_{\rm{idle}}$ is tuned to be close
to $\omega_{\rm{max}}$ in order to reduce the effect of flux noise. The energy
relaxation time $T_{1}$ for each qubit at the idle point is above 20 $\mu$s,
and the dephasing time $T_{2}^{\ast}$ ranges from 1.60 to 22.9 $\mu$s. The
single-qubit gate fidelities are about 99.9$\%$. The coupling strength between
the qubit and the coupler connected to it is $\sim$ 80 MHz, and the direct
coupling strength between two neighboring qubits is $\sim$ 9 MHz. The
effective coupling strength $g$ of the neighboring qubits ranges from 1.5 to
$-$50 MHz as determined by the vacuum Rabi oscillation measurement at the
resonance point. The two-qubit CZ gate fidelities are $\sim$ 99.3$\%$. To
achieve high performance of the single- and two-qubit gate operations, a
series of signal calibrations and corrections have been done, which are
detailed below.

Measurement setup. Our experiments are performed on a BlueFors XLD-1000
dilution refrigerator with a base temperature of 10 mK. The device is placed
in a superconducting sample box which is magnetically shielded in a double-
layer Cryoperm-10 cylinder. The details of the measurement setup are shown in
Fig. S1. On the left side, the purple, green, and blue lines indicate the
coupler control, the qubit control, and the readout control, respectively. On
the right side, the red lines represent the Josephson parametric amplifier
(JPA) control. All attenuators at each cryogenic temperature stage are
utilized to remove the excess thermal photons from the higher temperature
stage. The XY control line and fast Z line are combined by a duplexer at room
temperature, further infrared filtered by CR124, and finally combined with the
DC line by a bias-Tee into the device. The combination is considered due to
the fact that each qubit has only one control line in the device. Similarly,
the coupler control contains a flux line and DC line, which are also combined
by the bias-Tee at the lowest temperature stage. The readout control includes
the input line and the output line. When the multi-tone microwave signals are
sent to the transmission line via the input line, the output signals are
amplified successively by the Josephson parametric amplifier, high electron
mobility transistor (HEMT), and room-temperature microwave amplifiers in the
output line, which are ultimately demodulated by the analog-digital converter.

Calibration of Z pulse distortion. Due to the impedance mismatch in the Z
control line, there exist distortions with the Z pulse, such as overcharge,
ringing, as well as prolonged rising and falling edges [68]. The resulting
deviation of the qubit frequency would lead to the accumulation of additional
kinetic phases. Moreover, the Z pulse distortion makes the timing of the
required interactions difficult to control, which is a serious problem for the
realization of high-fidelity two-qubit gates. However, for a linear time-
invariant system, these waveform distortions can be corrected by the method of
deconvolution [69]. In this way, the Z pulse distortion can be calibrated to
achieve an accurate required pulse shape for the two-qubit gate
implementation.

The measurement and calibration of Z pulse distortion are insensitive to
decoherence [39]. Figure S2(a) illustrates the corresponding pulse sequence
for the calibration of the qubit. The sweet point is chosen as the idle point,
and the phase accumulated by the short and small Z pulse $z_{p}$ is used as
the probe to measure the distortion of the falling edge of the long and big Z
pulse $z_{t}$. Here $z_{p}$ corresponds to the flux-sensitive point. A Ramsey-
like pulse sequence is used in the probing process with $z_{p}$ sandwiched
between an $\text{X}/2$ gate and a tomography gate. We can fit the
experimental data with the above function,

$\delta\varphi_{t_{d}}=z_{t}D(z_{p})\sum_{i}\tau_{i}a_{i}(e^{-(t_{d}+t_{p})/\tau_{i}}-e^{-t_{d}/\tau_{i}}),$
(3)

in which $t_{d}$ is the delay between the falling edge and the start of the
probe pulse, $\varphi_{t_{d}}$ is the phase accumulated by $z_{p}$ and
$D(z_{p})=\frac{\mathrm{d}\omega}{\mathrm{d}z}|_{z_{p}}$. Besides, $a_{i}$ and
$\tau_{i}$ represent the amplitude and decay time of the exponential decay,
which are obtained by fitting the experimental results. The experimental
results and fits are shown in Fig. S2(b), taking Q1 as an example. The fitted
parameters are then used to adjust the input pulse shape such that the qubit
receives an accurate required Z pulse signal.

Figure S2: Calibrations of Z pulse distortion and timing of the single-qubit
XY and Z control lines. Subplots (a) and (b) are pulse sequence and
experimental result for the calibration of Z pulse distortion, respectively.
Subplots (c) and (d) are pulse sequence and experimental result for the
single-qubit control line timing calibration, respectively. The measurements
are performed on Q1.

Since the coupler does not have a readout resonator, its Z pulse distortion
calibration is different from the qubit, and it needs to be read out with the
help of qubit. Here we use the method in Ref. [70] to achieve the Z pulse
distortion calibration of the coupler by mapping the distortion of the coupler
to the qubits with the help of the ac-Stark effect between the coupler and the
qubit.

Calibration of timing. Due to the different response times of individual
channels of the Z pulse and microwave pulse sources, as well as different
lengths of various control lines, the time it takes for a signal to reach a
qubit from a source varies among different control lines, which reduces the
gate fidelity and needs to be calibrated and adjusted.

Figure S3: Timing calibrations between two neighboring qubits (a)-(b) and
between a qubit and the neighboring coupler (c)-(d). Subplots (a) and (c) are
pulse sequences for the calibrations. Subplots (b) and (d) are experimental
results and fits. The measurements are performed on Q1, Q2, and the coupler
C12 in between.

To calibrate the timing of the XY and Z control lines of a qubit, we apply an
X gate and a Z gate, and adjust the relative position of the two gates to
obtain the relative timing according to the excited state probability of the
qubit. Figure S2(c) shows the pulse sequence for the experiment and the result
measured on Q1 is shown in (d), in which a time difference of 0.1 ns between
the XY and Z lines is indicated.

We perform the vacuum Rabi experiment to calibrate the timing between two
adjacent qubits, and between a qubit and its adjacent coupler. For calibrating
the timing between two qubits interleaved by a coupler, we consider the Z
pulses of qubits Q1, Q2, and the coupler C12 between them. The pulse sequence
for calibrating the timing of the Z lines of qubits Q1 and Q2 is shown in Fig.
S3(a) and the experimental results are shown in (b). We prepare the initial
state $|100\rangle$ (or $|001\rangle$), the Z pulse lengths of the qubits and
the coupler are fixed at $t_{\rm iSWAP}=\pi/g$ and 2 $-$ 6 times of $t_{\rm
iSWAP}$, respectively, here $g$ is the effective coupling strength between the
two qubits (see Fig. S4(a) where $g$ can be tuned from 1.5 to $-$50 MHz). The
Z pulses of the two qubits are moved within the time range of the coupler Z
pulse to complete the timing calibration. The pulse sequence for calibrating
the timing of the Z lines of qubit Q1 and coupler C12 is shown in Fig. S3(c)
and the experimental results are presented in (d). In this case, the Z pulses
of the two qubits are aligned and their lengths are the same as that of the
coupler. We complete the timing calibration by moving the relative timing
between the coupler Z pulse and the qubit Z pulse.

In the actual calibration process, the timing calibration is performed using
the XY line of Q1 as the reference. Timing calibration is done sequentially
between the XY and Z lines of the qubit, between the Z lines of qubits Q1 and
Q2, and between the Z lines of qubit Q1 and its adjacent coupler C12.
Likewise, the calibrations can be applied to other couplers, such as Q2 and
Q3, Q4 and Q5, and so on.

Figure S4: Implementation of the CZ gate between Q1 and Q2. (a) Measurement of
the effective coupling strength between Q1 and Q2 as a function of the coupler
bias. Measurements of the leakage (b) and conditional phase (c) of the CZ gate
to find the optimal points of the pulse amplitudes for qubit (ampq) and
coupler (ampc). In (b) and (c), the upper parts show the pulse sequences for
the measurements.

Calibration of single-qubit gates and readout. The single-qubit gates
including X/2 and Y/2 are realized by 20 ns microwave pulses with cosine
envelopes, where the quadrature correction terms with DRAG coefficients are
implemented to minimize state leakage to higher levels [71]. We characterize
the performance of the single-qubit gates by randomized benchmarking (RB) and
cross-entropy benchmarking (XEB) at idle points. The measured averaged
fidelities are listed in Tab. SI. To implement single-qubit gates with an
arbitrary angle, we use
$U(\theta,\phi,\lambda)=\text{R}_{\text{Z}}(\phi-\pi/2)\text{R}_{X}(\pi/2)\text{R}_{\text{Z}}(\pi-\theta)\text{R}_{X}(\pi/2)\text{R}_{\text{Z}}(\lambda-\pi/2)$
where $\lambda$, $\phi$ and $\theta$ are arbitrary angles and
$\text{R}_{\text{Z}}$ is a virtual gate [43]. The parameterization of single-
qubit gates in this way does not increase gate count and gate operation time,
which is of great importance to raise the circuit fidelity to unity after RL
compilation, as discussed in the main text.

The duration of readout pulses for all qubits is around 1.0 $\mu$s. To achieve
high-visibility and low-error readouts, the power and frequency of the readout
pulses and the demodulation time are optimized. The readout fidelities of the
ground and first-excited states are around 95 $\%$ and 92 $\%$, respectively,
which are also presented in Tab. SI.

Calibration of two-qubit CZ gate. The CZ gate between two neighboring qubits
is implemented by adjusting the energy levels with an error-function shaped
pulse from idle frequency to working frequency, so that the $|101\rangle$ and
$|200\rangle$ (or $|002\rangle$) energy levels can be tuned to resonate for a
specific amount of time. At the same time, the coupling is turned on to
accumulate enough phase through resonance.

The calibration of a CZ gate is carried out in two steps. The first step is to
find the working points of the qubits and the coupler. For this we fix the CZ
gate time, and measure the leakage and the conditional phase angle, as is
shown in Figs. S4(b) and (c). To measure the leakage from the $|101\rangle$
state, we prepare the state by applying an X gate to both Q1 and Q2 and
measure the population of the $|101\rangle$ state after applying the CZ gate.
To get the conditional phase angle, we measure cross Ramsey for Q2 (Q1) when
Q1 (Q2) is applied with an I or X gate, which can be used to measure the
conditional accumulation of Q2 (Q1). We then find the optimal working points
by minimizing the probability of $1-P_{|101\rangle}$ and making the
conditional phase angle closest to $\phi=180^{\circ}$, as is shown in Figs.
S4(b) and (c) by the white dots. The Q1 and Q2 Z-pulse amplitudes as well as
the coupler Z-pulse amplitude are then determined. The second step is to
measure the single-qubit phase accumulation on Q1 and Q2 that accompanies the
CZ gate by QPT. We subsequently apply virtual $Z$ gates to offset the phase.

Figure S5: Comparison of RB and XEB in benchmarking the two-qubit CZ gate. (a)
Interleaved RB data to characterize the CZ gate on Q1 and Q2. (b) XEB data is
taken simultaneously on Q1 and Q2 to characterize the single-qubit gates
($\alpha$ is the sequence fidelity). (c) XEB data to characterize the CZ gate
between Q1 and Q2, where each cycle contains two single-qubit gates in
parallel and a CZ gate. The CZ Pauli errors extracted from RB and XEB are
0.57$\%$ and 0.51$\%$, respectively.

Based on the above method, we are able to implement the CZ gates between seven
qubit pairs, whose parameters are listed in Tab. SI. The total CZ gate time is
55 ns, where both the rising and falling edges of the pulse are 6 ns, the
flattening part for the accumulating phase is 39 ns (corresponding to
$g/2\pi\approx$ $-$9 MHz), and the buffer time is 4 ns. In Fig. S5, the
characterizations of the CZ gate between $\rm Q_{1}$ and $\rm Q_{2}$ by
randomized benchmarking (RB) and cross-entropy benchmarking (XEB) [72] at idle
points are characterized, with consistent Pauli errors. As an example, the
fidelities of the CZ gates between Q1 and Q2 and between Q1 and Q4,
characterized by XEB, are 99.49 $\%$ and 99.33 $\%$, respectively.

Cross entropy benchmarking (XEB). We characterize the single-qubit and two-
qubit gates (Fig. S5), as well as the performance of compiled circuits by XEB.
For single-qubit gates, each cycle in an XEB circuit consists of a $\pi/2$
rotation randomly chosen from the following set:
$\\{\text{R}_{\phi}(\pi/2)\big{|}\phi\in\\{0,\pi/4,\pi/2,3\pi/4,\pi,5\pi/4,3\pi/2,7\pi/4\\}\\}$,
where $\phi$ represents a specific angle. The rotation operator itself can be
expressed as
$\text{R}_{\phi}(\pi/2)=\exp({-i\frac{\pi}{4}(\cos{\phi}\text{X}+\sin{\phi}\text{Y})})$.
For two-qubit sequences, each cycle contains a layer of two single-qubit gates
$\text{R}_{\phi}(\frac{\pi}{2})$ followed by a CZ gate. We refer to Ref. [48]
for the calculation of fidelity using measured probabilities.

Quantum process tomography (QPT). We characterize the performance of compiled
circuits running on the quantum processor by QPT [73]. We select
$\bigotimes^{n}\\{\text{I},\text{X},\text{Y}/2,\text{X}/2\\}$ to prepare the
initial state, and select $\otimes^{n}\\{-\text{Y}/2,\text{X}/2,\text{I}\\}$
to complete the projection measurement of quantum states in different axial
directions. In order to suppress fluctuations caused by experimental errors,
each measurement is repeated 4000 times. The experimental fidelity is obtained
by
$F_{2}(\hat{U},\hat{V})=\text{Tr}(\mathcal{\hat{E}}^{\dagger}\hat{\mathcal{F}})$,
where $\mathcal{\hat{E}}$ and $\hat{\mathcal{F}}$ are the calculated and
measured process matrices of the target unitary $\hat{U}$ and the unitary
$\hat{V}$ of the compiled circuit, respectively.

##  Appendix SI B RL-BASED QUANTUM COMPILER

Quantum compiling. For quantum compiling with discrete native basis set
$\mathcal{G}$ and given the topology of the quantum processor, we define the
generalized basis set $\tilde{\mathcal{G}}$ whose element corresponds to the
executable operation on the processor. For example, for a three-qubit system
with chain structure Q1–Q2–Q3, if X gate is in $\mathcal{G}$, then the
generalized basis gate $\tilde{\mathcal{G}}$ contains
$\mathbb{I}_{4}\otimes\text{X}$, $\text{X}\otimes\mathbb{I}_{4}$, and
$\mathbb{I}_{2}\otimes\text{X}\otimes\mathbb{I}_{2}$. Similarly, when
$\text{CZ}\in\mathcal{G}$, the corresponding elements in $\tilde{\mathcal{G}}$
are $\text{CZ}\otimes\mathbb{I}_{2}$ and $\mathbb{I}_{2}\otimes\text{CZ}$.
This generalized basis set $\tilde{\mathcal{G}}$ is dubbed action space in the
context of RL-based quantum compiler, as denoted by
$\mathcal{A}=\\{A_{i}\\}_{i=1}^{d}$.

The purpose of quantum compiling is to find a sequence of quantum gates
$\\{A_{1}^{\prime},A_{2}^{\prime},\dots,A_{L}^{\prime}\\}\in\mathcal{A}^{N}$
with the minimum length $N$ such that the distance between the target unitary
$\hat{U}$ and $\hat{V}=\prod_{j=1}^{N}A_{j}^{\prime}$ is bounded by a pre-
defined error $\varepsilon$, i.e.,

$\displaystyle d\left(\hat{V},\hat{U}\right)<\varepsilon,$ (4)

where the distance metric $d(\cdot,\cdot)$ refers to two infidelity measures
$1-F_{1}(\hat{V},\hat{U})$ and $1-F_{2}(\hat{V},\hat{U})$ defined in the main
text.

Training data Generation. Here we elucidate the omitted construction rule of
training data in Methods. Recall that the cardinality of $\mathcal{W}_{l}$
exponentially scales with $l$, which may lead to memorization and computation
issues for a large $l$. To tackle this issue, a threshold value $l^{*}$ is
introduced. When $l\leq l^{*}$, the percepts $\mathcal{W}_{l}$ are generated
in a brute force manner; when $l>l^{*}$, the percepts are generated in a
random sampling manner. The setting of the threshold $l^{*}$ is heuristic,
depending on the available memory resources, the computation power, and the
maximum loop number $L$. We refer to Supplementary Information SI C and SI D
for the detailed setting of $l^{*}$.

RL-based quantum compiler. The pseudo-code of DQN employed in our proposal is
presented in Alg. 1. To improve the training stability, we adopt the technique
developed in Refs. [74, 63] to optimize DQN. Concretely, following notations
in the main text, the loss function at the $l$-th loop to be minimized takes
the form

$\displaystyle\mathcal{L}_{l}(\bm{\theta})=\frac{1}{|\mathcal{W}_{l}|\cdot|\mathcal{A}|}\sum_{W_{i}\in\mathcal{W}_{l}}\sum_{A\in\mathcal{A}}\left(Q_{\bm{\theta}}(W_{i},A)-\left(\max_{A^{\prime}\in\mathcal{A}}Q_{\bm{\theta}^{-}}(AW_{i},A^{\prime})-1\right)\right)^{2},$
(5)

where $W_{i}\in\mathcal{W}_{l}$, $Q_{\bm{\theta}}$ represents a DQN and
$Q_{\bm{\theta}}(W_{i},A)$ represents its estimation value of the percept
$AW_{i}$, and $Q_{\bm{\theta}^{-}}$ represents an auxiliary DQN as elaborated
below.

Compared to the loss defined in Methods, an auxiliary DQN
$Q_{\bm{\theta}^{-}}$ is introduced in Eq. (5) to replace $\hat{V}_{ji}$. The
two DQNs $Q_{\bm{\theta}^{-}}$ and $Q_{\bm{\theta}}$ have identical structures
and similar parameters. The only difference is that the original DQN
$Q_{\bm{\theta}}$ dynamically updates $\bm{\theta}$ using the back-
propagation, while the auxiliary DQN $Q_{\bm{\theta}^{-}}$ stays intact most
of the time and provides the estimation of the target values, i.e.,
$\max_{A^{\prime}\in\mathcal{A}}Q_{\bm{\theta}^{-}}(AW,A^{\prime})-1$.

The optimization of $\mathcal{L}_{l}(\bm{\theta})$ in Eq. (5) can be completed
by the gradient descent methods. For each $l\in[L]$, the training parameters
$\bm{\theta}$ of the original DQN are updated up to $T$ epochs (Line 3 in Alg.
1). An early termination is allowable if the loss $\mathcal{L}(\bm{\theta})$
is below a threshold $\delta$ (Line 8 in Alg. 1). Then, we move to the next
loop $l\rightarrow l+1$ and the parameters $\bm{\theta}^{-}$ are updated to
$\bm{\theta}$.

Input: Action space $\mathcal{A}$; Loss threshold $\delta$; Adam optimizer
with the initial learning rate $\gamma$; The maximum loop $L$; Epoch $T$;
Initialized the original and target DQNs with the parameters ${\bm{\theta}}$
and ${\bm{\theta}^{-}}$; Threshold $\delta$.

Output: The trained target DQN.

1 Set $l=1$

2 while  _$l\leqslant L$_ do

3 Generate the training data $\mathcal{W}_{l}$

4 for  _$i\leftarrow 1$ to $T$_ do

5 Sample random minibatch of percepts from $\mathcal{W}_{l}$ and $\mathcal{A}$

6 Calculate the loss $\mathcal{L}_{l}(\bm{\theta})$ in Eq. (5)

7 Compute the gradient $\nabla_{\bm{\theta}}\mathcal{L}_{l}(\bm{\theta})$

8 Update $\bm{\theta}$ based on the gradients

9 if  _loss $\leqslant\delta$_ then

10 $\bm{\theta}^{-}\leftarrow\bm{\theta}$

11 $l\leftarrow l+1$

12 end if

13

14 end for

15 $\bm{\theta}^{-}\leftarrow\bm{\theta}$

16 $l\leftarrow l+1$

17 end while

Algorithm 1 Training DQN assisted by the auxiliary DQN

Multi-DQN collaborative AQ* search. We have seen in the main text that DQNs
cannot learn the whole unitary space when the total number of loops $L$ is
limited. To improve the RL-based compiler in the inference stage, we adapt the
original AQ* search (Alg. 2) to the multi-DQN collaborative version. The
pseudo-code of the multi-DQN collaborative AQ* search algorithm is summarized
in Alg. 3.

To better understand Alg. 3, we consider $J$ DQN models ($J=3$ in the
presented example). These models are denoted as $Q_{\bm{\theta}_{1}}$,
$Q_{\bm{\theta}_{2}}$ and $Q_{\bm{\theta}_{3}}$, with each model being trained
on distinct generalized basis gate sets: $\mathcal{A}_{1}$, $\mathcal{A}_{2}$
and $\mathcal{A}_{3}$, respectively. The algorithm leveraging these DQN models
for collaborative AQ* search aims to find a gate sequence to approximate a
target unitary matrix $\hat{U}$ while adhering to an accuracy threshold of
$\varepsilon$. The core idea is a sequential collaboration between the DQN
models. Initially, we use $Q_{\bm{\theta}_{1}}$ to execute the standard AQ*
search algorithm (Alg. 2) and generates an initial guess, $\hat{V}_{1}$, to
approximate $\hat{U}$. Subsequently, $Q_{\bm{\theta}_{2}}$ takes over, using
the AQ* search to compile the product $\hat{V}_{1}^{\dagger}\hat{U}$ and
derive $\hat{V}_{2}$. This process iterates, with each DQN model
$Q_{\bm{\theta}_{j}}$ ($j=1,2,3$ in this case) focusing on a specific stage of
compiling the overall sequence. The final collaborative compiling target is
defined as $\hat{V}=\hat{V}_{1}\cdot\hat{V}_{2}\cdot\hat{V}_{3}$, where each
component originates from a different basis gate set. Intuitively, this
collaborative approach can enhance the accuracy of the compiled sequence at
the expense of increased resources.

Input: Target unitary matrix $\hat{U}$; The DQN model $Q_{\bm{\theta}}$; The
error tolerance $\varepsilon$; The search depth threshold $D$; The action
space $\mathcal{A}$; The initial percept $\mathbb{I}$.

Output: Compiling sequence $\hat{V}$ to approximate $\hat{U}$.

1

2Set $\hat{V}=\mathbb{I}$

3 for  _$i\leftarrow 1$ to $D$_ do

4 $A^{\star}=\arg\max_{A\in\mathcal{A}}f(\hat{V},A;\hat{U})$

5 Execute the matrix multiplication $\hat{V}=\hat{V}\cdot A^{\star}$

6 if  _$F_{1}(\hat{V},\hat{U})\leq\varepsilon$_ then

7 return $\hat{V}$

8 end if

9

10 end for

return $\hat{V}$

Algorithm 2 AQ* Search

Input: Target unitary matrix $\hat{U}$; DQN models $Q_{\bm{\theta}_{1}}$,
$Q_{\bm{\theta}_{2}}$ $\cdots$ $Q_{\bm{\theta}_{J}}$; The error tolerance
$\varepsilon$; The search depth threshold $D$; The action space $\mathcal{A}$;
The initial percept $\mathbb{I}$.

Output: Compiling sequence $\hat{V}$ to approximate $\hat{U}$.

1 Set $\hat{V}=\mathbb{I}$

2 for  _$j\leftarrow 1$ to $J$_ do

3 $\hat{V}$ = AQ* Search($\hat{U}$, $Q_{\bm{\theta}_{j}}$, $\varepsilon$, $D$,
$\mathcal{A}$, $\hat{V}$)

4 if  _$F_{1}(\hat{V},\hat{U})\leq\varepsilon$_ then

5 return $\hat{V}$

6 end if

7

8 end for

return $\hat{V}$

Algorithm 3 Multi-DQN collaborative AQ* search

Comparison with other RL-based compilers. Two previous works [34, 33] have
utilized the RL techniques for quantum compilation, showing varying
performance in accomplishing the quantum compiling tasks. The proposal in Ref.
[33] involves policy-based RL, which encounters the sparse reward problem. To
address the issue, our proposal utilizes value-based RL, which can be applied
to any universal basis set and multi-qubit systems. While the proposal in Ref.
[34] also belongs to the value-based RL, it requires prior knowledge in
compiling two-qubit gates to ensure performance in terms of fidelity. Our
proposal removes this requirement by adopting a more powerful model, DQN, and
a different loss function. As a result, it allows for a shorter gate sequence
length and a shorter inference time in compiling multi-qubit circuits.

##  Appendix SI C EXTENDED DISCUSSION AND RESULTS FOR TWO-QUBIT SYSTEM

In this section, we describe the construction rule of the training data and
test data, and the implementation details of the RL-based compiler, followed
by the presentation of more numerical and experimental results of compilation
for the two-qubit system. Finally, a summary of the results is presented.

###  SI C.1 Construction of training data and test data

Data generation. We exploit three basis sets for different two-qubit compiling
tasks. The first basis set is
$\mathcal{G}^{(2,1)}=\\{\pm\text{X}/2,\pm\text{Y}/2,\text{T},\text{T}^{\dagger},\text{CZ}\\}$,
where the superscript $(2,1)$ represents the two-qubit case under the first
basis set. Accordingly, the generalized basis set, or equivalently the action
space, yields

$\displaystyle\tilde{\mathcal{G}}^{(2,1)}\equiv\mathcal{A}^{(2,1)}=\Big{\\{}$
$\displaystyle\text{R}_{X}(\frac{\pi}{2})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{X}(\frac{\pi}{2}),\text{R}_{X}(-\frac{\pi}{2})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{X}(-\frac{\pi}{2}),\text{R}_{Y}(\frac{\pi}{2})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{Y}(\frac{\pi}{2}),$
(6)
$\displaystyle\text{R}_{Y}(-\frac{\pi}{2})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{Y}(-\frac{\pi}{2}),T\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes
T,T^{\dagger}\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes
T^{\dagger},\text{CZ}\Big{\\}},$

with the cardinal number $|\mathcal{A}^{(2,1)}|=13$.

The second basis set is
$\mathcal{G}^{(2,2)}=\\{\pm\text{X}/6,\pm\text{Y}/6,\pm\text{Z}/6\\}$, where
$\rm P/6$ represents rotation gate $\rm R_{P}(\pi/6)$ for $\rm
P\in\\{X,Y,Z\\}$. Accordingly, the action space with respect to the second
basis set yields

$\displaystyle\mathcal{A}^{(2,2)}=\Big{\\{}$
$\displaystyle\text{R}_{X}(\frac{\pi}{6})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{X}(\frac{\pi}{6}),\text{R}_{X}(-\frac{\pi}{6})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{X}(-\frac{\pi}{6}),\text{R}_{Y}(\frac{\pi}{6})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{Y}(\frac{\pi}{6}),\text{R}_{Y}(-\frac{\pi}{6})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{Y}(-\frac{\pi}{6}),$
(7)
$\displaystyle\text{R}_{\text{Z}}(\frac{\pi}{6})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{\text{Z}}(\frac{\pi}{6}),\text{R}_{\text{Z}}(-\frac{\pi}{6})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{\text{Z}}(-\frac{\pi}{6})\Big{\\}},$

with the cardinal number $|\mathcal{A}^{(2,2)}|=12$.

The third basis set is
$\mathcal{G}^{(2,3)}=\\{\pm\text{X}/128,\pm\text{Y}/128,\pm\text{Z}/128\\}$,
where $\rm P/128$ represents the rotation gate $\rm R_{P}(\pi/128)$ for $\rm
P\in\\{X,Y,Z\\}$. Accordingly, the action space with respect to the third
basis set yields

$\displaystyle\mathcal{A}^{(2,3)}=\Big{\\{}$
$\displaystyle\text{R}_{X}(\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{X}(\frac{\pi}{128}),\text{R}_{X}(-\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{X}(-\frac{\pi}{128}),\text{R}_{Y}(\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{Y}(\frac{\pi}{128}),$
(8)
$\displaystyle\text{R}_{Y}(-\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{Y}(-\frac{\pi}{128}),\text{R}_{\text{Z}}(\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{\text{Z}}(\frac{\pi}{128}),\text{R}_{\text{Z}}(-\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{R}_{\text{Z}}(-\frac{\pi}{128})\Big{\\}},$

with the cardinal number $|\mathcal{A}^{(2,3)}|=12$.

As explained in Sec. SI B and Methods, the initial percept $\mathbb{I}$ is
perturbed to enhance the algorithmic robustness. In the two-qubit compiling
tasks, the perturbed percept takes the form

$\mathbb{I}^{\prime}=\left(\cos{(\frac{\alpha}{2})}\mathbb{I}_{2}-i\sin{(\frac{\alpha}{2})}(j_{1}^{(1)}\text{X}+j_{2}^{(1)}\text{Y}+j_{3}^{(1)}\text{Z})\right)\otimes\left(\cos{(\frac{\alpha}{2})}\mathbb{I}_{2}-i\sin{(\frac{\alpha}{2})}(j_{1}^{(2)}\text{X}+j_{2}^{(2)}\text{Y}+j_{3}^{(2)}\text{Z})\right),$
(9)

where the tunable parameters
$j_{1}^{(1)},j_{2}^{(1)},j_{3}^{(1)},j_{1}^{(2)},j_{2}^{(2)},j_{3}^{(2)}\in\mathbb{R}$
satisfy $|j_{1}^{(1)}|^{2}+|j_{2}^{(1)}|^{2}+|j_{3}^{(1)}|^{2}=1$ and
$|j_{1}^{(2)}|^{2}+|j_{2}^{(2)}|^{2}+|j_{3}^{(2)}|^{2}=1$, and $\alpha$ is
randomly sampled from $[-0.002,0.002]$.

In generating the training data, the maximum loop number is set as $L=50$ and
the threshold for the number of loops is set as $l^{*}=3$.

Compiling targets generation. To verify the versatility of our RL-based
compiler, we benchmark its performance via the following four tasks. In Task
1, we construct $50$ target operations. Each target is constructed by randomly
sampling $83$ gates from $\mathcal{G}^{(2,1)}$ and deployed on a two-qubit
circuit in a specific order. Specifically, the compiling targets are generated
guided by the KAK decomposition method as shown in Fig. S6(a). We collect the
target unitaries by fixing three CNOT gates in a quantum circuit and operating
those single-qubit gates. And $80$ single-qubit gates uniformly sampled from
$\mathcal{G}^{(2,1)}$ are employed to form the single-qubit gates. Fig. S6(b)
exhibits the dissimilarity of the unitaries. In Task 2, 30 random targets are
sampled from SU(4). Fig. S6(c) exhibits the dissimilarity of these targets. In
Task 3, we compile the $\text{R}_{\text{ZZ}}$ gates. In Task 4, we simplify
the targets in Task 1 by sampling only 43 gates from $\mathcal{G}^{(2,1)}$ to
construct the targets with the same circuit structure.

Figure S6: Generation of compiling targets. (a) The KAK decomposition of a
template circuit. (b) The heat-map for the generated $50$ target unitaries in
the robustness verification task (Task 1). The correlation between compiling
targets $U_{1}$ and $U_{2}$ is calculated by $F_{1}(U_{1},U_{2})$. Both the
horizontal and vertical coordinates represent the target index. (c) The heat-
map for the $30$ targets sampled from SU(4) unitaries in the task of compiling
general unitary (Task 2).

###  SI C.2 Implementation of RL-based compiler

Classical Device. For the proposed RL-based compiler, both the precompilation
and inference processes related to the classical part are implemented on an
Nvidia Tesla V100 GPU.

Figure S7: The structure of the employed DQN model. The blue blocks represent
the hidden layers. Every two layers are fully connected. The output dimension
of the three-qubit DQN is determined by the dimension of the action space
$\mathcal{A}$.

Architecture of DQNs. The visualization of the employed DQN is shown in Fig.
S7, which is adapted from ResNet [44]. Specifically, it consists of an input
layer, two hidden layers, and six residual blocks, followed by
$|\mathcal{A}^{(2,t)}|$ output neurons with $t\in\\{1,2,3\\}$. The input layer
is of size $16$. The first two hidden layers are of sizes 6000 and 2000, and
each residual block is composed of two hidden layers with 2000 hidden neurons,
with the full connection. The leaky ReLU activation function and batch
normalization are applied to all hidden layers.

Hyper-parameters. We use Adam optimizer to optimize DQN, where the learning
rate is $\eta=10^{-3}$ without the weight decay [75]. The loss threshold
$\delta$ in Alg. 1 is set as $0.02$. The epoch $T$ depends on $l$, i.e.,
$T=100\cdot l$. In the training process, the optimized DQN models at each loop
$l$ under each action space $\mathcal{A}^{(2,t)}$, denoted by
$\\{Q^{(t,{l})}_{\mathbb{\theta}}\\}$, are stored to conduct multi-DQN
collaborative AQ* search. In the inference stage, we execute the multi-DQN
collaborative AQ* search guided by DQN model
$\\{Q^{(t,{l})}_{\mathbb{\theta}}\\}_{t,l}$. And the maximum AQ* search depth
is set as 200 and the inference time is restricted to be less than 10 minutes.

###  SI C.3 More numerical and experimental results

In this subsection, we provide more results in compiling two-qubit circuits to
further evaluate the performance of the RL-based compiler in three metrics,
i.e., fidelity, gate sequence length, and inference time (variational RL-based
compiler will be discussed in Sec SI E). We focus on exploring the capability
of the compiler and compare it with the KAK decomposition under four tasks
discussed below, where the KAK decomposition is implemented by Qiskit compiler
[76] whose basis gate set contains $\text{U}_{\text{3}}$ gate and CZ gate. As
shown in Fig. S6(a), any two-qubit quantum gate can be decomposed into three
CZ gates and multiple single-qubit gates. We perform experiments for the four
tasks and each decomposed gate sequence is performed five times repeatedly to
evaluate the statistical performance.

We point out that the Qiskit compiler is mainly used as a reference for
convenience to assess the performance of the RL-based compiler, and we refer
to other compilers that may outperform Qiskit compiler in some metrics such as
those employing heuristic search [24, 46, 47]. Our results show that the
experimental fidelity $F_{2}$ by RL-based compiler can possibly be higher than
that by Qiskit compiler even when the corresponding circuit fidelity $F_{1}$
by the former (latter) is lower (higher). This is unique for the NISQ
processor reflecting the interplay among circuit fidelity, circuit length and
structure, as well as gate errors and decoherence of the processor, which is
also seen in the discussion of the three-qubit QFT circuits in the main text.

Figure S8: Compilation of random targets in two-qubit system. (a) Simulation
($F_{1}$) and experimental ($F_{2}$) fidelity distributions of circuits by RL-
based compiler for 50 target unitaries. The abscissa representing the index of
targets is not shown. (b) Frequency under variations of experimental fidelity
$F_{2}$ and two-qubit gate count $N_{2}$ for circuits by RL-based compiler
(left panel) and Qiskit compiler (right panel). Compared with the Qiskit
compiler, the RL-based compiler requests fewer two-qubit gates and achieves a
higher fidelity on average.

Task 1: Compiling random target operations. In the main text, we have shown
the results of compiling the SWAP gate and $\text{R}_{\text{ZZ}}(\pi/2)$ gate.
Here we explore the performance of the RL-based compiler for more general
random target operations. As can be seen in Fig. S8, the RL-based compiler
outperforms the Qiskit compiler in terms of experimental fidelity $F_{2}$ and
two-qubit gate count $N_{2}$. On average, the RL-based compiler has
$N_{2}=2.80$ gates to accomplish the compiling task with $F_{2}=0.9840\pm
0.0049$, while the Qiskit compiler has $N_{2}=2.96$ gates with
$F_{2}=0.9790\pm 0.0041$. However, the inference time of the RL-based compiler
is much longer than that of the Qiskit compiler, i.e., $250$ versus $0.02$
seconds. The long inference time is caused by the constant calls to the DQN.
Exploring ways to reduce the inference time is left for future work.

Task 2: Compiling random operations from SU(4). We specify the universal basis
set $\mathcal{G}^{(2,2)}$ and use it to train three DQNs. The only difference
between the three DQNs is the varied total loop number $L\in\\{20,25,45\\}$.
The trained DQNs are applied to compile $50$ two-qubit gates whose
construction rule is identical to that in Task 1. To investigate the
performance of the RL-based compiler for more general two-qubit gates, we
employ the trained DQNs with universal basis set $\mathcal{G}^{(2,1)}$ to
compile $30$ two-qubit gates randomly sampled from SU(4).

Figure S9: Scaling behavior of DQN and compilation of random operations from
SU(4). (a) Compilation of 50 two-qubit gates with restricted complexity. The
left panel illustrates the performance of the RL-based compiler in the metric
of $F_{1}$ and $F_{2}$ when the loop number is set as $L=20$, $25$, and $45$,
respectively. The right panel shows the number of two-qubit gates $N_{2}$ in
the output gate sequence with different $L$. The corresponding number of
single-qubit gates $N_{1}$ on average is $10.69$, $16.96$, and $25.68$,
respectively. The inference time in all cases is around $250$ seconds. (b)
Compilation of 30 two-qubit gates sampled from SU(4). The left panel shows the
performance of the RL-based compiler with $L=45$ by $F_{1}$ (simulation) and
$F_{2}$ (experiment). The right panel presents the experimental fidelity
$F_{2}$ of RL-based and Qiskit compilers. The abscissa in the left panels
representing the index of targets is not shown.

The compiling and experimental results are depicted in Fig. S9. Figure S9(a)
indicates that the compiling accuracy of the RL-based compiler is steadily
improved by increasing $L$, in the measures of both $F_{1}$ and $F_{2}$. This
observation suggests that training DQN with a sufficiently large $L$ can
promise a satisfactory performance. Besides, the rightmost panel shows that
$N_{2}$ is independent with $L$, implying that the gate sequence length can be
well controlled for large $L$, as a desired feature for NISQ devices. Figure
S9(b) illustrates the results of compiling random unitaries sampled from
SU(4). On average, the RL-based compiler produces results less satisfactory
compared to the Qiskit compiler in terms of $F_{2}$, caused by the fact that
the trained DQN with $L=45$ is insufficient to explore the whole unitary
space. In practice, this gap can be mitigated by employing the variational RL-
based compiler as detailed in Sec SI E.

Task 3: Compiling $\text{R}_{\text{ZZ}}(\gamma)$ gate with different $\gamma$.
In the main text, we have presented the compiling and experimental results
with $\gamma=\pi/2$. Here we present the results for arbitrary angles with

$\gamma\in\left\\{\frac{5\pi}{180},\frac{15\pi}{180},\frac{25\pi}{180},\cdots,\frac{95\pi}{180}\right\\}.$

Figure S10: Compilation of $\text{R}_{\text{ZZ}}(\gamma)$ gates. (a) Frequency
under variations of experimental fidelity $F_{2}$ and two-qubit gate count
$N_{2}$ for circuits by RL-based compiler (left panel) and Qiskit compiler
(right panel). (b) Left side: target unitary; right side: gate sequence by RL-
based compiler with $F_{1}=0.9914$.

The experimental results are shown in Fig. S10(a). For 10 target gates, the
RL-based compiler reaches the accuracy with ${F_{1}}=0.9967\pm 0.0029$ and
$F_{2}=0.9615\pm 0.0153$ on average, where the corresponding two-qubit gate
count is $N_{2}=1.6$ on average. The average inference time of AQ* search is
around $480$ seconds. For the Qiskit compiler, the accuracy is $F_{1}=1$ and
$F_{2}=0.9633\pm 0.0036$, respectively, and the number of two-qubit gates is
$N_{2}=3$. Hence, except for one target unitary
$\text{R}_{\text{ZZ}}(25\pi/180)$, the RL-based compiler requires fewer two-
qubit gates than those required by the Qiskit compiler, while the achieved
compiling accuracy is comparable. We show the gate decomposition of
$\text{R}_{\text{ZZ}}(75\pi/180)$ returned by the RL-based compiler in Fig.
S10(b). Compared with the standard solution requiring three two-qubit gates,
the RL-based compiler is more resource-efficient and fidelity-warranted.

Figure S11: Compilation of special targets for robust verification. Frequency
under variations of experimental fidelity $F_{2}$ and two-qubit gate count
$N_{2}$ for circuits by RL-based compiler (left panel) and Qiskit compiler
(right panel).

Task 4: More special targets for robust verification. In Task 1 where the
targets are composed of gates randomly sampled from $\mathcal{G}^{(2,1)}$, the
RL-based compiler demonstrates a slight benefit over the Qiskit compiler in
terms of the exploited two-qubit gates. In Task 2 where the targets are
randomly sampled from SU(4), our proposal shows a slight disadvantage. Hence,
a question that naturally arises is whether the performance of the RL-based
compiler can be enhanced even further by reducing the complexity of target
unitaries. Here we answer this question by generating a new test dataset
containing $50$ target unitaries, i.e., the construction rule is the same as
the one used in the main text, while the only difference is randomly sampling
$40$ single-qubit gates from $\mathcal{G}^{(2,1)}$ rather than $80$ single-
qubit gates.

The experimental results are exhibited in Fig. S11. On average, the RL-based
compiler uses $N_{2}=2.38$ CZ gates to accomplish the compiling task with
fidelity $F_{2}=0.9824\pm 0.0058$, while the Qiskit compiler uses $N_{2}=2.9$
CZ gates with fidelity $F_{2}=0.9789\pm 0.0045$. The results indicate that a
well-trained DQN with more training data and deeper loop number $L$
contributes to enhancing the robustness in compiling more complicated
circuits.

###  SI C.4 Summary of results for two-qubit system

We summarize the results of compilation for the two-qubit system in Tab. SII.

Table SII:  Performance of RL-based compiler and Qiskit compiler for the two-qubit system. $F_{2}$(XEB) and $F_{2}$(QPF) are the outcomes from XEB measurement and QPT measurement, respectively. | Qiskit | RL
---|---|---
Target | $N_{2}$ | $N_{1}$ | $F_{1}$ |  $F_{2}$(XEB) |  $F_{2}$(QPT) | $N_{2}$ | $N_{1}$ | $F_{1}$ |  $F_{2}$(XEB) |  $F_{2}$(QPT)
SWAP | 3 | 12 | 1 | 0.951 | 0.985 | 3 | 6 | 1 | 0.969 | 0.986
$\text{R}_{\text{ZZ}}({\pi}/{2})$ | 2 | 8 | 1 | 0.973 | 0.981 | 1 | 4 | 1 | 0.982 | 0.988
40 basis gates$\rightarrow$ U | 2.9 | 15.6 | 0.9999997 | 0.956 | 0.979 | 2.38 | 13.1 | 0.999995 | 0.967 | 0.982
80 basis gates$\rightarrow$ U | 2.957 | 15.84 | 0.9999997 | 0.954 | 0.979 | 2.80 | 11.8 | 0.999991 | 0.966 | 0.984
Random SU(4) | 3 | 16.0 | 0.9999997 | 0.953 | 0.978 | 2.97 | 14.2 | 0.9911 | 0.933 | 0.963

##  Appendix SI D EXTENDED DISCUSSION AND RESULTS FOR THREE-QUBIT SYSTEM

In this section, we describe the construction rule of the training data and
test data, and the implementation details of the RL-based compiler. A summary
of the results of compilation for the three-qubit system is also presented.

###  SI D.1 Construction of training data and test data

Data generation. We exploit four basis sets for different three-qubit
compiling tasks. The first basis set is
$\mathcal{G}^{(3,1)}=\\{\pm\text{X}/2,\pm\text{Y}/2,\text{T},\text{T}^{\dagger},\text{CZ}\\}$,
where the superscript $(3,1)$ represents the three-qubit case under the first
basis set and the connectivity constraint is $\rm
Q_{1}\mbox{-}Q_{2}\mbox{-}Q_{3}$. Thus the CZ gate can only be implemented on
neighboring qubits. Accordingly, the generalized basis set, or equivalently
the action space, yields

$\displaystyle\tilde{\mathcal{G}}^{(3,1)}\equiv\mathcal{A}^{(3,1)}=\Big{\\{}$
$\displaystyle\text{R}_{X}(\frac{\pi}{2})\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes\text{R}_{X}(\frac{\pi}{2})\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes\text{R}_{X}(\frac{\pi}{2}),\text{R}_{X}(-\frac{\pi}{2})\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes\text{R}_{X}(-\frac{\pi}{2})\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes\text{R}_{X}(-\frac{\pi}{2}),$
(10)
$\displaystyle\text{R}_{Y}(\frac{\pi}{2})\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes\text{R}_{Y}(\frac{\pi}{2})\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes\text{R}_{Y}(\frac{\pi}{2}),\text{R}_{Y}(-\frac{\pi}{2})\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes\text{R}_{Y}(-\frac{\pi}{2})\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes\text{R}_{Y}(-\frac{\pi}{2}),$
$\displaystyle T\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
T\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
T,T^{\dagger}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
T^{\dagger}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
T^{\dagger},\text{CZ}\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{CZ}\Big{\\}},$

with the cardinal number $|\mathcal{A}^{(3,1)}|=20$.

The second basis set is formed by HRC gate set [77] with
$\mathcal{G}^{(3,2)}=\\{V_{1},V_{1}^{\dagger},V_{2},V_{2}^{\dagger},V_{3},V_{3}^{\dagger},\text{CZ}\\}$,
where

$\displaystyle V_{1}$
$\displaystyle=\frac{1}{\sqrt{5}}\left(\begin{array}[]{cc}1&2i\\\
2i&1\end{array}\right),V_{2}=\frac{1}{\sqrt{5}}\left(\begin{array}[]{cc}1&2\\\
-2&1\end{array}\right),V_{3}$
$\displaystyle=\frac{1}{\sqrt{5}}\left(\begin{array}[]{cc}1+2i&0\\\
0&1-2i\end{array}\right).$ (11)

Considering the same connectivity constraint, the corresponding action space
yields

$\displaystyle\mathcal{A}^{(3,2)}=\Big{\\{}$ $\displaystyle
V_{1}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
V_{1}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
V_{1},V_{1}^{\dagger}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
V_{1}^{\dagger}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
V_{1}^{\dagger},V_{2}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
V_{2}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
V_{2},V_{2}^{\dagger}\otimes\mathbb{I}_{4},$ (12)
$\displaystyle\mathbb{I}_{2}\otimes
V_{2}^{\dagger}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
V_{2}^{\dagger},V_{3}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
V_{3}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
V_{3},V_{3}^{\dagger}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
V_{3}^{\dagger}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
V_{3}^{\dagger},\text{CZ}\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{CZ}\Big{\\}},$

with the cardinality $|\mathcal{A}^{(3,2)}|=20$.

For the third basis set, we consider the ideal setting by removing the
topological constraint of $\mathcal{G}^{(3,2)}$, with
$\mathcal{G}^{(3,3)}=\mathcal{G}^{(3,2)}=\\{V_{1},V_{1}^{\dagger},V_{2},V_{2}^{\dagger},V_{3},V_{3}^{\dagger},H,\text{CZ}\\}$.
Accordingly, the action space with respect to the third basis set yields

$\displaystyle\mathcal{A}^{(3,3)}=\Big{\\{}$ $\displaystyle
V_{1}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
V_{1}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
V_{1},V_{1}^{\dagger}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
V_{1}^{\dagger}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
V_{1}^{\dagger},V_{2}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
V_{2}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes V_{2},$ (13) $\displaystyle
V_{2}^{\dagger}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
V_{2}^{\dagger}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
V_{2}^{\dagger},V_{3}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
V_{3}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes
V_{3},V_{3}^{\dagger}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes
V_{3}^{\dagger}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes V_{3}^{\dagger},$
$\displaystyle\text{H}\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes\text{H}\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes\text{H},\text{CZ}\otimes\mathbb{I}_{2},\mathbb{I}_{2}\otimes\text{CZ},\ket{0}\bra{0}\otimes\mathbb{I}_{4}+\ket{1}\bra{1}\otimes\mathbb{I}_{2}\otimes\text{Z}\Big{\\}},$

with the cardinality $|\mathcal{A}^{(3,3)}|=24$.

The fourth gate set is formed by the single-qubit rotation gate with
$\mathcal{G}^{(3,4)}=\\{\pm\text{X}/128,\pm\text{Y}/128,\pm\text{Z}/128\\}$.
Accordingly, the action space for the fourth basis gate set yields

$\displaystyle\mathcal{A}^{(3,4)}=\Big{\\{}$
$\displaystyle\text{R}_{X}(\frac{\pi}{128})\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes\text{R}_{X}(\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes\text{R}_{X}(\frac{\pi}{128}),\text{R}_{X}(-\frac{\pi}{128})\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes\text{R}_{X}(-\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes\text{R}_{X}(-\frac{\pi}{128}),$
(14)
$\displaystyle\text{R}_{Y}(\frac{\pi}{128})\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes\text{R}_{Y}(\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes\text{R}_{Y}(\frac{\pi}{128}),\text{R}_{Y}(-\frac{\pi}{128})\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes\text{R}_{Y}(-\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes\text{R}_{Y}(-\frac{\pi}{128}),$
$\displaystyle\text{R}_{\text{Z}}(\frac{\pi}{128})\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes\text{R}_{\text{Z}}(\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes\text{R}_{\text{Z}}(\frac{\pi}{128}),\text{R}_{\text{Z}}(-\frac{\pi}{128})\otimes\mathbb{I}_{4},\mathbb{I}_{2}\otimes\text{R}_{\text{Z}}(-\frac{\pi}{128})\otimes\mathbb{I}_{2},\mathbb{I}_{4}\otimes\text{R}_{\text{Z}}(-\frac{\pi}{128})\Big{\\}}$

with the cardinality $|\mathcal{A}^{(3,4)}|=18$.

Compiling targets generation. The construction rule of three-qubit targets is
similar to that of the two-qubit case. As shown in Fig. S12(a), we consider
the two-qubit gates operated on $\rm Q_{1}$ and $\rm Q_{3}$. As such, we
collect the three-qubit targets by fixing three CNOT gates operated on $\rm
Q_{1}$ and $\rm Q_{3}$, and operating those single-qubit gates on $\rm Q_{1}$
and $\rm Q_{3}$. Particularly, 20 single-qubit gates uniformly sampled from
$\tilde{\mathcal{G}}^{(3,1)}$ are employed to form the single-qubit gates in
the circuit. Note that due to the topological constraint $\rm
Q_{1}\mbox{-}Q_{2}\mbox{-}Q_{3}$, this task belongs to the three-qubit gate
compiling. Fig. S12(b) exhibits the dissimilarity of the employed 10 unitaries
used in the three-qubit gate compiling tasks.

Figure S12: Three-qubit compiling targets generation. (a) Template circuit for
compiling target generation. (b) The heatmap of the similarity between 10
target unitaries employed in the three-qubit task. The $F_{1}$ metric
quantifies the similarity.

###  SI D.2 Implementation of RL-based compiler

Classical Device. Both the precompilation and inference processes related to
the classical part are implemented on the Nvidia Tesla V100 GPU.

Architecture of DQNs. The implementations of DQNs in three-qubit gate
compiling tasks are the same as those used in two-qubit tasks except for the
model size. The visualization of the employed DQN is shown in Fig. S7.
Specifically, it consists of an input layer, two hidden layers, and six
residual blocks, followed by $|\mathcal{A}^{(3,t)}|$ output neurons with
$t\in\\{1,2,3,4\\}$. The input layer is of size $64$. The first two hidden
layers are of sizes 8000 and 3000, and each residual block is composed of two
fully connected layers with 3000 hidden neurons. The leaky ReLU activation
function and batch normalization are applied to all hidden layers.

Hyper-parameters. We use Adam optimizer to optimize DQN, where the learning
rate is $\eta=10^{-3}$ without the weight decay [75]. The loss threshold
$\delta$ in Alg. 1 is set as $0.02$. The epoch $T$ depends on $l$, i.e.,
$T=100\cdot l$. In the training process, the optimized DQN models at each loop
$l$ under each action space $\mathcal{A}^{(3,t)}$, denoted by
$\\{Q^{(t,{l})}_{\bm{\theta}}\\}$, are stored to conduct multi-DQN
collaborative AQ* search. In the inference stage, we execute the multi-DQN
collaborative AQ* search guided by DQN model
$\\{Q^{(t,{l})}_{\bm{\theta}}\\}$. And the maximum AQ* search depth is set as
500 and the inference time is restricted to be less than 20 minutes.

The total variation distance (TVD). We exploit the total variation distance to
measure the distance between two distributions in the main text. To be
concrete, the total variation distance (TVD) is a standard metric to measure
the statistical distance between two probability distributions:

$d_{TV}(P_{e},P_{i})=\frac{1}{2}\sum_{x}|P_{e}(x)-P_{i}(x)|,$ (15)

where in this context, $P_{i}(x)$ and $P_{e}(x)$ denote the distributions of
the ideal and experimental quantum states in the measure of computational
basis, respectively.

###  SI D.3 Summary of results for three-qubit system

We summarize the results of compilation for the three-qubit system in Tab.
SIII.

Table SIII:  Performance of Qiskit, RL-based, and VRL-based compilers for three-qubit system. | Qiskit | RL | VRL
---|---|---|---
Target | $N_{2}$ | $N_{1}$ | $F_{1}$ |  $F_{2}$(QPT) | $N_{2}$ | $N_{1}$ | $F_{1}$ |  $F_{2}$(QPT) | $N_{2}$ | $N_{1}$ | $F_{1}$ |  $F_{2}$(QPT)
QFT | 15 | 40 | 1 | 0.739 | 7 | 28 | 0.94 | 0.834 | 7 | 28 | 1-10-7 | 0.932
Topologically constrained | 35.7 | 65.3 | 1 | 0.626 | 6.36 | 20.2 | 1 | 0.934 | $-$ | $-$ | $-$ | $-$

##  Appendix SI E VARIATIONAL RL-BASED QUANTUM COMPILER

The variational principle is widely used in the studies of many-body physics
and chemistry [42]. The key concept of the variational methods is considering
trial states from a physically motivated small subset of the exponentially
large Hilbert space, which effectively avoids the exponential space problem.
Examples include variational quantum algorithms [78, 7, 79, 80, 81, 82, 83]
and tensor networks [84, 85, 86, 87, 88].

We have combined the variational methods with the RL-based compiler using a
toolkit to optimize the single-qubit gate parameters classically. Below we
show that such variational RL-based (VRL-based) compiler can reach a higher
circuit fidelity, albeit with a trade-off of the increased inference time.

Implementation. The implementation of the VRL-based compiler consists of two
steps. In the first step, the optimized RL-based compiler described in Alg. 1
is utilized to search for a gate sequence
$\\{\tilde{G}_{i}\\}_{i=1}^{N}\in\tilde{\mathcal{G}}^{N}$ that approximates
the target $M$-qubit unitary $\hat{U}$. The estimated unitary is denoted by
$\hat{V}=\prod_{i=N}^{1}\tilde{G}_{i}\in\mathbb{C}^{2^{M}\times 2^{M}}$. In
the second step, the searched gate sequence $\\{\tilde{G}_{i}\\}_{i=1}^{N}$ is
transformed into a variational ansatz, i.e., $\hat{V}_{\bm{\gamma}}$, by
parametrizing the quantum gates in $\\{\tilde{G}_{i}\\}_{i=1}^{N}$. Note that
the way of parameterization is flexible and a possible approach is detailed
below. Once the parameterization is completed, the trainable parameters
$\bm{\gamma}$ are updated to minimize the loss function that measures the
discrepancy between the estimated unitary $\hat{V}_{\bm{\gamma}}$ and the
target unitary $\hat{U}$. The explicit form of the loss function is

$\displaystyle\min_{\bm{\gamma}}\mathcal{L}(\hat{V}_{\bm{\gamma}},\hat{U})=2^{M}-\text{Tr}\left(\hat{U}^{\dagger}\cdot\hat{V}_{\bm{\gamma}}\right).$
(16)

Construction of the variational ansatz $\hat{V}_{\bm{\gamma}}$. Given the
searched gate sequence $\\{\tilde{G}_{i}\\}_{i=1}^{N}$, the construction rule
of the variational ansatz $\hat{V}_{\bm{\gamma}}$ is heuristic. All the
single- and two-qubit gates can be parameterized to form a variational ansatz.
Practically, we only need to parameterize the single-qubit gates in
$\\{\tilde{G}_{i}\\}_{i=1}^{N}$ for higher circuit fidelity and keep the two-
qubit gate parameters fixed.

Results. We have seen in the main text that for the three-qubit QFT operation,
the fidelity of the RL compiled circuit is $F_{1}=0.94$. In combining the
variational method, the settings of the RL quantum compiler are identical to
those introduced in the main text. The Adam optimizer [75] with the initial
learning rate $0.01$ is employed to minimize the loss function in Eq. (16).
The learning steps are fixed to be $500$ and the process is repeated five
times to suppress the effects of randomness. Figure S13 illustrates the gate
sequence returned by the VRL-based compiler. In Tab. SIII, we find that the
fidelity of the circuit by the VRL-based compiler is $F_{1}=1-10^{-7}$ and the
corresponding experimental fidelity is $F_{2}=0.932$. The inference time
increases from about 800 seconds to about 900 seconds.

Figure S13: Circuit of three-qubit QFT compiled by VRL-based compiler. Without
the $\rm U_{3}$ gates, the circuit is returned by the RL-based compiler.