# CollComm: Enabling Efficient Collective Quantum Communication Based on EPR
buffering

Anbang Wu
<EMAIL_ADDRESS>
UC, Santa Barbara  Yufei Ding
<EMAIL_ADDRESS>
UC, Santa Barbara  Ang Li
<EMAIL_ADDRESS>
Pacific Northwest National Laboratory

###### Abstract

The noisy and lengthy nature of quantum communication hinders the development
of distributed quantum computing. The inefficient design of existing compilers
for distributed quantum computing worsens the situation. Previous compilation
frameworks couple communication hardware with the implementation of expensive
remote gates. However, we discover that the efficiency of quantum
communication, especially collective communication, can be significantly
boosted by decoupling communication resources from remote operations, that is,
the communication hardware would be used only for preparing remote
entanglement, and the computational hardware, the component used to store
program information, would be used for conducting remote gates. Based on the
observation, we develop a compiler framework to optimize the collective
communication happening in distributed quantum programs. In this framework, we
decouple the communication preparation process in communication hardware from
the remote gates conducted in computational hardware by buffering EPR pairs
generated by communication hardware in qubits of the computational hardware.
Experimental results show that the proposed framework can almost halve the
communication cost of various distributed quantum programs, compared to the
state-of-the-art compiler for distributed quantum computing.

##  I Introduction

Distributed quantum computing (DQC) architecture is being actively explored
[1, 2] to scale up the monolithic near-term quantum processing unit (QPU)
whose system size is small (up to a few hundred qubits) due to the fabrication
limitation [3]. In a DQC system, multiple small QPUs (a.k.a compute nodes) are
coordinated by inter-node communications to form a large quantum computing
(QC) system. A general DQC system may interface with a hierarchical quantum
network and contain several computing levels, e.g., the computing cluster
driven by quantum chiplets [1] on short-range networks [4, 5] and the quantum
data-center including computing clusters linked by the long-range network [6,
7, 8].

Inter-node communication is inevitable when executing programs on DQC
hardware. Inter-node communication relies on EPR pairs generated by
communication qubits (qubits specifically designed for EPR generation) and
makes gates on inter-node data qubits (qubits used to store program
information) executable. Compared to in-node communication between data
qubits, inter-node communication is far more error-prone and time-consuming
[9, 10] and should be carefully optimized when distributing quantum programs
by DQC compilers. Existing DQC compilers can be divided into two categories—
either ignore the low-level implementation of quantum communication and
perform program optimizations on the logical level [11, 12, 13, 14, 15, 16]
(equivalent to assuming an unbounded number of communication qubits) or
consider a limited number of communication qubits due to the hardware
fabrication difficulty [17, 18, 19]. Works in the former category focus on
distributing circuits to compute nodes as the parallelism between inter-node
communication and the feasibility of executing any inter-node circuit blocks
are naturally enabled by the unbounded communication qubits. Thus it is hard
to guarantee the performance of works in the first category when applied to
DQC hardware in the near future where limited communication resource is
expected on compute nodes.

Works in the latter category target optimizing communication on DQC hardware
with limited communication qubits but fall short in implementing multi-node
operations (i.e., collective communication). For example, if the four qubits
of a CCCX gate are evenly distributed on four compute nodes, with the few
($\leq 2$) communication qubits considered in [17, 18, 19], we cannot
implement the CCCX gate collectively by sharing all the three control qubits
to communication qubits that are in the same node as the target qubit and then
executing the CCCX gate locally. Actually, for works in the second category to
implement multi-node gates (or unitary blocks), they need to decompose the
target operation into smaller unitary blocks or basis gates (e.g., CX+U3
[20]), which would inevitably incur higher communication costs than
implementing the multi-node operation collectively. Besides the inefficiency
in performing collective communication, these works admit low communication
throughput. They usually perform inter-node communications with quite limited
parallelism and even sequentially.

Thus, can we overcome the inefficiency
cast by limited communication qubits while
respecting the constraints on DQC hardware?

We observe that it’s the inefficient use of communication qubits rather than
the amount of them that results in the unpromising communication performance
of works in the second category. In these works, communication qubits are
coupled with inter-node operations that we cannot use them to prepare new EPR
pairs for the next communication when they are being used for implementing
inter-node operations, let alone to accommodate the multi-node gates and
concurrent inter-node communication. Our answer to the above question is thus
YES with our key insight:

Rather than coupling inter-node operations
with communication qubits as in previous works,
decoupling them gives more benefits for communication.

Figure 1: The communication buffer which buffers the EPR pair produced by
communication qubits.

Specifically, we invent a new versatile computing component called
communication buffer based on idle data qubits of compute nodes to decouple
inter-node operations from communication qubits. In detail, communication
qubits are always used for preparing EPR pairs and once an EPR pair is
generated, we would swap it from communication qubits to data qubits in the
communication buffer, as shown in Figure 1. In this way, we can use the EPR
pairs stored in the communication buffer to conduct inter-node operations. The
communication buffer essentially provides an abstraction or intermediate layer
that is able to approximate the ideal DQC hardware (the one with unlimited
communication qubits) on near-term DQC hardware which is expected to have few
communication qubits. As long as the communication buffer is large enough, we
can implement multi-qubit inter-node operations collectively and admit a large
number of concurrent inter-node communication requests. On the other hand,
since the communication buffer exploits idle data qubits in DQC hardware, it
also boosts the device utilization rate.

Figure 2: The design overview of CollComm.

Based on the proposed communication buffer, we further develop the first
buffer-based communication optimization framework, CollComm as shown in Figure
2. CollComm provides excellent optimizations for collective communication
which is unexplored by existing DQC compilers. CollComm consists of three key
compiler passes. Firstly, given a qubit mapping, the first pass would inspect
the communication characteristics (e.g., the maximal number of concurrent
inter-node communication and the largest inter-node circuit block) of each
compute node, and then produce the communication buffer allocation that
potentially enables maximal throughput and fastest EPR generation rate on each
node. As in Figure 2, the buffer allocation would be co-optimized with the
qubit mapper to achieve a good balance in the number between data qubits and
the qubits in the buffer. With the well-tuned communication buffer, the second
pass performs buffer-based communication transformations to reduce the EPR
pair consumption. This pass would try to fuse multi-node circuit blocks as
decomposed operations usually incur higher communication resource consumption
than implemented collectively. This pass also explores several optimizations
which dynamically convert qubits in the communication buffer back to normal
data qubits to reduce the communication cost. Finally, the communication
routing pass exploits the communication buffer to boost the communication
throughput while reducing the communication cost induced by the hierarchical
DQC network. This pass tries to squeeze out all EPR pairs in the communication
buffer for concurrent inter-node communication. This pass further reduces
hardware-induced communication costs by smartly relaying communications
between distant (intra-cluster and inter-cluster) nodes.

Our contributions are summarized as follows:

  * •

We invent the communication buffer which provides an abstraction to
approximate ideal DQC hardware and reveals vast communication optimization
opportunities.

  * •

We propose an efficient communication buffer allocation pass by inspecting the
communication characteristics of the distributed quantum program. This pass
lays the foundation for communication optimizations.

  * •

We propose two buffer-based communication optimization passes which reduce
both program-intrinsic and hardware-induced communication costs remarkably.

  * •

Compared to the state-of-the-art baseline [19], CollComm significantly reduces
the inter-cluster communication resource consumption and the latency of
various programs by 50.4% and 47.6% on average, respectively.

##  II Background

In this section, we only introduce the essential background knowledge of
distributed quantum computing (DQC). We recommend [21] for the basics of
quantum computing. Without ambiguity, quantum/remote communication in the
following sections specifically refers to inter-node communication. And for
simplicity, we use block to denote a series of gates.

#### Quantum entanglement and DQC:

Quantum communication relies on the remote EPR (Einstein–Podolsky–Rosen)
entanglement in a pair of qubits that belongs to different quantum nodes. An
EPR entangled qubit pair (a.k.a an EPR pair) holds the entangled two-qubit
state $\frac{\left|00\right\rangle+\left|11\right\rangle}{\sqrt{2}}$. Other
commonly-used entangled states in DQC include the GHZ state
$\frac{\left|000\right\rangle+\left|111\right\rangle}{\sqrt{2}}$ and the cat-
state $\frac{\left|0\right\rangle^{\otimes n}+\left|1\right\rangle^{\otimes
n}}{\sqrt{2}}$, a generalization of the GHZ state on $n$ qubits.

The remote EPR pair between two nodes can be generated, e.g., via microwave
[4], or by interfering photons emitted from separate nodes [6]. The former way
is used to create short-range interconnect networks for quantum chiplets [1],
while the latter way may achieve long-range quantum networks like the quantum
internet [22]. The Fidelity of EPR pairs on the short-range network is higher:
the on-chip quantum link/EPR pair between nodes can achieve a fidelity of at
most 98.6% [5] while the fidelity of a long-range quantum link is at most 90%
in the recent studies [7, 8]. The preparation of a long-range quantum link is
also much more time-consuming than preparing a short-range link. Similar to
classical distributed computing, quantum compute nodes can also form a
hierarchical network topology where compute nodes connected by a short-range
network form a computing cluster, and clusters are inter-connected by the
long-range network.

Not all physical qubits on a DQC node can be directly used to establish the
remote EPR entanglement with another node [13]. Qubits able to construct
remote EPR pairs are called communication qubits [23] and serve as the base of
quantum communication. To distinguish the communication qubits from other
qubits in the node, we would refer to other qubits that are not designed for
quantum communication as data qubits or computational qubits. For example, in
Figure 3, $q_{0}$ and $q_{0}^{\prime}$ are communication qubits while $q_{1}$
and $q_{1}^{\prime}$ are data qubits. In the programming aspect, data qubits
used to store the information of the quantum program are also called program
qubits.

(a) (b)

Figure 3: (a) The Cat-Comm protocol. (b) The TP-Comm protocol. For the gate
(e.g. the final $Z$ gate) conditioned on the Z measurement, it is executed
only if the post-measured state is $\left|1\right\rangle$, corresponding to
the classical bit 1.

#### Quantum communication protocols:

The quantum no-clone theorem [21] makes quantum data not replicable between
compute nodes. To overcome this limitation, quantum communication protocols
rely on EPR pairs for data sharing. There are mainly two communication
protocols: the Cat-Comm shown in Figure 3(a) and the TP-Comm shown in Figure
3(b). In Figure 3(a), Cat-Comm utilizes the cat-entangler and -disentangler to
first share the state of $q_{1}$ to $q_{0}^{\prime}$, perform the target
controlled-unitary block and then revoke the sharing. Cat-Comm is used to
implement many commonly-used multi-node/collective operations [24], e.g.,
broadcast (sharing one qubit to all other compute nodes), multicast (sharing
one qubit to multiple other compute nodes), and reduce (equivalent to
broadcast up to local H gates). TP-Comm, on the other hand, exploits quantum
teleportation [21] to move qubits between compute nodes. For the TP-Comm in
Figure 3(b), it first moves $q_{1}$ to $q_{0}^{\prime}$ and then performs the
target unitary block. The differences between Cat-Comm and TP-Comm come from
two aspects: Cat-Comm can only be used to implement remote controlled-unitary
blocks while TP-Comm can implement any inter-node unitary blocks; After Cat-
Comm, the qubit being shared (i.e., $q_{1}$ in Figure 3(a)) will recover its
state while TP-Comm will leave the teleported state at the new qubit
($q_{0}^{\prime}$ in Figure 3(b)) and destroy the original qubit ($q_{1}$). In
a distributed quantum program, both TP-Comm and Cat-Comm may appear in (be a
part of) a general collective communication (the communication involving
multiple nodes). To avoid ambiguity, in the following sections, when we say
sharing one qubit to another node, we mean using Cat-Comm; when we say moving
or teleporting one qubit to another node, we mean using TP-Comm.

##  III Problem and Framework Design

###  III-A Collective Communication in DQC

(a) (b)

Figure 4: (a) The comparison of DQC and monolithic QC by applying Qiskit [20]
to the QFT program. Gate fidelity and latency are derived from [25, 18, 5] and
IBM’s device data. The time for preparing the EPR pair is ignored. (b) The
average percentages of multi-qubit gates from various quantum circuits that
involve more than a certain number of nodes, using the qubit-node mapping by
METIS [26]. Circuits are collected from [20, 27].

Like classical distributed computing, quantum communication bottlenecks the
performance of DQC, both in fidelity and latency. Figure 4(a) shows the
comparison of the DQC version quantum Fourier transformation (QFT) and the
monolithic QC version QFT (both compiled by Qiskit [20] with remote
communications treated in the same way as local communications). As shown in
Figure 4(a), the ESP (Estimated Probability of Success) and latency of
programs run on DQC hardware will be dozens of times worse than the ones
executed on a monolithic QC device. The considerable performance discrepancy
in Figure 4(a) results not only from the noisy and tedious nature of quantum
communication [21] but also from the bad adaptability of monolithic QC
compilers in addressing inter-node communication overheads. While the hardware
properties of quantum communication are hard to improve significantly in the
near future, the software compilation of distributed quantum programs should
try its best to mitigate the detrimental effects of quantum communication.
Efficient compilation support is critical to making DQC a feasible solution to
overcome the awful scalability of monolithic quantum devices [1, 3].

We observe that the efficient implementation of collective communication (the
communication involving multiple nodes) in distributed quantum programs is
critical to promoting DQC’s computational potential. By collecting statistics
on various quantum circuits (QFT, arithmetic functions, encoding circuits,
etc.) from existing quantum benchmarks [20, 27], we observe that 77.6%, 20.2%,
and 10.6% of quantum gates involve more than 3, 6, and 9 qubits, respectively,
on average. This demonstrates the potential existence of abundant collective
communications when executing these circuits on DQC hardware. Figure 4(b)
shows the statistics of collective communication in distributed quantum
circuits, with the METIS algorithm [26] to map qubits to compute nodes. As
shown in Figure 4(b), there are about 28.4% of multi-qubit gates require
communications between more than 3 nodes, when running collected circuits on 4
compute nodes. The percentage grows up when we run these circuits on 8 compute
nodes, where 49.8% of multi-qubit gates involve computation on more than 3
nodes. Therefore, to promote the efficiency of DQC, we must optimize the
implementation of collective communication wisely, e.g., reducing the
communication latency and the number of used EPR pairs.

###  III-B Communication Buffer and Framework Design

Figure 5: The buffering protocols which transfer the remote EPR pair on
communication qubits ($q_{0}$ and $q_{0}^{\prime}$) to data qubits ($q_{1}$
and $q_{1}^{\prime}$).

The optimization of collective communication still remains unexplored by
current DQC compilers. Due to the shortage of communication qubits in near-
term DQC hardware, it is often impractical to execute one multi-node operation
collectively, i.e., teleport/share all involved qubits to one node and then
implement the target operation locally, let alone explore specialized circuit
transformation and routing strategies for optimizing the cost of collective
communication. For example, if the three qubits of a Toffoli gate are
distributed in three fully-occupied nodes and each node only has one
communication qubit, then this remote Toffoli gate cannot be executed unless
it is decomposed into a series of CX gates and single-qubit gates with the
cost of consuming more EPR pairs. Thus to enable efficient collective
communication in DQC, we need to address challenges coming from both hardware
incapability and software inefficiency.

We observe that idle data qubits can be used to store remote EPR pairs
generated by the communication qubit, as indicated by buffering protocols in
Figure 5. We call data qubits with remote EPR pairs stored virtual
communication qubits. Virtual communication qubits can also be used for
implementing inter-node operations. That’s to say, by buffering protocols in
Figure 5, one physical communication qubit can emulate many virtual
communication qubits, thus overcoming the hardware incapability. Virtual
communication qubits constitute a software-defined communication facility and
we refer to it as a communication buffer. The communication buffer provides an
abstraction layer that decouples the remote EPR entanglement preparation
process from the execution of distributed quantum programs. A communication
buffer can hold any number of remote EPR pairs as long as there are sufficient
data qubits. The emergence of the communication buffer defines a new
communication model for DQC: the physical communication qubit serves as the
factory of virtual communication qubits and the communication buffer serves as
the component for conducting inter-node communication. Compared to the
communication model adopted by existing DQC compilers where communication
qubits are directly involved in the implementation of remote operations, the
communication buffer-based communication model can overcome the shortage of
‘physical communication qubits’ and pave the way for optimizing collective
communication.

Equipped with the communication buffer, we are now able to address the
software inefficiency in optimizing collective communication. Specifically, we
design a buffer-based framework with three key passes, for an end-to-end
compiler optimization for collective communications.

Firstly, we design a communication-aware buffer allocations pass in Section V
which determines the size of each node’s communication buffer according to the
communication characteristics (e.g., the maximum communication throughout and
the longest remote operation) of the distributed quantum circuit. The design
insight is that we can hide the latency of EPR preparation behind the
execution of remote operations. A well-tuned communication buffer should
enable a large EPR generation rate (thus high communication throughput) and
not occupy too many data qubits which could alternatively be used as program
qubits (to potentially achieve less node occupancy and fewer inter-node
communication). This pass utilizes high-level communication characteristics to
produce a good answer to the trade-off between virtual communication qubits
and program qubits (data qubits that stores program information).

Secondly, we design a buffer-based communication transformation pass in
Section IV-A. This pass would fuse remote unitary blocks to form a large
multi-node block (a.k.a collective communication block) in a greedy way (as
long as fewer EPR pairs are consumed). The insight behind such a design is
that the EPR pair consumption would likely decrease when implementing remote
operations collectively. On the other hand, for a better trade-off between the
virtual communication qubits and program qubits on each node, this pass would
dynamically convert a virtual communication qubit to a program qubit as long
as fewer EPR pair consumption is expected.

Finally, we design a communication routing pass that efficiently arranges
remote communications on a hierarchical DQC system with two network levels
where we need to choose the communication path for communication between
distant nodes. The design of this pass is based on the observation that
shortest path-based strategies for routing two-node or in-node communication
cannot lead to an optimal footprint of collective communication. One insight
in this pass is that Cat-Comm-based collective communication could be
optimized by the minimum spanning tree of involved nodes. Also, there is a
trade-off between parallelism and the footprint size of collective
communications, especially for inter-cluster communication where EPR pairs are
hard to prepare. For such a case, higher parallelism may be more favorable
than always sticking to the minimal footprint of the collective communication.

##  IV  Buffer-based Communication Optimizations

In this section, we introduce the buffer-based communication optimizations:
communication transformation to reduce the EPR pairs resulting from
distributed computing and communication routing to minimize the communication
overhead caused by the non-fully connected DQC network topology.

###  IV-A Communication Transformation

Figure 6: One example circuit extracted from quantum arithmetic circuits [20],
which is distributed over three compute nodes, with each node having one
communication qubit.

The basic benefit of the communication buffer is to preallocate EPR pairs
before implementing inter-node operations, similar to the persistent
communication in MPI [28]. The communication buffer can also be used to store
the teleported qubits in TP-Comm-based remote operations. In this way, we can
perform lazy teleportation that we do not need to move the teleported qubit
immediately back to its original nodes but leave it in the communication
buffer (equivalent to converting one virtual communication qubit into a
program qubit) since in the communication buffer, the teleported qubit cannot
stall the EPR preparation process.

Besides these basic usages of the communication buffer, there are more
advanced communication optimizations with communication buffers, as introduced
in the following paragraphs. The communication transformation pass is a loop
in which each iteration would run all these communication optimizations one by
one. This loop continues until no more improvement on EPR pair consumption is
reported.

#### Communication Fusion

Collective communication, if implemented efficiently, may consume fewer EPR
pairs than decomposed two-node communications. Unfortunately, collective
communication is not well optimized by state-of-the-art DQC compilers [17, 18,
19] due to the lack of communication qubits. Using the virtual communication
qubits in the communication buffer, we can reduce the EPR pair consumption by
exploiting the potential opportunity for collective communication. As an
example, we consider the distributed circuit shown in Figure 6. To implement
this circuit on the DQC hardware in which each compute node only has one
communication qubit, five EPR pairs are needed with existing DQC compilers,
e.g., [19]. However, with a communication buffer that occupies at least two
qubits in node C, we can implement all two-node gates in Figure 6 together,
requiring only need two EPR pairs to share $q_{1}$ and teleport $q_{2}$ to the
communication buffer of node C, and then executing the circuit in Figure 6
locally in node C. The optimization described, which combines several inter-
node operations into a collective communication block and then executes the
collective block by collecting involved qubits into the same node with the
help of the communication buffer, is called communication fusion. By fusing
remote unitary blocks with the communication buffer, we can significantly
reduce the EPR pairs required to implement these remote blocks.

We adopt a greedy strategy to discover and utilize communication fusion
opportunities. Given a preprocessed distributed circuit (we assume the circuit
is preprocessed by [19], i.e., two-node gates that can be implemented by one
EPR pair are already aggregated into a two-node block), for example the one in
Figure 6 in which each circled number denotes one two-node block, we describe
the greedy strategy for the communication fusion starting from the two-node
block \raisebox{-.9pt} {1}⃝ as follows:

  1. 1.

Check the following two-qubit block \raisebox{-.9pt} {2}⃝. If \raisebox{-.9pt}
{2}⃝ and \raisebox{-.9pt} {1}⃝ shares qubits, we proceed to Step 2. Otherwise,
\raisebox{-.9pt} {2}⃝ commutes with \raisebox{-.9pt} {1}⃝. We exchange the
position of these two blocks and restart from Step 1.
– In this example, since block \raisebox{-.9pt} {2}⃝ and block
\raisebox{-.9pt} {1}⃝ share qubit $q_{2}$, block \raisebox{-.9pt} {2}⃝ is a
potential block for fusion and we proceed to Step 2.

  2. 2.

Compute the EPR pair cost of executing block \raisebox{-.9pt} {1}⃝,
\raisebox{-.9pt} {2}⃝ collectively, i.e., by communication fusion.

  3. 3.

Check whether the computed EPR pair cost in Step 2 surpasses the EPR pair cost
of executing these blocks independently or not. If does not surpass, we fuse
block \raisebox{-.9pt} {1}⃝ and block \raisebox{-.9pt} {2}⃝ and use the fused
block as the new starting point of communication fusion. Otherwise, we use
block \raisebox{-.9pt} {2}⃝ as the new starting point.
– For the example circuit in Figure 6, the computed EPR pair cost in Step 2 is
2 and does not surpass the EPR pair cost of executing them independently.
Thus, for this example, we will fuse block \raisebox{-.9pt} {1}⃝ and block
\raisebox{-.9pt} {2}⃝.

  4. 4.

Repeat this process until no more fusion can be done.

It’s easy to see that after performing the above greedy procedure on the
example circuit in Figure 6, we can successfully fuse block \raisebox{-.9pt}
{1}⃝–\raisebox{-.9pt} {5}⃝ into a collective communication block that can be
implemented with only two EPR pairs. In the communication transformation pass,
we would invoke the greedy communication fusion procedure several times to
squeeze out opportunities for collective communication in the underlying
distributed program.

#### Communication Splitting

Figure 7: The circuit identity for splitting the multi-controlled unitary
block with one ancillary qubit.

We need to split the collective communication (multi-node unitary block) when
one compute node is not large enough to accommodate all involved qubits. For
example, we consider the multi-controlled-unitary block. If control qubits of
the multi-controlled-unitary block are scattered in compute nodes and the
number of them is larger than the node size, we cannot directly implement the
related collective communication. One simple way to overcome this issue is to
decompose the multi-controlled-unitary block into basis gates as in existing
DQC compilers [17, 19]. However, this method would incur overwhelming
communication costs. Even for a four-node CCCX gate, the decomposed block
would require at least 11 EPR pairs for implementation.

Figure 8: Recursive steps (i.e., (a)&(b)) of splitting a four-node CCCCCX
gate.

To avoid the heavy communication cost of the simple decomposition method, we
design an efficient communication splitting method by utilizing the circuit
identity in Figure 7. Our communication splitting method first preserves
qubits in the communication buffer of involved compute nodes, and then
recursively applies the circuit identity in Figure 7 to split the multi-
controlled-unitary block. As an example, we can split the CCCCCX gate in
Figure 8 as follows,

  1. 1.

For the node with $\geq 2$ control qubits (e.g., node A), use one qubit of the
buffer (e.g., $q_{c1}$) to act as the ancilla in Figure 7, as shown in Figure
8(a);

  2. 2.

For the node with only control qubit, combining this control qubit with other
control qubits to apply the splitting in Figure 7, like the $q_{c1}$ and
$q_{c2}$ in Figure 8(b).
– In this example, a four-node CCCX gate at most uses 6 EPR pairs (each
Toffoli in Figure 8(b) uses two EPR pairs), almost half of that by trivial
decomposition.

Note that the above recursive splitting procedure should be stopped once we
find the resulted communications are all executable, in order to exploit the
benefit of large collective communication. A multi-controlled unitary of a
size large than the buffer size could also be executed collectively as we can
use the communication buffer to swap qubits between nodes.

To some extent, our communication splitting optimization is based on the
dynamical conversion between virtual communication qubits and program qubits
(see $q_{c1}$ in Figure 8). Our communication splitting method is generally
applicable as any collective communication block is essentially a group of
multi-controlled-unitary blocks [29].

###  IV-B Communication Routing

Figure 9: A hierarchical DQC system, which consists of two network levels: compute nodes forming clusters representing short-range networks and clusters forming a long-range quantum network. One communication channel denotes one EPR pair. Numbers on edges denote the fidelity of the related EPR pairs. TABLE I: Quantitative data of operations in DQC (from [25, 7, 8, 30], with latencies normalized to CX counts). Fidelity is the averaged data and may not be uniform in the DQC system. Operation | Latency | Fidelity
---|---|---
Single-qubit gates |  $t_{1q}$ $\sim$ 0.1 CX |  $f_{1q}\approx$ 99.99%
CX and CZ gates |  $t_{2q}=$ 1 CX |  $f_{2q}\approx$ 99.80%
Measure |  $t_{ms}$ $\sim$ 5 CX |  $f_{ms}\approx$ 99.60%
Intra-cluster EPR preparation |  $t_{iep}$ $\sim$ 12 CX |  $f_{iep}\approx$ 98%
Intra-cluster classical comm |  $t_{icb}$ $\sim$ 1 CX |  —
Inter-cluster EPR preparation |  $t_{oep}$ $\sim 1000$ CX |  $f_{oep}\approx$ 90%
Inter-cluster classical comm |  $t_{ocb}$ $\sim 100$ CX |  —

In this section, we consider the hierarchical DQC system shown in Figure 9 for
demonstrating the buffer-based routing of collective communication. The
quantitative data of operations in the hierarchical DQC system are summarized
in Table I. When routing collective communication operations on the DQC
computing system in Figure 9, extra nodes are needed for relaying the
communication as the EPR pairs between any two nodes are not always readily
available in the communication buffer, either consumed or not able to be
established directly. For example, sharing qubits in node A to node D may need
both the EPR pairs between nodes A, B and nodes B, D, i.e., using node B for
communication relaying. More nodes in the communication (relay) path would
incur higher communication overhead, w.r.t fidelity and latency.

Overall, we expect to minimize the extra communication overhead induced by the
non-fully connected DQC network topology. Specifically, we focus on optimizing
the routing of Cat-Comm-based collective communication as each TP-Comm usually
involves two nodes and can be well optimized by using the shortest
communication paths between involved nodes.

#### Intra-cluster communication routing

For optimizing intra-cluster communication, we should reduce the overall
communication footprint of the collective operation, instead of simply
shortening paths between any two communication endpoints. To demonstrate the
difference between these two optimization targets, we consider the example in
Figure 10 where we are multicasting $q_{1}$ to nodes B, D. The shortest path
for establishing the cat-state on $q_{1}$ and $q_{4}$ is by $q_{1}\to q_{3}\to
q_{4}$ as it has the highest overall fidelity along the path. However, this
shortest path does not lead to the minimal overall fidelity of the multicast
operation. As shown in Figure 10, using the blue paths $q_{1}\to q_{2}\to
q_{4}$ in (b), we can establish the cat-state on $\\{q_{1},q_{2},q_{4}\\}$
with only two EPR pairs. In contrast, using the paths $q_{1}\to q_{3}\to
q_{4}$ and $q_{1}\to q_{2}$ in (a) would need only one more EPR pair to build
the cat-state on $\\{q_{1},q_{2},q_{4}\\}$. Moreover, the fidelity of the
paths in (a) is 0.98*0.98*0.98 which is worse than 0.98*0.97, i.e., the
fidelity of paths in (b).

Thus, using intermediate nodes in the collective communication operation for
relaying the communication can greatly improve the fidelity and latency of the
collective communication. Specifically, we can optimize the collective
communication by finding the minimum spanning tree (MST) of involved qubits,
which can be efficiently constructed by Kruskal’s algorithms [31]. Besides
being used for building the MST, we can exploit the communication buffer to
enable concurrent (collective) communication, in order to reduce the overall
communication latency.

Figure 10: The data flow (blue paths) of (intra-cluster) multicasting $q_{1}$
to node B, D: (a) by the shortest path; (b) by the minimum spanning tree.
$q_{1},q_{2},q_{3},q_{4}$ are qubits in node A, B, C, D, respectively. Numbers
on edges denote the fidelity of the related EPR pairs.

#### Inter-cluster communication routing

Figure 11: Multicasting $q_{1}$ to node E, G, and $q_{2}$ to node E: (a) by
the minimum spanning tree; (b) by maximizing the inter-cluster parallelism.
Assuming only one currently available communication channel between node B and
node E.

The optimization of inter-cluster communications is different from optimizing
intra-cluster communications as it is much harder and more time-consuming to
establish a communication channel between nodes in a different cluster than
nodes in the same cluster.

On the one hand, similar to the MST solution in intra-cluster communication
routing, when sharing the qubit $q_{1}$ in Figure 11 to nodes E, G in the
other cluster, we first communicate $q_{1}$ to node E and use the EPR pair
between node E and node G for sharing $q_{1}$ to node G. If we use the path
$q_{1}\to D\to G$, we would need another inter-cluster EPR pair which would
harm both the fidelity and latency of the collective communication being
routed. In summary, when communicating one qubit from one cluster to another
cluster, we should use one node in another cluster as the communication relay
to reduce the inter-cluster EPR pairs needed.

On the other hand, we cannot simply apply the intra-cluster communication
routing to the inter-cluster case. For example in Figure 11, we are
multicasting $q_{1}$ to nodes E, G and $q_{2}$ to node E, concurrently. There
are two ways to fulfill these inter-cluster communications, as shown in Figure
11(a)(b). The communication path in Figure 11(a) is simply generated by
solving the MST problem. The shortcoming of this communication routing is that
we have to wait for a long inter-cluster EPR preparation time for multicasting
$q_{2}$ to node E as the only available communication channel between node B
and E is consumed previously by communicating $q_{1}$. To reduce the overall
communication latency and improve the information throughput, we propose to
use the communication routing shown in Figure 11(b). That is, we should
exploit all available inter-cluster channels for data transfer, instead of
sticking to the routing generated by the intra-cluster communication
optimization. The reason for this routing design is that communication
channels between intra-cluster nodes are easier to prepare (thus relatively
abundant) while the number of inter-cluster channels is very limited and takes
an extremely long time to prepare. For inter-cluster communication,
unnecessary stalling would greatly degrade the communication throughput and
lead to very high decoherence error.

##  V  Buffer-aware Distributed Compilation

In this section, we assemble compilation components to form the pipeline for
generating communication-efficient distributed quantum programs, and discuss
the optimization of the communication buffer.

Figure 12: The communication buffer-based compilation pipeline for distributed
quantum programs. The compiled distributed circuit is further improved by the
parameter optimizer.

###  V-A Compilation Pipeline Overview

In the compilation pipeline shown in Figure 12, we consider the following
basic components:

  1. 1.

Communication buffer allocation, which produces buffer-aware device graph $G$
by reserving some data qubits in each node as the communication buffer. The
size of communication buffer $s=(s_{1},s_{2},\cdots,s_{i},\cdots)$ on each
node $n_{i}$ is a tunable parameter;

  2. 2.

Distributed circuit partition algorithm $P$, which takes a quantum circuit
$circ$ and $G$ as inputs, and generates a distributed version of $circ$ on
$G$. The qubit mapping $\pi$ of $P$ is a tunable parameter;

  3. 3.

The buffer-based collective communication optimization pass
$coll\\_comm\\_opt\\_pass$ proposed in Section IV;

  4. 4.

The circuit scheduling pass, which generates the schedule for operations in
the distributed circuit optimized by $coll\\_comm\\_opt\\_pass$. Specifically,
for EPR preparation, we would keep establishing EPR pairs among computing
nodes for inter-node blocks going to happen and buffer established EPR pairs
in the communication buffer for future use. The timing for EPR preparation is
adjusted to minimize program stall and EPR pair decaying (due to decoherence).

  5. 5.

Cost function $C$, which takes the distributed circuit $dcirc$ generated by
the scheduling pass as input, and outputs the estimated program fidelity based
on statistics of $dcirc$, e.g., the total number of EPR pairs needed, the
overall distributed circuit latency.

Specifically, we define the cost model as follows:

$C=f_{1q}^{N_{1q}}f_{2q}^{N_{2q}}f_{ms}^{N_{ms}}f_{iep}^{N_{iep}}f_{oep}^{N_{oep}}(1-e^{-\frac{t}{T}})$
(1)

where $f_{1q}$, $f_{2q}$, $f_{ms}$, $f_{iep}$, $f_{oep}$ are defined in Table
I, $N_{1q}$, $N_{2q}$, $N_{ms}$, $N_{iep}$, $N_{oep}$ are counts of the
operations, $t$ is the program latency, and $T$ is the coefficient for
decoherence error.

The two tunable parameters $s$ (communication buffer size on compute nodes)
and $\pi$ (qubit mapping) in the compilation pipeline affect the communication
efficiency of the compiled distributed program, and we adopt the simulated
annealing algorithm to optimize them, with $-\log C$ as the energy function,
$s$ being optimized first and $\pi$ the second. The initial values of $\pi$
and $s$ greatly affect the iteration cost of simulated annealing and the
values found. We initialize $\pi$ with the commonly-used graph partition
algorithm– Overall Extreme Exchange (OEE) by Park and Lee [32]. This algorithm
can be used to find the qubit mapping that minimizes the interaction between
compute nodes by exchanging qubits. To ease the design of the communication
buffer, we preserve one data qubit in each node. The generated qubit mapping
serves as a good initialization as it provides a lower up bound for
communication overhead.

In the next section, we discuss the initialization of the buffer parameter
$s$, which plays a critical role in the proposed buffer-aware compilation
pipeline.

###  V-B Communication Buffer Design

Figure 13: Communication buffer size initialization example. Node $n_{i+1}$ is
in the same cluster as the node $n_{i}$.

For communication buffer initialization, we consider the distributed circuit
mapped by the initial $\pi$ and optimized by the burst communication technique
proposed in [19]. For an arbitrary non-idle node $n_{i}$ (i.e., $\exists$
program qubits mapped into $n_{i}$), we compute initial $s_{i}$ as follows:

$s_{i}=2*\max(\max_{B}\lfloor\frac{t_{B}}{t^{B}_{ep}+t_{buf}}\rfloor,R_{pb}(n_{i}))$
(2)

Here $B$ is an inter-node block (involving two compute nodes) with $n_{i}$
being the communication target node, i.e., the block $B$ would be locally
computed in $n_{i}$ with related quantum data shared/teleported to $n_{i}$.
$t^{B}_{ep}$ is the latency for preparing EPR pairs between the endpoints of
$B$. If the endpoints of $B$ are in the same cluster as $n_{i}$, then
$t^{B}_{ep}=t_{iep}$, otherwise, $t^{B}_{ep}=t_{oep}$. $t_{buf}$ is the
latency of the selected EPR buffering protocol, e.g. $t_{buf}=2$ for the one
in Figure 5(b). $R_{pb}(n_{i})$ is the maximum number of parallel inter-node
blocks involving $n_{i}$. The ‘2*’ in Equ (2) means that we would use half of
the communication buffer in each node for handling communication requests, and
the other half for buffering EPR pairs. To give a concrete example of
computing $s_{i}$, we consider the example in Figure 13. Since there are three
concurrent inter-node blocks involving $n_{i}$, we have $R_{pb}(n_{i})=3$. On
the other hand,
$\max_{B}\lfloor\frac{t_{B}}{t^{B}_{ep}+t_{buf}}\rfloor=\lfloor\frac{50}{12+2}\rfloor=3$
(note that block $b_{1}$ is the largest remote block involving $n_{i}$).
Therefore, for this example, we just $s_{i}=2*3=6$.

This communication buffer design could achieve the maximal EPR generation rate
of $n_{i}$, i.e., the communication qubit of $n_{i}$ is not stalled by a
specific lengthy communication request while keeping $n_{i}$ always ready for
concurrent communication requests. On the other hand, a large communication
buffer resulting from $s_{i}$ can accommodate more aggressive collective
communication optimization discussed in Section IV.

In practice, it is possible that the sum $\sum_{i}s_{i}$ is larger than the
number of idle qubits in non-idle nodes, say $N_{idleq}$. In such a case, we
adjust $s_{i}$ proportionally to make $\sum_{i}s_{i}\leq N_{idleq}$. This
constraint on the communication buffer size $s$ is also enforced when we pick
the neighborhood of $s$ in simulated annealing, which is adopted to achieve a
good trade-off between program qubits and virtual communication qubits in each
node.

##  VI Evaluation

In this section, we compare the performance of CollComm to the baseline [19]
and analyze the effect of optimization passes proposed in CollComm.

###  VI-A Experiment Setup

#### Platforms

All experiments are executed on a Ubuntu 20.04 server which is equipped with
one 28-core Intel Xeon Platinum 8280 processor and 1TB RAM. The used software
includes Python 3.9.13 and Qiskit 0.37.0 [20].

#### Benchmark programs

The benchmark programs used in the evaluation are summarized in Table II and
are derived from IBM Qiskit [20] and RevLib [27]. Programs in Table II are
divided into two categories. The first category consists of building blocks of
quantum applications, including multi-controlled gate (MCTR), quantum ripple
carry adder (RCA) and cat-state preparation (CSP). The benchmarks in the first
category are small circuits. The second category contains relatively large
programs that represent various quantum applications, e.g., quantum Fourier
transformation (QFT), Bernstein-Vazirani (BV) algorithm, and Quantum
Approximate Optimization algorithm (QAOA). For BV, we consider 1000 randomized
secret strings which each contain $0.7*\\#\,qubit$ nonzeros. For QAOA, we
consider the maxcut problem on graphs where each vertex connects
$30+\lfloor\frac{\\#qubit}{12}\rfloor$ vertices on average.

#### Baseline

As the baseline, we implement the DQC compiler AutoComm [19]. AutoComm groups
a series of remote CX gates between two compute nodes into a burst
communication block and implements the burst communication block (all remote
CX gates in it) with at most two EPR pairs. AutoComm represents currently the
best effort in optimizing quantum communication overhead in distributing
quantum programs, as far as we know. However, without the communication
buffer, AutoComm cannot optimize more general collective communication that
may involve more than two compute nodes. In the following evaluation, we adopt
the same circuit partition algorithm—OEE algorithm (ref. Section V) for both
CollComm and AutoComm, in order to eliminate the difference caused by the
circuit partition.

#### DQC network model

For evaluation, we adopt the DQC computing system in Figure 9. Specifically,
we assume two clusters in the DQC network, with each cluster containing four
compute nodes and each compute node containing 40 data qubits and 1
communication qubit. The EPR pairs/channels that can be directly established
between two compute nodes are depicted in Figure 9. The inter-cluster EPR pair
and intra-cluster EPR pair differ from each other significantly in terms of
fidelity and preparation time, as shown in Table I. Like in the baseline, to
focus on communication optimization, we assume trapped-ion style compute nodes
[33] where any two local qubits can communicate with each other directly. Note
that our work can be easily extended to superconducting compute nodes by
integrating with existing single-node routing passes [20].

#### Metric

To characterize the communication overhead of the compiled distributed
circuits, we consider the following metrics:

  1. 1.

The number of consumed EPR pairs (‘# EPR’). More EPR pairs consumed would
naturally lead to lower program fidelity and longer latency. Thus, a lower
value of ‘# EPR’ is preferred.

  2. 2.

The number of consumed inter-cluster EPR pairs (‘# CROSS EPR’). ‘# CROSS EPR’
should be minimized due to the low fidelity of inter-cluster EPR pairs and the
long latency to prepare them.

  3. 3.

The latency of the compiled distributed circuit (normalized to CX counts).
Longer latency would lead to higher decoherence error, thus a lower value is
preferred.

  4. 4.

The size of allocated communication buffers to model the resource consumption
induced by using communication buffers. Specifically, we consider the average
number of qubits in communication buffers on compute nodes (‘AVG CB #q’).

To model the relative performance of CollComm to the baseline, we consider the
following metrics (expected as large as possible):

  1. 1.

‘EPR-DEC (%)’, which is defined by ‘1 - (# EPR by CollComm / # EPR by
baseline)*100%’.

  2. 2.

‘CROSS-EPR-DEC (%)’, which is defined by ‘1 - (# CROSS EPR by CollComm / #
CROSS EPR by baseline)*100%’.

  3. 3.

‘LAT-DEC (%)’, which is defined by ‘1 - (latency by CollComm / latency by
baseline)*100%’.

  4. 4.

‘FD-INC (%)’, which is defined ‘(ES-FD by CollComm / ES-FD by baseline -
1)*100%’. The ‘ES-FD’ (estimated fidelity) is computed according to Equ (1).

TABLE II: The compilation results of CollComm and its relative performance to
the baseline on benchmark programs.

Type |  Name |  # qubit | CollComm | Compared to AutoComm
---|---|---|---|---
# node |  AVG CB #q |  # EPR |  # CROSS EPR |  Latency |  EPR-DEC |  CROSS-EPR-DEC |  LAT-DEC |  FD-INC
Building Blocks |  Multi-Controlled Gate (MCTR) |  100 |  3 |  1.333 |  2 |  0 |  2521.3 |  0.6 |  0 |  0.045 |  0.109
200 |  6 |  1.667 |  5 |  2 |  2547.7 |  0.8 |  0.333 |  0.263 |  1.195
300 |  8 |  1.75 |  7 |  4 |  2917.3 |  0.811 |  0.636 |  0.398 |  4.499
Ripple-Carry Adder (RCA) |  100 |  3 |  1.333 |  4 |  0 |  872.6 |  0.5 |  0 |  0.081 |  0.139
200 |  6 |  1.667 |  10 |  2 |  1775.8 |  0.375 |  0.5 |  0.151 |  0.470
300 |  8 |  1.75 |  14 |  2 |  2664.6 |  0.5 |  0.5 |  0.143 |  0.900
Cat-State Preparation (CSP) |  100 |  3 |  0.333 |  2 |  0 |  63.4 |  0 |  0 |  0.427 |  0.005
200 |  6 |  0.667 |  5 |  1 |  158.4 |  -0.25 |  0 |  0.547 |  -0.011
300 |  8 |  0.75 |  7 |  1 |  164.4 |  0 |  0.75 |  0.792 |  0.374
Real World Appli-cations |  Quantum Fourier Transformation (QFT) |  100 |  3 |  6 |  98 |  0 |  3903.9 |  -0.225 |  0 |  0.638 |  0.148
200 |  6 |  6 |  492 |  64 |  9790.5 |  -0.23 |  0.6 |  0.772 |  $>>1e^{3}$
300 |  8 |  2.5 |  1058 |  152 |  38822.2 |  -0.080 |  0.729 |  0.913 |  $>>1e^{9}$
Bernstein Vazirani (BV) |  100 |  3 |  0 |  1 |  0 |  55.3 |  0 |  0 |  0.002 |  0.000
200 |  5 |  0.333 |  3 |  0 |  73.4 |  0 |  0 |  0.583 |  0.010
300 |  8 |  0.5 |  5 |  1 |  160.4 |  0 |  0.5 |  0.661 |  0.123
QAOA |  100 |  3 |  4 |  60 |  0 |  1323.0 |  0 |  0 |  0.476 |  0.128
200 |  6 |  4 |  300 |  40 |  3545.0 |  0 |  0.75 |  0.726 |  $>>1e^{3}$
300 |  8 |  2.5 |  560 |  80 |  8294.0 |  0 |  0.75 |  0.958 |  $>>1e^{9}$

###  VI-B Compared to the baseline

In this section, we analyze the relative performance of CollComm compared to
the baseline and discuss the effect of each optimization pass in CollComm.
Table II summarizes the relative performance of CollComm compared to the
baseline.

Firstly, compared to the baseline, CollComm on average reduces ‘# EPR’ (the
total number of EPR pairs consumed) by 14.6%, 11.6%, and 20.5% on 100-qubit,
200-qubit, 300-qubit programs, respectively. The optimization of ‘# EPR’ in
CollComm is affected by two factors: the communication buffer and the
communication transformation pass. The communication buffer alone has a
negative impact on reducing ‘# EPR’. This is because the communication buffer
may lead to more occupied nodes and more scattered qubit layouts when
distributing the circuit and thus more inter-node communications. For example,
for the 200-qubit CSP program, CollComm requires 6 nodes while AutoComm
requires 5 nodes. The one more node usage leads to one more EPR pair by
CollComm.

On the other hand, the communication transformation pass could reduce the EPR
pair consumption remarkably. For example, for RCA, the major communication is
TP-Comm based. ‘# EPR’ used by CollComm is smaller due to the lazy
teleportation technique in the communication transformation pass which
utilizes the communication buffer to temporarily store the teleported qubit
instead of immediately moving the teleported qubit back to its original node
as in AutoComm. For MCTR, CollComm largely reduces ‘# EPR’ (73.0% on average)
because of the communication fusion technique which can optimize multi-party
communication using the communication buffer. The MCTR by AutoComm is already
optimized by the communication splitting technique in CollComm. Otherwise, to
deal with the multi-controlled gate, AutoComm has to decompose it into the
CX+U3 basis [20] which would incur thousands of EPR pair consumption, making
AutoComm even worse compared to CollComm.

Secondly, the communication routing pass in CollComm greatly reduces inter-
cluster communication costs. Except for 100-qubit programs which do not
involve inter-cluster communication, CollComm reduces ‘# CROSS EPR’ (the total
number of inter-cluster EPR consumed) on average by 36.4% and 64.4%, on
200-qubit and 300-qubit programs respectively. The reduction of ‘# CROSS EPR’
on 200-qubit programs is smaller than that on 300-qubit programs because
inter-cluster communication is relatively limited on 200-qubit programs.
Overall, we observe that the significant reduction of CollComm on inter-
cluster EPR pairs owes to the optimization in communication routing pass that
converts inter-cluster communications into inter-node communications by using
one compute node in the (communication) target cluster for communication
relaying. To illustrate this point clearly, though the ‘# EPR’ by CollComm for
CSP, QFT, BV, and QAOA is more than (or equal to) that by AutoComm, ‘# CROSS
EPR’ by CollComm on these programs is still considerably lower than that by
AutoComm.

Moreover, CollComm significantly reduces the latency of distributed programs.
Compared to the baseline, CollComm reduces the program latency on average by
27.8%, 50.7%, and 64.4% on 100-qubit, 200-qubit, and 300-qubit programs,
respectively. For 100-qubit programs, there is no inter-cluster communication
and the latency reduction is mainly due to the high communication
parallelism/throughput enabled by a large number of virtual communication
qubits in communication buffers. The latency reduction by CollComm grows as
inter-cluster communication increases due to the great ability of CollComm in
optimizing inter-cluster communication.

Not surprisingly, CollComm also boosts the fidelity of compiled programs
greatly. For 100-qubit programs where the number of inter-node communication
is relatively limited, CollComm still achieves an 8.8% fidelity improvement
over AutoComm on average. For larger programs, the fidelity improvement by
CollComm is even larger. Except for QFT and QAOA, CollComm reduces the program
latency on average by 41.6% and 147.4% on 200-qubit and 300-qubit programs,
respectively compared to AutoComm. For QFT and QAOA, CollComm provides
overwhelming fidelity improvement due to the significantly reduced inter-
cluster EPR pairs and program latency.

In summary, CollComm is obviously better than state-of-the-art DQC compilers
in optimizing collective communication on the hierarchical and heterogeneous
DQC network.

##  VII Related Work

#### Compilers for monolithic QC:

Previous quantum compilers [20, 34, 35, 36, 37] compile quantum programs to a
monolithic QC hardware. While we can extend these works to DQC by treating the
transformation and routing remote communications like local/in-node
communications, this simple extension cannot fully expose the computational
potential of DQC. This is because, unlike this paper, these compilers do not
optimize collective communications widely existing in distributed quantum
programs.

#### Compilers for DQC:

Compilers for DQC can be mainly divided into two categories. Works in the
first category [11, 12, 13, 14, 15, 9, 24, 16] ignore the low-level
implementation of quantum communication and perform program optimizations on
the logical level, equivalent to assuming unlimited communication qubits.
These works mostly focus on the circuit partition and do not consider advanced
transformation and routing strategies for collective communication as
efficient communication is naturally enabled by the unbounded communication
resource. Thus we cannot guarantee the performance of these works on a
communication-resource-constrained DQC system. While Häner et al. [24]
provided optimizations for broadcast and reduce operations in DQC, their work
optimizes these collective operations as library functions and does not
consider the program context and the underlying hardware topology, which are
essential for compiler optimizations in this paper.

Works in the second category [17, 18, 19] consider communication optimization
on DQC hardware with limited communication qubits as in this paper. However,
these works fall short in implementing multi-node operations due to the
shortage of communication qubits, let alone exploring the transformation and
routing of collective communication. To execute multi-node operations, these
works would decompose them into small multi-node circuit blocks which would
inevitably incur higher communication costs. Besides the inefficiency in
performing collective communication, these works admit low communication
throughput. They usually perform inter-node communications with quite limited
parallelism and even sequentially, due to the lack of efficient utilization of
the constrained communication resource. All these drawbacks make these works
less efficient than this paper in optimizing collective communication.

Finally, there are also compilers [38, 39] proposed to execute distributed
quantum programs in a way without remote communications. These works cut large
circuits into smaller ones and then run these small circuits on different
devices. The drawback of these works is that they require unscalable classical
post-processing for large-scale DQC. Instead, our paper does not rely on
classical post-processing and can be scaled to the large-scale DQC.

##  VIII Discussion and Future Work

To the best of our knowledge, this paper is the first attempt that formalizes
a concept of the communication buffer in DQC. The communication buffer leads
to a new computing model lying between the one on a logical DQC network and
the other one on a physical DQC system and unveils new optimization
opportunities for collective communication in DQC. We believe this paper would
open new research topics for communication optimization in DQC.

Although we show that the proposed framework significantly surpasses existing
works in optimizing the collective communication of distributed quantum
programs, there is still much space left for potential future works.

#### Exploring more usage of communication buffer

This paper mainly focuses on the benefit of the communication buffer in
communication transformation and routing. There is actually more usage of the
communication buffer. For example, we can also use the communication buffer
for distilling the EPR pairs, especially the inter-cluster EPR pairs.
Optimizing collective communication along with EPR pair distillation presents
new research opportunities. Besides, the communication buffer can also be used
to implement more complicated communication protocols beyond the Cat-Comm and
TP-Comm. For example, we may replace a physical qubit in quantum communication
with a logical qubit so that the logical qubit serves as a bridge to two
compute nodes with communication channels hidden from programming and mainly
used for the error detection on the logical qubit.

#### Extending to fault-tolerant DQC program

This paper only considers the collective communication of unprotected
distributed quantum programs. For future DQC, it is expected programs will be
built upon quantum error correction (QEC) codes to account for the noise from
inter-node communications. If we consider the fault-tolerant architecture [2]
which constructs one logical qubit upon multiple nodes, basic QEC operations
such as magic state distillation, logical gates, and stabilizer measurements
will all inevitably incur collective communication, making the fault-tolerant
DQC program more demanding for collective communication optimization. It’s
thus interesting to extend our work to optimize fault-tolerant DQC programs.

#### Adapting to more DQC architectures

The proposed framework mainly explores the network hierarchy of DQC.
Considering recent progress in constructing quantum networks [40, 41, 42, 43,
44, 45, 46, 47, 48, 49, 50, 51], we can inspect more hardware levels in DQC
architecture. For example, a DQC system may consist of multiple physically
distinguished kinds of qubits [40]: memory qubits, communication qubits, etc.
We can use memory qubits that have a long coherence time to build the
communication buffer and use the native operations on the memory qubit as the
new instruction set on the communication buffer to construct QEC in the
communication buffer or to optimize collective communication. For program
qubits, we may adopt a different type of qubits from memory qubits, e.g.
superconducting qubits which are relatively accurate for quantum computation.
In such a case, the communication between the communication buffer and program
qubits becomes a new optimization problem to explore.

##  IX Conclusion

To promote the computational power of DQC, we invent the communication buffer
to decouple the execution of inter-node operations from communication qubits.
This design paves the way for collective communication optimization. We then
propose CollComm, a buffer-based compiler framework targeting the optimization
of collective communication in distributed quantum programs. Experimental
results on a hierarchical DQC system show that the proposed CollComm can
reduce the most expensive inter-node communication request and the program
latency by 50.4% and 47.6% on average, respectively.

## Acknowledgment

This material is based upon work supported by the U.S. Department of Energy,
Office of Science, National Quantum Information Science Research Centers,
Quantum Science Center. The Pacific Northwest National Laboratory is operated
by Battelle for the U.S. Department of Energy under Contract DE-
AC05-76RL01830. This work was also supported in part by NSF 2048144.

## References

  * [1] Nicholas Laracuente, Kaitlin N. Smith, Poolad Imany, Kevin L. Silverman, and Fred Chong.  Short-range microwave networks to scale superconducting quantum computation.  ArXiv, abs/2201.08825, 2022.
  * [2] Hamza Jnane, Brennan Undseth, Zhenyu Cai, Simon C. Benjamin, and Bálint Koczor.  Multicore quantum computing.  2022\.
  * [3] Gushu Li, Yufei Ding, and Yuan Xie.  Towards efficient superconducting quantum processor architecture design.  Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, 2020.
  * [4] P. Magnard, Simon Storz, P. Kurpiers, Josua Schär, Fabian Marxer, J Lütolf, Theo Walter, Jean-Claude Besse, Mihai S. Gabureac, Kevin Reuer, A. Akin, Baptiste Royer, Alexandre Blais, and Andreas Wallraff.  Microwave quantum link between superconducting circuits housed in spatially separated cryogenic systems.  Physical review letters, 125 26:260502, 2020.
  * [5] Alysson Gold, J.-P. Paquette, Anna Stockklauser, Matthew Reagor, M. Sohaib Alam, Andrew J Bestwick, Nicolas Didier, Ani Nersisyan, Feyza B. Oruç, Armin Razavi, Ben Scharmann, Eyob A. Sete, Biswajit Sur, Davide Venturelli, Cody James Winkleblack, Filip A. Wudarski, M. Harburn, and Chad T. Rigetti.  Entanglement across separate silicon dies in a modular superconducting qubit device.  npj Quantum Information, 7:1–10, 2021.
  * [6] Christopher R. Monroe, Robert Raussendorf, A Ruthven, Kenneth R. Brown, Peter Maunz, Luming Duan, and J. Kim.  Large-scale modular quantum-computer architecture with atomic memory and photonic interconnects.  Physical Review A, 89:022317, 2014.
  * [7] Raju Valivarthi, Samantha I. Davis, Cristian Pena, Si Xie, Nikolai Lauk, Lautaro Narváez, Jason P. Allmaras, Andrew D. Beyer, Yewon Gim, Meraj Hussein, George Iskander, Hyunseong Linus Kim, Boris A. Korzh, Andrew Mueller, Mandy Kathleen Rominsky, Matthew D. Shaw, Dawn Tang, Emma E. Wollman, Christoph Simon, Panagiotis Spentzouris, Daniel Oblak, Neil Sinclair, and Maria Spiropulu.  Teleportation systems toward a quantum internet.  PRX Quantum, 2020.
  * [8] Jelena V. Rakonjac, Dario Lago-Rivera, Alessandro Seri, Margherita Mazzera, Samuele Grandi, and Hugues de Riedmatten.  Entanglement between a telecom photon and an on-demand multimode solid-state quantum memory.  Physical Review Letters, 2021.
  * [9] Jonathan M. Baker, Casey Duckering, Alexander Hoover, and Frederic T. Chong.  Time-sliced quantum circuit partitioning for modular architectures.  Proceedings of the 17th ACM International Conference on Computing Frontiers, 2020.
  * [10] Christopher Young, Akbar Safari, Preston Huft, J. Zhang, Eun Oh, Ravikumar Chinnarasu, and Mark Saffman.  An architecture for quantum networking of neutral atom processors.  2022\.
  * [11] Robert Beals, Stephen Brierley, Oliver Gray, Aram Wettroth Harrow, Samuel Kutin, Noah Linden, Dan J. Shepherd, and Mark J. Stather.  Efficient distributed quantum computing.  Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 469, 2013.
  * [12] Mariam Zomorodi Moghadam, Monireh Houshmand, and Mahboobeh Houshmand.  Optimizing teleportation cost in distributed quantum circuits.  International Journal of Theoretical Physics, 57:848–861, 2016\.
  * [13] Pablo Andr’es-Mart’inez and Chris Heunen.  Automated distribution of quantum circuits via hypergraph partitioning.  Physical Review A, 2019.
  * [14] Zohreh Davarzani, Mariam Zomorodi Moghadam, Mahboobeh Houshmand, and Mostafa Nouri.  A dynamic programming approach for distributing quantum circuits by bipartite graphs.  Quantum Inf. Process., 19:360, 2020.
  * [15] Omid Daei, Keivan Navi, and Mariam Zomorodi-Moghadam.  Optimized quantum circuit partitioning.  International Journal of Theoretical Physics, 59(12):3804–3820, 2020.
  * [16] Davood Dadkhah, Mariam Zomorodi, Seyed Ebrahim Hosseini, Pawel Plawiak, and Xujuan Zhou.  Reordering and partitioning of distributed quantum circuits.  IEEE Access, 10:70329–70341, 2022.
  * [17] Davide Ferrari, Angela Sara Cacciapuoti, Michele Amoretti, and Marcello Caleffi.  Compiler design for distributed quantum computing.  IEEE Transactions on Quantum Engineering, 2:1–20, 2021.
  * [18] Stephen Diadamo, Marco Ghibaudi, and James R. Cruise.  Distributed quantum computing and network control for accelerated vqe.  IEEE Transactions on Quantum Engineering, 2:1–21, 2021.
  * [19] Anbang Wu, Hezi Zhang, Gushu Li, Alireza Shabani, Yuan Xie, and Yufei Ding.  Autocomm: A framework for enabling efficient communication in distributed quantum programs.  arXiv preprint arXiv:2207.11674, 2022.
  * [20] MD SAJID ANIS, Héctor Abraham, AduOffei, Rochisha Agarwal, Gabriele Agliardi, Merav Aharoni, Ismail Yunus Akhalwaya, Gadi Aleksandrowicz, Thomas Alexander, Matthew Amy, Sashwat Anagolum, Eli Arbel, Abraham Asfaw, Anish Athalye, Artur Avkhadiev, Carlos Azaustre, PRATHAMESH BHOLE, Abhik Banerjee, Santanu Banerjee, Will Bang, Aman Bansal, Panagiotis Barkoutsos, Ashish Barnawal, George Barron, George S. Barron, Luciano Bello, Yael Ben-Haim, M. Chandler Bennett, Daniel Bevenius, Dhruv Bhatnagar, Arjun Bhobe, Paolo Bianchini, Lev S. Bishop, Carsten Blank, Sorin Bolos, Soham Bopardikar, Samuel Bosch, Sebastian Brandhofer, Brandon, Sergey Bravyi, Nick Bronn, Bryce-Fuller, David Bucher, Artemiy Burov, Fran Cabrera, Padraic Calpin, Lauren Capelluto, Jorge Carballo, Ginés Carrascal, Adam Carriker, Ivan Carvalho, Adrian Chen, Chun-Fu Chen, Edward Chen, Jielun (Chris) Chen, Richard Chen, Franck Chevallier, Kartik Chinda, Rathish Cholarajan, Jerry M. Chow, Spencer Churchill, CisterMoke, Christian Claus, Christian Clauss, Caleb Clothier, Romilly Cocking, Ryan Cocuzzo, Jordan Connor, Filipe Correa, Abigail J. Cross, Andrew W. Cross, Simon Cross, Juan Cruz-Benito, Chris Culver, Antonio D. Córcoles-Gonzales, Navaneeth D, Sean Dague, Tareq El Dandachi, Animesh N Dangwal, Jonathan Daniel, Marcus Daniels, Matthieu Dartiailh, Abdón Rodríguez Davila, Faisal Debouni, Anton Dekusar, Amol Deshmukh, Mohit Deshpande, Delton Ding, Jun Doi, Eli M. Dow, Eric Drechsler, Eugene Dumitrescu, Karel Dumon, Ivan Duran, Kareem EL-Safty, Eric Eastman, Grant Eberle, Amir Ebrahimi, Pieter Eendebak, Daniel Egger, ElePT, Emilio, Alberto Espiricueta, Mark Everitt, Davide Facoetti, Farida, Paco Martín Fernández, Samuele Ferracin, Davide Ferrari, Axel Hernández Ferrera, Romain Fouilland, Albert Frisch, Andreas Fuhrer, Bryce Fuller, MELVIN GEORGE, Julien Gacon, Borja Godoy Gago, Claudio Gambella, Jay M. Gambetta, Adhisha Gammanpila, Luis Garcia, Tanya Garg, Shelly Garion, James R. Garrison, Tim Gates, Leron Gil, Austin Gilliam, Aditya Giridharan, Juan Gomez-Mosquera, Gonzalo, Salvador de la Puente González, Jesse Gorzinski, Ian Gould, Donny Greenberg, Dmitry Grinko, Wen Guan, Dani Guijo, John A. Gunnels, Harshit Gupta, Naman Gupta, Jakob M. Günther, Mikael Haglund, Isabel Haide, Ikko Hamamura, Omar Costa Hamido, Frank Harkins, Kevin Hartman, Areeq Hasan, Vojtech Havlicek, Joe Hellmers, Łukasz Herok, Stefan Hillmich, Hiroshi Horii, Connor Howington, Shaohan Hu, Wei Hu, Junye Huang, Rolf Huisman, Haruki Imai, Takashi Imamichi, Kazuaki Ishizaki, Ishwor, Raban Iten, Toshinari Itoko, Alexander Ivrii, Ali Javadi, Ali Javadi-Abhari, Wahaj Javed, Qian Jianhua, Madhav Jivrajani, Kiran Johns, Scott Johnstun, Jonathan-Shoemaker, JosDenmark, JoshDumo, John Judge, Tal Kachmann, Akshay Kale, Naoki Kanazawa, Jessica Kane, Kang-Bae, Annanay Kapila, Anton Karazeev, Paul Kassebaum, Josh Kelso, Scott Kelso, Vismai Khanderao, Spencer King, Yuri Kobayashi, Kovi11Day, Arseny Kovyrshin, Rajiv Krishnakumar, Vivek Krishnan, Kevin Krsulich, Prasad Kumkar, Gawel Kus, Ryan LaRose, Enrique Lacal, Raphaël Lambert, Haggai Landa, John Lapeyre, Joe Latone, Scott Lawrence, Christina Lee, Gushu Li, Jake Lishman, Dennis Liu, Peng Liu, Abhishek K M, Liam Madden, Yunho Maeng, Saurav Maheshkar, Kahan Majmudar, Aleksei Malyshev, Mohamed El Mandouh, Joshua Manela, Manjula, Jakub Marecek, Manoel Marques, Kunal Marwaha, Dmitri Maslov, Paweł Maszota, Dolph Mathews, Atsushi Matsuo, Farai Mazhandu, Doug McClure, Maureen McElaney, Cameron McGarry, David McKay, Dan McPherson, Srujan Meesala, Dekel Meirom, Corey Mendell, Thomas Metcalfe, Martin Mevissen, Andrew Meyer, Antonio Mezzacapo, Rohit Midha, Daniel Miller, Zlatko Minev, Abby Mitchell, Nikolaj Moll, Alejandro Montanez, Gabriel Monteiro, Michael Duane Mooring, Renier Morales, Niall Moran, David Morcuende, Seif Mostafa, Mario Motta, Romain Moyard, Prakash Murali, Jan Müggenburg, Tristan NEMOZ, David Nadlinger, Ken Nakanishi, Giacomo Nannicini, Paul Nation, Edwin Navarro, Yehuda Naveh, Scott Wyman Neagle, Patrick Neuweiler, Aziz Ngoueya, Johan Nicander, Nick-Singstock, Pradeep Niroula, Hassi Norlen, NuoWenLei, Lee James O’Riordan, Oluwatobi Ogunbayo, Pauline Ollitrault, Tamiya Onodera, Raul Otaolea, Steven Oud, Dan Padilha, Hanhee Paik, Soham Pal, Yuchen Pang, Ashish Panigrahi, Vincent R. Pascuzzi, Simone Perriello, Eric Peterson, Anna Phan, Kuba Pilch, Francesco Piro, Marco Pistoia, Christophe Piveteau, Julia Plewa, Pierre Pocreau, Alejandro Pozas-Kerstjens, Rafał Pracht, Milos Prokop, Viktor Prutyanov, Sumit Puri, Daniel Puzzuoli, Jesús Pérez, Quant02, Quintiii, Rafey Iqbal Rahman, Arun Raja, Roshan Rajeev, Isha Rajput, Nipun Ramagiri, Anirudh Rao, Rudy Raymond, Oliver Reardon-Smith, Rafael Martín-Cuevas Redondo, Max Reuter, Julia Rice, Matt Riedemann, Rietesh, Drew Risinger, Marcello La Rocca, Diego M. Rodríguez, RohithKarur, Ben Rosand, Max Rossmannek, Mingi Ryu, Tharrmashastha SAPV, Nahum Rosa Cruz Sa, Arijit Saha, Abdullah Ash-Saki, Sankalp Sanand, Martin Sandberg, Hirmay Sandesara, Ritvik Sapra, Hayk Sargsyan, Aniruddha Sarkar, Ninad Sathaye, Bruno Schmitt, Chris Schnabel, Zachary Schoenfeld, Travis L. Scholten, Eddie Schoute, Mark Schulterbrandt, Joachim Schwarm, James Seaward, Sergi, Ismael Faro Sertage, Kanav Setia, Freya Shah, Nathan Shammah, Rohan Sharma, Yunong Shi, Jonathan Shoemaker, Adenilton Silva, Andrea Simonetto, Deeksha Singh, Divyanshu Singh, Parmeet Singh, Phattharaporn Singkanipa, Yukio Siraichi, Siri, Jesús Sistos, Iskandar Sitdikov, Seyon Sivarajah, Magnus Berg Sletfjerding, John A. Smolin, Mathias Soeken, Igor Olegovich Sokolov, Igor Sokolov, Vicente P. Soloviev, SooluThomas, Starfish, Dominik Steenken, Matt Stypulkoski, Adrien Suau, Shaojun Sun, Kevin J. Sung, Makoto Suwama, Oskar Słowik, Hitomi Takahashi, Tanvesh Takawale, Ivano Tavernelli, Charles Taylor, Pete Taylour, Soolu Thomas, Kevin Tian, Mathieu Tillet, Maddy Tod, Miroslav Tomasik, Caroline Tornow, Enrique de la Torre, Juan Luis Sánchez Toural, Kenso Trabing, Matthew Treinish, Dimitar Trenev, TrishaPe, Felix Truger, Georgios Tsilimigkounakis, Davindra Tulsi, Wes Turner, Yotam Vaknin, Carmen Recio Valcarce, Francois Varchon, Adish Vartak, Almudena Carrera Vazquez, Prajjwal Vijaywargiya, Victor Villar, Bhargav Vishnu, Desiree Vogt-Lee, Christophe Vuillot, James Weaver, Johannes Weidenfeller, Rafal Wieczorek, Jonathan A. Wildstrom, Jessica Wilson, Erick Winston, WinterSoldier, Jack J. Woehr, Stefan Woerner, Ryan Woo, Christopher J. Wood, Ryan Wood, Steve Wood, James Wootton, Matt Wright, Lucy Xing, Jintao YU, Bo Yang, Unchun Yang, Daniyar Yeralin, Ryota Yonekura, David Yonge-Mallo, Ryuhei Yoshida, Richard Young, Jessie Yu, Lebin Yu, Christopher Zachow, Laura Zdanski, Helena Zhang, Iulia Zidaru, and Christa Zoufal.  Qiskit: An open-source framework for quantum computing, 2021.
  * [21] Michael A Nielsen and Isaac Chuang.  Quantum computation and quantum information, 2002.
  * [22] Stephanie Wehner, David Elkouss, and Ronald Hanson.  Quantum internet: A vision for the road ahead.  Science, 362, 2018.
  * [23] Marcello Caleffi, Daryus Chandra, Daniele Cuomo, Shima Hassanpour, and Angela Sara Cacciapuoti.  The rise of the quantum internet.  Computer, 53:67–72, 2020.
  * [24] Thomas Häner, Damian S. Steiger, Torsten Hoefler, and Matthias Troyer.  Distributed quantum computing with qmpi.  Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021.
  * [25] Nemanja Isailovic, Yatish Patel, Mark Whitney, and John Kubiatowicz.  Interconnection networks for scalable quantum computers.  In 33rd International Symposium on Computer Architecture (ISCA’06), pages 366–377. IEEE, 2006.
  * [26] George Karypis and Vipin Kumar.  A fast and high quality multilevel scheme for partitioning irregular graphs.  SIAM J. Sci. Comput., 20:359–392, 1998.
  * [27] R. Wille, D. Große, L. Teuber, G. W. Dueck, and R. Drechsler.  RevLib: An online resource for reversible functions and reversible circuits.  In Int’l Symp. on Multi-Valued Logic, pages 220–225, 2008.  RevLib is available at http://www.revlib.org.
  * [28] Lyndon Clarke, Ian Glendinning, and Rolf Hempel.  The mpi message passing interface standard.  In Programming environments for massively parallel distributed systems, pages 213–218. Springer, 1994.
  * [29] Dorit Aharonov.  A simple proof that toffoli and hadamard are quantum universal.  arXiv: Quantum Physics, 2003.
  * [30] Roberto Sanchez Correa and Jean Pierre David.  Ultra-low latency communication channels for fpga-based hpc cluster.  Integration, 63:41–55, 2018.
  * [31] Joseph B Kruskal.  On the shortest spanning subtree of a graph and the traveling salesman problem.  Proceedings of the American Mathematical society, 7(1):48–50, 1956\.
  * [32] Taehoon Park and Chae Y. Lee.  Algorithms for partitioning a graph.  Computers & Industrial Engineering, 28:899–909, 1995.
  * [33] Colin D Bruzewicz, John Chiaverini, Robert McConnell, and Jeremy M Sage.  Trapped-ion quantum computing: Progress and challenges.  Applied Physics Reviews, 6(2):021314, 2019.
  * [34] Gushu Li, Yufei Ding, and Yuan Xie.  Tackling the qubit mapping problem for nisq-era quantum devices.  Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, 2019.
  * [35] Matthew Amy and Vlad Gheorghiu.  staq—a full-stack quantum processing toolkit.  arXiv: Quantum Physics, 2019.
  * [36] Seyon Sivarajah, Silas Dilkes, Alexander Cowtan, Will Simmons, Alec Edgington, and Ross Duncan.  t—ket⟩: a retargetable compiler for nisq devices.  Quantum Science and Technology, 2020.
  * [37] Nader Khammassi, Imran Ashraf, J. van Someren, Răzvan Nane, A. M. Krol, M. A. Rol, Lingling Lao, Koen Bertels, and Carmen Garcia Almudever.  Openql : A portable quantum programming framework for quantum accelerators.  ACM J. Emerg. Technol. Comput. Syst., 18:13:1–13:24, 2022.
  * [38] Wei Tang, Teague Tomesh, Martin Suchara, Jeffrey Larson, and Margaret Martonosi.  Cutqc: using small quantum computers for large quantum circuit evaluations.  In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pages 473–486, 2021.
  * [39] Tianyi Peng, Aram Wettroth Harrow, Maris A. Ozols, and Xiaodi Wu.  Simulating large quantum circuits on a small quantum computer.  Physical review letters, 125 15:150504, 2020.
  * [40] S. L. N. Hermans, Matteo Pompili, H. K. C. Beukers, S. Baier, Johannes Borregaard, and Ronald Hanson.  Qubit teleportation between non-neighbouring nodes in a quantum network.  Nature, 605:663 – 668, 2022.
  * [41] David L Moehring, Peter Maunz, Steve Olmschenk, Kelly C Younge, Dzmitry N Matsukevich, L-M Duan, and Christopher Monroe.  Entanglement of single-atom quantum bits at a distance.  Nature, 449(7158):68–71, 2007.
  * [42] Stephan Ritter, Christian Nölleke, Carolin Hahn, Andreas Reiserer, Andreas Neuzner, Manuel Uphoff, Martin Mücke, Eden Figueroa, Joerg Bochmann, and Gerhard Rempe.  An elementary quantum network of single atoms in optical cavities.  Nature, 484(7393):195–200, 2012.
  * [43] Julian Hofmann, Michael Krug, Norbert Ortegel, Lea Gérard, Markus Weber, Wenjamin Rosenfeld, and Harald Weinfurter.  Heralded entanglement between widely separated atoms.  Science, 337(6090):72–75, 2012.
  * [44] LJ Stephenson, DP Nadlinger, BC Nichol, S An, P Drmota, TG Ballance, K Thirumalai, JF Goodwin, DM Lucas, and CJ Ballance.  High-rate, high-fidelity entanglement of qubits across an elementary quantum network.  Physical review letters, 124(11):110501, 2020.
  * [45] Hannes Bernien, Bas Hensen, Wolfgang Pfaff, Gerwin Koolstra, Machiel S Blok, Lucio Robledo, Tim H Taminiau, Matthew Markham, Daniel J Twitchen, Lilian Childress, et al.  Heralded entanglement between solid-state qubits separated by three metres.  Nature, 497(7447):86–90, 2013.
  * [46] Peter C Humphreys, Norbert Kalb, Jaco PJ Morits, Raymond N Schouten, Raymond FL Vermeulen, Daniel J Twitchen, Matthew Markham, and Ronald Hanson.  Deterministic delivery of remote entanglement on a quantum network.  Nature, 558(7709):268–273, 2018.
  * [47] Aymeric Delteil, Zhe Sun, Wei-bo Gao, Emre Togan, Stefan Faelt, and Ataç Imamoğlu.  Generation of heralded entanglement between distant hole spins.  Nature Physics, 12(3):218–223, 2016.
  * [48] Robert Stockill, MJ Stanley, Lukas Huthmacher, E Clarke, M Hugues, AJ Miller, C Matthiesen, Claire Le Gall, and Mete Atatüre.  Phase-tuned entangled state generation between distant spin qubits.  Physical review letters, 119(1):010503, 2017.
  * [49] P Maunz, S Olmschenk, D Hayes, DN Matsukevich, L-M Duan, and C Monroe.  Heralded quantum gate between remote quantum memories.  Physical review letters, 102(25):250502, 2009.
  * [50] Severin Daiss, Stefan Langenfeld, Stephan Welte, Emanuele Distante, Philip Thomas, Lukas Hartung, Olivier Morin, and Gerhard Rempe.  A quantum-logic gate between distant quantum-network modules.  Science, 371(6529):614–617, 2021.
  * [51] Norbert Kalb, Andreas A Reiserer, Peter C Humphreys, Jacob JW Bakermans, Sten J Kamerling, Naomi H Nickerson, Simon C Benjamin, Daniel J Twitchen, Matthew Markham, and Ronald Hanson.  Entanglement distillation between solid-state quantum network nodes.  Science, 356(6341):928–932, 2017.