# Efficient Intra-Rack Resource Disaggregation for HPC Using Co-Packaged DWDM
Photonics

George Michelogiannakis
Lawrence Berkeley National Laboratory
<EMAIL_ADDRESS>
&Yehia Arafa
Qualcomm Technologies, Inc
<EMAIL_ADDRESS>
&Brandon Cook
Lawrence Berkeley National Laboratory
<EMAIL_ADDRESS>
&Liang Yuan Dai
Columbia University
<EMAIL_ADDRESS>
&Abdel-Hameed Badawy
New Mexico State University
<EMAIL_ADDRESS>
&Madeleine Glick
Columbia University
<EMAIL_ADDRESS>
&Yuyang Wang
Columbia University
<EMAIL_ADDRESS>
&Keren Bergman
Columbia University
<EMAIL_ADDRESS>
&John Shalf
Lawrence Berkeley National Laboratory
<EMAIL_ADDRESS>

###### Abstract

The diversity of workload requirements and increasing hardware heterogeneity
in emerging high performance computing (HPC) systems motivate resource
disaggregation. Disaggregation separates servers into their constituent
compute and memory resources so that they can be allocated as required to each
workload. Previous work has shown the potential of intra-rack resource
disaggregation, but it is not clear how to realize these gains and cost-
effectively meet the stringent bandwidth and latency requirements of HPC
applications. To that end, we describe how modern photonic components can be
co-designed with modern HPC racks to implement flexible intra-rack resource
disaggregation and fully meet the high escape bandwidth of contemporary multi
chip module (MCM) packages and all chip types in modern HPC racks with
negligible power overhead. We show how to use distributed indirect routing to
meet these demands without the need for significant complexity for
reconfiguration that spatial optical switches require. We then show that
intra-rack resource disaggregation implemented using emerging photonics and
parallel optical wavelength-selective switches satisfies bit error rate (BER)
and bandwidth constraints and provides an average application speedup of 23.9%
for 31 out-of-order CPU and 61.5% for 27 GPU benchmarks compared to a similar
system that instead uses modern electronic switches for disaggregation, due to
their higher latency. Using observed resource usage from a production system,
we estimate that an iso-performance intra-rack disaggregated HPC system using
photonics would require 4$\times$ fewer memory modules and 2$\times$ fewer
NICs than a non-disaggregated baseline.

_Keywords_ Photonics $\cdot$ disaggregation $\cdot$ AWGR $\cdot$ spatial

##  1 Introduction

Leading HPC systems are steadily embracing heterogeneity of compute and memory
resources as a means to preserve performance scaling and reduce system power
Liu et al. (2012); Top (2018); Ujaldón (2016). This trend is already apparent
with the integration of GPUs Mittal and Vetter (2015); Tiwari et al. (2015);
Gao and Zhang (2016) and is expected to continue with fixed-function or
reconfigurable accelerators such as field programmable gate arrays, Milojicic
(2020); Asaadi and Chapman (2017); Segal et al. (2014); Hogervorst et al.
(2021); Lant et al. (2020); Dimond et al. (2011); Ramirez-Gargallo et al.
(2019), emerging customized accelerators, and heterogeneous memory Venkata et
al. (2017). In addition, key HPC workloads show considerable diversity in
computational and memory access patterns Michelogiannakis et al. (2022);
Rodrigo et al. (2016).

This expectation of resource heterogeneity, workload diversity, and today’s
method of allocating resources to applications in units of statically-
configured nodes where every node is identical and unused resources are left
to idle, raises the concern of resource underutilization (referred to as
“marooned resources”). These marooned resources increase both capital and
operational costs without improving performance. This has led to the emergence
of resource disaggregation. Disaggregation refers to decomposing servers into
their constituent compute and memory resources so that these can be allocated
as required according to the needs of each workload. Hyperscale datacenters
have readily embraced resource disaggregation and have demonstrated that it
significantly improves utilization of GPUs and memory Guleria et al. (2019a);
Peng et al. (2020); Taylor (2015); Li et al. (2022); Koh et al. (2019);
Papaioannou et al. (2016); Gonzalez et al. (2020); Cheng et al. (2019a);
Guleria et al. (2019b).

Although file storage is routinely disaggregated in modern systems Per ;
Michelogiannakis et al. (2022); Petersen and Bent (2017), HPC has been slow to
embrace disaggregation of compute and memory resources Glick et al. (2020);
Guo et al. (2021) due to the sensitivity of HPC workloads to bandwidth and
latency that cannot be met by current PCIe/CXL or Ethernet link technologies
used in contemporary disaggregated architectures. Studies showed that
disaggregation only among resources in the same rack (i.e., intra-rack
resource disaggregation) in HPC could reduce resources by 5.36% to 69.01%
while avoiding the overhead of full-system disaggregation Michelogiannakis et
al. (2022), but the impact of increased memory latency and specific
architectural trade offs have not been explored. Thus, although disaggregation
using electronic networks have been demonstrated in hyperscale datacenters Lin
et al. (2020); Papaioannou et al. (2016); Call et al. (2020), minimizing
adverse effects to and addressing the stringent bandwidth density and latency
demands of HPC workloads requires a thorough investigation.

In this work, our contributions are as follows. Firstly, we describe how to
use emerging photonic links and switches to design modern and practical
resource-disaggregated HPC racks based on an existing GPU-accelerated HPE/Cray
EX supercomputer Per . Secondly, we show how state of the art commercially
available photonics and advanced packaging MCMs meet BER requirements, impose
a negligible power overhead, and deliver sufficient bandwidth to satisfy the
escape bandwidth of all chips in modern HPC racks. Thirdly, we show how to use
distributed indirect routing and arrayed waveguide grating routers Liu et al.
(2020); Zhang et al. (2019) to satisfy all bandwidth requirements without the
overhead and latency for reconfiguration that spatial Seok et al. (2019a);
Ding et al. (2016) and wave-selective Huang et al. (2020) switches require.
Moreover, having demonstrated negligible adverse impact to all other metrics,
we show that intra-rack disaggregation using emerging photonics provides an
average application speedup of 23.9% for 31 out-of-order (OOO) CPU and 61.5%
for 27 GPU benchmarks compared to a similar system that instead uses state of
the art electronic switches, which also increase power overhead by three
orders of magnitude. Finally, based on observed resource usage, we estimate
that a system based on racks using photonics for resource disaggregation can
have 43% fewer overall chips compared to a non-disaggregated system with the
same computational throughput.

##  2 Related Work

Hyperscale datacenters predominantly focus on full-system resource
disaggregation where applications can allocate fine-grain resources of
different types, today typically graphical processing units Guleria et al.
(2019a) and memory Peng et al. (2020); Lim et al. (2009); Koh et al. (2019);
Gonzalez et al. (2020), which are located anywhere in the system or within a
group of racks Lin et al. (2020); Papaioannou et al. (2016); Call et al.
(2020). In such a system, resources of the same type are typically placed in
the same rack.

However, full-system, flexible, and fine-grain disaggregation introduces
significant overhead because of the higher latency and lower bandwidth density
of contemporary hardware that is used to implement resource disaggregation –
typically PCIe, 100Gig Ethernet, and eventually compute express link (CXL) Van
Doren (2019) over electronic links. This overhead does not simply increase
power and procurement cost, but rather adds potentially substantial latency
between key resources such as central processing units and memory that
traditionally exhibit latency-sensitive communication. The aforementioned
studies quote a several orders of magnitude increase in network and memory
latency due to full-system resource disaggregation to improve resource
utilization by 35% at most Zervas et al. (2018). Another study found that
application performance degradation depends on both network bandwidth and
latency, but can still reach up to 40% even with high bandwidth, low-latency
networks Gao et al. (2016). Work on SPEC and commercial benchmarks also found
an up to 27% application slowdown due to the additional memory latency Abali
et al. (2015). A study on Microsoft’s Azure found a range of performance
slowdowns up to 30% from an extra 65 ns to access main memory Li et al.
(2022). SDN based on electrical networks fare no better in terms of overhead
Gao et al. (2016); Han et al. (2013); Call et al. (2020).

Hybrid full-system photonic–electronic approaches have also been proposed that
rely on circuit switching Zervas et al. (2018) for reconfiguration. As a
result, a few studies call for intra-rack disaggregation in datacenters Taylor
(2015); Lim et al. (2009); Guleria et al. (2019b). Even the low latency and
high bandwidth density of modern photonics cannot fully satisfy the bandwidth,
energy, and latency requirements of full system disaggregation. This makes
system-wide disaggregation impractical in many cases Lin et al. (2020); Zervas
et al. (2018); Cheng et al. (2019a, 2018).

Recent full system approaches in HPC rely on optics to connect CPUs and
memory, and electronic switches for hard disk drives to increase resource CPU
utilization by 36.6% and memory 21.5% Guo et al. (2021). In contrast, another
study confirms that production HPC systems can reduce resources from 5.36% to
69.01% with intra-rack disaggregation and still satisfy the worst-case average
rack utilization Michelogiannakis et al. (2022). Similar to datacenters,
intra-rack disaggregation in HPC promises the lowest overhead and impact to
applications Glick et al. (2020); Taylor (2015); Guleria et al. (2019b).

Related work has research other aspects necessary to make resource
disaggregation practical in a system, such as job scheduling Fan et al.
(2019); Agosta et al. (2018); Amaral et al. (2021); Domeniconi et al. (2019),
how the operating system (OS) and runtime should adapt Maccabe (2017); Hwu et
al. (2015); Shan et al. (2018), programming of code portability in
heterogeneous systems Gioiosa et al. (2020); Agosta et al. (2018),
partitioning of application data Khaleghzadeh et al. (2020), fault tolerance
Hussain (2020), how to fairly compare the performance of different
heterogeneous systems Jamieson et al. (2018), and the impact of heterogeneous
resources to application performance Tang et al. (2017); Lastovetsky (2015);
Venkata et al. (2017). These are important but out of scope topics for our
study.

###  2.1 Under-utilization in Production Systems

We use NERSC’s Cori system as an exemplar production HPC system, while
recognizing workload requirements on other systems may differ. In NERSC’s
Cori, before Perlmutter came online and thus Cori was runnign the full NERSC
workload, three quarters of the time Haswell nodes use less than 17.4% of
memory capacity (50.1% for KNL nodes) and less than 0.46 GB/s of memory
bandwidth Michelogiannakis et al. (2022). These observations are similar to
observations collected on LANL clusters Peng et al. (2020) and Alibaba
machines that execute batch jobs. Likewise, half of the time Cori nodes use no
more than half of their compute cores and three quarters of the time 1.25% of
available network interface controller (NIC) bandwidth. Similarly, in Lawrence
Livermore National Laboratory (LANL) clusters, approximately 75% of the time,
no more than 20% of memory capacity is used Peng et al. (2020). Alibaba’s
published data Guo et al. (2019) show that memory is underutilized similarly
to Cori for machines that execute batch jobs. Data from Google systems shows
that memory and disk capacity of tasks is spread over three orders of
magnitude and typically underutilized Han et al. (2013). Azure reports 25% of
memory under-utilization Li et al. (2022). Datacenters have also reported 28%
to 55% CPU idle in the case of Google trace data Patel et al. (2015) and 20%
to 50% most of the time in Alibaba Guo et al. (2019). Early studies also
suggest GPU under-utilization Li et al. (2015); Jeon et al. (2019); Li et al.
(2011).

##  3 Photonics for Resource Disaggregation

Here we walk through available optical link and switch technologies and argue
that photonics today meet the strict performance and error rate requirements
to efficiently implement intra-rack resource disaggregation in HPC.

###  3.1 Memory Technologies and Requirements

IO systems in HPC are already largely disaggregated over conventional system-
scale interconnects since the underlying technologies (disk or SSD) are
relatively high latency and lower bandwidth Michelogiannakis et al. (2022);
Terzenidis et al. (2018). By contrast, memory technologies (particularly high
bandwidth memorys needed by GPUs) are much higher bandwidth and much less
tolerant of latency and require much lower BERs. Given that memory
disaggregation imposes the most challenging constraints among other resources
in today’s compute nodes, we will use DDR and HBM memory technology to set our
performance target. A typical DDR4 memory has a response latency of
approximately 90 ns and for HBM the average response latency is 90-140 ns Wang
et al. (2020). Still, any added latency between the CPU and memory from
resource disaggregation may penalize application performance as we quantify
later. Server-class memories typically require BERs of less than $10^{-18}$ to
achieve tolerable failures in time (FIT) rates with conventional single-error-
correct/double-error-detect (SEC-DED) protection Meza et al. (2015); Sridharan
et al. (2015). FEC can reduce the BER, but with additional latency Luyi et al.
(2012).

###  3.2 Optical Link Technologies

We consider a range of photonic link technologies that include conventional
100 Gbps Ethernet physical interfaces that represent the current baseline link
technology for memory disaggregation. We also introduce a range of cutting-
edge dense wavelength division multiplexing (DWDM) link technologies that are
either demonstrated as research prototypes or are commercially available. The
photonic components all come from existing commercial technologies (100 Gbps,
400 Gbps, Ayar TeraPhy) and some research prototypes from DARPA PIPES (the 1-2
Tb link technologies). These higher performance link technologies must be co-
packaged in order to achieve their bandwidth density. These link technologies
are summarized in Table 1. The technology for the optical links is depicted in
Figure 1. Delivering multiple channels of laser light to the package has been
challenging to scale cost-effectively if each "color" of light were to require
a separate laser source. This concern was alleviated by the emergence of
quantum dot and soliton comb laser sources shown in Figure 4 that can produce
hundreds of usable light frequencies with wall-plug efficiencies of up to 41%
Kim et al. (2019a).

Figure 1: Logical schematic of a DWDM link using ring resonator technology and
a comb-laser source. Each ring is tuned to a different frequency of light and
can be used to modulate that specific wavelength of light (a channel). Comb
laser sources provide a comb of frequencies of light to provide those
wavelengths for encoding. All of the encoded optical channels share the same
optical fiber and are decoded using the rings on the receiving side to route
channels to the photodetectors.

((a)) Active Optical MCM ((b)) Blade ((c)) Rack/Pod

Figure 2: Overall physical structure of Rack/Pod scale resource disaggregation
from photonically connected MCMs to the Rack/Pod scale pooling of
disaggregated resources. The conversion from CXL-over-fiber to HBM or NVM
electrical protocol is implemented in the active interposer for the photonics
MCM.

Figure 3: Copackaged optics are required for DWDM link technologies to achieve
the kind of bandwidth density required to operate at native memory bandwidths.

###  3.3 Active Photonic MCMs

Many CPUs and GPUs do not have the necessary off-chip bandwidth for full
utilization of their compute resources because operating their I/O pins at
higher bandwidth incurs a power cost Chen et al. (2017); Jouppi et al. (2017).
Using emerging high-speed optical links directly to the MCM, illustrated in
Figure 3 provides to the order of 10$\times$ gains in escape bandwidth Glick
et al. (2020); Wade (2019); Bergman et al. (2018); Maniotis et al. (2021).
This is a necessary property to enable efficient resource disaggregation as
well as handle changing bandwidth requirements of key applications such as
machine learning that drastically shifts bandwidth between inter-GPUs and off-
chip from inference to training.

MCMs with integrated photonics have been demonstrated in both 2.5D and 3D
interposer platforms Glick et al. (2020); Minkenberg et al. (2021); Sutono et
al. (1998); Abrams et al. (2020). They can use different die-to-die link
standards such as UCIe. Active interposer platforms combine the photonic
integrated circuit (PIC) and interposer into a single integrated substrate.
The active interposer allows photonic components to be fabricated and directly
integrated with through silicon vias and additional metal redistribution
layers. Electronic circuits are flip-chipped on top of active interposers
using copper pillars Dittrich et al. (2017). Further work has embedded
photonic switch fabrics within MCM platforms with a crosstalk suppression and
extinction ratio of $>$50dB and on-chip loss as low $<$1.8dB Glick et al.
(2020). This was further scaled up to support more than 100 ports with
microring resonators using a scalable switch fabric that combined switching in
the space domain with wavelength-selectivity to define fine-grained
connectivity for node disaggregation Huang et al. (2020); Glick et al. (2020).

Figure 4: Comb laser sources provide the many discrete optical frequencies for the DWDM link with up to 41% experimentally measured conversion efficiency. BW (Gbps) |  Energy (pJ/bit) |  Link Gbps $\times$ Channels |  #Links (2 TB/s escape) |  Agg. Ws (2 TB/s escape) |  Ref.
---|---|---|---|---|---
100 |  30 |  25 $\times$ 4 |  160 |  480 |  Fathololoumi et al. (2021); Agrell et al. (2016)
400 |  30 |  100 $\times$ 4 |  40 |  197 |  Wei et al. (2015)
768 |  $<$ 1 |  32 $\times$ 24 |  21 |  14.4 |  Wade (2019)
1,024 |  0.45 |  16 $\times$ 64 |  16 |  7.2 |  Kim et al. (2019b)
2,048 |  0.3 |  16 $\times$ 128 |  8 |  4.8 |  Kim et al. (2019b)
Table 1: A range of WDM photonic link technologies.

####  3.3.1 Link Protocol

We adopt CXL as our link protocol Van Doren (2019). CXL is an overlay on the
PCIe-Gen6 physical layer, it includes guaranteed ordering of events and is a
broadly adopted industry standard with published specifications. However, our
study does not rely on any features of any particular protocol so alternatives
such as UCIe also apply.

####  3.3.2 Link Propagation and Encoding/Decoding Latency

The target reach for an intra-rack disaggregation solution is approximately
1-4 meters. Given the speed of light $c$ and light propagating through optical
material that has an index of refraction that is near r1.5, the effective
latency of propagating through an optical fiber at nominally 0.75$c$ is
approximately 5 ns per meter. Therefore, rack-scale disaggregation adds 5-15
ns to our latency budget, less than 20% of the typical DRAM latency. The link
latency for SERDES and photonic ring modulation is negligible. Intra-rack
fiber lengths up to 4 meters require no intervening Electrical Optical (OEO)
conversions.

####  3.3.3 Bit Error Rates and FEC

To achieve $10^{-18}$ BER required for memory technologies, forward error
correction (FEC) Luyi et al. (2012) will likely be required. Using the
lightweight FEC scheme that is proposed for CXL Van Doren (2019) and PCIe Gen6
Sharma (2020) as an example, the all-inclusive latency for FEC can be as low
as 2 ns. Therefore, for 200 Gbps, serialization delay is 10 ns and the FEC
calculations add 2-3 ns. At 400 Gbps and above, the net latency for FEC would
be 5 ns plus 2-3 ns. Of note, this approach to achieving these BER targets is
achievable with less than a 0.1% bandwidth loss.

In terms of impact on BER, this PCIe/CXL-like correction scheme corrects all
single bursts of up to 16 bits. Double bursts will likely be mis-corrected,
but the chance of a bad flit decreases quadratically (e.g., a flit BER of
$10^{-6}$ becomes $10^{-12}$ as you need two error bursts per flit to fail).
Each flit is protected with a strong 64-flit CRC such that the flit FIT rate
(CRC escapes) is significantly less than one part per billion. Lastly, the FEC
escapes become link retransmissions and the ASIC-to-ASIC connection sees close
to zero errors. As a result, emerging memory fabric protocols such as CXL,
which could be run over our evaluated physical links, _are capable_ of
achieving a BER rate that meets the stringent memory system requirements and
minimizes performance loss due to retransmission.

###  3.4 Optical Circuit Switch Technologies

Motivated by minimizing latency, our vision for a disaggregated rack is to
have photonically-enabled MCMs that are connected via an optical circuit
switch, as shown in Figure 2. Compute and memory chips would be in the center
of the MCM and the edge of the MCM would contain co-packaged optical silicon
in-package photonics. Switches with all-optical paths include spatial- and
wave-selective approaches, shown in Table 2.

Switch Type |  Radix |  Wave-lengths per port |  B/W per channel (wavelength) |  Insertion Loss |  Crosstalk
---|---|---|---|---|---
Mach-Zehnder based Ikeda et al. (2020) |  32$\times$32 |  1 |  439 Gbps |  12.8 dB |  -26.6 dB
MEMS-actuated Seok et al. (2019b) |  240$\times$240 |  1 |  – |  9.8 dB |  -70 dB
Microring resonator Khope et al. (2017); Cheng et al. (2019b) |  8$\times$8 (128$\times$128) |  8 (128) |  100 Gbps (42 Gbps) |  5dB (10dB) |  (-35 dB)
Casc. AWGRs Sato (2018) |  370$\times$370 |  370 |  25 Gbps |  15 dB |  -35 dB
Table 2: High-radix CMOS-compatible photonic switches.

####  3.4.1 Spatial Optical Switches

In recent years, the primary switching cells investigated are
microelectromechanical systems (MEMS) actuated couplers, Mach-Zehnder
interferometers, and microring resonators. Taking after their free-space
counterpart, photonic MEMS-actuated switches are broadband spatial switches
that have demonstrated radix scaling up to 240$\times$240 Seok et al. (2019b).
However, MEMS switching cells generally require high driving voltages (greater
than 20 V), which make them less attractive for co-integration with electronic
drivers, but typically offer low inter-channel cross-talk and low optical
losses. Spatial switches can also use mirrors Cal , photonic integrated
circuits Ding et al. (2016), or tiled planar silicon photonics Seok et al.
(2019a). MZI switches are more co-integration friendly compared to MEMS but
have only been shown to scale up to 32$\times$32 Ikeda et al. (2020). This
limit can be seen as a consequence of the higher insertion-loss scaling
resulting from cascaded MZI cells, as well as the susceptibility of popular
MZI topologies to first-order crosstalk.

The challenge for scaling-up the spatial approach is the quantization of
package and MCM escape bandwidth and reduced configuration options. For
example, at 768 Gbps (the Ayar TeraPhy Wade (2019)), the number of fibers
escaping the package is 21 fibers, which means the package can be connected
only up to 21 different potential destinations using a spatial switch.

####  3.4.2 Wavelength Selective Optical Switches and AWGRs

The inherent wavelength-selectivity of MRR switching cells allows for the
straightforward implementation of wavelength-selective switching (WSS)
topologies. This enables one to establish all-to-all networks by leveraging
wavelength-division multiplexing (WDM). Currently, MRR-based switches with the
largest radix include the 8$\times$8 crossbar Khope et al. (2017) and switch-
and-select Nikolova et al. (2017), but have been experimentally emulated to
include a 16$\times$16 Clos Dai et al. (2020). The metrics in Dai et al.
(2020) can be seen to correlate very closely with the scaling proposed in
Cheng et al. (2019b), making a practical case for the 128$\times$128 shown in
Table 2.

All-to-all networks via WDM signals can also be achieved by AWGRs Liu et al.
(2020); Zhang et al. (2019); Proietti et al. (2013); Lea (2015); Terzenidis et
al. (2018). As AWGRs are passive optical elements, no reconfiguration is
possible within the routing fabric itself. Instead, fast wavelength-tunable
lasers must be leveraged at the transmitter of every node if it wishes to
address a different destination since AWGRs shuffle the light frequencies such
that one lambda goes to each endpoint from each source. AWGRs enable us to
implement an N$\times$N all-to-all topology using just $\mathcal{O}(N)$ fibers
(each carrying N frequencies of light) whereas an implementation using copper
would require N2 wires. Although the cost of fast wavelength-tunable lasers is
still an ongoing research topic Dhoore, Sören and Roelkens, Günther and
Morthier, Geert (2019), AWGRs are mature, commercially available, and well
established in literature FSp .

In AWGRs, only a limited number of ports can be practically supported due to
the walk-off of passband center frequencies from the carrier wavelength grid
and the worse crosstalk associated with a larger number of ports ($N$). A
feasible implementation of AWGR-based optical switches with large $N$ has been
demonstrated utilizing cascaded small-size AWGRs Sato (2018). Specifically,
$N$ $M\times M$ AWGRs (front-AWGRs) are interconnected with $M$ $N\times N$
AWGRs (rear-AWGRs) to effectively act as an $MN\times MN$ AWGR. Each output
port of a front-AWGR is connected to an input port of a rear-AWGR, where the
interconnection pattern can be optimized with knowledge of port-specific
insertion losses to minimize the worst-case insertion loss of the aggregated
AWGR. Further up-scaling of the switch radix can be achieved by
interconnecting small $K\times K$ delivery-coupling switchs with multiple
copies of the $MN\times MN$ AWGRs, yielding a $KMN\times KMN$ switching
capability. This architecture has been verified by hardware prototypes of
$270\times 270$ and $1440\times 1440$ Sato et al. (2013); Ueda et al. (2016),
showing $\sim$15 dB insertion loss and below $-35$ dB crosstalk suppression.
In order to accommodate the 350 MCMs of our rack, a reasonable configuration
is $KMN=3\times 12\times 11=396$. This results in 370 ports and 370
wavelengths per port (Table 2). Since AWGRs typically have a 25 GHz optical
bandwidth if the wavelength grid is 50 GHz, with PAM4 we assume 25 Gbps per
wavelength Dai et al. (2020); Bhoja (2017).

Wave-selective switches Huang et al. (2020); Marom et al. (2017) can steer any
subset of wavelengths to a given destination, not just all (spatial) or one
(AWGR). Dynamic programming methods can avoid sending the same frequency of
light from two different sources to the same destination. Since this is a
relatively new technology, we constructed a model shown in Table 2 that
projects the performance of a larger radix switch that is comprised of smaller
demonstrated building blocks.

####  3.4.3 Reconfiguration Time

Spatial and wave-selective switches typically require centralized scheduling
Teh et al. (2020) to reach a steady globally optimal solution. The
reconfiguration time can range from tens of nanoseconds to tens of
milliseconds. In production HPC systems, multi-node jobs start every few
seconds and last from minutes to hours Michelogiannakis et al. (2019, 2022).
Also, job resource usage and communication becomes predictable early, does not
change fast, and typically remains predictable throughout a job’s execution
time Michelogiannakis et al. (2022, 2019); Shalf et al. (2005); Vetter and
Mueller (2002). Therefore, even milliseconds of reconfiguration time is ample.

##  4 Control Logic

Here we describe how we can perform indirect routing to increase point-to-
point bandwidth using only per-source logic.

Figure 5: With an AWGR, endpoint 1 has one wavelength directly connecting it
to endpoint 3. If it desires more bandwidth, it can route through another
intermediate endpoint (indirect routing) chosen in a Valiant fashion Liu et
al. (2020); Teh et al. (2020). Here, the link from 1 to 7 is available (green)
but the link from 7 to 3 is not (red). The chosen path is from 1 to 6 to 3
because both links are available.

###  4.1 Indirect routing in AWGRs

AWGRs dedicate exactly one wavelength between any source–destination pair. If
a source–destination pair requests more bandwidth than what a single
wavelength can satisfy, sources can use indirect routing an example of which
is shown in Figure 5. Sources can split traffic to $N$ intermediate
destinations in parallel in order to use the bandwidth of $N$ wavelengths.
This does not consume additional power in the photonic components assuming
lasers are constantly powered. Sources consider indirect paths only if the
direct (single-hop) bandwidth to their desired destination does not suffice. A
source considers indirect destinations for which the direct bandwidth from the
source is available and whose wavelengths from the intermediate hop to the
desired final destination is available. Among potentially multiple candidates,
sources choose one in a Valiant fashion Liu et al. (2020); Teh et al. (2020);
Domke et al. (2019). This is done on a per-flow basis in order to avoid out of
order packet delivery. This routing logic can be modelled as an allocator
problem and implemented with a low latency and area penalty Ma et al. (2014);
Becker and Dally (2009).

Indirect routing relies on sources knowing which other sources attached to the
same AWGR are utilizing their local wavelengths in order to identify a
productive intermediate destination. For instance, in Figure 5 endpoint 1
should know whether the wavelengths from 7 to 3 and 6 to 7 are occupied. For
that, we rely on piggybacking where traffic between a source and a destination
periodically includes the state of the sources’s wavelengths as a way to
broadcast local state to the rest of the endpoints attached to the same AWGR
Jiang et al. (2009). In the case of a N$\times$N AWGR, each source uses N bits
to encode which of its N local wavelengths it is using with one-hot encoding.
Even if we piggyback this information multiple times a second, the bandwidth
impact is negligible. For instance, if we multiplex multiple flows into a
wavelength and therefore denote 8 bits per wavelength, the status vector per
source becomes $256\times 8=2048bits=256bytes$. If, due to stale information,
sources pick an intermediate destination whose wavelength direct to the final
destination is not available, the intermediate destination performs indirect
routing through a second intermediate destination, and so on. If no data is
exchanged between a pair, thus presenting no opportunity for piggybacking,
that pair can exchange a separate control message with the same information.

###  4.2 Spatial and Wave-Selective Switches

Spatial and wave-selective switches can use indirect routing in tandem with
reconfiguration. Indirect routing reduces the need for reconfiguration, but
intermediate hops should be chosen among hops that already have a direct
connection with the final destination; otherwise, the intermediate hop itself
may trigger a reconfiguration. The synergy between indirect routing and switch
reconfiguration was explored in Teh et al. (2020).

##  5 Disaggregated Rack Design

For the rest of our study, we will model an HPC rack based on a GPU-
accelerated HPE/Cray EX Supercomputer Per  where a rack contains 128 GPU-
accelerated nodes. Each node of our model system contains an AMD Milan CPU
that has eight memory controllers each supporting a 3200MHz DDR4 module.
Therefore, each CPU has 256 GB of memory with a maximum bandwidth of 204.8
GBps. A compute node also has four NVIDIA Ampere A100 GPUs. Each GPU supports
12 third generation NVLink links each supporting 25 GBps per direction. Each
GPU also has 40 GB of co-located HBM with a bandwidth of 1555.2 GBps. Each
node also has four 31.5 GBps PCI Gen4 links to connect each GPU to the CPU.
The CPU also connects to four Slingshot 11 NICs with 200 Gbps per direction De
Sensi et al. (2020a). Note that our photonic disaggregation hardware is
orthogonal to and thus does not impair past work related to disaggregation
such as runtimes, OS support, endpoint sharing management, and security.

###  5.1 MCMs and Escape Bandwidth

We organize chips within each rack into an MCMs package. For simplicity, we
restrict all MCMs to have the same escape bandwidth and we place chips of only
the same type in MCMs. We then make conservative assumptions for next
generation photonics that are entering the market today based on our analysis
of Section 3. In particular, each MCM has 32 optical fibers attached to it, a
conservative assumption compared to the five arrays of 24 fibers demonstrated
Hosseini et al. (2021). Each fiber supports 64 wavelengths (channels) of 25
Gbps each for a 6400 GBps escape bandwidth per MCM. We vary the number of
chips per MCM such that each chip enjoys the same escape bandwidth as in our
baseline rack Per . Therefore, our photonic architecture _does not restrict
chip escape bandwidth_. Table 3 shows the number of chips per MCM and the
total number of MCMs containing chips of that type to satisfy chip escape
bandwidth. Each MCM contains a controller chip that interfaces the native
protocol of the disaggregated resource to the CXL protocol over the photonic
links. CXL’s overhead and its associated FEC is included in our model of the
overall architecture.

Chip type | Chips per MCM | # MCMs per rack
---|---|---
CPU | 14 | 10
GPU | 3 | 171
NIC | 203 | 3
HBM | 4 | 128
DDR4 | 27 | 38
Total |  | 350
Table 3: The number of chips ((CPU, GPU, NIC, HBM, or DDR4 module) per MCM and
MCMs in a rack assuming 32 fibers per MCM, 64 wavelengths of 25 Gbps per
fiber. The target BER to and from memory is $10^{-18}$ (Section 3.1).

###  5.2 Optical Switches

The radix and wavelengths per port of optical switches dictate number of MCMs
we can fully connect optically with a single switch as well as the amount of
direct (single-hop) bandwidth. From Section 3.4, we pick state-of-the-art
representatives of wave-selective, cascaded AWGRs, and spatial optical
switches. Their parameters are shown in Table 4. Even though spatial Seok et
al. (2019b) and wave-selective switches Huang et al. (2020) are capable of 100
Gbps per wavelength, most links available widely today do not support that
(Table 1). In addition, we show that we can still satisfy bandwidth demands
with the conservative assumption of 25 Gbps per wavelength.

| Switch type | State of the art
---|---|---
Switch radix | Cascaded AWGRs Sato (2018) | 370
Spatial Seok et al. (2019b) | 240
Wave-Selective Huang et al. (2020) | 256
Gbps per wavelength | All switches | 25
Wavelengths per port | Cascaded AWGRs Sato (2018) | 370
Spatial Seok et al. (2019b) | 240
Wave-Selective Huang et al. (2020) | 256
Table 4: Switch configuration for our study.

To connect our 350 MCMs using 370$\times$370 AWGRs, we can combine MCM fibers
in five groups of six and connect each group to one port of five parallel
AWGRs. However, this would require each AWGR port to handle 384 wavelengths.
To respect the per port 370 wavelength limitation of our AWGR configuration
but still satisfy the full escape bandwidth of MCMs, we combine the remaining
14 wavelengths along with the remaining two fibers per MCM ($128+14=142$
wavelengths total) that were left unconnected into an extra parallel AWGR, for
a total of six parallel AWGRs. We then connect MCM fibers to AWGRs in a
staggered manner such that each MCM connects to each other MCM using at least
five 25 Gbps direct-path wavelengths, for a direct MCM–MCM bandwidth of
$25\times 5=125$ Gbps.

For simplicity, because of their relative small difference and because wave-
selective switches can also achieve configurations that spatial switches can,
we treat both wave-selective and spatial switches as 256 ports with 256
wavelengths per port. Each MCM can connect to $\frac{2048}{256}=8$ parallel
switches. However, because the radix of optical switches is lower than the
number of MCMs, we instantiate 11 optical switches and connect MCMs in a
staggered manner such that optical switch with an index $I$ connects to MCMs
that have an index starting from $(32\times I)\mod 350$ until $(I+255)\mod
350$. This way, a small number of optical switch ports are left unconnected in
order to not exceed the 32 fibers per MCM. Similar to AWGRs, these ports can
support future larger racks. If the switches configure appropriately, each MCM
has at least three direct paths to any other MCM. Each path has 256
wavelengths, thus the direct MCM bandwidth is $256\times 3\times 25=2304$
Gbps.

##  6 Evaluation

Having previously evaluated in Section 3.3.3 that photonic switches satisfy
BER requirements, in this Section we analyze the impact of photonic-based
intra-rack resource disaggregation to bandwidth, latency, and power. We then
compare against electronic switches and estimate system-wide savings.

###  6.1 Bandwidth Evaluation

We distinguish two test cases based on Section 5.2: (A) Six parallel AWGRs and
(B) 11 parallel wave-selective switches.

####  6.1.1 Available Bandwidth

Using indirect routing and switch reconfiguration, any one particular MCM can
use its full escape bandwidth to reach a single destination MCM. In test case
(A), all wavelengths escaping an MCM can reach the same destination MCM using
indirect routing. In test case (B), $768$ wavelengths can be configured to
route directly to a destination MCM and the other $2048-768=1280$ wavelengths
can be configured to route indirectly through intermediate MCMs. This assumes
that other MCMs will not contend for bandwidth that may disrupt indirect
routing or complicate switch reconfiguration. While the direct (single-hop)
bandwidth between cases (A) and (B) has a large difference, case (A) always
provides that direct bandwidth between MCMs whereas a spatial or wave-
selective switch requires a scheduler and leaves the majority of input–output
combinations unconnected at any one time, thus also has to use indirect
routing to compensate.

Based on system profiling data of a production open-science HPC system
Michelogiannakis et al. (2022), the 125 Gbps direct bandwidth between MCMs in
test case (A) suffices over 99.5% of the time between CPUs and main memory
(DDR4) and virtually all the time between memory and NICs. In addition, the
bandwidth of a single AWGR wavelength of 25 Gbps suffices 97% of the time
between CPUs and memory as well as between memory and NICs. This means that
with a 97% probability, four of the five wavelengths between a memory and CPUs
or NICs and memory pair are available to use for indirect routing in case the
direct 125 Gbps bandwidth does not suffice between another memory–CPU or
NIC–memory pair. Therefore, the probability at any one time that the direct
bandwidth does not suffice for a number of CPU–memory and NIC–memory pairs
large enough such that they cannot find unused bandwidth in other pairs to use
for indirect routing is multiple orders of magnitude less than 0.1% and thus
negligible. To further reduce the probability, congested pairs can use direct
paths from CPUs to CPUs that communicate minimally and NICs to other NICs that
do not communicate at all Michelogiannakis et al. (2022). Therefore, test case
(A) satisfies bandwidth between CPUs, NICs, and main memory (DDR4).

Figure 6: Average and maximum slowdown for each suite and input set size. The
slowdown is for an additional 35ns of latency between the LLC and main memory
from the additional photonic components. Left: in-order pipeline compute
cores. Right: Out of order (OOO) compute cores.

For GPUs, in test case (A) with indirect routing a single GPU can use a total
of $125\times 512=8000$ GBps to access any one HBM or more in case a GPU is
allocated more than one HBMs. This well satisfies the 1555.2 GBps that NVIDIA
Ampere A100 GPUs in our model rack Per  access HBMs with today and leaves
$8000-1555.2=6444.8$ GBps unused per GPU. In addition, in the worst case, an
MCM containing three GPUs will communicate at full bandwidth (12 NVLink links
of 25 GBps per each of the three GPU equals 900 GBps) to other MCMs containing
GPUs. Here, if all GPUs in the rack acts similarly, we cannot rely on indirect
routing from a GPU through an intermediate GPU to reach a destination GPU. The
direct 125 Gbps bandwidth between GPU MCMs do not suffice. Therefore, each GPU
can use the 6444.8 GBps of unused bandwidth to and from HBMs for indirect
routing to well cover the 900 GBps bandwidth that would otherwise use NVLink
GPU–GPU links. This leaves $6444.8-900=5544.8$ GBps per GPU that can support
direct HBM–HBM communication such as due to GPUDirect RDMA, indirect routing
for other MCMs, or simply increase available bandwidth to memory. Of note, our
analysis does not use direct optical paths from GPUs to main memory (DDR4).
Future protocols may use for these paths or they can be used to provide even
more indirect routing bandwidth.

Our analysis shows that test case (A) with AWGRs more than satisfies bandwidth
demands and avoids the need for a scheduler to reconfigure spatial and wave-
selective switches that would otherwise add overhead and reduce reaction time.

###  6.2 Latency Evaluation

Intra-rack resource disaggregation based on modern photonics increases the
latency significantly less than full system disaggregation. For intra-rack
disaggregation we assume an additional latency between MCMs of 35 ns. That
additional latency covers 15 ns for electrical–optical–electrical conversion
and 4 meters of photonic propagation at 5 ns per meter, which covers round-
trip distance of typical two-meter tall racks (Section 3.3.2). The small
impact of distance to latency with photonics practically makes MCMs in a rack
equi-distant, thus mitigating a traditional queuing delay versus locality
tradeoff in job scheduling Jeon et al. (2019). Indirect routing would increase
latency by a few extra ns, but the probability of routing indirectly is low.
Because 35 ns is orders of magnitude lower than system-wide network latency,
we do not consider the effect of the additional 35 ns to inter-rack
communication through NICs.

####  6.2.1 CPU Evaluation

We experimentally quantify the impact to application performance with in-order
pipeline and OOO compute cores. In-order cores provide insight of the impact
of memory latency when the compute core does not mask latency, whereas OOO
cores are representative of modern cores. We use full system simulation in
Gem5 Binkert et al. (2011) of x86 compute cores running an Ubuntu 18.4 guest
OS. We configure the cache hierarchy to match the CPUs of our model HPC rack
Per . We calculate the slowdown of application execution time when we add 35
ns of latency between the LLC and main memory, compared to a baseline system
with no additional latency to memory. Latency is the only potential source of
application slowdown since our architecture satisfies the full escape
bandwidth for each chip.

We evaluate the impact in three benchmark suites: PARSEC 3.1 Bienia et al.
(2008), NAS parallel benchmarks 3.4.1 Bailey et al. (1992), and Rodinia Che et
al. (2009). For PARSEC we evaluate small, medium, and large input sets. For
NAS, we evaluate input sizes “A”, “B”, and “C”. For Rodinia we use the single
default input set. These benchmark suites have been widely used and contain a
large variety of computation kernels that are representative of key HPC
applications such as stencils, graph processing, linear algebra, computational
mathematics, grid, sorting, and many others that have been observed to be
important workloads in NERSC’s systems N10_workload_analysis. Overall, we use
58 benchmarks to provide a wide representation. We use a single compute core
to better focus on the effect of the additional latency to memory.

Figure 6 shows slowdown percentages for benchmarks across our three suites for
an in-order core on the left and OOO core on the right. As shown, NAS
benchmarks are negligibly affected by the increased latency. Rodinia
benchmarks have an average slowdown of 15% with in-order cores and 13% for OOO
cores. However, a single benchmark (NW) has a slowdown of approximately 76%
for in-order cores. The largest slowdown for the rest of Rodinia benchmarks
across in-order and OOO cores is 12%. Finally, PARSEC benchmarks are impacted
the most, but the average slowdown remains below 25% except for large inputs
using OOO cores. OOO cores typically tolerate memory access latency better,
but they also produce more memory accesses per unit of time compared to in-
order cores.

Figure 7: Results for individual PARSEC benchmarks with large inputs.

Figure 7 shows slowdown for individual PARSEC benchmarks for large inputs and
in-order cores. As shown, only three benchmarks exceed a 25% slowdown, while
eight benchmarks have a slowdown of no more approximately 3.5%. Therefore, our
experiments show that while some benchmarks (three in PARSEC and one in
Rodinia) experience important slowdowns, the majority of benchmarks are
impacted minimally even without mitigation strategies. This is the case with
all of the NAS benchmarks we used, eight PARSEC, and all but one Rodinia
benchmarks. For benchmarks that are more affected, there is a range of
hardware and software techniques Mutlu et al. (2006); Parcerisa and Gonzalez
(2001); Mowry et al. (1998); Nekkalapu et al. (2008) to increase memory
tolerance that we can apply to further reduce application slowdown.

Figure 8: Percentage of slowdown that we can recover with LLC modifications.

####  6.2.2 Recovering Performance

To gauge the effectiveness of strategies to recover application performance,
we test the impact of the following remedies applied one at a time: (i) 256
miss status handling registers in the LLC, (ii) doubling the LLC size with the
default number of 16 MSHRs, and (iii) default LLC configuration but a strided
prefetcher with a larger stride than the default four. Figure 8 shows the
slowdown percentage that we were able to recover for PARSEC benchmarks through
these three techniques at the best, average, and worst case. This figure is
the only one that includes these remedies in this results of this section. As
shown, about a 20% performance loss for small and large inputs is recovered by
average. The most effective remedy is doubling the LLC size. The reason for
the smaller speedup for medium is that due the particular LLC size, memory
access patterns, and input sizes in PARSEC, medium experienced a smaller
benefit from a larger LLC. These findings motivate future work to mitigate the
latency impact of the disaggregation hardware, similar to mitigating the
increased latency to access emerging memory technologies Mittal and Vetter
(2016).

Figure 9: Percentage slowdown for 25ns, 30ns, and 35ns of additional
LLC–memory latency for in order cores.

####  6.2.3 Sensitivity to Latency

To show the sensitivity of application performance to the amount of additional
latency, Figure 9 shows application slowdown for 25 ns, 30 ns, and 35 ns for
in-order cores (OOO cores show comparable trends). As shown, reducing the
additional latency to 25 ns from 35 ns reduces application slowdown by as much
as half. This motivates latency improvements in photonic components or shorter
rack distances.

The overall average slowdown across all benchmarks is approximately _13% for
both in-order cores and OOO cores_ without architectural remedies, for large
PARSEC inputs, and “B” size NAS inputs. This considerably less than slowdowns
quoted in past work for full-system disaggregation, furthering the case for
intra-rack disaggregation.

####  6.2.4 GPU Evaluation

To evaluate the impact of the additional latency between GPUs and HBMs or DDR4
main memory, we extend the publicly available version of PPT-GPU Arafa et al.
(2021) toolkit to account for the additional latency between the main memory
of the GPU and the LLC. In our evaluation, we modeled one NVIDIA A100 GPU
Choquette and Gandhi (2020) running a total of 27 applications that have a
total of 2133 kernels from different benchmark suites. We run 13 applications
from Rodinia Che et al. (2009) and 10 applications from Polybench Grauer-Gray
et al. (2012). Polybench applications are linear algebra applications that
stress the GPU cache and main memory. Furthermore, we run AlexNet, CifarNet,
GRU, and LSTM from the Tango deep network Karki et al. (2019) benchmark suite.
We use the default input sizes and configuration that came with the
benchmarks, detailed in Arafa et al. (2021). We run applications using the
“SASS” model, where we extract memory and instruction traces for each
application.

Figure 10 shows the effect of different latencies on the performance of our
GPU benchmarks. We compare performance in terms of the total predicted cycles.
As shown, the highest average slowdown is 24% for Polybench. The overall
average slowdowns across the 27 applications is only 8%, 10%, and 12% for the
25 ns, 30 ns, and 35 ns additional latency, respectively. For these
benchmarks, doubling the LLC size recovers an average of 8% of the performance
loss.

Figure 10: Percentage slowdown for 25ns, 30ns, and 35ns of additional
LLC–memory latency for different GPU benchmark suites.

####  6.2.5 CPU–GPU Comparison

Figure 11: Percentage slowdown for CPU and GPU Rodinia benchmarks.

We illustrate the difference in memory latency tolerance of in-order CPUs, OOO
CPUs, and GPUs in Figure 11 for the intersection of Rodinia benchmarks that
correctly ran on both CPU and GPU with their default input sets. As shown,
GPUs tolerate the additional 35 ns latency significantly better with a maximum
slowdown of 3.3%. This is promising for resource disaggregation given the
steady growth of GPU presence in HPC systems.

###  6.3 Power Overhead

We calculate the per-rack power overhead of our photonic solution for 350 MCMs
with 2048 escape wavelengths from each MCM and 25 Gbps per wavelength. If we
use a DFB laser array demonstrated in Rahimi et al. (2022) with a 11% wall
plug efficiency (WPE) at 10 dDm, a total of $256\times 256$ such lasers
consumes 64.5 kW. For the components and distances in our study, the required
optical power per wavelength is 10 dBm. Furthermore, $350\times 2048$ of the
modulators and receivers of Sun et al. (2020) that consume 0.8 and 2.12 pJ/bit
at 25 Gbps respectively result in a total additional power of 52.5 kW.
Finally, the switches of Table 2 consume no more than 1 kW at the worst case.
In summary, the total power overhead taking into account parallel switches is
no more than 150 kW. Our analysis assumes the components are constantly on.
Considering that the maximum power consumption of a single A100 GPU is a few
hundreds of Ws and our modelled rack contains 512 such GPUs, the power
overhead for our photonic solution is _negligible_.

###  6.4 Comparison With Electronic Switches

Electronic SERDES signalling rate per wire is only 112 Gbps for a short reach.
Also, typical CXL or PCIe signaling rates top out at 35 GHz/wire. In fact, as
SERDES rates increase, the distance that those signals can reach reduces down
to even a few millimeters due to the resistance and capacitance of copper
wires. Photonics break the reach limitations of copper and with co-packaging
can achieve 4 Tbps per mm of shoreline on the chip die.

Focusing on electronic switches, Rosetta De Sensi et al. (2020b) and
Infiniband Katebzadeh et al. (2020) have a measured per hop latency of no less
than approximately 200 ns. Emerging PCIe Gen5 switches add just 10 ns per hop
Vasa et al. (2020), but only support 100 lanes per switch. To fully connect
our disaggregated rack, we consider a two-level tree network with four hops
(the top level is composed of an internal two-hop subnetwork). These four hops
will be in addition to the 35 ns we previously evaluated for FEC and
propagation (propagation delay is comparable between copper and photonic for
rack distances), since our photonic solution uses switches with negligible
traversal latency. Therefore, the additional latency for disaggregation in the
PCIe case becomes 85 ns compared to 35 ns for our photonic architecture.
Finally, we also consider the latency through one hop of an Anton 3 network,
which is approximately 90 ns by average Shim et al. (2022), though scaling up
to match our rack size would require multiple hops. These latencies represent
the best case for electronic packet switches because scheduler decisions or
congestion can cause higher worst-case (tail) latencies that may further
penalize application performance. This assumes that we connect only one lane
per endpoint which carries 32 Gbps for PCIe Gen5 and 29 Gbps for Anton 3. This
is multiple times less than the per-chip bandwidth of photonics our photonic
architecture.

Figure 12 shows the speedup of a system that implements intra-rack
disaggregation with emerging photonics with an additional 35 ns latency to and
from DDR4 and HBM memory compared to a similar system that uses modern
electronic switches instead. 85 ns is the lowest case for electronic switches
and corresponds to a four-hop PCIe Gen5 network or a single-hop Anton 3
network. As shown, for CPU benchmarks if we only take into account “medium”
from PARSEC to avoid counting PARSEC benchmarks three times, the average
speedup for in-order CPUs is 12.7% and the maximum 76%. For OOO compute cores,
the average is 23.9% and maximum 78.3%. For GPUs, the average and maximum are
both 61.5%. This analysis clearly shows the adverse impact of the additional
latency of electronic switches and further motivates the use of photonics for
intra-rack resource disaggregation. Furthermore, the four electronic switches
of this analysis consume at least many tens of Watts of power, which is
multiple orders of magnitude higher than our photonic solution.

Figure 12: Speedup of a system that uses emerging photonics to implement
intra-rack resource disaggregation that adds 35 ns of additional latency to
and from memory compared to a similar system that uses modern electronic
switches and adds 85 ns of memory latency instead.

###  6.5 Iso-Performance Comparison

Based on our performance evaluations, in order to preserve system-wide average
computational throughput as our baseline GPU-accelerated HPE/Cray EX system
Per , our photonically-disaggregated system requires 13% more CPUs and 8% more
GPUs. However, intra-rack resource disaggregation allows our rack to have an
average 4$\times$ fewer memory modules and 2$\times$ fewer NICs
Michelogiannakis et al. (2022). Combining the two effects, our disaggregated
rack has 1082 total modules compared to 1920 in the baseline system, a 43%
reduction. Alternatively, we can preserve all rack resources and instead add
128 of a combination of CPUs and GPUs (with their HBMs), which is only a 7%
chip increase across the rack. Doing so _doubles_ computational throughput.

##  7 Conclusion

We have designed a resource disaggregated HPC rack that uses modern photonic
links and switches to meet BER and bandwidth requirements of HPC applications,
has a negligible power impact, uses distributed indirect routing instead of
complex switch reconfiguration, and provides a 23.9% for CPUs or 61.5% for
GPUs speedup compared to a similar disaggregated rack implemented with modern
electronic switches. Our architecture enables a disaggregated system to
preserve its performance but use _43% fewer overall chips_.

## References

  * Liu et al. [2012] B. Liu, D. Zydek, H. Selvaraj, and L. Gewali.  Accelerating high performance computing applications: Using cpus, gpus, hybrid CPU/GPU, and fpgas.  In _2012 13th International Conference on Parallel and Distributed Computing, Applications and Technologies_ , pages 337–342, 2012.
  * Top [2018] The top500 HPC list, November 2018.  URL https://www.top500.org/green500/lists/2018/11/.
  * Ujaldón [2016] M. Ujaldón.  Hpc accelerators with 3d memory.  In _2016 IEEE Intl Conference on Computational Science and Engineering (CSE) and IEEE Intl Conference on Embedded and Ubiquitous Computing (EUC) and 15th Intl Symposium on Distributed Computing and Applications for Business Engineering (DCABES)_ , pages 320–328, 2016.  doi:10.1109/CSE-EUC-DCABES.2016.203.
  * Mittal and Vetter [2015] Sparsh Mittal and Jeffrey S. Vetter.  A survey of cpu-gpu heterogeneous computing techniques.  _ACM Comput. Surv._ , 47(4), July 2015.  ISSN 0360-0300.  doi:10.1145/2788396.
  * Tiwari et al. [2015] D. Tiwari, S. Gupta, J. Rogers, D. Maxwell, P. Rech, S. Vazhkudai, D. Oliveira, D. Londo, N. DeBardeleben, P. Navaux, L. Carro, and A. Bland.  Understanding gpu errors on large-scale hpc systems and the implications for system design and operation.  In _2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA)_ , pages 331–342, 2015.  doi:10.1109/HPCA.2015.7056044.
  * Gao and Zhang [2016] Y. Gao and P. Zhang.  A survey of homogeneous and heterogeneous system architectures in high performance computing.  In _2016 IEEE International Conference on Smart Cloud (SmartCloud)_ , pages 170–175, 2016.  doi:10.1109/SmartCloud.2016.36.
  * Milojicic [2020] D. Milojicic.  Accelerators for artificial intelligence and high-performance computing.  _Computer_ , 53(2):14–22, 2020.
  * Asaadi and Chapman [2017] H. Asaadi and B. Chapman.  Comparative study of deep learning framework in hpc environments.  In _2017 New York Scientific Data Summit (NYSDS)_ , pages 1–7, 2017\.  doi:10.1109/NYSDS.2017.8085040.
  * Segal et al. [2014] O. Segal, N. Nasiri, M. Margala, and W. Vanderbauwhede.  High level programming of FPGAs for HPC and data centric applications.  In _2014 IEEE High Performance Extreme Computing Conference (HPEC)_ , pages 1–3, 2014.  doi:10.1109/HPEC.2014.7040979.
  * Hogervorst et al. [2021] Tom Hogervorst, Tong Dong Qiu, Giacomo Marchiori, Alf Birger, Markus Blatt, and Razvan Nane.  Hardware acceleration of HPC computational flow dynamics using HBM-enabled FPGAs, 2021.
  * Lant et al. [2020] J. Lant, J. Navaridas, M. Luján, and J. Goodacre.  Toward fpga-based HPC: Advancing interconnect technologies.  _IEEE Micro_ , 40(1):25–34, 2020.  doi:10.1109/MM.2019.2950655.
  * Dimond et al. [2011] Rob Dimond, Sebastien Racanière, and Oliver Pell.  Accelerating large-scale hpc applications using fpgas.  In _2011 IEEE 20th Symposium on Computer Arithmetic_ , pages 191–192, 2011.  doi:10.1109/ARITH.2011.34.
  * Ramirez-Gargallo et al. [2019] G. Ramirez-Gargallo, M. Garcia-Gasulla, and F. Mantovani.  Tensorflow on state-of-the-art HPC clusters: A machine learning use case.  In _2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)_ , pages 526–533, 2019.  doi:10.1109/CCGRID.2019.00067.
  * Venkata et al. [2017] M. G. Venkata, F. Aderholdt, and Z. Parchman.  SharP: Towards programming extreme-scale systems with hierarchical heterogeneous memory.  In _2017 46th International Conference on Parallel Processing Workshops (ICPPW)_ , pages 145–154, 2017.  doi:10.1109/ICPPW.2017.32.
  * Michelogiannakis et al. [2022] George Michelogiannakis, Benjamin Klenk, Brandon Cook, Min Yee Teh, Madeleine Glick, Larry Dennison, Keren Bergman, and John Shalf.  A case for intra-rack resource disaggregation in HPC.  _ACM Trans. Archit. Code Optim._ , jan 2022.  ISSN 1544-3566.  doi:10.1145/3514245.  Just Accepted.
  * Rodrigo et al. [2016] G. P. Rodrigo, P. Östberg, E. Elmroth, K. Antypas, R. Gerber, and L. Ramakrishnan.  Towards understanding job heterogeneity in hpc: A nersc case study.  In _2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid)_ , pages 521–526, 2016.
  * Guleria et al. [2019a] A. Guleria, J. Lakshmi, and C. Padala.  Quadd: Quantifying accelerator disaggregated datacenter efficiency.  In _2019 IEEE 12th International Conference on Cloud Computing (CLOUD)_ , pages 349–357, 2019a.  doi:10.1109/CLOUD.2019.00064.
  * Peng et al. [2020] I. Peng, R. Pearce, and M. Gokhale.  On the memory underutilization: Exploring disaggregated memory on HPC systems.  In _2020 IEEE 32nd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)_ , pages 183–190, 2020\.
  * Taylor [2015] J. Taylor.  Facebook’s data center infrastructure: Open compute, disaggregated rack, and beyond.  In _2015 Optical Fiber Communications Conference and Exhibition (OFC)_ , pages 1–1, 2015.
  * Li et al. [2022] Huaicheng Li, Daniel S. Berger, Stanko Novakovic, Lisa Hsu, Dan Ernst, Pantea Zardoshti, Monish Shah, Ishwar Agarwal, Mark D. Hill, Marcus Fontoura, and Ricardo Bianchini.  First-generation memory disaggregation for cloud platforms.  arXiv, 2022.  doi:10.48550/ARXIV.2203.00241.
  * Koh et al. [2019] K. Koh, K. Kim, S. Jeon, and J. Huh.  Disaggregated cloud memory with elastic block management.  _IEEE Transactions on Computers_ , 68(1):39–52, 2019.
  * Papaioannou et al. [2016] A. D. Papaioannou, R. Nejabati, and D. Simeonidou.  The benefits of a disaggregated data centre: A resource allocation approach.  In _2016 IEEE Global Communications Conference (GLOBECOM)_ , pages 1–7, 2016.  doi:10.1109/GLOCOM.2016.7842314.
  * Gonzalez et al. [2020] J. Gonzalez, A. Gazman, M. Hattink, M. G. Palma, M. Bahadori, R. Rubio-Noriega, L. Orosa, M. Glick, O. Mutlu, K. Bergman, and R. Azevedo.  Optically connected memory for disaggregated data centers.  In _2020 IEEE 32nd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)_ , pages 43–50, 2020.  doi:10.1109/SBAC-PAD49847.2020.00017.
  * Cheng et al. [2019a] Qixiang Cheng, Keren Bergman, Yishen Huang, Hao Yang, Meisam Bahadori, Nathan Abrams, Xiang Meng, Madeleine Glick, Yang Liu, and Michael Hochberg.  Silicon photonic switch topologies and routing strategies for disaggregated data centers.  _IEEE Journal of Selected Topics in Quantum Electronics_ , PP:1–1, 12 2019a.  doi:10.1109/JSTQE.2019.2960950.
  * Guleria et al. [2019b] A. Guleria, J. Lakshmi, and C. Padala.  Emf: Disaggregated gpus in datacenters for efficiency, modularity and flexibility.  In _2019 IEEE International Conference on Cloud Computing in Emerging Markets (CCEM)_ , pages 1–8, 2019b.  doi:10.1109/CCEM48484.2019.000-5.
  * [26] NERSC’s Perlmutter configuration.  URL https://docs.nersc.gov/systems/perlmutter/.
  * Petersen and Bent [2017] Torben Kling Petersen and John Bent.  Hybrid flash arrays for hpc storage systems: An alternative to burst buffers.  In _2017 IEEE High Performance Extreme Computing Conference (HPEC)_ , pages 1–7, 2017.  doi:10.1109/HPEC.2017.8091092.
  * Glick et al. [2020] M. Glick, N. C. Abrams, Q. Cheng, M. Y. Teh, Y. H. Hung, O. Jimenez, S. Liu, Y. Okawachi, X. Meng, L. Johansson, M. Ghobadi, L. Dennison, G. Michelogiannakis, J. Shalf, A. Liu, J. Bowers, A. Gaeta, M. Lipson, and K. Bergman.  PINE: Photonic integrated networked energy efficient datacenters (enlitened program) [invited].  _IEEE/OSA Journal of Optical Communications and Networking_ , 12(12):443–456, 2020.  doi:10.1364/JOCN.402788.
  * Guo et al. [2021] Xiaotao Guo, Xuwei Xue, Bitao Pan, Fulong Yan, Georgios Exarchakos, and Nicola Calabretta.  Fosda: A hybrid disaggregated hpc architecture based on distributed nanoseconds optical switches.  In _Future computing_ , 2021.  ISBN 978-1-61208-846-4.
  * Lin et al. [2020] R. Lin, Y. Cheng, M. D. Andrade, L. Wosinska, and J. Chen.  Disaggregated data centers: Challenges and trade-offs.  _IEEE Communications Magazine_ , 58(2), 2020.
  * Call et al. [2020] Aaron Call, Jordà Polo, David Carrera, Francesc Guim, and Sujoy Sen.  Disaggregating non-volatile memory for throughput-oriented genomics workloads.  _CoRR_ , abs/2007.02813, 2020.
  * Liu et al. [2020] Gengchen Liu, Roberto Proietti, Marjan Fariborz, Pouya Fotouhi, Xian Xiao, and S. J. Ben Yoo.  Architecture and performance studies of 3d-hyper-flex-lion for reconfigurable all-to-all hpc networks.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , SC ’20. IEEE Press, 2020\.  ISBN 9781728199986.
  * Zhang et al. [2019] Y. Zhang, X. Xiao, K. Zhang, S. Li, A. Samanta, Y. Zhang, K. Shang, R. Proietti, K. Okamoto, and S. J. B. Yoo.  Foundry-enabled scalable all-to-all optical interconnects using silicon nitride arrayed waveguide router interposers and silicon photonic transceivers.  _IEEE Journal of Selected Topics in Quantum Electronics_ , 25(5):1–9, 2019.  doi:10.1109/JSTQE.2019.2910415.
  * Seok et al. [2019a] Tae Joon Seok, Kyungmok Kwon, Johannes Henriksson, Jianheng Luo, and Ming C. Wu.  Wafer-scale silicon photonic switches beyond die size limit.  _Optica_ , 6(4):490–494, Apr 2019a.  doi:10.1364/OPTICA.6.000490.  URL http://opg.optica.org/optica/abstract.cfm?URI=optica-6-4-490.
  * Ding et al. [2016] Yunhong Ding, Valerija Kamchevska, Kjeld Dalgaard, Feihong Ye, Rameez Asif, Simon Gross, Michael Withford, Michael Galili, Toshio Morioka, and L.K. Oxenlowe.  Reconfigurable sdm switching using novel silicon photonic integrated circuit.  _Scientific Reports_ , 6, 08 2016.  doi:10.1038/srep39058.
  * Huang et al. [2020] Yishen Huang, Qixiang Cheng, Anthony Rizzo, and Keren Bergman.  Push—pull microring-assisted space-and-wavelength selective switch.  _Opt. Lett._ , 45(10):2696–2699, May 2020.  doi:10.1364/OL.392482.
  * Lim et al. [2009] Kevin Lim, Jichuan Chang, Trevor Mudge, Parthasarathy Ranganathan, Steven K. Reinhardt, and Thomas F. Wenisch.  Disaggregated memory for expansion and sharing in blade servers.  _SIGARCH Comput. Archit. News_ , 37(3):267–278, jun 2009.  ISSN 0163-5964.  doi:10.1145/1555815.1555789.
  * Van Doren [2019] S. Van Doren.  Abstract - hoti 2019: Compute express link.  In _2019 IEEE Symposium on High-Performance Interconnects (HOTI)_ , pages 18–18, 2019.  doi:10.1109/HOTI.2019.00017.
  * Zervas et al. [2018] G. Zervas, H. Yuan, A. Saljoghei, Q. Chen, and V. Mishra.  Optically disaggregated data centers with minimal remote memory latency: Technologies, architectures, and resource allocation [invited].  _IEEE/OSA Journal of Optical Communications and Networking_ , 10(2):A270–A285, 2018.  doi:10.1364/JOCN.10.00A270.
  * Gao et al. [2016] Peter X. Gao, Akshay Narayan, Sagar Karandikar, Joao Carreira, Sangjin Han, Rachit Agarwal, Sylvia Ratnasamy, and Scott Shenker.  Network requirements for resource disaggregation.  In _Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation_ , OSDI’16. USENIX Association, 2016.  ISBN 9781931971331.
  * Abali et al. [2015] Bulent Abali, Richard J. Eickemeyer, Hubertus Franke, Chung-Sheng Li, and Marc A. Taubenblatt.  Disaggregated and optically interconnected memory: when will it be cost effective?, 2015.
  * Han et al. [2013] Sangjin Han, Norbert Egi, Aurojit Panda, Sylvia Ratnasamy, Guangyu Shi, and Scott Shenker.  Network support for resource disaggregation in next-generation datacenters.  In _Proceedings of the Twelfth ACM Workshop on Hot Topics in Networks_ , HotNets-XII, 2013.  ISBN 9781450325967.  doi:10.1145/2535771.2535778.
  * Cheng et al. [2018] Y. Cheng, M. De Andrade, L. Wosinska, and J. Chen.  Resource disaggregation versus integrated servers in data centers: Impact of internal transmission capacity limitation.  In _2018 European Conference on Optical Communication (ECOC)_ , pages 1–3, 2018.  doi:10.1109/ECOC.2018.8535214.
  * Fan et al. [2019] Yuping Fan, Zhiling Lan, Paul Rich, William E. Allcock, Michael E. Papka, Brian Austin, and David Paul.  Scheduling beyond CPUs for HPC.  _Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing_ , Jun 2019.
  * Agosta et al. [2018] Giovanni Agosta, William Fornaciari, Giuseppe Massari, Anna Pupykina, Federico Reghenzani, and Michele Zanella.  Managing heterogeneous resources in HPC systems.  PARMA-DITAM ’18, page 7–12, New York, NY, USA, 2018. Association for Computing Machinery.  ISBN 9781450364447.
  * Amaral et al. [2021] Marcelo Amaral, Jorda Polo, David Carrera, Nelson Gonzalez, Chih-Chieh Yang, Alessandro Morari, Bruce D’Amora, Alaa Youssef, and Malgorzata Steinder.  DRMaestro: orchestrating disaggregated resources on virtualized data-centers.  _Journal of Cloud Computing_ , 10, 03 2021.  doi:10.1186/s13677-021-00238-6.
  * Domeniconi et al. [2019] G. Domeniconi, E. Lee, Vanamala Venkataswamy, and Swaroopa Dola.  CuSH: Cognitive scheduler for heterogeneous high performance computing system.  2019\.
  * Maccabe [2017] Arthur B. Maccabe.  Operating and runtime systems challenges for hpc systems.  In _Proceedings of the 7th International Workshop on Runtime and Operating Systems for Supercomputers ROSS 2017_ , ROSS ’17, 2017.  ISBN 9781450350860.  doi:10.1145/3095770.3095771.
  * Hwu et al. [2015] W. Hwu, L. Chang, H. Kim, A. Dakkak, and I. El Hajj.  Transitioning hpc software to exascale heterogeneous computing.  In _2015 Computational Electromagnetics International Workshop (CEM)_ , pages 1–2, 2015.  doi:10.1109/CEM.2015.7237412.
  * Shan et al. [2018] Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang.  Legoos: A disseminated, distributed os for hardware resource disaggregation.  In _Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation_ , OSDI’18, page 69–87. USENIX Association, 2018.  ISBN 9781931971478.
  * Gioiosa et al. [2020] Roberto Gioiosa, Ozcelik Burcu Mutlu, Seyong Lee, S. Jeffrey Vetter, Giulio Picierro, and Marco Cesati.  The minos computing library - efficient parallel programming for extremely heterogeneous systems.  _GPGPU@PPoPP_ , pages 1–10, 2020.
  * Khaleghzadeh et al. [2020] H. Khaleghzadeh, R. R. Manumachu, and A. Lastovetsky.  A hierarchical data-partitioning algorithm for performance optimization of data-parallel applications on heterogeneous multi-accelerator numa nodes.  _IEEE Access_ , 8:7861–7876, 2020.
  * Hussain [2020] Zaeem Hussain.  Heterogeneity aware fault tolerance for extreme scale computing.  August 2020.  URL http://d-scholarship.pitt.edu/39456/.
  * Jamieson et al. [2018] P. Jamieson, A. Sanaullah, and M. Herbordt.  Benchmarking heterogeneous hpc systems including reconfigurable fabrics: Community aspirations for ideal comparisons.  In _2018 IEEE High Performance extreme Computing Conference (HPEC)_ , pages 1–6, 2018.
  * Tang et al. [2017] K. Tang, D. Tiwari, S. Gupta, S. S. Vazhkudai, and X. He.  Effective running of end-to-end hpc workflows on emerging heterogeneous architectures.  In _2017 IEEE International Conference on Cluster Computing (CLUSTER)_ , pages 344–348, 2017.  doi:10.1109/CLUSTER.2017.22.
  * Lastovetsky [2015] Alexey Lastovetsky.  Heterogeneous parallel computing: from clusters of workstations to hierarchical hybrid platforms.  _Supercomputing Frontiers and Innovations_ , 1(3), 2015\.  ISSN 2313-8734.
  * Guo et al. [2019] Jing Guo, Zihao Chang, Sa Wang, Haiyang Ding, Yihui Feng, Liang Mao, and Yungang Bao.  Who limits the resource efficiency of my datacenter: An analysis of alibaba datacenter traces.  In _Proceedings of the International Symposium on Quality of Service_ , IWQoS ’19, 2019.  ISBN 9781450367783.  doi:10.1145/3326285.3329074.
  * Patel et al. [2015] J. Patel, V. Jindal, I. Yen, F. Bastani, J. Xu, and P. Garraghan.  Workload estimation for improving resource management decisions in the cloud.  In _2015 IEEE Twelfth International Symposium on Autonomous Decentralized Systems_ , pages 25–32, 2015.  doi:10.1109/ISADS.2015.17.
  * Li et al. [2015] Teng Li, Vikram K. Narayana, and Tarek A. El-Ghazawi.  Efficient resource sharing through GPU virtualization on accelerated high performance computing systems.  _CoRR_ , abs/1511.07658, 2015.
  * Jeon et al. [2019] Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, unjie Qian, Wencong Xiao, and Fan Yang.  Analysis of large-scale multi-tenant gpu clusters for dnn training workloads.  In _Proceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference_ , USENIX ATC ’19, page 947–960, USA, 2019. USENIX Association.  ISBN 9781939133038.
  * Li et al. [2011] Teng Li, Vikram K. Narayana, Esam El-Araby, and Tarek El-Ghazawi.  Gpu resource sharing and virtualization on high performance computing systems.  In _2011 International Conference on Parallel Processing_ , pages 733–742, 2011.  doi:10.1109/ICPP.2011.88.
  * Terzenidis et al. [2018] N. Terzenidis, M. Moralis-Pegios, G. Mourgias-Alexandris, T. Alexoudi, K. Vyrsokinos, and N. Pleros.  High-port and low-latency optical switches for disaggregated data centers: The hipolaos switch architecture.  _IEEE/OSA Journal of Optical Communications and Networking_ , 10(7):102–116, 2018.  doi:10.1364/JOCN.10.00B102.
  * Wang et al. [2020] Zeke Wang, Hongjing Huang, Jie Zhang, and Gustavo Alonso.  Benchmarking high bandwidth memory on FPGAs, 2020.
  * Meza et al. [2015] Justin Meza, Qiang Wu, Sanjeev Kumar, and Onur Mutlu.  Revisiting memory errors in large-scale production data centers: Analysis and modeling of new trends from the field.  In _45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2015, Rio de Janeiro, Brazil, June 22-25, 2015_ , pages 415–426. IEEE Computer Society, 2015.  doi:10.1109/DSN.2015.57.
  * Sridharan et al. [2015] Vilas Sridharan, Nathan DeBardeleben, Sean Blanchard, Kurt B. Ferreira, Jon Stearley, John Shalf, and Sudhanva Gurumurthi.  Memory errors in modern systems: The good, the bad, and the ugly.  In Özcan Özturk, Kemal Ebcioglu, and Sandhya Dwarkadas, editors, _Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’15, Istanbul, Turkey, March 14-18, 2015_ , pages 297–310. ACM, 2015\.  doi:10.1145/2694344.2694348.
  * Luyi et al. [2012] S. Luyi, F. Jinyi, and Y. Xiaohua.  Forward error correction.  In _2012 Fourth International Conference on Computational and Information Sciences_ , pages 37–40, 2012.  doi:10.1109/ICCIS.2012.158.
  * Kim et al. [2019a] Bok Young Kim, Yoshitomo Okawachi, Jae K. Jang, Mengjie Yu, Xingchen Ji, Yun Zhao, Chaitanya Joshi, Michal Lipson, and Alexander L. Gaeta.  Turn-key, high-efficiency kerr comb source.  _Opt. Lett._ , 44(18):4475–4478, Sep 2019a.  doi:10.1364/OL.44.004475.
  * Chen et al. [2017] Shaoming Chen, Samuel Irving, Lu Peng, Yue Hu, Ying Zhang, and Ashok Srivastava.  Using switchable pins to increase off-chip bandwidth in chip-multiprocessors.  _IEEE Trans. Parallel Distrib. Syst._ , 28(1):274–289, jan 2017.  ISSN 1045-9219.  doi:10.1109/TPDS.2016.2546246.
  * Jouppi et al. [2017] Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon.  In-datacenter performance analysis of a tensor processing unit.  In _Proceedings of the 44th Annual International Symposium on Computer Architecture_ , ISCA ’17, pages 1–12, New York, NY, USA, 2017. ACM.  ISBN 978-1-4503-4892-8.  doi:10.1145/3079856.3080246.
  * Wade [2019] M. Wade.  TeraPHY: A chiplet technology for low-power, high-bandwidth in-package optical i/o.  In _2019 IEEE Hot Chips 31 Symposium (HCS)_ , pages i–xlviii, 2019\.  doi:10.1109/HOTCHIPS.2019.8875658.
  * Bergman et al. [2018] Keren Bergman, John Shalf, George Michelogiannakis, Sebastien Rumley, Larry Dennison, and Monia Ghobadi.  Pine: An energy efficient flexibly interconnected photonic data center architecture for extreme scalability.  In _2018 IEEE Optical Interconnects Conference (OI)_ , pages 25–26, 2018.  doi:10.1109/OIC.2018.8422036.
  * Maniotis et al. [2021] Pavlos Maniotis, Laurent Schares, Marc A. Taubenblatt, and Daniel M. Kuchta.  Co-packaged optics for HPC and data center networks.  In Henning Schröder and Ray T. Chen, editors, _Optical Interconnects XXI_ , volume 11692, pages 17 – 22. International Society for Optics and Photonics, SPIE, 2021.  doi:10.1117/12.2579066.
  * Minkenberg et al. [2021] Cyriel Minkenberg, Rajagopal Krishnaswamy, Aaron Zilkie, and David Nelson.  Co-packaged datacenter optics: Opportunities and challenges.  _IET Optoelectronics_ , 15(2):77–91, 2021.  doi:https://doi.org/10.1049/ote2.12020.
  * Sutono et al. [1998] A. Sutono, A. Pham, J. Laskar, and W.R. Smith.  Investigations of multi-layer ceramic-based mcm technology.  In _IEEE 7th Topical Meeting on Electrical Performance of Electronic Packaging (Cat. No.98TH8370)_ , pages 83–86, 1998.  doi:10.1109/EPEP.1998.733854.
  * Abrams et al. [2020] Nathan C. Abrams, Qixiang Cheng, Madeleine Glick, Moises Jezzini, Padraic Morrissey, Peter O’Brien, and Keren Bergman.  Silicon photonic 2.5d multi-chip module transceiver for high-performance data centers.  _Journal of Lightwave Technology_ , 38(13):3346–3357, 2020.  doi:10.1109/JLT.2020.2967235.
  * Dittrich et al. [2017] Michael Dittrich, Andy Heinig, Fabian Hopsch, and Robert Trieb.  Heterogeneous interposer based integration of chips with copper pillars and c4 balls to achieve high speed interfaces for adc application.  In _2017 IEEE 67th Electronic Components and Technology Conference (ECTC)_ , pages 643–648, 2017.  doi:10.1109/ECTC.2017.221.
  * Fathololoumi et al. [2021] S. Fathololoumi, D. Hui, S. Jadhav, J. Chen, K. Nguyen, M. N. Sakib, Z. Li, H. Mahalingam, S. Amiralizadeh, N. N. Tang, H. Potluri, M. Montazeri, H. Frish, R. A. Defrees, C. Seibert, A. Krichevsky, J. K. Doylend, J. Heck, R. Venables, A. Dahal, A. Awujoola, A. Vardapetyan, G. Kaur, M. Cen, V. Kulkarni, S. S. Islam, R. L. Spreitzer, S. Garag, A. C. Alduino, R. Chiou, L. Kamyab, S. Gupta, B. Xie, R. S. Appleton, S. Hollingsworth, S. McCargar, Y. Akulova, K. M. Brown, R. Jones, D. Zhu, T. Liljeberg, and L. Liao.  1.6 tbps silicon photonics integrated circuit and 800 gbps photonic engine for switch co-packaging demonstration.  _Journal of Lightwave Technology_ , 39(4):1155–1161, 2021.  doi:10.1109/JLT.2020.3039218.
  * Agrell et al. [2016] Erik Agrell, Magnus Karlsson, A R Chraplyvy, David J Richardson, Peter M Krummrich, Peter Winzer, Kim Roberts, Johannes Karl Fischer, Seb J Savory, Benjamin J Eggleton, Marco Secondini, Frank R Kschischang, Andrew Lord, Josep Prat, Ioannis Tomkos, John E Bowers, Sudha Srinivasan, Mae Brandt-Pearce, and Nicolas Gisin.  Roadmap of optical communications.  _Journal of Optics_ , 18(6):063002, may 2016.  doi:10.1088/2040-8978/18/6/063002.
  * Wei et al. [2015] J. Wei, Q. Cheng, R. V. Penty, I. H. White, and D. G. Cunningham.  400 gigabit ethernet using advanced modulation formats: Performance, complexity, and power dissipation.  _IEEE Communications Magazine_ , 53(2):182–189, 2015.  doi:10.1109/MCOM.2015.7045407.
  * Kim et al. [2019b] Bok Young Kim, Yoshitomo Okawachi, Jae K. Jang, Mengjie Yu, Xingchen Ji, Yun Zhao, Chaitanya Joshi, Michal Lipson, and Alexander L. Gaeta.  Turn-key, high-efficiency kerr comb source.  _Opt. Lett._ , 44(18):4475–4478, Sep 2019b.  doi:10.1364/OL.44.004475.
  * Sharma [2020] D. D. Sharma.  Pci express® 6.0 specification at 64.0 gt/s with pam-4 signaling: a low latency, high bandwidth, high reliability and cost-effective interconnect.  In _2020 IEEE Symposium on High-Performance Interconnects (HOTI)_ , pages 1–8, 2020.  doi:10.1109/HOTI51249.2020.00016.
  * Ikeda et al. [2020] Kazuhiro Ikeda, Keijiro Suzuki, Ryotaro Konoike, Shu Namiki, and Hitoshi Kawashima.  Large-scale silicon photonics switch based on 45-nm cmos technology.  _Optics Communications_ , 466:125677, 2020.  ISSN 0030-4018.  doi:https://doi.org/10.1016/j.optcom.2020.125677.
  * Seok et al. [2019b] Tae Joon Seok, Kyungmok Kwon, Johannes Henriksson, Jianheng Luo, and Ming C. Wu.  240×240 wafer-scale silicon photonic switches.  In _2019 Optical Fiber Communications Conference and Exhibition (OFC)_ , pages 1–3, 2019b.
  * Khope et al. [2017] Akhilesh S. P. Khope, Andrew M. Netherton, Takako Hirokawa, Nicolas Volet, Eric J. Stanton, Clint Schow, Roger Helkey, Adel A. M. Saleh, John E. Bowers, and Rod C. Alferness.  Elastic wdm optoelectronic crossbar switch with on-chip wavelength control.  In _Advanced Photonics 2017 (IPR, NOMA, Sensors, Networks, SPPCom, PS)_ , page PTh1D.3. Optica Publishing Group, 2017.  doi:10.1364/PS.2017.PTh1D.3.
  * Cheng et al. [2019b] Qixiang Cheng, Meisam Bahadori, Yu-Han Hung, Yishen Huang, Nathan Abrams, and Keren Bergman.  Scalable microring-based silicon clos switch fabric with switch-and-select stages.  _IEEE Journal of Selected Topics in Quantum Electronics_ , 25(5):1–11, 2019b.  doi:10.1109/JSTQE.2019.2911421.
  * Sato [2018] Ken-ichi Sato.  Realization and Application of Large-Scale Fast Optical Circuit Switch for Data Center Networking.  _Journal of Lightwave Technology_ , 36(7):1411–1419, April 2018.  ISSN 1558-2213.  doi:10.1109/JLT.2018.2801308.
  * [87] S series optical circuit switch.  URL https://www.calient.net/products/s-series-photonic-switch/.
  * Nikolova et al. [2017] Dessislava Nikolova, David M Calhoun, Yang Liu, Sébastien Rumley, Ari Novack, Tom Baehr-Jones, Michael Hochberg, and Keren Bergman.  Modular architecture for fully non-blocking silicon photonic switch fabric.  _Microsystems & nanoengineering_, 3(1):1–9, 2017.
  * Dai et al. [2020] Liang Yuan Dai, Yu-Han Hung, Qixiang Cheng, and Keren Bergman.  Experimental demonstration of pam-4 transmission through microring silicon photonic clos switch fabric.  In _2020 Optical Fiber Communications Conference and Exhibition (OFC)_ , pages 1–3, 2020.
  * Proietti et al. [2013] R. Proietti, Y. Yin, R. Yu, C. J. Nitta, V. Akella, C. Mineo, and S. J. B. Yoo.  Scalable optical interconnect architecture using awgr-based tonak lion switch with limited number of wavelengths.  _Journal of Lightwave Technology_ , 31(24):4087–4097, 2013.  doi:10.1109/JLT.2013.2285883.
  * Lea [2015] C. Lea.  A scalable awgr-based optical switch.  _Journal of Lightwave Technology_ , 33(22):4612–4621, Nov 2015.  ISSN 0733-8724.  doi:10.1109/JLT.2015.2479296.
  * Dhoore, Sören and Roelkens, Günther and Morthier, Geert [2019] Dhoore, Sören and Roelkens, Günther and Morthier, Geert.  Fast wavelength-tunable lasers on silicon.  _IEEE JOURNAL OF SELECTED TOPICS IN QUANTUM ELECTRONICS_ , 25(6):1500908:1–1500908:8, 2019.  ISSN 1077-260X.
  * [93] Optical circuit switch product availability.  https://www.fs.com/products/66601.html.
  * Sato et al. [2013] Ken-ichi Sato, Hiroshi Hasegawa, Tomonobu Niwa, and Toshio Watanabe.  A large-scale wavelength routing optical switch for data center networks.  _IEEE Communications Magazine_ , 51(9):46–52, September 2013.  ISSN 1558-1896.  doi:10.1109/MCOM.2013.6588649.
  * Ueda et al. [2016] Koh Ueda, Yojiro Mori, Hiroshi Hasegawa, Hiroyuki Matsuura, Kiyo Ishii, Haruhiko Kuwatsuka, Shu Namiki, Toshio Watanabe, and Ken-ichi Sato.  Demonstration of 1,440×1,440 fast optical circuit switch for datacenter networking.  In _2016 21st OptoElectronics and Communications Conference (OECC) Held Jointly with 2016 International Conference on Photonics in Switching (PS)_ , pages 1–3, July 2016.
  * Bhoja [2017] Sudeep Bhoja.  Pam4 signaling for intra-data center and data center to data center connectivity (dci).  In _2017 Optical Fiber Communications Conference and Exhibition (OFC)_ , pages 1–54, 2017.
  * Marom et al. [2017] D. M. Marom, P. D. Colbourne, A. D’errico, N. K. Fontaine, Y. Ikuma, R. Proietti, L. Zong, J. M. Rivas-Moscoso, and I. Tomkos.  Survey of photonic switching architectures and technologies in support of spatially and spectrally flexible optical networking [invited].  _IEEE/OSA Journal of Optical Communications and Networking_ , 9(1):1–26, 2017.  doi:10.1364/JOCN.9.000001.
  * Teh et al. [2020] Min Yee Teh, Yu-Han Hung, George Michelogiannakis, Shijia Yan, Madeleine Glick, John Shalf, and Keren Bergman.  Tago: Rethinking routing design in high performance reconfigurable networks.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–16, 2020.  doi:10.1109/SC41405.2020.00029.
  * Michelogiannakis et al. [2019] George Michelogiannakis, Yiwen Shen, Min Yee Teh, Xiang Meng, Benjamin Aivazi, Taylor Groves, John Shalf, Madeleine Glick, Manya Ghobadi, Larry Dennison, and Keren Bergman.  Bandwidth steering in HPC using silicon nanophotonics.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , SC ’19, 2019.  ISBN 9781450362290.  doi:10.1145/3295500.3356145.
  * Shalf et al. [2005] John Shalf, Shoaib Kamil, Leonid Oliker, and David Skinner.  Analyzing ultra-scale application communication requirements for a reconfigurable hybrid interconnect.  In _Proceedings of the ACM/IEEE SC2005 Conference on High Performance Networking and Computing, November 12-18, 2005, Seattle, WA, USA, CD-Rom_ , page 17, 2005.  doi:10.1109/SC.2005.12.
  * Vetter and Mueller [2002] Jeffrey S. Vetter and Frank Mueller.  Communication characteristics of large-scale scientific applications for contemporary cluster architectures.  In _16th International Parallel and Distributed Processing Symposium (IPDPS 2002), 15-19 April 2002, Fort Lauderdale, FL, USA, CD-ROM/Abstracts Proceedings_ , 2002.  doi:10.1109/IPDPS.2002.1015504.
  * Domke et al. [2019] Jens Domke, Satoshi Matsuoka, Ivan R. Ivanov, Yuki Tsushima, Tomoya Yuki, Akihiro Nomura, Shin’ichi Miura, Nie McDonald, Dennis L. Floyd, and Nicolas Dubé.  Hyperx topology: First at-scale implementation and comparison to the fat-tree.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , SC ’19, 2019.  ISBN 9781450362290.  doi:10.1145/3295500.3356140.
  * Ma et al. [2014] Sheng Ma, Zhiying Wang, Natalie Enright Jerger, Li Shen, and Nong Xiao.  Novel flow control for fully adaptive routing in cache-coherent nocs.  _IEEE Transactions on Parallel and Distributed Systems_ , 25(9):2397–2407, 2014.  doi:10.1109/TPDS.2013.166.
  * Becker and Dally [2009] Daniel U. Becker and William J. Dally.  Allocator implementations for network-on-chip routers.  In _Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis_ , pages 1–12, 2009.  doi:10.1145/1654059.1654112.
  * Jiang et al. [2009] Nan Jiang, John Kim, and William J. Dally.  Indirect adaptive routing on large scale interconnection networks.  In _Proceedings of the 36th Annual International Symposium on Computer Architecture_ , ISCA ’09, pages 220–231, 2009.  ISBN 978-1-60558-526-0.  doi:10.1145/1555754.1555783.
  * De Sensi et al. [2020a] Daniele De Sensi, Salvatore Di Girolamo, Kim H. McMahon, Duncan Roweth, and Torsten Hoefler.  An in-depth analysis of the slingshot interconnect.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–14, 2020a.  doi:10.1109/SC41405.2020.00039.
  * Hosseini et al. [2021] Kaveh Hosseini, Edwin Kok, Sergey Y. Shumarayev, Chia-Pin Chiu, Arnab Sarkar, Asako Toda, Yanjing Ke, Allen Chan, Daniel Jeong, Mason Zhang, Sangeeta Raman, Thungoc Tran, Kumar Abhishek Singh, Pavan Bhargava, Chong Zhang, Haiwei Lu, Ravi Mahajan, Xiaoqian Li, Nitin Deshpande, Conor O’Keeffe, Tim Tri Hoang, Uma Krishnamoorthy, Chen Sun, Roy Meade, Vladimir Stojanovic, and Mark Wade.  8 tbps co-packaged FPGA and silicon photonics optical IO.  In _2021 Optical Fiber Communications Conference and Exhibition (OFC)_ , pages 1–3, 2021.
  * Binkert et al. [2011] Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K. Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R. Hower, Tushar Krishna, Somayeh Sardashti, Rathijit Sen, Korey Sewell, Muhammad Shoaib, Nilay Vaish, Mark D. Hill, and David A. Wood.  The gem5 simulator.  _SIGARCH Comput. Archit. News_ , 39(2):1–7, aug 2011.  ISSN 0163-5964.  doi:10.1145/2024716.2024718.
  * Bienia et al. [2008] Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li.  The parsec benchmark suite: Characterization and architectural implications.  In _2008 International Conference on Parallel Architectures and Compilation Techniques (PACT)_ , pages 72–81, 2008.
  * Bailey et al. [1992] D.H. Bailey, E. Barszcz, L. Dagum, and H.D. Simon.  Nas parallel benchmark results.  In _Supercomputing ’92:Proceedings of the 1992 ACM/IEEE Conference on Supercomputing_ , pages 386–393, 1992.  doi:10.1109/SUPERC.1992.236665.
  * Che et al. [2009] Shuai Che, Michael Boyer, Jiayuan Meng, David Tarjan, Jeremy W. Sheaffer, Sang-Ha Lee, and Kevin Skadron.  Rodinia: A benchmark suite for heterogeneous computing.  In _2009 IEEE International Symposium on Workload Characterization (IISWC)_ , pages 44–54, 2009.  doi:10.1109/IISWC.2009.5306797.
  * Mutlu et al. [2006] O. Mutlu, Hyesoon Kim, and Y.N. Patt.  Efficient runahead execution: Power-efficient memory latency tolerance.  _IEEE Micro_ , 26(1):10–20, 2006.  doi:10.1109/MM.2006.10.
  * Parcerisa and Gonzalez [2001] J.-M. Parcerisa and A. Gonzalez.  Improving latency tolerance of multithreading through decoupling.  _IEEE Transactions on Computers_ , 50(10):1084–1094, 2001.  doi:10.1109/12.956093.
  * Mowry et al. [1998] T.C. Mowry, C.Q.C. Chan, and A.K.W. Lo.  Comparative evaluation of latency tolerance techniques for software distributed shared memory.  In _Proceedings 1998 Fourth International Symposium on High-Performance Computer Architecture_ , pages 300–311, 1998.  doi:10.1109/HPCA.1998.650569.
  * Nekkalapu et al. [2008] Satyanarayana Nekkalapu, Haitham Akkary, Komal Jothi, Renjith Retnamma, and Xiaoyu Song.  A simple latency tolerant processor.  In _2008 IEEE International Conference on Computer Design_ , pages 384–389, 2008.  doi:10.1109/ICCD.2008.4751889.
  * Mittal and Vetter [2016] Sparsh Mittal and Jeffrey S. Vetter.  A survey of software techniques for using non-volatile memories for storage and main memory systems.  _IEEE Trans. Parallel Distrib. Syst._ , 27(5):1537–1550, may 2016.  ISSN 1045-9219.  doi:10.1109/TPDS.2015.2442980.
  * Arafa et al. [2021] Yehia Arafa, Abdel-Hameed Badawy, Ammar ElWazir, Atanu Barai, Ali Eker, Gopinath Chennupati, Nandakishore Santhi, and Stephan Eidenbenz.  Hybrid, scalable, trace-driven performance modeling of GPGPUs.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , SC ’21, 2021.  ISBN 9781450384421.  doi:10.1145/3458817.3476221.
  * Choquette and Gandhi [2020] Jack Choquette and Wish Gandhi.  NVIDIA A100 GPU: Performance & innovation for GPU computing.  In _2020 IEEE Hot Chips 32 Symposium (HCS)_ , pages 1–43, 2020.  doi:10.1109/HCS49909.2020.9220622.
  * Grauer-Gray et al. [2012] S. Grauer-Gray, L. Xu, R. Searles, S. Ayalasomayajula, and J. Cavazos.  Auto-tuning a high-level language targeted to gpu codes.  In _Innovative Parallel Computing_ , InPar ’12, page 1–10, 2012\.  doi:10.1109/InPar.2012.6339595.
  * Karki et al. [2019] A. Karki, C. P. Keshava, S. M. Shivakumar, J. Skow, G. M. Hegde, and H. Jeon.  Detailed characterization of deep neural networks on gpus and fpgas.  In _Proceedings of the 12th Workshop on General Purpose Processing Using GPUs_ , GPGPU ’19, page 12–21, 2019.  doi:https://doi.org/10.1145/3300053.3319418.
  * Rahimi et al. [2022] Javad Rahimi, Joris Van Kerrebrouck, Bahawal Haq, Johan Bauwelinck, Gunther Roelkens, and Geert Morthier.  Demonstration of a high-efficiency short-cavity iii-v-on-si c-band dfb laser diode.  _IEEE Journal of Selected Topics in Quantum Electronics_ , 28(3):1–6, 2022.  doi:10.1109/JSTQE.2021.3122552.
  * Sun et al. [2020] Chen Sun, Daniel Jeong, Mason Zhang, Woorham Bae, Chong Zhang, Pavan Bhargava, Derek Van Orden, Shahab Ardalan, Chandarasekaran Ramamurthy, Erik Anderson, Austin Katzin, Haiwei Lu, Sidney Buchbinder, Behrooz Beheshtian, Anatoly Khilo, Michael Rust, Chen Li, Forrest Sedgwick, John Fini, Roy Meade, Vladimir Stojanović, and Mark Wade.  TeraPHY: An o-band wdm electro-optic platform for low power, terabit/s optical i/o.  In _2020 IEEE Symposium on VLSI Technology_ , pages 1–2, 2020.  doi:10.1109/VLSITechnology18217.2020.9265012.
  * De Sensi et al. [2020b] Daniele De Sensi, Salvatore Di Girolamo, Kim H. McMahon, Duncan Roweth, and Torsten Hoefler.  An in-depth analysis of the slingshot interconnect.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , SC ’20. IEEE Press, 2020b.  ISBN 9781728199986.
  * Katebzadeh et al. [2020] M. R. Siavash Katebzadeh, Paolo Costa, and Boris Grot.  Evaluation of an infiniband switch: Choose latency or bandwidth, but not both.  In _2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_ , pages 180–191, 2020.  doi:10.1109/ISPASS48437.2020.00033.
  * Vasa et al. [2020] Mallikarjun Vasa, Chun-Lin Liao, Sanjay Kumar, Ching-Huei Chen, and Bhyrav Mutnury.  Pcie gen-5 design challenges of high-speed servers.  In _2020 IEEE 29th Conference on Electrical Performance of Electronic Packaging and Systems (EPEPS)_ , pages 1–3, 2020.  doi:10.1109/EPEPS48591.2020.9231458.
  * Shim et al. [2022] Keun Sup Shim, Brian Greskamp, Brian Towles, Bruce Edwards, J. P. Grossman, and David E. Shaw.  The specialized high-performance network on anton 3, 2022.

  *[HPC]: high performance computing
  *[MCM]: multi chip module
  *[BER]: bit error rate
  *[CPU]: central processing unit
  *[GPU]: graphical processing unit
  *[NICs]: network interface controller
  *[GPUs]: graphical processing unit
  *[MCMs]: multi chip module
  *[OOO]: out-of-order
  *[CXL]: compute express link
  *[CPUs]: central processing unit
  *[OS]: operating system
  *[NIC]: network interface controller
  *[BERs]: bit error rate
  *[HBM]: high bandwidth memory
  *[FIT]: failures in time
  *[SEC-DED]: single-error-correct/double-error-detect
  *[DWDM]: dense wavelength division multiplexing
  *[NVM]: non-volatile memory
  *[PIC]: photonic integrated circuit
  *[WDM]: wavelength-division multiplexing
  *[OEO]: Electrical Optical
  *[FEC]: forward error correction
  *[MEMS]: microelectromechanical systems
  *[MZI]: Mach-Zehnder interferometer
  *[MRR]: microring resonator
  *[WSS]: wavelength-selective switching
  *[AWGRs]: arrayed waveguide grating router
  *[AWGR]: arrayed waveguide grating router
  *[LLC]: last level cache
  *[HBMs]: high bandwidth memory
  *[MSHRs]: miss status handling register
  *[ OOO]: out-of-order
  *[WPE]: wall plug efficiency