# Muchisim: A Simulation Framework for Design Exploration of Multi-Chip
Manycore Systems

Marcelo Orenes-Vera, Esin Tureci, Margaret Martonosi, David Wentzlaff
Princeton University, Princeton, New Jersey, USA
{movera, esin.tureci, mrm, wentzlaf} @princeton.edu

###### Abstract

Current design-space exploration tools cannot accurately evaluate
communication-intensive applications whose execution is data-dependent (e.g.,
graph analytics and sparse linear algebra) on scale-out manycore systems, due
to either lack of scalability or lack of detail in modeling the network. This
paper presents Muchisim, a novel parallel simulator designed to address the
challenges in exploring the design space of distributed multi-chiplet manycore
architectures for communication-intensive applications. We evaluate Muchisim
at simulating systems with up to a million interconnected processing elements
(PEs) while modeling data movement and communication in a cycle-accurate
manner. In addition to performance, Muchisim reports the energy, area, and
cost of the simulated system, and it comes with a benchmark application suite
and two data visualization tools.

Muchisim supports various parallelization strategies and communication
primitives such as task-based parallelization and message passing, making it
highly relevant for architectures with software-managed coherence and
distributed memory. Via a case study, we show that Muchisim helps users
explore the balance between memory and computation units and the constraints
related to chiplet integration and inter-chip communication. Muchisim enables
scaling up the systems in which new techniques or design parameters are
evaluated, opening the gate for further research in this area.

##  I Introduction

The end of Moore’s law and the increase in dataset sizes to be processed have
driven the recent surge in development of scale-out manycore systems [15, 29,
91, 38, 17, 101, 43, 94]. Design-space exploration for these architectures at
large scales employs roofline and analytical models to estimate performance,
by relying on the deterministic nature of communications as well as
computations for many machine learning and dense applications. Alternatively,
for data-dependent communication-intensive (DCI) applications, such as graph
analytics, sparse linear algebra, and database operations, the communication
is not deterministic, requiring simulation tools to accurately estimate
performance. However, current simulators cannot evaluate DCI applications at
scale due to the limited scalability of model processing units in detail.
Because of that, recent work on architecture designs for DCI applications are
evaluated on relatively small scales, and thus, their network design—whose
performance impact would be exacerbated at larger scales—is not well explored
[18, 64, 61, 60, 30, 56, 19, 52].

Current manycore architectures typically emphasize speeding up computation,
dedicating significant chip area to it at the expense of memory and network
resources. However, when the workload is highly parallelized, data movement
starts to become the limiting factor for performance. This is especially true
for DCI applications but also other application domains such as spectral
methods [65], which necessitate a careful balance between computational,
memory and network resources. Thus, in manycore systems with thousands to
millions of cores, factors such as the number of cores per silicon die, the
number of chiplet dies per package, memory hierarchy (integration and
coherence), network-on-chip (NoC) and inter-chip interconnect, become critical
design considerations. Lack of ability to consider these at large scale can
create significant gaps between designed systems and their actual
realizations.

To address these issues, we introduce Muchisim, 111Muchisim is derived from
the Spanish word muchisimo (very much), alluding to its ability to simulate a
large number of PEs in a multi-chip setting. Muchisim is pronounced as ”moo-
chee-sim”. a parallel simulator that allows exploration of the manycore
architecture design space, with a special focus on communication-intensive
applications.222At large scale, memory-intensive applications lead to a
communication bottleneck, so we englobe them in the communication-intensive
category. Muchisim scales up to millions of interconnected PEs by relying on
user instrumentation for code runtime on the compute units while simulating
data movement and communication in a cycle-accurate manner. It also includes
energy, area and cost models for multi-chip distributed systems, which are
critical to reason about design tradeoffs. In addition, Muchisim includes an
application benchmark suite and data visualization tools to facilitate the
analysis of its outputs.

As the industry leans towards explicit data movement for large manycores due
to the prohibitive cost of hardware-based coherence, Muchisim becomes
particularly relevant. Muchisim is applicable for architectures with software-
managed coherence and distributed-memory, providing an accurate representation
of modern scale-out systems. Muchisim facilitates exploring the integration
granularity and balance between memory and computation units, as well as
constraints related to chiplet integration and inter-chip communication. With
a multifaceted analysis aimed at expanding current understanding of manycore
architectures for communication-intensive applications, Muchisim offers a
flexible yet powerful framework for exploring many design configurations,
thereby contributing to the design and optimization of future large-scale
systems.

Our main technical contributions are:

  * •

A novel performance modeling approach for distributed manycore architectures
that allows for scalable simulations of DCI applications across millions of
PEs.

  * •

Performance, energy, area and cost modeling of multi-chip module (MCM) and
interposer-based integrations, critical in the design of scalable manycore
systems.

  * •

Support for different parallelization strategies (do-all and task-based) and
communication primitives (e.g., message-passing and reduction trees).

  * •

A benchmark suite of eight applications and ten datasets of different sizes
and characteristics.

  * •

Two visualization tools to compare certain metrics for different datapoints
(DUT configurations, applications and datasets) or analyze them for every PE
throughout the execution of a single datapoint.

We evaluate Muchisim and demonstrate that:

  * •

Muchisim is the first open-source simulator that accurately simulates DCI
applications with billion-element datasets parallelized across a million PEs
within tens of hours.

  * •

Its parallelization achieves linear speedups up to the thread count equal to
the columns of the PE grid being simulated.

  * •

Muchisim accurately models the runtime and area, as validated with the
Cerebras Wafer-Scale Engine.

  * •

Muchisim is a powerful design-exploration tool that helps identify optimal
configurations for different metrics (e.g., performance-per-dollar) and
applications

The rest of the paper is organized as follows: section II provides background
on existing simulation approaches and motivates our work; section III details
the architecture class that Muchisim simulates, its design and capabilities,
and the benchmark and visualization tool that it includes; section IV presents
the validation and scalability analysis of Muchisim, in addition to a case
study that uses Muchisim to study different memory integrations; and section V
concludes the paper and discusses future work.

##  II Background and Motivation

As the number of processing elements (PEs) used to parallelize a workload
increases, the network bandwidth and topology become important design
considerations. Depending on the architecture, the bottleneck may also shift
or aggravate with the level of parallelism. Thus, detailed simulation of large
manycore systems is critical for design-space exploration, to avoid making
suboptimal design decisions that may be hard to correct later in the design
process.

The computer architecture community has seen a lot of promising research on
hardware-software co-designs and optimizations for data-dependent
communication-intensive (DCI) applications [90, 64, 30, 78, 72, 1, 60, 61, 18,
76, 35, 96, 69, 66, 23]. However, these ideas are often evaluated on a limited
number of PEs (hundreds to a few thousand) or relatively small datasets (up to
millions of data elements), as increasing either of these dimensions increases
the simulation time significantly. With Muchisim, we want to enable the
exploration of these ideas on larger scales, as well as offer key parameters
to explore the balance of the hardware resources and power allocated to
memory, network and compute units. For the rest of this section, we outline
the desired features of a scale-out manycore system simulator (and their
potential trade-offs) and review existing simulators.

Accuracy is one of the most important features of an architecture simulator,
as it dictates the reliability of design experiments. For systems of different
scales and target application domains, focusing on modeling some of their
components in detail may become necessary when building a simulator that
balances simulator performance and accuracy [99].

Full versus application-level simulation: Full-system simulators offer the
ability to evaluate the performance of a target design in the context of other
system components as well as a complete software stack including the operating
system (OS). However, this comes at a great expense of speed and scalability
even for small-scale systems. Nevertheless, when the execution significantly
relies on OS and I/O processes, full-system simulators are essential when
evaluating the performance of a design. Simulators such as GEM5, SimFlex,
COTSon, RAMP Gold are some of the most commonly used full-system simulators
with varying scope and capabilities [12, 31, 7, 92]. In the manycore
architectures we are considering in this study, PEs are not expected to run
the OS but rather behave as accelerators to host CPUs, and they do not
implement hardware coherence in order to save area and power but instead have
an explicit view of the memory. In addition, PEs rely on software to
orchestrate the data movement—optimized based on application specifics [15,
69, 70, 21, 94]. This makes an application/user-level simulator for these
systems more appropriate. Application-level simulators can scale to higher
degrees of parallelism than full-system simulators by simulating up to a
couple of thousand PEs [55, 80, 26, 3, 7]. However, these simulators cannot
efficiently simulate systems beyond this size, and some of them lack detailed
network modeling, which significantly affects the accuracy of larger system
simulations. BigSim [102] is a simulator originally developed to simulate Blue
Gene that performs detailed network simulation after the emulation of the
program on a real system to account for network traffic, achieving scaling in
tens to hundreds of thousands of cores. BigSim relies on the program to be
deterministic, but certain DCI workloads like graph algorithms do not converge
deterministically unless additional synchronization steps are employed.

Simulating DCI applications on manycores: Our goal is to model a class of
tiled, distributed manycore architectures. This class of architectures is
composed of an interconnected number of machine nodes, each with a board of
chip packages, further subdivided into chiplets and PEs. Therefore the entire
compute system can be viewed as a hierarchically connected grid of PEs.
Muchisim is built as a tool to explore the design space of this class of
architectures, with a focus on DCI applications. The challenge with these
applications is that their data-dependent execution requires functional
simulation to precisely capture performance and energy usage. When
parallelizing DCI applications across thousands of PEs and billions of data
elements, faithfully simulating all architecture components every cycle
becomes infeasible.

PE detail level: Since the aforementioned literature has identified the memory
bandwidth and inter-PE communication to be the main bottlenecks of these
applications, Muchisim focuses on modeling the network and memory systems,
while executing the task on the host and relying on the user to provide a
performance model for the compute time (e.g., instructions per cycle). A
similar approach has been previously used by PriME [26], but PriME simplifies
the PE model by simulating each instruction as 1 cycle, whereas Muchisim can
use that or a detailed user-provided model (section III). Because the PEs used
in scale-out manycores do not interact with the OS and have a simple view of
the memory system, it is possible to model the timing of execution relatively
accurately using user-provided performance models and without detailed core
simulations. Simulators such as SimpleScalar [8], SimFlex [31], and Gem5 [12]
offer detailed core modeling with a wide range of supported ISAs. In addition,
simulators such as MosaicSim [53] and SimTRaX [84] provide LLVM-based
instruction sets. When precise PE models are desired, these simulators can be
used to provide the PE- and task-specific cycle counts to be incorporated into
Muchisim simulations.

Detailed Network simulation: Since inter-PE communication becomes a
significant performance factor for large-scale manycore systems, the
capability to model different configurations of the network-on-chip (NoC) and
the inter-chip interconnect in detail is essential when modeling these
systems. While there are several cycle-accurate NoC simulators such as Xpipes
[10], NOXIM [14], SICOSYS [77], BookSim [36], among others [32], most of the
existing multicore system simulators [87, 26, 55] do not model the NoC in
detail. Although some implementations incorporate these NoC models within
system simulators (e.g. Xpipes with Gem5), a functional architecture simulator
(with cycle-accurate NoC modeling) able to simulate beyond a thousand PEs was
not offered so far.

Scope: Muchisim comes to fill this gap; the class of architectures for which
Muchisim is applicable includes—but not limited to—tiled accelerators [69, 68,
103, 4, 18] and dataflow machines [15, 21, 29, 94, 25]. Muchisim can also be
used to simulate systolic array architectures like Google’s TPU [38], but
these designs usually execute deterministic workloads with near-neighbor
communication, and thus they would be modeled much faster using analytical
models. Unlike general-purpose simulators for HPC such as SST [79], focusing
on this architectures class allows Muchisim to be a highly-scalable, parallel,
light-weight simulator while providing detailed simulations of critical
aspects of DCI workloads such as NoC modeling at flit-level granularity.
Muchisim could also potentially be used to model the performance of workloads
running on more general-purpose manycores [101, 50, 27, 20] and server-class
chip [22, 5, 58], provided that they use explicit data movement or message
passing instead of shared-memory across PEs.

##  III Muchisim Simulation Framework

In section III-A we first present the hardware components of Muchisim’s target
architecture class, and how they are hierarchically organized. Then, section
III-B explains the execution models supported in Muchisim and how to describe
applications. Next, section III-C elaborates on how Muchisim simulates the
execution of the applications on the target architecture. Muchisim is a
parallel simulator written in C/C++. Muchisim does not use external libraries
beyond threading, so it can be compiled with any C++ compiler that includes
OpenMP (-fopenmp) or pthread support (-lpthread). The code can be found in our
repository[67] under the src folder, where the configs subfolder contains the
files to configure the DUT to simulate, and its latency, energy, area and cost
parameters (described in section III-D and section III-E). The gui and plot
folders contain the visualization tools to analyze and compare simulation logs
(section III-F). The apps and datasets folders contain the benchmark suite of
applications and datasets (section III-G). In addition, the repository
includes a Readme file explaining how to configure and run Muchisim for
different experiments.

###  III-A The Target Architecture Class

Figure 1:  Hierarchical overview of how tiles can be organized on the
Muchisim’s target architecture. The board of a cluster node may contain one or
multiple chip packages, each with one or multiple chiplets. Packages can be
composed of only compute chiplets (with a grid of tiles), or also DRAM
chiplets (adjacent to the compute chiplets). Chiplets also include the PHY for
inter-chiplet communication. A tile contains one or more processing units
(PUs), a network router (R), the task scheduling unit (TSU), and an SRAM
scratchpad (SPM). The task queues are mapped into the SPM.

Throughout the paper, we refer to the simulated architecture as the design
under test (DUT). We borrow this acronym from the hardware verification
community, where it is used to refer to the design that is being tested. We
call the machine that runs the simulator, the host, and use DUT’s host to
refer to the machine for which the DUT may behave as an accelerator.

fig. 1 shows the hierarchy of Muchisim’s target architecture class. A cluster
is composed of interconnected boards ($\sim$nodes) that contain one or more
chip packages ($\sim$sockets). Each package is composed of one or several
compute chiplets of any size (from a few mm2 to as big as a wafer), optionally
integrating DRAM chiplets with the compute chiplets. Each compute chiplet has
a 2D array of tiles, where each tile contains one or more processing units
(PUs), an SRAM scratchpad memory (SPM), a router, and a task scheduling unit
(TSU).

Processing Units (PUs): A PU can be an ISA-programmable core, a CGRA unit, or
a hardware accelerator, depending on the performance, energy and area model
provided. We have only evaluated cases where the PUs are homogeneous across
the chiplet, but Muchisim could potentially simulate heterogeneous manycores
[27, 21, 16], since a different performance model could be provided for
different PU IDs.

Network Routers: Routers generally have five bidirectional ports: North,
South, East, West, and the PU ports (connected to the PU’s input and output
queues). When configuring hierarchical inter-chiplet networks [66] or ruche
channels [71, 39], some or all routers have an additional set of cardinal
ports (for a total of nine). Muchisim supports 2D mesh and 2D folded torus
topologies[24] (with dimension-ordered routing) across tiles and chiplets, up
to the level of a node. Across nodes, Muchisim currently supports a mesh
interconnect, but it could be extended to support other cluster-level
topologies [11, 47, 41, 9, 2]. In addition, Muchisim supports configuring
reduction trees of different degrees [70] .

Task Scheduling Unit (TSU): To support task-based parallelizations and
message-triggered tasks (see section III-B), Muchisim considers a TSU per
tile, which takes messages from the input queues (one per task type) and
schedules them to a PU. The TSU decides which task to schedule next—from the
ones available at the queues—based on a scheduling policy. The policies
currently supported are round-robin, priority-based (e.g., prioritizing tasks
from a particular queue), or occupancy-based (to prevent full queues from
backpressuring the network).

SRAM scratchpad memories (SPMs): The size of each tile’s memory is specified
in Kibibytes (KiB); it is the same for all the tiles. Muchisim supports using
the SPM as a write-back cache (direct-mapped or associative) when DRAM is
integrated on-package or as a scratchpad (given the tile’s local address
space) when the chip’s main memory is the tile-distributed SRAM. In cache
mode, we model the area and energy of the tags (and valid/dirty bits) using
part of the SPM; the cacheline width equals the bitline width of the DRAM
memory controller (512 by default). Cache misses go to the on-chip memory
controller, which fetches the full cacheline from DRAM directly since there is
no hardware coherence in our target DUT. Dirty lines are written back to DRAM
upon eviction.

Queues are currently modeled as circular FIFOs mapped into the SPM, which
allows for compile-time configuration on the DUT. Thus, in our energy model,
queue reads and writes are similar to load and store operations.
Alternatively, one could change the model to consider them as separate
structures, with the corresponding tradeoff in area and energy.

Prefetching: Muchisim supports modeling next-line and pointer-indirection data
prefetching from DRAM into the SPM (used as a cache). The latter is enabled by
the fact that tasks can be split at pointer indirection and the TSU can
prefetch the data that tasks (waiting in the input queue) will use.

Chiplet Integration: The network routers at the edges of the compute chiplets
are connected to the PHYs for inter-chiplet communication [6]. When DRAM is
integrated on-package, Muchisim models the pairs of DRAM and compute chiplets
to be connected via a silicon interposer, while these interposers lay on an
organic substrate (details in section III-D). Alternatively, Muchisim can
model the compute chiplets to be connected via a single silicon or organic
interposer. In our model, the choice of the interposer mostly affects maximum
communication bandwidth, area of the PHY, and energy per bit (see table I).

Interconnect links: Muchisim offers parameters to set the width or the number
of interconnect links across chiplets or chip packages. Across cluster nodes,
one can set the multiplexing factor of the links between nodes, i.e., how many
edge tiles share an inter-node link.

###  III-B Describing and Mapping an Application

The code for an application and its potential DUT compile-time configurations
are described as C header files in a structured manner using a template that
defines a series of functions. The file for the application to simulate
(src/apps) is included via preprocessor macros so that the simulator code is
compiled together with the application code.

Configuration functions: The functions with the prefix config_ are invoked
only once at the beginning of the simulation. They allow overriding software
parameters of the DUT for which the configuration files have set default
values. These parameters include the sizes of the queues per task, as well as
the parameters for prioritizing the scheduling of certain task IDs. There are,
however, other software parameters of the DUT are macros that need to be set
when compiling the simulator. The reason why some parameters are macros
(instead of being defined in the configuration file) is that they activate
different parts of the simulator code that are evaluated every clock cycle. In
other words, Muchisim uses pre-processor macros to save branches and improve
performance.

Address space and dataset layout: By default, the memory address space is
distributed equally and contiguously across the tiles of the DUT. Then, the
dataset arrays for a given application are allocated and mapped in memory (at
config_app) depending on how a user or mapping technique [95] decides so. On
our application suite, the dataset is scattered so that each tile has an equal
chunk of it.

Message-triggered tasks (MTTs): The functions associated with an identifier
(ID), e.g., task1, describe the different types of MTTs that are invoked upon
receiving a message into their corresponding input queue (IQ). Defining these
functions enables simulating communication primitives like message passing,
active messages, and non-interrupting remote-procedure calls (RPCs). Tasks can
invoke other tasks by placing messages into another task’s IQ (if the task is
going to be executed on the same PE) or into a channel queue (CQ) if the task
is meant to be executed on a different PE. This section later elaborates on
the relation between queues and tasks.

Init task: The function with the suffix _init is invoked once at the beginning
of each kernel. The kernel_count variable can be used to distinguish behavior
between different kernels. Multiple kernels can be simulated in sequence (with
global barrier synchronization between them) to compose an application. The
combination of init and MTTs per kernel enables simulating a variety of
programming models. However, recursion is not supported. On the one extreme,
the init task can be empty if the entire kernel only comprises MTTs invoking
each other, starting from a seed invocation coming from the DUT’s host. On the
other extreme, a kernel can be entirely included in the init task (not
invoking remote tasks).

All task functions include the identifier of the PE for which the task is
being simulated (and the size of the grid of PEs is macro), so kernels can be
parallelized across PEs similarly as with pthreads. section III-C describes
how Muchisim iterates over every PE to check whether there are tasks ready to
execute.

Association of queues and tasks: Each MTT is associated with an IQ matching
its ID. If a task wants to invoke other tasks, it can do so by placing
messages into the IQ of the target task ID on the same PE or into the CQs.
Each CQ drains into its associated logical channel in the network, and each
channel is associated with an IQ ID.

Task dependencies: To avoid network deadlocks, loops in the association
between MTTs are not allowed (e.g., task1 invoking task2 and task2 invoking
task1). In other words, the dependency chain must end on a leaf task (a task
not invoking other tasks); a new phase of computation can start after a local
or global barrier. Recall that it is not compulsory for a kernel to use MTTs,
but they are the mechanism for the simulator to support message passing and
RPCs.

Result-check function: Since Muchisim is a functional simulator, the
application results can be compared with reference values. The compare_out
function can be used to compare the outputs directly or write them into a file
to be compared outside the simulator.

###  III-C Simulating the Application Runtime

Although the tiles may be organized hierarchically, for parallelization
purposes, the simulator considers a global grid of tiles, of which each host
thread simulates one or more columns. There are two types of threads, the ones
that simulate the TSU and the PU tasks (_execution threads_) and the ones that
simulate the network (_router threads_).

The execution threads iterate over every tile in their slice of the grid of
tiles, checking if there are tasks ready to be executed in the tile’s IQs, in
addition to the init task. If there are, the TSU selects the next task type to
execute based on the configured policy. Then, the function associated with
that task is executed, and the task delay is obtained from the latency-
instrumented code. The codes of the application suite included in Muchisim are
instrumented considering simple in-order PUs, where the delay is the number of
instructions executed (CPI=1); for memory operations, Muchisim offers a
special dcache function that returns the latency to fetch a given memory
address, depending on whether it hits in the data cache, and the configured
memory hierarchy. The function call simulating the task may have pushed
messages into other IQs or CQs, with the corresponding timestamps, so they
cannot be routed until the router thread is executed for that timestamp. Once
the task function returns, the clock for that PU advances as many cycles as
the task delay, and the execution thread continues evaluating the next PU. The
execution thread keeps iterating over the PUs until there are no more tasks to
be done, and the network is empty (i.e., no more messages to be routed). This
models a hardware-based termination condition based on idleness, and by
default, we set its latency to two times the diameter of the network.

The router threads iterate every cycle over every router in their slide of the
grid. For every input port, they check whether there is a message to be routed
and a free buffer slot in the destination port (the buffer size is also
parameterizable). When two or more input ports want to route to the same
output, the priority is determined via a round-robin policy (static priority
could also be configured). A message is only routable if the timestamp (set at
injection and updated every hop) is less or equal to the current router cycle;
the timestamps do not exist in the DUT, but they are used to allow PUs and
routers to be simulated in parallel. The synchronization between the execution
and router threads is done based on their wall clock time, i.e., a router can
never be simulated ahead of its tile’s PUs. In addition, a PU cannot start
executing a new task until the router thread has caught up with the clock time
of the slowest PU in the tile. Thanks to this type of synchronization,
Muchisim can seamlessly support different frequencies for the PUs and the
network, with any ratio between them.

Frequency: The configuration file of Muchisim distinguishes between the target
frequency established at design for the PU and the routers and their operating
frequency. The first is going to affect the area in our current model, while
the second affects the power consumption, due to the voltage scaling
(described in section III-D). The default target and operating frequency for
all PUs, NoC and memory is 1GHz.

SRAM model: The default latency, area and energy parameters of the SRAM
memories are based on 7nm technology at 1 Ghz [98] (see table I). Unlike the
PU and NoC, SRAM memories have by design a narrower operating voltage and
frequency range, so we do not support changing that since we only have a
reference for our default values. Muchisim models scaling the size of SRAM
memories by increasing the number of banks. Only the active banks consume
static energy, and we model the energy of the multiplexor tree that selects
the bank to grow by 50% at each doubling step. The access latency is also
modeled to increase by 1ns at quadrupling steps beyond 512KiB. All these
modeling options and values can be changed in the simulator parameters. On the
DUT configurations with multiple PUs per tile, the SRAM is shared among them,
but bank conflicts are not modeled yet for this case.

DRAM model: Each compute chiplet may be paired with a memory device of a given
number of channels. By default, Muchisim considers an HBM2E device with eight
64GB/s channels. On the 2.5D integration shown in fig. 1, the memory
controller sits on one edge of the chiplet, adjacent to the closely integrated
DRAM device. The bus connecting the tiles to the memory controller is separate
from the task-communication network. Since many tiles share each memory
channel, the contention is modeled by considering that the memory channel can
only take one request per cycle, and keeping the count of the transactions of
each channel. For example, if a request is done at cycle $X$, but the memory
channel has received $Y$ transactions (where $Y>X$), then the delay if this
request is $Y-X+$ the round-trip to the memory channel. Increasing the
capacity of HBM increases the cost linearly (see section III-E) but not the
device area since it is a 3D stack. The HBM bandwidth available to the
processing units can be defined by changing the number of channels or devices
integrated with each compute chiplet, or by changing the number of tiles or
PUs per tile of these chiplets.

###  III-D Energy and Area Model

table I summarizes some of the energy, latency, and area parameters for
communication links and memory technology, while the full set of default
energy, area and performance parameters of Muchisim can be found in our
repo[67]. The simulator periodically logs performance and energy during the
simulation, in what we call frames, at a rate that can be configured. This
name will become more apparent in section III-F, where we describe our
visualization tool, and the key goal of frames is allowing the user to see the
progress of the simulation (both at the current frame and aggregated). At the
end of the simulation, the simulator also outputs a series of averaged metrics
like throughput, average power, and network traffic at the different levels of
the hierarchy.

The files starting with calc_ under the src/common folder contain the
functions that calculate the performance metrics, energy, area and cost of the
different DUT components based on the simulation of the application execution.

Because for many energy and cost parameters, we do not have a ground truth but
rather assumptions or estimates, Muchisim allows post-processing a given
simulation to re-calculate the energy and cost with different parameters. In
addition to the execution log file, Muchisim creates a separate file with
counters collected during simulation, such as messages hops and memory
accesses at different levels, and instructions executed for each type. The
counters file is then provided as an argument to our separate post-processing
executable (which uses the same configuration and parameter files as the
simulator, with potentially new values), to re-calculate the energy usage of
the execution and the DUT area and cost.

Voltage and area scaling (with frequency) are some of the aspects that can be
re-evaluated during post-processing. By default, Muchisim simplistically
models the area of the PUs and routers to scale by 50% of the increase in
their target frequency. This is a simplistic model that could be refined by
synthesizing particular RTL components with different target frequencies. For
our voltage scaling model, we fit a ridge regression to the frequency and
voltage data from the shmoo plots of chips with 5, 7 and 12nm technology [82,
81, 27]. The current model grows voltage with $0.06+0.13*freq+0.06*node$,
which could adjusted by adding data to src/voltage_model.py.

DRAM integration: The standard integration of DRAM and compute chiplets in
Muchisim is a 2.5D integration via a passive interposer [17, 59]. However,
since the adjacent compute chiplet has dedicated access to the DRAM device,
another possible integration would be having the DRAM device on top of the
compute chiplet, which would need to be redimensioned to behave as an active
interposer, in addition to posing a power and thermal challenge [13, 33].
Muchisim supports studying this mode by adjusting the area, cost, and wire
energy with the DRAM integration, but we do not consider a different latency.
Muchisim reports power density, which can be used to estimate the thermal
feasibility of a 3D integration.

Chiplet PHY and Package I/O are affected by the width, frequency and topology
of the on-chip and off-chip network, respectively. Muchisim configures one
physical network by default, but up to three independent NoCs have been
evaluated (one for each of the task types of the benchmark suite, section
III-G).

TABLE I: Default values of energy (E), bandwidth, latency, and area parameters of links and memory devices modeled in Muchisim. Memory Model Parameters | Values
---|---
SRAM Density |  3.5 MB/mm2 [98]
SRAM R/W Latency & E. |  0.82ns & 0.18 / 0.28 pJ/bit [98]
Cache Tag Read & cmp. E. |  6.3 pJ [98, 100]
HBM2E 4-high Density |  8GB/110mm2 (75 MB/mm2) [46]
Mem.Channels & Bandwidth |  8 x 64 GB/s [46]
Mem.Ctrl-to-HBM RW Latency & E. |  50ns & 3.7pJ/bit [73, 40]
Bitline Refresh Period & E. |  32ms & 0.22pJ/bit [86, 28]
Wire & Link Model Parameters | Values
MCM PHY Areal Density |  690 Gbits/mm2 [6]
MCM PHY Beachfront Density |  880 Gbits/mm [6]
Si. Interposer PHY Areal Density |  1070 Gbits/mm2 [6]
Si. Interposer PHY Beachfront Density |  1780 Gbits/mm [6]
Die-to-Die Link Latency & E. |  4ns & 0.55pJ/bit ($<$25mm) [63]
NoC Wire Latency & E. |  50 ps/mm & 0.15pJ/bit/mm [42]
NoC Router Latency & E. | 500ps & 0.1pJ/bit
I/O Die RX-TX Latency |  20ns [83]
Off-Package Link E. |  1.17pJ/bit (upto 80mm) [97]

###  III-E Cost Model

We also added a cost model to the simulator to study the cost-effectiveness of
different architecture configurations. The cost model—similarly to the energy
model—is decoupled from the runtime simulation process. This is useful to
study how price variations can change the cost-effectiveness of different DUT
configurations.

Silicon cost: By default, we consider 7nm technology for transistors; we
assume that a 300mm wafer with this transistor process costs $6,047 [37]. We
obtain the cost per die by dividing the wafer cost by the number of good dies,
which we calculate using 0.2mm scribes, 4mm edge loss, and 0.07 defects per
$mm^{2}$. We integrate and validate [34] die yield calculations in our cost
model using Murphy’s model. When comparing the cost-effectiveness of the
simulated architecture, we do not include the Non-Recurring Engineering (NRE)
cost of the compute dies since all the options use the same technology.

Packaging cost: We model the cost of the 65nm silicon interposer connecting a
compute die with a DRAM device (including bonding) to be 20% of the price of a
compute die [93], and the cost of an organic substrate to be 10% of the price
of an equal-sized compute die, and the bonding to add an additional 5%
overhead [88, 45].

DRAM cost: While the cost of HBM2 is not disclosed, we made an educated guess
using public sources [37, 75]. By default, the cost models assumes 7.5$/GB,
which is more affordable than when HBM was released in 2017. One could expect
this price to decrease over time as more vendors fabricate HBM or the process
matures [54, 46, 62, 74]. Muchisim’s post-processing tool allows evaluating
the performance-per-dollar of a given simulation in the light of different
DRAM cost scenarios.

###  III-F Visualization Tools

Muchisim comes with two data visualization tools: a command-line interface
(CLI) tool to generate plots including multiple application executions, and a
graphical user interface (GUI) tool—coded in PyQt5—to visualize Muchisim
metrics over time for a single execution. Both tools are written in Python and
use matplotlib to generate the plots.

The CLI tool allows plotting multiple metrics for combinations of DUT
configurations and sizes, and different applications and datasets. These
metrics include runtime, throughput (FLOPS or TEPS), energy (and its
breakdown), cost (in USD), simulator time, arithmetic intensity (FLOPS divided
by data loads or network traffic), overall network traffic (in message hops),
and cache hit-rate, among others. They can be plotted as absolute numbers (as
in fig. 3) or normalized to a baseline configuration (as in fig. 5).

The GUI tool allows visualizing performance counters throughout the
application execution, such as router port collisions and utilization, end-
point contention, PU utilization, cache hit-rate and memory controller
requests and average latency. For these metrics, we can plot various
statistics such as the average, maximum and minimum values across all tiles as
well as boxplot and standard deviation, to observe the work distribution
throughout the execution. This time series also provides insights into the
length and challenges of the tail of the execution, where the maximum and
minimum values are far apart. In addition to plotting statistics about these
performance counters, the GUI can also generate a heatmap of the tile grid,
where the color represents the percentage of the duration of the frame
($\sim$microseconds) that the counter was activated. The frames are then
played in sequence (by creating a gif) to visualize the evolution of that
metric on the tile grid over time. For example, fig. 2 shows that for the
routing activity when using a Mesh (top), a Torus (middle), and a Torus with
reduction trees (bottom). The left panels show the routing activity and the
right panels show the core (PU) activity. Overall, Muchisim’s visualization
tool helps identify the bottlenecks of different applications and datasets,
and how they change with different DUT configurations.

Verbosity: Muchisim supports four levels of output verbosity: (v=0) only
reports aggregated statistical metrics at the end of the execution; (v=1)
reports these metrics for each time frame; (v=2) reports the metrics for each
tile in the grid, which is required to plot heatmaps; (v=3) also shows the
capacity of the input and output queues of each task type for every tile.
Higher verbosity levels increase the log file size and the simulator runtime,
proportionally to the configured frame rate.

|
[width=110mm]3figs/HEAT0_2_64_Kron22_Router_Active/heatmap_Router_Active_frame_149
---
|
[width=110mm]3figs/HEAT0_2_64_Kron22_Core_Active/heatmap_Core_Active_frame_149
---
|
[width=110mm]3figs/HEAT1_2_64_Kron22_Router_Active/heatmap_Router_Active_frame_127
---
|
[width=110mm]3figs/HEAT1_2_64_Kron22_Core_Active/heatmap_Core_Active_frame_127
---
|
[width=110mm]3figs/HEAT5_2_64_Kron22_Router_Active/heatmap_Router_Active_frame_115
---
|
[width=110mm]3figs/HEAT5_2_64_Kron22_Core_Active/heatmap_Core_Active_frame_115
---

Figure 2:  Animation of the router activity when running BFS with R22 on a
Mesh (top), a Torus (middle), and a Torus with Reduction trees (bottom). The
left panels show the routing activity and the right panels show the core (PU)
activity. No router activity can mean that the router has no messages to
route, or that the NoC is clogged and messages are stuck. The animation is
composed of snapshots at a rate of a frame every 40 microseconds (this rate is
configurable prior to simulation). Since this rate is the same for these
plots, the number of frames (50, 28, and 16, from top to bottom) is
proportional to the execution time. The animation can be visualized by opening
this PDF with Adobe (for convenience, we have also added these animations as
gifs in repository[67].) Visualizing the router and core activity
simultaneously helps understand the effect of NoC congestion on core
utilization. In addition, plotting the destination port collisions helps
understand the router activity. We evaluated the version of BFS with barrier
synchronization at the end of each epoch to showcase the effect of the network
on the tail execution time (3 major epochs can be seen). A finer time
resolution allows observing the evolution of the execution in more detail, but
it increases the size of the gif.

###  III-G Benchmark Suite

Applications: Muchisim includes four graph algorithms, two sparse linear
algebra, and two HPC kernels [89]. Breadth-First Search (BFS) determines the
number of hops from a root vertex to all vertices reachable from it; Single-
Source Shortest Path (SSSP) finds the shortest path from the root to each
reachable vertex; PageRank ranks websites based on the potential flow of users
to each page [44]; Weakly Connected Components (WCC) finds and labels each set
of vertices reachable from one to all others in at least one direction (using
graph coloring [85]); Sparse Matrix-Vector Multiplication (SPMV) multiplies a
square sparse matrix with a dense vector. Sparse Matrix-Matrix Multiplication
(SPMM) multiplies a square sparse matrix with a dense matrix and stores the
result in a dense matrix. 3D Fast Fourier Transform (FFT) computes the Fourier
Transform of a 3D tensor. Histogram counts the values that fall within a
series of intervals. The implementation of BFS, SSSP and WCC, supports running
with local or global barrier synchronization at the end of each epoch, where
the next vertices to explore are stored in the frontier.

As throughput, Muchisim reports FLOPS (considering the dataset as arrays of
FP32) and traversed edges per second (TEPS) as $TEPS=m/time$ where $m$ is the
number of edges connected to the vertices in the graph starting from the
search key. When reporting TEPS for SPMV, SPMM, FFT and Histogram, Muchisim
considers the non-zero elements to multiply, and elements to process,
respectively.

Datasets: Muchisim includes ten datasets, including six sizes of the RMAT [48]
graphs—standard on the Graph500 list [57]—RMAT-16, 21, 22, 25, 26, and 27,
which are named after their scale. For example, RMAT-26 (abbreviated as R26 in
section IV-B) contains $2^{26}$, i.e., 67M vertices (V) and 1.3B edges (E),
and has a memory footprint of 12GB. We also include the real-world graphs from
Wikipedia (V=4.2M, E=101M), LiveJournal (V=5.3M, E=79M), Amazon (V=262K,
E=1.2M) and Twitter (V=81K, E=2.4M) [49]. For SPMV and SPMM, the graphs are
seen as a square sparse matrix with V rows and columns and E non-zero
elements. The graphs (as sparse matrices) are stored in the Compressed Sparse
Row (CSR) format without any partitioning, i.e, the dataset contains three
input arrays, one for the values of the non-zeros, one for the column indices
of those non-zeros, and one for the pointers to the beginning of each row in
the previous two arrays. For Histogram, the output array is the count of the
column indices of the non-zeros.

##  IV Results

This section first presents our validation of Muchisim by simulating the
Cerebras Wafer Scale Engine (WSE) [51] and comparing the results with an
existing experimental evaluation [65]. Then, section IV-B shows the speedup of
Muchisim with the number of host threads and its throughput when simulating up
to a million tiles. Finally, section IV-C presents a case study of Muchisim to
explore the optimal memory integration for a given application domain, for
several target metrics.

We include additional experiments and case studies, that we do not have the
space for in the current paper, in our repo[67], where we explore the impact
of (1) the NoC topology, width and frequency, (2) the reduction-tree size (3)
the number of PUs per tile and the PU frequency, (4) integrating DRAM or not,
which sets the minimum parallelization required for a given dataset, and (5)
the size of the tile input and output queues, among others.

###  IV-A Validation

We compare the runtimes reported on the WSE for Fast Fourier Transforms (FFTs)
[65] with the runtimes obtained with our simulator. We compare that with all
their reported datapoints, i.e., parallelizing FFT of $n^{3}$ tensors across
$n^{2}$ processors, for $n$ equal to 32, 64, 128, 256, and 512. In doing so,
we show that Muchisim has high level of scalability and the accuracy is not
impacted by the size of the DUT.

The WSE mounted on the CS-2 [51] is a 7nm-based 46,225mm2 monolithic die with
850,000 cores and 40GB of SRAM. To model the WSE, we configure the DUT as a
package with a single chiplet (32-bit 2D mesh NoC) with no integrated DRAM
[51]. The WSE has a circuit-switched NoC with traffic filtering, while our
simulator models a packet-switched NoC. The advantage of the circuit-switched
NoC is to avoid message headers and simplify the routers, but it requires more
synchronization when setting up a path to send data. To overcome this
difference, we do not consider the destination header in this case, and rather
consider the same amount of data sent by the PUs in both cases. The
performance model for the PUs is set based on their reported numbers [65].

Runtime: The runtimes of the $n^{3}$ FFT problems parallelized across $n^{2}$
tiles reported on the WSE [65] are $1.2\times$ of the runtimes reported by
Muchisim, consistently, for $n$ ranging from 32 to 512 (i.e., from a thousand
to a quarter million tiles). We believe the reason for this slightly
optimistic runtime is that Muchisim does not model the synchronization
overhead of the circuit-switched NoC. We also simulated these datapoints with
up to 32 host threads to measure the speedup of the simulator, shown in fig. 3
for FFT and other applications.

In terms of chip area, the area reported by the simulator is 8.8% larger than
the area of the WSE. We attribute this to WSE’s NoC, where their routers are
likely smaller than the default area of Muchisim’s routers. Regarding energy,
the simulator reports an average power of the processing tiles of a bit over
1KW. While Morenes et. al. [65] did not report energy for the FFT evaluation,
we know that the entire CS-2 system (including cooling) draws up to 20KW [51].
Considering that 512x512 is less than a third of the CS-2, and that PU
utilization is low ($\sim$30%) because of the communication bottleneck of FFT,
the energy estimation seems reasonable.

###  IV-B Parallelization Speedup and Scaling Throughput

Figure 3:  Ratio between the simulator and the DUT runtime, for two DUT sizes
(32x32 and 64x64 tiles, monolithic, connected via a 64-bit Torus), evaluated
with an increasing number of host threads to process the same R22 dataset. The
DUT time is considered as the aggregated runtime of all tiles. The simulator
runtime decreases close to linearly with the number of host threads.

We show the speedup of Muchisim with the number of host threads in fig. 3, for
two DUT sizes, as the ratio between the simulator and the DUT runtime. The
ratio of the host’s simulation time to the simulated runtime of executing
these applications on the DUT decreases from a geomean ratio of 614 to 43 (a
12$\times$ speedup) across the datapoints, when scaling from 2 to 32 host
threads (Intel Xeon Gold 6342 at 2.8Ghz). The speedup is Muchisim’s
parallelization is perfectly linear until the point where each host thread
only processes a couple of tile columns (32-thread 32x32 DUT). fig. 3
demonstrates the efficiency of Muchisim and its parallelization. On a
32-thread host, simulating the execution of applications on the DUT takes only
$43\times$ (on geomean, and down to $21\times$ for FFT) longer than the
expected runtime of every tile of the DUT. This ratio is remarkable
considering that Muchisim performs a functional simulation of data-dependent
workloads, where all NoC routers are evaluated every cycle at the flit level.

Figure 4:  Simulation time (in host seconds) and throughput in DUT operations
and NoC message fits routed per second, for scaling DUT sizes from a thousand
to a million tiles (32x32 tiles per chiplet, connected via a 64-bit
hierarchical 2D Torus) when processing the R26 dataset. The $2^{10}$ and
$2^{12}$ datapoints are evaluated with 16 and 32 host threads, respectively,
of a single-socket Intel Xeon Gold 6342 at 2.8Ghz, the datapoints from
$2^{14}$ to $2^{18}$ with 64 and $2^{20}$ with 128 threads, respectively, of a
4-socket Intel Xeon Gold 6230 at 2.1GHz.

fig. 4 shows the simulation time and throughput of Muchisim (in host seconds)
when using increasingly large DUT configurations (from a thousand to a million
tiles) to execute the R26 dataset. The first three datapoints see a decreased
or nearly constant simulation time because we are increasing the
parallelization of Muchisim from 16 to 64 host threads. The NoC simulation
throughput ranges from a few million message flits routed per second
(PageRank) to over 40 million (SSSP). Note that this NoC throughput does not
count the attemps to routed that fail due to contention (two input ports
compete for the same output) or backpressure (the buffer of the output port is
full). The throughput of operations—natively executed on the host—reaches up
to a few billion.

The absolute runtimes (fig. 4, blue) show that even the simulations of
million-tile DUT configurations on a billion-element dataset—unattainable
before Muchisim—are simulated in under half a day for BFS, SPMV and Histo, and
in up to two days for SSSP and PageRank, on a single server-class chip. FFT
(not shown in fig. 4 since it does not use a graph but a $n^{3}$ tensor for
$n^{2}$ parallelization) is simulated under 100 seconds for up to $n=2^{7}$,
i.e, $2^{14}$ tiles, but it also takes a day to simulate a million tiles on a
billion-element ($1024^{3}$) tensor, where the all-to-all communication of FFT
dominates the runtime.

###  IV-C Case Study: Memory Integration

Figure 5: Performance, energy efficiency, and performance per dollar
improvements of the DUT using different SRAM sizes and number of tiles per HBM
channel, over a baseline of 64KB SRAM and 128 tiles per HBM channel (T/C). In
this study, a chiplet is always attached to a single 8-channel HBM device, and
thus, the number of tiles per chiplet (16x16 or 32x32) determines the ratio of
tiles per HBM channel. The R25 dataset is studied on a DUT with 1024 tiles;
the dataset footprint per tile ranges from 4MB (Histogram) to 8MB per tile
(SPMV).

As a case study, we studied the performance, energy efficiency, and
performance per dollar improvements of different SRAM sizes and numbers of
tiles per chiplet (same number of tiles in total) and show the results in fig.
5. Since all applications but SPMM have a very low arithmetic intensity with
respect to the data movement, they require a large amount of memory bandwidth.
We observe, (fig. 5 top), a strong performance increase with SRAM size,
$3.5\times$ on geomean when increasing from 64KB to 256KB, with the same
chiplet configuration of 32x32 tiles, and an additional $2\times$ increase
when using 16x16-tile chiplets (32T/C), for the same SRAM size of 256KB. The
hit-rate of the data cache (not shown) increases from 83% to 95% on geomean
with the SRAM size increase, but since the effective bandwidth of a tile is
$SRAM\\_bandwidth\times hit\\_rate+DRAM\\_bandwidth\times(1-hit\\_rate)$, the
hit-rate has a larger impact on the bandwidth when more tiles share the same
DRAM channel (128T/C).

The throughput per dollar is lower with 16x16-tile chiplets for all
applications but SSSP and SPMV, due to the extra cost of having four times
more HBM devices. SPMM, with over an order of magnitude more arithmetic
intensity than the rest of the applications for the dataset evaluated (K25),
showcases that when targetting performance per dollar, the optimal memory
integration depends on the application domain.

##  V Conclusion

Muchisim allows evaluating tiled manycore architectures with different
hierarchical organizations, network topologies, and memory integrations. It
also supports various parallelization strategies (do-all and task-based) and
communication primitives (e.g., message-passing and reduction trees).
Moreover, Muchisim enables exploring the optimal balance between resources
dedicated to compute, memory and network, for different target metrics and
application domains. Muchisim comes with a benchmark of eight memory- and
communication-intensive applications and ten datasets of several sizes and
characteristics. Muchisim also includes two data visualization tools to
facilitate the analysis of the simulation results.

We have created the scripts for the artifact evaluation of the experiments
presented in this paper. Our repository[67] includes the Muchisim simulation
framework, applications, datasets and visualization tools; we are planning to
document these further with the publication of this paper to allow researchers
to explore the design space of scale-out manycore architecture as well as
extend it with their own novel techniques. Future work on the simulator itself
includes multi-node MPI parallelization and support for low-diameter cluster
interconnects, to enable simulating even larger systems.

## Acknowledgements

This material is based upon work supported in part by National Science
Foundation (NSF) award No. 1763838, and based on research sponsored by the Air
Force Research Laboratory (AFRL) and Defense Advanced Research Projects Agency
(DARPA) under agreement FA8650-18-2-7862. In addition, Martonosi received
separate NSF support while serving at NSF as an IPA rotator. 333The U.S.
Government is authorized to reproduce and distribute reprints for Governmental
purposes notwithstanding any copyright notation thereon. The views and
conclusions contained herein are those of the authors and should not be
interpreted as necessarily representing the official policies or endorsements,
either expressed or implied, of NSF, AFRL, DARPA or the U.S. Government.

## References

  * [1] M. Abeydeera and D. Sanchez, “Chronos: Efficient speculative parallelism for accelerators,” in _Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2020, pp. 1247–1262.
  * [2] N. R. Adiga, G. Almási, G. S. Almasi, Y. Aridor, R. Barik, D. Beece, R. Bellofatto, G. Bhanot, R. Bickford, M. Blumrich _et al._ , “An overview of the bluegene/l supercomputer,” in _SC’02: Proceedings of the 2002 ACM/IEEE Conference on Supercomputing_. IEEE, 2002, pp. 60–60.
  * [3] J. H. Ahn, S. Li, S. O, and N. P. Jouppi, “Mcsima+: A manycore simulator with application-level+ simulation and detailed microarchitecture modeling,” in _2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_ , 2013, pp. 74–85.
  * [4] J. Ahn, S. Hong, S. Yoo, O. Mutlu, and K. Choi, “A scalable processing-in-memory accelerator for parallel graph processing,” in _Proceedings of the 42nd Annual International Symposium on Computer Architecture_ , 2015, pp. 105–117.
  * [5] Ampere Computing, “Ampereone 192-core server processor,” https://amperecomputing.com/products/processors.
  * [6] S. Ardalan, B. Vinnikota, T. Arabi, and E. Alon, “What is the right die-to-die interface? a comparison study,” 2022, https://www.opencompute.org/events/past-events/hipchips-chiplet-workshop-isca-conference.
  * [7] E. Argollo, A. Falcón, P. Faraboschi, M. Monchiero, and D. Ortega, “Cotson: Infrastructure for full system simulation,” _SIGOPS Oper. Syst. Rev._ , vol. 43, no. 1, p. 52–61, jan 2009. [Online]. Available: https://doi.org/10.1145/1496909.1496921
  * [8] T. Austin, E. Larson, and D. Ernst, “Simplescalar: an infrastructure for computer system modeling,” _Computer_ , vol. 35, no. 2, pp. 59–67, 2002.
  * [9] A. Benner, “Optical interconnect opportunities in supercomputers and high end computing,” in _OFC/NFOEC_. IEEE, 2012, pp. 1–60.
  * [10] D. Bertozzi and L. Benini, “Xpipes: a network-on-chip architecture for gigascale systems-on-chip,” _IEEE Circuits and Systems Magazine_ , vol. 4, no. 2, pp. 18–31, 2004.
  * [11] M. Besta and T. Hoefler, “Slim fly: A cost effective low-diameter network topology,” in _SC’14: proceedings of the international conference for high performance computing, networking, storage and analysis_. IEEE, 2014, pp. 348–359.
  * [12] N. Binkert, B. Beckmann, G. Black, S. K. Reinhardt, A. Saidi, A. Basu, J. Hestness, D. R. Hower, T. Krishna, S. Sardashti, R. Sen, K. Sewell, M. Shoaib, N. Vaish, M. D. Hill, and D. A. Wood, “The gem5 simulator,” _SIGARCH Comput. Archit. News_ , vol. 39, no. 2, p. 1–7, aug 2011. [Online]. Available: https://doi.org/10.1145/2024716.2024718
  * [13] B. Black, “Die stacking is happening,” in _Intl. Symp. on Microarchitecture, Davis, CA_ , 2013.
  * [14] V. Catania, A. Mineo, S. Monteleone, M. Palesi, and D. Patti, “Cycle-accurate network on chip simulation with noxim,” _ACM Trans. Model. Comput. Simul._ , vol. 27, no. 1, aug 2016. [Online]. Available: https://doi.org/10.1145/2953878
  * [15] Cerebras Systems Inc., “The second generation wafer scale engine,” https://cerebras.net/wp-content/uploads/2021/04/Cerebras-CS-2-Whitepaper.pdf.
  * [16] T.-J. Chang, A. Li, F. Gao, T. Ta, G. Tziantzioulis, Y. Ou, M. Wang, J. Tu, K. Xu, P. J. Jackson, A. Ning, G. Chirkov, M. Orenes-Vera, S. Agwa, X. Yan, E. Tang, J. Balkind, C. Batten, and D. Wentzlaff, “CIFER: A 12nm, 16mm2, 22-core soc with a 1541 lut6/mm2 1.92 mops/lut, fully synthesizable, cachecoherent, embedded fpga,” in _2023 IEEE Custom Integrated Circuits Conference (CICC)_ , 2023, pp. 1–2. [Online]. Available: https://doi.org/10.1109/CICC57935.2023.10121294
  * [17] J. Choquette and W. Gandhi, “Nvidia A100 GPU: Performance & innovation for GPU computing,” in _2020 IEEE Hot Chips 32 Symposium (HCS)_. IEEE Computer Society, 2020, pp. 1–43.
  * [18] V. Dadu, S. Liu, and T. Nowatzki, “Polygraph: Exposing the value of flexibility for graph processing accelerators,” in _2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA)_. IEEE, 2021, pp. 595–608.
  * [19] V. Dadu and T. Nowatzki, “Taskstream: accelerating task-parallel workloads by recovering program structure,” in _Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2022, pp. 1–13.
  * [20] S. Davidson, S. Xie, C. Torng, K. Al-Hawai, A. Rovinski, T. Ajayi, L. Vega, C. Zhao, R. Zhao, S. Dai, A. Amarnath, B. Veluri, P. Gao, A. Rao, G. Liu, R. K. Gupta, Z. Zhang, R. Dreslinski, C. Batten, and M. B. Taylor, “The celerity open-source 511-core risc-v tiered accelerator fabric: Fast architectures and design methodologies for fast chips,” _IEEE Micro_ , vol. 38, no. 2, pp. 30–41, 2018.
  * [21] M. Emani, V. Vishwanath, C. Adams, M. E. Papka, R. Stevens, L. Florescu, S. Jairath, W. Liu, T. Nama, and A. Sujeeth, “Accelerating scientific applications with sambanova reconfigurable dataflow architecture,” _Computing in Science & Engineering_, vol. 23, no. 2, pp. 114–119, 2021.
  * [22] Esperanto Technologies, “Esperanto’s et-minion on-chip risc-v cores,” https://www.esperanto.ai/technology/.
  * [23] A. Feldmann and D. Sanchez, “Spatula: A hardware accelerator for sparse matrix factorization,” in _Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture_ , ser. MICRO ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 91–104. [Online]. Available: https://doi.org/10.1145/3613424.3623783
  * [24] K. Feng, Y. Ye, and J. Xu, “A formal study on topology and floorplan characteristics of mesh and torus-based optical networks-on-chip,” _Microprocessors and Microsystems_ , vol. 37, no. 8, pp. 941–952, 2013.
  * [25] D. Fox, J. M. Diaz, and X. Li, “A gem5 implementation of the sequential codelet model: Reducing overhead and expanding the software memory interface,” in _Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis (SC-W 2023), November 12–17, 2023, Denver, CO, USA_ , 2023.
  * [26] Y. Fu and D. Wentzlaff, “PriME: A parallel and distributed simulator for thousand-core chips,” in _ISPASS_. IEEE Press, March 2014.
  * [27] F. Gao, T.-J. Chang, A. Li, M. Orenes-Vera, D. Giri, P. J. Jackson, A. Ning, G. Tziantzioulis, J. Zuckerman, J. Tu _et al._ , “DECADES: A 67mm 2, 1.46 tops, 55 giga cache-coherent 64-bit risc-v instructions per second, heterogeneous manycore soc with 109 tiles including accelerators, intelligent storage, and efpga in 12nm finfet,” in _2023 IEEE Custom Integrated Circuits Conference (CICC)_. IEEE, 2023, pp. 1–2.
  * [28] S. Ghose, A. G. Yaglikçi, R. Gupta, D. Lee, K. Kudrolli, W. X. Liu, H. Hassan, K. K. Chang, N. Chatterjee, A. Agrawal _et al._ , “What your dram power models are not telling you: Lessons from a detailed experimental study,” _Proceedings of the ACM on Measurement and Analysis of Computing Systems_ , vol. 2, no. 3, pp. 1–41, 2018.
  * [29] L. Gwennap, “Groq rocks neural networks,” _Microprocessor Report, Tech. Rep., jan_ , 2020.
  * [30] T. J. Ham, L. Wu, N. Sundaram, N. Satish, and M. Martonosi, “Graphicionado: A high-performance and energy-efficient accelerator for graph analytics,” in _Proceedings of the 49th Annual International Symposium on Microarchitecture_ , ser. MICRO, 2016. [Online]. Available: https://doi.org/10.1109/MICRO.2016.7783759
  * [31] N. Hardavellas, S. Somogyi, T. F. Wenisch, R. E. Wunderlich, S. Chen, J. Kim, B. Falsafi, J. C. Hoe, and A. G. Nowatzyk, “Simflex: A fast, accurate, flexible full-system simulation framework for performance evaluation of server architecture,” _SIGMETRICS Perform. Eval. Rev._ , vol. 31, no. 4, p. 31–34, mar 2004. [Online]. Available: https://doi.org/10.1145/1054907.1054914
  * [32] M. Horro, G. Rodríguez, and J. Touriño, “Simulating the network activity of modern manycores,” _IEEE Access_ , vol. 7, pp. 81 195–81 210, 2019.
  * [33] X. Hu, D. Stow, and Y. Xie, “Die stacking is happening,” _IEEE micro_ , vol. 38, no. 1, pp. 22–28, 2018.
  * [34] Isine, “Die yield calculator,” https://isine.com/resources/die-yield-calculator/.
  * [35] M. C. Jeffrey, S. Subramanian, C. Yan, J. Emer, and D. Sanchez, “A scalable architecture for ordered parallelism,” in _Proceedings of the 48th International Symposium on Microarchitecture_ , ser. MICRO-48. New York, NY, USA: Association for Computing Machinery, 2015, p. 228–241. [Online]. Available: https://doi.org/10.1145/2830772.2830777
  * [36] N. Jiang, D. U. Becker, G. Michelogiannakis, J. Balfour, B. Towles, D. E. Shaw, J. Kim, and W. J. Dally, “A detailed and flexible cycle-accurate network-on-chip simulator,” in _2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_ , 2013, pp. 86–96.
  * [37] S. W. Jones, “Lithovision: Economics in the 3d era,” https://semiwiki.com/wp-content/uploads/2020/03/Lithovision-2020.pdf.
  * [38] N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers _et al._ , “In-datacenter performance analysis of a tensor processing unit,” in _Proceedings of the 44th annual international symposium on computer architecture_ , 2017, pp. 1–12.
  * [39] D. C. Jung, S. Davidson, C. Zhao, D. Richmond, and M. B. Taylor, “Ruche networks: Wire-maximal, no-fuss nocs: Special session paper,” in _2020 14th IEEE/ACM International Symposium on Networks-on-Chip (NOCS)_. IEEE, 2020, pp. 1–8.
  * [40] D.-H. Kim, B. Song, H.-a. Ahn, W. Ko, S. Do, S. Cho, K. Kim, S.-H. Oh, H.-Y. Joo, G. Park, J.-H. Jang, Y.-H. Kim, D. Lee, J. Jung, Y. Kwon, Y. Kim, J. Jung, S. O, S. Lee, J. Lim, J. Son, J. Min, H. Do, J. Yoon, I. Hwang, J. Park, H. Shim, S. Yoon, D. Choi, J. Lee, S. Woo, E. Hong, J. Choi, J.-S. Kim, S. Han, J. Bang, B. Park, J. Kim, S.-K. Choi, G.-H. Han, Y.-C. Sung, W.-I. Bae, J.-D. Lim, S. Lee, C. Yoo, S. J. Hwang, and J. Lee, “A 16gb 9.5gb/s/pin lpddr5x sdram with low-power schemes exploiting dynamic voltage-frequency scaling and offset-calibrated readout sense amplifiers in a fourth generation 10nm dram process,” in _2022 IEEE International Solid- State Circuits Conference (ISSCC)_ , vol. 65, 2022, pp. 448–450.
  * [41] J. Kim, W. J. Dally, S. Scott, and D. Abts, “Technology-driven, highly-scalable dragonfly topology,” _ACM SIGARCH Computer Architecture News_ , vol. 36, no. 3, pp. 77–88, 2008.
  * [42] S. Kim, S. Kim, K. Cho, T. Shin, H. Park, D. Lho, S. Park, K. Son, G. Park, and J. Kim, “Processing-in-memory in high bandwidth memory (pim-hbm) architecture with energy-efficient and low latency channels for high bandwidth system,” in _2019 IEEE 28th Conference on Electrical Performance of Electronic Packaging and Systems (EPEPS)_ , 2019, pp. 1–3.
  * [43] S. Knowles, “Graphcore,” in _2021 IEEE Hot Chips 33 Symposium (HCS)_. IEEE, 2021, pp. 1–25.
  * [44] P. Lawrence, B. Sergey, R. Motwani, and T. Winograd, “The pagerank citation ranking: Bringing order to the web,” Stanford University, Technical Report, 1998.
  * [45] C.-C. Lee, C. Hung, C. Cheung, P.-F. Yang, C.-L. Kao, D.-L. Chen, M.-K. Shih, C.-L. C. Chien, Y.-H. Hsiao, L.-C. Chen, M. Su, M. Alfano, J. Siegel, J. Din, and B. Black, “An overview of the development of a gpu with integrated hbm on silicon interposer,” in _2016 IEEE 66th Electronic Components and Technology Conference (ECTC)_ , 2016, pp. 1439–1444.
  * [46] D. U. Lee, H. S. Cho, J. Kim, Y. J. Ku, S. Oh, C. D. Kim, H. W. Kim, W. Y. Lee, T. K. Kim, T. S. Yun _et al._ , “22.3 a 128gb 8-high 512gb/s hbm2e dram with a pseudo quarter bank structure, power dispersion and an instruction-based at-speed pmbist,” in _2020 IEEE International Solid-State Circuits Conference-(ISSCC)_. IEEE, 2020, pp. 334–336.
  * [47] C. E. Leiserson, “Fat-trees: Universal networks for hardware-efficient supercomputing,” _IEEE transactions on Computers_ , vol. 100, no. 10, pp. 892–901, 1985.
  * [48] J. Leskovec, D. Chakrabarti, J. Kleinberg, C. Faloutsos, and Z. Ghahramani, “Kronecker graphs: An approach to modeling networks,” _Journal of Machine Learning Reseach (JMLR)_ , vol. 11, pp. 985–1042, Mar. 2010.
  * [49] J. Leskovec and A. Krevl, “SNAP Datasets: Stanford large network dataset collection,” http://snap.stanford.edu/data, Jun. 2014.
  * [50] A. Li, T.-J. Chang, F. Gao, T. Ta, G. Tziantzioulis, Y. Ou, M. Wang, J. Tu, K. Xu, P. Jackson, A. Ning, G. Chirkov, M. Orenes-Vera, S. Agwa, X. Yan, E. Tang, J. Balkind, C. Batten, and D. Wentzlaff, “Cifer: A cache-coherent 12nm 16mm2 soc with four 64-bit risc-v application cores, 18 32-bit risc-v compute cores, and a 1541 lut6/mm2 synthesizable efpga,” _IEEE Solid-State Circuits Letters_ , pp. 1–1, 2023.
  * [51] S. Lie, “Multi-million core, multi-wafer AI cluster,” in _2021 IEEE Hot Chips 33 Symposium (HCS)_. IEEE Computer Society, 2021, pp. 1–41.
  * [52] A. Manocha, T. Sorensen, E. Tureci, O. Matthews, J. L. Aragón, and M. Martonosi, “Graphattack: Optimizing data supply for graph applications on in-order multicore architectures,” _ACM Transactions on Architecture and Code Optimization (TACO)_ , vol. 18, no. 4, pp. 1–26, 2021.
  * [53] O. Matthews, A. Manocha, D. Giri, M. Orenes-Vera, E. Tureci, T. Sorensen, T. J. Ham, J. L. Aragon, L. P. Carloni, and M. Martonosi, “Mosaicsim: A lightweight, modular simulator for heterogeneous systems,” in _2020 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_. IEEE, 2020, pp. 136–148.
  * [54] Micron, “High Bandwidth Memory with ECC,” 2018, https://media-www.micron.com/-/media/client/global/documents/products/data-sheet/dram/hbm2e/8gb_and_16gb_hbm2e_dram.pdf.
  * [55] J. E. Miller, H. Kasture, G. Kurian, C. Gruenwald, N. Beckmann, C. Celio, J. Eastep, and A. Agarwal, “Graphite: A distributed parallel simulator for multicores,” in _HPCA_. IEEE Press, 2010.
  * [56] F. Muñoz-Martínez, R. Garg, M. Pellauer, J. L. Abellán, M. E. Acacio, and T. Krishna, “Flexagon: A multi-dataflow sparse-sparse matrix multiplication accelerator for efficient dnn processing,” in _Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3_ , 2023, pp. 252–265.
  * [57] R. C. Murphy, K. B. Wheeler, B. W. Barrett, and J. A. Ang, “Introducing the Graph 500,” http://www.graph500.org/specifications, Cray User’s Group (CUG), 2010.
  * [58] S. Naffziger, N. Beck, T. Burd, K. Lepak, G. H. Loh, M. Subramony, and S. White, “Pioneering chiplet technology and design for the amd epyc™ and ryzen™ processor families,” in _Proceedings of the 48th Annual International Symposium on Computer Architecture_ , ser. ISCA ’21. IEEE Press, 2021, p. 57–70.
  * [59] N. Nassif, A. O. Munch, C. L. Molnar, G. Pasdast, S. V. Lyer, Z. Yang, O. Mendoza, M. Huddart, S. Venkataraman, S. Kandula _et al._ , “Sapphire rapids: The next-generation intel xeon scalable processor,” in _2022 IEEE International Solid-State Circuits Conference (ISSCC)_ , vol. 65. IEEE, 2022, pp. 44–46.
  * [60] Q. M. Nguyen and D. Sanchez, “Pipette: Improving core utilization on irregular applications through intra-core pipeline parallelism,” in _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 2020, pp. 596–608.
  * [61] Q. M. Nguyen and D. Sanchez, “Fifer: Practical acceleration of irregular applications on reconfigurable architectures,” in _MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture_ , ser. MICRO ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 1064–1077. [Online]. Available: https://doi.org/10.1145/3466752.3480048
  * [62] C.-S. Oh, K. C. Chun, Y.-Y. Byun, Y.-K. Kim, S.-Y. Kim, Y. Ryu, J. Park, S. Kim, S. Cha, D. Shin _et al._ , “22.1 a 1.1 v 16gb 640gb/s hbm2e dram with a data-bus window-extension technique and a synergetic on-die ecc scheme,” in _2020 IEEE International Solid-State Circuits Conference-(ISSCC)_. IEEE, 2020, pp. 330–332.
  * [63] Open Compute Group, “Bunch of wires phy specification,” https://opencomputeproject.github.io/ODSA-BoW/bow_specification.html.
  * [64] M. Orenes-Vera, A. Manocha, J. Balkind, F. Gao, J. L. Aragón, D. Wentzlaff, and M. Martonosi, “Tiny but mighty: designing and realizing scalable latency tolerance for manycore socs.” in _ISCA_ , 2022, pp. 817–830.
  * [65] M. Orenes-Vera, I. Sharapov, R. Schreiber, M. Jacquelin, P. Vandermersch, and S. Chetlur, “Wafer-scale fast fourier transforms,” in _Proceedings of the 37th International Conference on Supercomputing_ , ser. ICS’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 180–191. [Online]. Available: https://doi.org/10.1145/3577193.3593708
  * [66] M. Orenes-Vera, E. Tureci, M. Martonosi, and D. Wentzlaff, “DCRA: A distributed chiplet-based reconfigurable architecture for irregular applications,” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2311.15443
  * [67] M. Orenes-Vera, E. Tureci, M. Martonosi, and D. Wentzlaff, “MuchiSim simulation framework and artifacts,” 2023, https://github.com/PrincetonUniversity/muchisim.git.
  * [68] M. Orenes-Vera, E. Tureci, D. Wentzlaf, and M. Martonosi, “Massive data-centric parallelism in the chiplet era,” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2304.09389
  * [69] M. Orenes-Vera, E. Tureci, D. Wentzlaff, and M. Martonosi, “Dalorex: A data-local program execution and architecture for memory-bound applications,” in _2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2023, pp. 718–730.
  * [70] M. Orenes-Vera, E. Tureci, D. Wentzlaff, and M. Martonosi, “Tascade: Hardware support for atomic-free, asynchronous and efficient reduction trees,” 2023. [Online]. Available: https://doi.org/10.48550/arxiv.2311.15810
  * [71] Y. Ou, S. Agwa, and C. Batten, “Implementing low-diameter on-chip networks for manycore processors using a tiled physical design methodology,” in _2020 14th IEEE/ACM International Symposium on Networks-on-Chip (NOCS)_. IEEE, 2020, pp. 1–8.
  * [72] M. M. Ozdal, S. Yesil, T. Kim, A. Ayupov, J. Greth, S. Burns, and O. Ozturk, “Energy efficient architecture for graph analytics accelerators,” _ACM SIGARCH Computer Architecture News_ , vol. 44, no. 3, pp. 166–177, 2016.
  * [73] M. O’Connor, N. Chatterjee, D. Lee, J. Wilson, A. Agrawal, S. W. Keckler, and W. J. Dally, “Fine-grained dram: Energy-efficient dram for extreme bandwidth systems,” in _2017 50th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 2017, pp. 41–54.
  * [74] M.-J. Park, H. S. Cho, T.-S. Yun, S. Byeon, Y. J. Koo, S. Yoon, D. U. Lee, S. Choi, J. Park, J. Lee _et al._ , “A 192-gb 12-high 896-gb/s hbm3 dram with a tsv auto-calibration scheme and machine-learning-based layout optimization,” in _2022 IEEE International Solid-State Circuits Conference (ISSCC)_ , vol. 65. IEEE, 2022, pp. 444–446.
  * [75] A. Patrizio, “High-bandwidth memory (hbm) delivers impressive performance gains,” https://semiengineering.com/whats-next-for-high-bandwidth-memory/.
  * [76] G. Posluns, Y. Zhu, G. Zhang, and M. C. Jeffrey, “A scalable architecture for reprioritizing ordered parallelism,” in _Proceedings of the 49th Annual International Symposium on Computer Architecture_ , ser. ISCA ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 437–453. [Online]. Available: https://doi.org/10.1145/3470496.3527387
  * [77] V. Puente, J. Gregorio, and R. Beivide, “Sicosys: an integrated framework for studying interconnection network performance in multiprocessor systems,” in _Proceedings 10th Euromicro Workshop on Parallel, Distributed and Network-based Processing_ , 2002, pp. 15–22.
  * [78] S. Rahman, N. Abu-Ghazaleh, and R. Gupta, “Graphpulse: An event-driven hardware accelerator for asynchronous graph processing,” in _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 2020, pp. 908–921.
  * [79] A. F. Rodrigues, K. S. Hemmert, B. W. Barrett, C. Kersey, R. Oldfield, M. Weston, R. Risen, J. Cook, P. Rosenfeld, E. Cooper-Balis, and B. Jacob, “The structural simulation toolkit,” _SIGMETRICS Perform. Eval. Rev._ , vol. 38, no. 4, p. 37–42, mar 2011. [Online]. Available: https://doi.org/10.1145/1964218.1964225
  * [80] D. Sanchez and C. Kozyrakis, “Zsim: Fast and accurate microarchitectural simulation of thousand-core systems,” _ACM SIGARCH Computer architecture news_ , vol. 41, no. 3, pp. 475–486, 2013.
  * [81] D. Schor, “TSMC demonstrates a 7nm ARM-based chiplet design for HPC,” 2019, https://fuse.wikichip.org/news/2446/tsmc-demonstrates-a-7nm-arm-based-chiplet-design-for-hpc/.
  * [82] D. Schor, “TSMC Details 5 nm,” 2020, https://fuse.wikichip.org/news/3398/tsmc-details-5-nm/.
  * [83] D. D. Sharma, “Pci express 6.0 specification: A low-latency, high-bandwidth, high-reliability, and cost-effective interconnect with 64.0 gt/s pam-4 signaling,” _IEEE Micro_ , vol. 41, no. 1, pp. 23–29, 2020.
  * [84] K. Shkurko, T. Grant, E. Brunvand, D. Kopta, J. Spjut, E. Vasiou, I. Mallett, and C. Yuksel, “Simtrax: Simulation infrastructure for exploring thousands of cores,” in _Proceedings of the 2018 on Great Lakes Symposium on VLSI_ , 2018, pp. 503–506.
  * [85] G. M. Slota, S. Rajamanickam, and K. Madduri, “BFS and coloring-based parallel algorithms for strongly connected components and related problems,” in _2014 IEEE 28th International Parallel and Distributed Processing Symposium, Phoenix, AZ, USA, May 19-23, 2014_. IEEE Computer Society, 2014, pp. 550–559. [Online]. Available: https://doi.org/10.1109/IPDPS.2014.64
  * [86] K. Sohn, W.-J. Yun, R. Oh, C.-S. Oh, S.-Y. Seo, M.-S. Park, D.-H. Shin, W.-C. Jung, S.-H. Shin, J.-M. Ryu, H.-S. Yu, J.-H. Jung, H. Lee, S.-Y. Kang, Y.-S. Sohn, J.-H. Choi, Y.-C. Bae, S.-J. Jang, and G. Jin, “A 1.2 v 20 nm 307 gb/s hbm dram with at-speed wafer-level io test scheme and adaptive refresh considering temperature distribution,” _IEEE Journal of Solid-State Circuits_ , vol. 52, no. 1, pp. 250–260, 2017.
  * [87] T. Sorensen, A. Manocha, E. Tureci, M. Orenes-Vera, J. L. Aragón, and M. Martonosi, “A simulator and compiler framework for agile hardware-software co-design evaluation and exploration,” in _2020 IEEE/ACM International Conference On Computer Aided Design (ICCAD)_. IEEE, 2020, pp. 1–9.
  * [88] D. Stow, Y. Xie, T. Siddiqua, and G. H. Loh, “Cost-effective design of scalable high-performance systems using active and passive interposers,” in _2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_ , 2017, pp. 728–735.
  * [89] J. A. Stratton, C. Rodrigues, I.-J. Sung, N. Obeid, L.-W. Chang, N. Anssari, G. D. Liu, and W.-m. Hwu, “Parboil: A revised benchmark suite for scientific and commercial throughput computing,” University of Illinois at Urbana-Champaign, Tech. Rep. IMPACT-12-01, 2012.
  * [90] N. Talati, K. May, A. Behroozi, Y. Yang, K. Kaszyk, C. Vasiladiotis, T. Verma, L. Li, B. Nguyen, J. Sun, J. M. Morton, A. Ahmadi, T. Austin, M. O’Boyle, S. Mahlke, T. Mudge, and R. Dreslinski, “Prodigy: Improving the memory latency of data-indirect irregular workloads using hardware-software co-design,” in _2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. IEEE, 2021, pp. 654–667.
  * [91] E. Talpes, D. Williams, and D. D. Sarma, “Dojo: The microarchitecture of tesla exa-scale computer,” in _2022 IEEE Hot Chips 34 Symposium (HCS)_. IEEE Computer Society, 2022, pp. 1–28.
  * [92] Z. Tan, A. Waterman, R. Avizienis, Y. Lee, H. Cook, D. Patterson, and K. Asanović, “Ramp gold: An fpga-based architecture simulator for multiprocessors,” in _Proceedings of the 47th Design Automation Conference_ , ser. DAC ’10. New York, NY, USA: Association for Computing Machinery, 2010, p. 463–468. [Online]. Available: https://doi.org/10.1145/1837274.1837390
  * [93] T. Tang and Y. Xie, “Cost-aware exploration for chiplet-based architecture with advanced packaging technologies,” _arXiv preprint arXiv:2206.07308_ , 2022.
  * [94] J. Vasiljevic, L. Bajic, D. Capalija, S. Sokorac, D. Ignjatovic, L. Bajic, M. Trajkovic, I. Hamer, I. Matosevic, A. Cejkov, U. Aydonat, T. Zhou, S. Z. Gilani, A. Paiva, J. Chu, D. Maksimovic, S. A. Chin, Z. Moudallal, A. Rakhmati, S. Nijjar, A. Bhullar, B. Drazic, C. Lee, J. Sun, K.-M. Kwong, J. Connolly, M. Dooley, H. Farooq, J. Y. T. Chen, M. Walker, K. Dabiri, K. Mabee, R. S. Lal, N. Rajatheva, R. Retnamma, S. Karodi, D. Rosen, E. Munoz, A. Lewycky, A. Knezevic, R. Kim, A. Rui, A. Drouillard, and D. Thompson, “Compute substrate for software 2.0,” _IEEE Micro_ , vol. 41, no. 2, pp. 50–55, 2021.
  * [95] Z. Wang, C. Liu, N. Beckmann, and T. Nowatzki, “Affinity alloc: Taming not-so near-data computing,” in _Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture_ , ser. MICRO ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 784–799. [Online]. Available: https://doi.org/10.1145/3613424.3623778
  * [96] T. Wei, N. Turtayeva, M. Orenes-Vera, O. Lonkar, and J. Balkind, “Cohort: Software-Oriented Acceleration for Heterogeneous SoCs,” ser. ASPLOS 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 105–117. [Online]. Available: https://doi.org/10.1145/3582016.3582059
  * [97] J. Wilson, “High-bandwidth density, energy-efficient, short-reach signaling that enables massively scalable parallelism,” 2022, https://www.opencompute.org/events/past-events/hipchips-chiplet-workshop-isca-conference.
  * [98] Y. Yokoyama, M. Tanaka, K. Tanaka, M. Morimoto, M. Yabuuchi, Y. Ishii, and S. Tanaka, “A 29.2 mb/mm2 ultra high density sram macro using 7nm finfet technology with dual-edge driven wordline/bitline and write/read-assist circuit,” in _2020 IEEE Symposium on VLSI Circuits_ , 2020, pp. 1–2.
  * [99] J. Zarrin, R. L. Aguiar, and J. P. Barraca, “Manycore simulation for peta-scale system design: Motivation, tools, challenges and prospects,” _Simulation Modelling Practice and Theory_ , vol. 72, pp. 168–201, Mar. 2017. [Online]. Available: https://doi.org/10.1016/j.simpat.2016.12.014
  * [100] F. Zaruba and L. Benini, “The cost of application-class processing: Energy and performance analysis of a linux-ready 1.7-ghz 64-bit risc-v core in 22-nm fdsoi technology,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , vol. 27, no. 11, pp. 2629–2640, Nov 2019, https://github.com/openhwgroup/cva6.
  * [101] F. Zaruba, F. Schuiki, and L. Benini, “Manticore: A 4096-core risc-v chiplet architecture for ultraefficient floating-point computing,” _IEEE Micro_ , vol. 41, no. 2, pp. 36–42, 2020.
  * [102] G. Zheng, G. Kakulapati, and L. Kale, “Bigsim: a parallel simulator for performance prediction of extremely large parallel machines,” in _18th International Parallel and Distributed Processing Symposium, 2004. Proceedings._ , 2004, pp. 78–.
  * [103] Y. Zhuo, C. Wang, M. Zhang, R. Wang, D. Niu, Y. Wang, and X. Qian, “Graphq: Scalable pim-based graph processing,” in _Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture_ , 2019, pp. 712–725.