# Development of a highly pixelated direct charge sensor, _Topmetal-I_ , for
ionizing radiation imaging

Yan Fan  Chaosong Gao  Guangming Huang  Xiaoting Li  Yuan Mei<EMAIL_ADDRESS>Hua Pei  Quan Sun  Xiangming Sun<EMAIL_ADDRESS>Dong Wang  Zhen Wang
Le Xiao  Ping Yang  Central China Normal University, Wuhan, Hubei 430079,
China  Nuclear Science Division, Lawrence Berkeley National Laboratory,
Berkeley, California 94720, USA  Institute of Acoustics, Chinese Academy of
Sciences, Beijing 100190, China

###### Abstract

Using industrial standard $0.35\text{\,}\mathrm{\SIUnitSymbolMicro m}$ CMOS
Integrated Circuit process, we realized a highly pixelated sensor that
directly collects charge via metal nodes placed on the top of each pixel and
forms two dimensional images of charge cloud distribution. The first version,
_Topmetal-I_ , features a $64\times 64$ pixel array of
$80\text{\,}\mathrm{\SIUnitSymbolMicro m}$ pitch size. Direct charge
calibration reveals an average capacitance of
$210\text{\,}\mathrm{f}\mathrm{F}$ per pixel. The charge collection noise is
near the thermal noise limit. With the readout, individual pixel channels
exhibit a most probable equivalent noise charge of
$330\text{\,}\mathrm{e}^{-}$.

###### keywords:

Topmetal , pixel , charge sensor

††journal: Nuclear Instruments and Methods in Physics Research A

##  1 Introduction

Charge sensors are at the heart of many ionizing radiation detectors. Modern
imaging systems often require a simultaneous determination of the amount,
position and time structure of charges from ionization. In these cases, a
charge sensor of sub-millimeter spatial resolution is highly desirable.

Solid state detector systems, mainly segmented silicon and germanium crystals
coupled to CMOS pixel sensors, are widely deployed. CMOS pixel sensors such as
Medipix/Timepix[1, 2] and FE-I3/FE-I4 [3] are readily available. They have
pixel sizes of tens to hundreds of microns. They are coupled to the crystals
through processes such as the flip-chip bump bonding.

On the other hand, for gaseous and liquid detectors, wire and Printed Circuit
Board (PCB) are still the most popular readout schemes. Those detectors are
often configured as Time Projection Chambers (TPCs [4]). Noticeable imaging
systems of such kind are LXeGRIT[5] and $\mu$-PIC[6]. For practical reasons,
it is difficult to realize a multi-wire readout with a wire pitch smaller than
a millimeter. Novel designs using PCB have reached a pitch of hundreds of
$\mathrm{\SIUnitSymbolMicro}\mathrm{m}$ [6]. However, such designs still face
challenges of signal readout multiplexing and speed.

Gaseous and liquid detectors have distinct advantages over solid state
detectors: they are more easily scalable to large mass and are resilient to
radiation damage. Only with an equally scalable yet high resolution charge
readout scheme, their superior properties can be exploited in imaging
applications. The CMOS pixel sensor is an excellent candidate for this task,
because of the fine pixel size, as well as the possibility of embedding
complex circuitry for signal paths.

There have been attempts towards the use of CMOS pixel sensors in a TPC to
read the charge signal, with limited success. The D3 experiment [7] uses
FE-I3/FE-I4 sensors developed for the ATLAS [8] experiment, behind a gaseous
electron multiplication stage, to detect charge tracks resulting from
potential dark matter interactions. A similar attempt was made to use the
Timepix sensor as the readout for a TPC [9].

The development of CMOS sensors is usually dictated by the requirements of
large experiments or the consensus of consortiums. Existing sensors either
have some high level processing already built-in to handle the massive data
rate common in collider experiments, or are lack of low noise analog channels.
The readout for gaseous and liquid detectors are often slower, but have more
stringent requirement on noise performance.

We set out to realize a CMOS sensor that is uniquely suitable for charge
readout in gaseous and liquid detectors. We implemented a direct charge sensor
with $80\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}$ pitch between pixels
using the industrial standard
$0.35\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}$ CMOS process. Its
behavior in charging collection is well characterized.

##  2 Sensor Structure and Operation

A photograph of one fully fabricated and wire-bonded _Topmetal-I_ sensor is
shown in Fig. 1. The sensor is implemented in a
$6\times$8\text{\,}\mathrm{m}\mathrm{m}^{2}$$ silicon real-estate area. With
$80\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}$ pitch distance between
pixels, the $64\times 64$ square pixel array makes up a
$5.12\times$5.12\text{\,}\mathrm{m}\mathrm{m}^{2}$$ charge sensitive region.
Readout interface logic and analog buffers are placed around the pixel array.

Figure 1: Photograph of a _Topmetal-I_ sensor (left). The sensor chip is
placed on a PCB and gold wire-bonded. The $64\times 64$ pixel array, with
$80\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}$ pitch distance between
pixels, constitutes an approximately
$5\times$5\text{\,}\mathrm{m}\mathrm{m}^{2}$$ charge sensitive area in the
center of the sensor. The sensor is divided into four $64\times 16$ sub-arrays
with dedicated analog buffer inside each sub-array. Two test pixel columns are
implemented in the left-most sub-array (right).

The pixel array is divided into $4$ sub-arrays for efficient analog readout.
As shown in Fig. 1, each sub-array is of $64$ rows by $16$ columns in size and
has one dedicated analog output buffer per sub-array. A single external clock
drives the analog row/column multiplexing readout circuitry in each of the $4$
sub-arrays synchronously.

The internal structure of a single pixel is illustrated in Fig. 2. The
_Topmetal_ is implemented with the PROBEPAD component in the standard CMOS
process, which is a patch of metal in the topmost layer and has a part of it
not covered by the passivation layer. We designed the metal patch to be
$25\times$25\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}^{2}$$ in size, with
the central $15\times$15\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}^{2}$$
exposed hence sensitive to external charge. The metal patch is large enough to
be mechanically and chemically stable, while small enough to suppress the
cross-talk between neighboring pixels to a negligible level.

Charges arrived at the sensor and collected by the _Topmetal_ result in an
electric potential change at the “Node”, which is a detectable signal. The
potential of the “Node”, $V_{\text{node}}$, is captured by means of a source
follower that drives the signal through row/column multiplexer for analog
readout. The equivalent capacitance of _Topmetal_ to ground is measured to be
approximately $210\text{\,}\mathrm{f}\mathrm{F}$. It means $1000$ electrons
collected by _Topmetal_ would correspond to $0.8\text{\,}\mathrm{m}\mathrm{V}$
voltage drop on the “Node”.

The resetting circuit for the “Node” allows controlled removal of charge
collected by the _Topmetal_ and the initial potential setting of the node.
When a logic high is applied to the reset gate, the “Node” potential is set to
$V_{\text{reset}}$ and any excess charge on the “Node” is neutralized by the
power supply driving $V_{\text{reset}}$.

Two test columns are implemented in the leftmost sub-array (Fig. 1). Pixels in
the first test column have identical structure as the rest of the pixels,
except that the “Node” is internally connected to ground. They serve as
markers in the multiplexed readout. Since their $V_{\text{node}}$ is fixed
internally, they are immune to the noise generated by the _Topmetal_
capacitance. Noise measurements on the first test column measures the noise
due to the analog readout path only, decoupled from that from the _Topmetal_.
Pixels in the second test column have the “Node” capacitively coupled to an
external pin. They were designed to test the pixel response to pulses injected
from the external pin. The pin is tied to ground during normal operations.

Figure 2: Internal structure of a single pixel. The _Topmetal_ is a metal
patch exposed to the environment external to the sensor to collect charge
directly. The components enclosed in the dotted box is unique to each
individual pixel, the column selection transistor is shared by all the pixels
in the same column, and the output buffer is shared by all the pixels in the
same sub-array. The capacitor $C$ and the resistor $R$, connected by dashed
lines, are not real components implemented in the sensor. They illustrate the
equivalent circuitry of the _Topmetal_. Its behavior is discussed in the text.
Only the analog channel is shown.

Charge signal in each pixel is also converted into digital values by means of
a threshold discriminator (comparator), a delayed resetter, and a 5-bit
counter. Muxing readout for digital data at array level is also implemented.
The digital data chain is suitable for charge event counting applications.
However, in this letter, we focus on the physical behavior and analog
characteristics of the device.

##  3 Signal Formation

The _Topmetal_ can be modeled as a resistor with large resistance $R$ and a
capacitor with small capacitance $C$ connected in parallel (Fig. 2). The time
constant $\tau=RC$ is on the order of
$10\text{\,}\mathrm{s}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{d}\mathrm{s}$.
During charge collection, the equivalent charge current $i$ will run through
the resistor while charging up the capacitor at the same time. The following
differential equation gives the general behavior of the circuitry:

$i(t)=\frac{U}{R}+C\frac{dU}{dt}\;.$ (1)

$U$ is the voltage across the capacitor, which is then measured.

The solution to the differential equation is

$U(t)=\\\ \begin{cases}U_{0}e^{-t/\tau}&i=0\\\
(U_{0}-iR)e^{-t/\tau}+iR&i=\text{constant}\\\ \displaystyle
e^{-t/\tau}\left(U_{0}+\frac{1}{C}\int_{0}^{t}i(x)e^{x/\tau}dx\right)&i(t)\end{cases}$
(2)

The time-dependent $i(t)$ solution is the most generic. The solutions for
$i=0$ and $i=\text{constant}$ are two special cases. The boundary conditions
are set such that no matter whether there is a charge current $i$ or not, at
$t=0$, the output voltage $U(t=0)$ is set to a defined reset value $U_{0}$.
When collecting electrons (negative charge), $i<0$, otherwise $i>0$.

The sudden arrival of a charge pulse can be modeled as
$i(t)=Q\delta(t-t_{q})$, where $t_{q}$ is the arrival time of the charge
pulse. At $t=t_{q}$, the voltage decay curve $U(t)$ drops by the amount of
$Q/C$, as shown in Fig. 3.

Ideally, we can build an algorithm to find the sudden drops and reconstruct
their amplitudes to measure the charge collected. However, due to the noise,
as seen in Fig. 3, such an approach is unlikely to be robust. Also, in the
case of continuous charge current, there will be no sudden drop. Instead, to
extract the charge information, we employ a “double subtraction” scheme.

Prior to the charge measurement, we remove the charge source, but operate the
sensor in the same condition, to establish a baseline. The baseline,
$U(t)|_{i=0}$, is a time-series measurement of node voltage without charge
input. During the charge measurement, we record $U(t)|_{i}$ in a similar
fashion. The baseline and charge measurements are synchronized by a periodical
reset signal. The time between two resets defines an event window, and
$t_{2}-t_{1}$ is the charge integration time. We subtract the voltage right
after reset, $U(t_{1})$, from the voltage at the end of the window,
$U(t_{2})$, in each case, then subtract the difference in baseline from that
in charge measurement. The quantity $\Delta U$ is shown in Eq. (3).

$\displaystyle\Delta U$
$\displaystyle=\left[U(t_{2})-U(t_{1})\right]_{i}-\left[U(t_{2})-U(t_{1})\right]_{i=0}$
(3) $\displaystyle=-iR\left(e^{-t_{2}/\tau}-e^{-t_{1}/\tau}\right)\,.$

In the limit where $t/\tau\rightarrow 0$,

$e^{-t_{2}/\tau}-e^{-t_{1}/\tau}\approx-(t_{2}-t_{1})/\tau\,,$ (4)

hence

$\Delta U\approx\frac{i(t_{2}-t_{1})R}{RC}=\frac{Q}{C}\,.$ (5)

Eqs. (3–5) are derived assuming a constant current $i$. It can be shown that
the same relation Eq. (5) holds for sudden charge arrivals.

We operate at a charge integration time window of tens of milliseconds, so
that Eq. (4) is satisfied. Therefore we can use the simple proportional
relation Eq. (5) to compute the charge value. We also refer to one $\Delta U$
measurement an “event”.

Figure 3: Temporal behavior of the “Node” voltage in a pixel. One example in
the real data. Reset happens at $t=0$. The node voltage at reset is about
$1.4\text{\,}\mathrm{V}$ (off scale). The curve with filled circles shows node
voltage during charge collection. A cluster of charge suddenly arrives at
$t_{q}\approx$21\text{\,}\mathrm{m}\mathrm{s}$$. The thin curve is the
baseline, showing the voltage is decaying exponentially with a large time
constant.

The node voltage behavior in a characteristic pixel is shown in Fig. 3. When
the reset is activated ($t=0$), the node is set to the voltage of
$V_{\text{reset}}$. As soon as the reset is deactivated, due to charge
injection on the MOS transistor [10], the node voltage drops to a
deterministic value. In this example, the node voltage right after reset is
about $1.295\text{\,}\mathrm{V}$. Up to a small fluctuation, the baseline
curve behaves identically after each reset for the same pixel. Then, when
there is no charge being collected, the node voltage changes exponentially
towards a stable value. When the charges arrive, the voltage curve shows a
sudden drop.

Due to the inhomogeneity of the CMOS process, the baseline curves behave
differently among pixels. The most significant variation is in the decay time
constant. Nevertheless, it is only necessary to calibrate the baseline curves
for every pixel once prior to the charge detection.

To image 2D distributions of the charge cloud, the above procedure is
performed on each pixel in the array. The temporal behavior of each pixel is
decoded from the multiplexed readout signal. The interval of sampling is the
pixel clock period times the number of pixels in a sub-array (1024). The time
between resets is set to be slightly larger than the desired charge
integration time. $\Delta U$ is then computed for each pixel and the 2D
distribution is established.

##  4 Charge Calibration

To demonstrate that the _Topmetal-I_ sensor is capable of directly detecting
charge, and to calibrate for charge measurements, we constructed a rudimentary
drift chamber, using an ${}^{241}\text{Am}$ alpha source to ionize air, then
drift the negative charge towards the sensor in an electric field.

Figure 4: Setup for _Topmetal-I_ sensor charge calibration. Not drawn to
scale. The sensor is placed on a PCB. They are on an electrical potential
within $3.3\text{\,}\mathrm{V}$ of ground. An ${}^{241}\text{Am}$ source, its
collimator and a metal plate are held at negative high voltage and placed
above the _Topmetal-I_ sensor, parallel to the PCB. An electric field of
$200\text{\,}\mathrm{V}\mathrm{/}\mathrm{c}\mathrm{m}$ is created between the
metal plate and the PCB. Alphas from the ${}^{241}\text{Am}$ source ionize air
and the negative charge is drifted towards the sensor.

We wire-bonded a _Topmetal-I_ sensor onto a PCB and placed supporting circuity
on the PCB to drive the sensor and to transmit analog signals out via coax
cables. On the other end of the coax cables, signals are digitized by a 14-bit
high speed ADC. To create a uniform drift field with field lines perpendicular
to the sensor, we placed a large aluminum plate
$5\text{\,}\mathrm{c}\mathrm{m}$ above, and parallel to the sensor, serving as
the cathode. While the sensor substrate and the PCB are placed close to ground
potential, the aluminum plate is held at $-1000\text{\,}\mathrm{V}$, creating
an electric field of $200\text{\,}\mathrm{V}\mathrm{/}\mathrm{c}\mathrm{m}$.

A collimated ${}^{241}\text{Am}$ source embedded in the cathode emits alpha
particles perpendicular to the sensor. The alphas ionize air along their
paths. Negative charge generated during the process is drifted towards the
sensor and eventually collected by the sensor. Alphas from the
${}^{241}\text{Am}$ source have a range of about
$4\text{\,}\mathrm{c}\mathrm{m}$ in air at standard temperature and pressure
[11]. The distance between the cathode and the sensor
($5\text{\,}\mathrm{c}\mathrm{m}$) was chosen so that alphas would deposit
their full energy in air instead of hitting the sensor. An averaged image of
the charge cloud from alphas seen by the sensor is shown in Fig. 5.

Figure 5: Charge cloud distribution imaged by the _Topmetal-I_ sensor. The
image is an average of events mostly containing one alpha in each event. Each
pixel is $80\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}$ in size. Alpha
particles traverse perpendicular to the sensor surface, but are stopped in air
before hitting the sensor. The charge is then drifted towards the sensor and
collected. The integration time is $32\text{\,}\mathrm{m}\mathrm{s}$ for each
event.

The event time window is $33.5\text{\,}\mathrm{m}\mathrm{s}$ and the
integration time is $32\text{\,}\mathrm{m}\mathrm{s}$. Effectively we are
collecting data at an event rate of $30\text{\,}\mathrm{H}\mathrm{z}$. The
collimator limits the alpha rate to be much lower than the data event rate.
The conditions are chosen so that within a data event, there can be either
zero or one alpha, and the probability of two or more alphas in the same event
becomes negligible. Meanwhile, the integration time is long enough to fully
collect the charge after an alpha interaction, since it takes only
microseconds for electrons to drift for $5\text{\,}\mathrm{c}\mathrm{m}$ [12].

Figure 6: Voltage summation of pixels within the selection circle. A sum of
two Gaussian functions (thick curve) is used to fit the histogram. The dashed
lines show the constituents of the fit function. The fit yields a
$\chi^{2}/\text{NDF}=53.8/60$. Events containing no alpha fill the high peak
centered around 0. The mean of the single-alpha peak is $-0.120\pm
0.008\text{\,}\mathrm{V}$

On average, each alpha particle emitted from the ${}^{241}\text{Am}$ source
deposits $5.45\text{\,}\mathrm{M}\mathrm{e}\mathrm{V}$ [13] energy in air
resulting in ionization. The air has an ionization $W$-value of
$35\text{\,}\mathrm{e}\mathrm{V}$ [14]. Since alphas are fully stopped before
hitting the sensor, and electrons are fully collected during the integration
time, each alpha event results in a mean value of approximately
$1.56\text{\times}{10}^{5}\text{\,}\mathrm{e}^{-}$. Since each event has
either zero or one alpha, if we plot a histogram of the voltage signal due to
charge, we should be able to identify a peak corresponding to single alphas,
and use the relation

$C=Q\Big{/}\sum_{mn}\Delta U_{mn}$ (6)

to compute the capacitance $C$ of a single pixel. $Q$ is the total charge and
the summation is over pixels that see charge.

$\Delta U$’s of pixels within a
$17\text{\,}\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}$ radius (Fig.
5) are summed up to fill the histogram in Fig. 6. The region is chosen such
that $90\text{\,}\mathrm{\char 37\relax}$ of charge is contained in the region
given the signal is modeled by a two-dimensional Gaussian distribution. We
will not attempt to correct for the potential loss of
$10\text{\,}\mathrm{\char 37\relax}$ charge, instead, we treat it as a
systematic uncertainty of the measurement. The single-alpha peak is clear in
Fig. 6 and is well characterized by a Gaussian function. From the mean voltage
value of the single-alpha peak, we measure the pixel capacitance
$C=$207\text{\,}\mathrm{f}\mathrm{F}$$. The uncertainty, a combination of
uncertainty from the fit and the systematic uncertainty, is found to be
$12\text{\,}\mathrm{\char 37\relax}$.

To validate the result, we used a less stringent collimator to increase the
total ionization per unit time. Under the same electric field, we measured the
ionization current with a picoammeter to be around
$10\text{\,}\mathrm{p}\mathrm{A}$. The ionization current is then collected by
the _Topmetal-I_ sensor operating under exactly the same condition. The
measurement yields a similar capacitance value, with larger uncertainty.

To evaluate the noise performance of the sensor, we use the baseline dataset
alone, and compute the Root-Mean-Square (RMS) of voltage drop
$\left[U(t_{2})-U(t_{1})\right]_{i=0}$ of each pixel across all events. The
noise is then calculated as
$C\times\text{RMS}(\left[U(t_{2})-U(t_{1})\right]_{i=0})$, where $C$ is the
above measured pixel capacitance. The RMS voltage drop distribution of the
entire pixel array is shown in Fig. 7. The distribution has three distinct
clusters. The cluster with the lowest RMS value contains the test pixels in
the first column. Those pixels have their voltages tied to a fixed value at
all times. The RMS value of them comes solely from the analog readout noise.
It provides a measurement of the analog readout noise independent of the noise
from the _Topmetal_ capacitance. The second test column is capacitively
coupled to an external pin therefore has higher noise. The RMS of the main
array has its most probable value at $0.253\text{\,}\mathrm{m}\mathrm{V}$,
which corresponds to about $327\text{\,}\mathrm{e}^{-}$. The uncertainty of
this value is solely determined by the uncertainty of capacitance measurement.

Figure 7: RMS fluctuation distribution of pixels. The left most two peaks
correspond to test pixels in the first and second column respectively. ENC is
computed assuming $C=$207\text{\,}\mathrm{f}\mathrm{F}$$.

During normal air-cooled operations, the sensor temperature stabilizes at
about $57\text{\,}\mathrm{\SIUnitSymbolCelsius}$. A
$207\text{\,}\mathrm{f}\mathrm{F}$ capacitor at such temperature has an
intrinsic thermal noise of $192\text{\,}\mathrm{e}^{-}$ [15]. Using the first
test pixel column, we determine that the analog readout system has an ENC of
$254\text{\,}\mathrm{e}^{-}$. The quadratic summation of the two noise values
yields $318\text{\,}\mathrm{e}^{-}$, which is very close to the most probable
ENC value of the main pixel array. It indicates that the equivalent capacitor
of the _Topmetal_ is operating near its thermal noise limit.

##  5 Summary

We have demonstrated the possibility of implementing a highly pixelated sensor
for direct charge collection using industrial standard
$0.35\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}$ CMOS technology. No post-
processing is necessary. Although in this first attempt, _Topmetal-I_ , the
sensor noise is moderately high, its capability in observing charge from
single-alpha events in air directly already shows its potential in
applications.

The design and production of CMOS IC are traditionally perceived as
inhibitively expensive. However, recent advancement in the industry has made
the mature technologies such as the
$0.35\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}$ process affordable. Also,
the direct charge collection capability, without the need of any post-
processing, makes the overall integration cost much lower than for example
technologies used in producing hybrid solid-state detectors.

To improve beyond _Topmetal-I_ , besides reducing the pixel size, we can
further reduce the noise and record the charge arrival time. By placing a
charge sensitive amplifier in each pixel, and reducing the _Topmetal_ size
hence its capacitance, we foresee an ENC better than
$10\text{\,}\mathrm{e}^{-}$ per pixel should be achievable. Information on
charge arrival time can be obtained using an individually addressable pixel
structure. We will explore these options in future series of _Topmetal_
sensors.

## Acknowledgments

This work is supported, in part, by the Thousand Talents Program at Central
China Normal University and by the National Natural Science Foundation of
China under Grant No. 11375073. We also acknowledge the support from LBNL for
hosting the physical measurements of the sensor. We would like to thank
Christine Hu-Guo and Nu Xu for fruitful discussions.

## References

  * Ballabriga et al. [2011] R. Ballabriga, M. Campbell, E. Heijne, X. Llopart, L. Tlustos, W. Wong, Medipix3: A 64k pixel detector readout chip working in single photon counting mode with improved spectrometric performance, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 633, Supplement 1 (0) (2011) S15 – S18, ISSN 0168-9002, URL http://www.sciencedirect.com/science/article/pii/S0168900210012982.
  * Llopart et al. [2007] X. Llopart, R. Ballabriga, M. Campbell, L. Tlustos, W. Wong, Timepix, a 65k programmable pixel readout chip for arrival time, energy and/or photon counting measurements, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 581 (1–2) (2007) 485 – 494, ISSN 0168-9002, URL http://www.sciencedirect.com/science/article/pii/S0168900207017020.
  * Garcia-Sciveres et al. [2010] M. Garcia-Sciveres, et al., The FE-I4 pixel readout integrated circuit, Nucl. Instr. and Meth. A 636 (2010) S155–S159.
  * Marx and Nygren [1978] J. Marx, D. Nygren, The Time Projection Chamber, Phys. Today 31 (10) (1978) 46\.
  * Aprile et al. [1996] E. Aprile, V. Egorov, F. Xu, E. L. Chupp, P. P. Dunphy, T. Doke, J. Kikuchi, G. J. Fishman, G. N. Pendleton, K. Masuda, T. Kashiwagi, Liquid Xenon Gamma-Ray Imaging Telescope (LXeGRIT) for medium energy astrophysics, URL http://dx.doi.org/10.1117/12.253996, 1996.
  * Nagayoshi et al. [2004] T. Nagayoshi, H. Kubo, K. Miuchi, R. Orito, A. Takada, A. Takeda, T. Tanimori, M. Ueno, O. Bouianov, M. Bouianov, Development of $\upmu$-PIC and its imaging properties, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 525 (1–2) (2004) 20–27, ISSN 0168-9002, URL http://www.sciencedirect.com/science/article/pii/S0168900204003419.
  * Vahsen et al. [2012] S. E. Vahsen, H. Feng, M. Garcia-Sciveres, I. Jaegle, J. Kadyk, Y. Nguyen, M. Rosen, S. Ross, T. Thorpe, J. Yamaoka, The Directional Dark Matter Detector (D3), EAS Publications Series 53 (2012) 43–50.
  * Aad et al. [2008] G. Aad, et al., The ATLAS Experiment at the CERN Large Hadron Collider, JINST 3 (2008) S08003.
  * Brezina et al. [2012] C. Brezina, K. Desch, J. Kaminski, M. Killenberg, T. Krautscheid, Operation of a GEM-TPC With Pixel Readout, IEEE Transactions on Nuclear Science 59 (6) (2012) 3221\.
  * Shieh et al. [1987] J.-H. Shieh, M. Patil, B. J. Sheu, Measurement and Analysis of Charge Injection in MOS Analog Switches, IEEE Journal of Solid-State Circuits SC-22 (2) (1987) 277.
  * Berger et al. [????] M. J. Berger, J. S. Coursey, M. A. Zucker, Stopping-power and range tables for electrons, protons, and helium ions, National Institute of Standards and Technology URL http://physics.nist.gov/PhysRefData/Star/Text/ASTAR.html.
  * Hegerberg and Reid [1980] R. Hegerberg, I. D. Reid, Electron drift velocities in air, Aust. J. Phys. (1980) 227–230.
  * Basunia [2006] M. S. Basunia, Nuclear Data Sheets 107 (2006) 3323.
  * Giesen and Beck [2013] U. Giesen, J. Beck, New Measurements of W-Values for Protons and Alpha Particles, Radiation Protection Dosimetry (2013) 1–4.
  * Sarpeshkar et al. [1993] R. Sarpeshkar, T. Delbruck, C. A. Mead, White noise in MOS transistors and resistors, IEEE Circuits Devices Mag. (1993) 23–29.


# Soft Tiles: Capturing Physical Implementation Flexibility for Tightly-
Coupled Parallel Processing Clusters

Gianna Paulin1, Matheus Cavalcante1, Paul Scheffler1, Luca Bertaccini1, Yichao
Zhang1,
Frank Gürkaynak1 and Luca Benini12
1Integrated Systems Laboratory, ETH Zurich
8092 Zurich, Switzerland
Email:
<EMAIL_ADDRESS>2University of Bologna
40126 Bologna, Italy

###### Abstract

Modern high-performance computing architectures (Multicore, GPU, Manycore) are
based on tightly-coupled clusters of processing elements, physically
implemented as rectangular tiles. Their size and aspect ratio strongly impact
the achievable operating frequency and energy efficiency, but they should be
as flexible as possible to achieve a high utilization for the top-level die
floorplan. In this paper, we explore the flexibility range for a high-
performance cluster of RISC-V cores with shared L1 memory used to build
scalable accelerators, with the goal of establishing a hierarchical
implementation methodology where clusters can be modeled as soft tiles to
achieve optimal die utilization.

###### Index Terms:

Floorplanning, Soft Blocks, VLSI Architectures.

##  I Introduction and Related Work

Floorplanning, the process of designing the physical layout of a chip, has a
big impact on the performance, energy efficiency, time-to-market, and
fabrication cost of VLSI chips. While the main objective of floorplanning has
been chip area reduction—which directly translates into lowering production
costs—the floorplanning process must also optimize wirelength, delays, thermal
stability, and energy efficiency [1]. With the decreasing feature size of
advanced nodes, the overall number of transistors per chip has skyrocketed.
Higher transistor densities have enabled larger chip designs, directly
increasing the turnaround time of naïve iterative floorplan refinement. To
counteract this trend, researchers have explored new paradigms to accelerate
the floorplanning process, leveraging Graphics Processing Units (GPUs) [2] and
artificial intelligence [3].

Despite such advanced paradigms, the high cell counts of today’s high-
performance chips make a hierarchical implementation flow a necessity. They
can be tackled by following a top-down or a bottom-up approach. Top-down flows
start by partitioning the die and allocating subregions of the chip layout to
specific sub-blocks, generating constraints on their dimensions and aspect
ratios. When implementing the sub-blocks, such requirements might be
unfeasible or lead to sub-optimal quality of results (QoR), requiring lengthy
iterations to converge to a feasible design. A bottom-up flow would start by
fully implementing and hardening blocks at lower hierarchy levels before
implementing the next hierarchy level based on these tiles. However, building
upon hard tiles—which have fixed dimensions and even a fixed orientation in
advanced technology nodes—might result in sub-optimal overall placement
results, as they limit the possible top-level floorplans and make it more
challenging to achieve a high utilization of the die area.

External Intellectual Property (IP) providers often deliver their modules as
placed-and-routed hard tiles. However, for IPs owned by the chip designer,
soft tiles are usually a better choice as their density and aspect ratios can
be refined during placement. Additionally, introducing soft tiles enables a
broader range of possible floorplans with higher overall utilization,
especially when the die dimensions are fixed (fixed-outline floorplanning) and
the designer needs to fulfill stringent requirements such as tight frequency
and power constraints [4, 5]. Various works have investigated algorithms
tackling the floorplanning challenges for designs with hard and soft tiles,
e.g., Cull-and-Aggregate Bottom-Up Floorplanner (CABF) [1] or Iterative
Merging Packing (IMP) [6]. However, the correlation between a soft tile’s
physical shape (e.g., aspect ratio and macro placement) in floorplanning and
its QoR after physical implementation has barely been analyzed so far.

Typically, modern architectures build upon a base compute cluster, combining
multiple processing elements (PEs) sharing access to an L1 cache or Scratchpad
Memory (SPM) via a low-latency interconnect. We call such a latency-critical
cluster a tile. This tile is then replicated and interconnected with a
latency-tolerant Network-on-Chip (NoC) to build a larger high-performance
system. For example, considering multicore processors, Fujitsu’s A64FX [7]
combines four interconnected Core Memory Groups (CGMs). Each CGM couples
twelve compute (and one control) superscalar out-of-order vector-capable cores
with a fast data cache. In addition, each CGM has a designated High Bandwidth
Memory (HBM) controller enabling a bandwidth of
$256\text{\,}\mathrm{G}\mathrm{B}\mathrm{/}\mathrm{s}$ to
$8\text{\,}\mathrm{GiB}\text{/}$ of HBM. As an example in the GPU field,
NVIDIA’s A100 [8] architecture contains 108 Streaming Multiprocessors (SMs)
grouped into Texture Processing Clusters, which in turn are grouped into GPU
Processing Clusters. Each SM has a combined L1 data cache, a shared memory of
$192\text{\,}\mathrm{KiB}\text{/}$, and four warps. Each warp includes a
tensor core, 16 INT32 and FP32 cores, eight FP64 cores, and a large shared
$64\text{\,}\mathrm{KiB}\text{/}$ register file. Finally, Esperanto’s ET-SoC-1
[9], an exemplary manycore accelerator, couples more than one thousand energy-
efficient RISC-V vector processors, each including a software-configurable L1
data cache or SPM, with four high-performance Linux-capable out-of-order
Central Processing Units (CPUs). Esperanto uses a highly regular tiled
architecture to fit all these cores on a chip. Eight energy-efficient RISC-V
processors with a $32\text{\,}\mathrm{KiB}\text{/}$ shared instruction cache
(I$) form a neighborhood, and four neighborhoods with a
$4\text{\,}\mathrm{MiB}\text{/}$ L2 Static Random-Access Memory (SRAM) form a
minion shire. Finally, 34 minion shires with
$136\text{\,}\mathrm{MiB}\text{/}$ of on-die memory and four Linux-capable
CPUs form the full ET-SoC chip.

Finding an optimal floorplan with good post-place-and-route QoR for the
tightly interconnected tile (multicore, GPU, manycore) is crucial for
achieving high performance and efficiency of the overall design. The tile’s
operating frequency ultimately determines the performance of the overall
architecture. Moreover, the L1-to-PE latency is critical for high Instructions
Per Cycle (IPC) and cannot easily be increased by pipelining to simplify
implementation. In this scenario, the soft tile and system-level QoR need to
be improved through physically-aware design approaches [10, 11].

This paper focuses on an open-source, high-performance cluster tile with eight
compute (and one control) RISC-V cores connected to a shared L1 SPM through a
low-latency interconnect [12]. Similarly to the state-of-the-art
architectures, the cluster tile is then replicated to build a scaled-up high-
performance acceleration system [13]. We explore the QoR of the physical
implementation of this cluster as a soft tile based on a flexible range of
aspect ratio and memory macro placement styles. The contributions of this
paper are:

  * •

We propose three different memory macro placement styles for a cluster of
RISC-V cores with a shared L1 SPM. For each, we place and route the cluster
and evaluate its QoR in terms of achievable operating frequency, Total
Negative Slack (TNS), number of violating paths, cell density, total routed
wirelength (RtWL), and number of inserted buffers.

  * •

For all three proposed placement styles, we explore and evaluate the impact of
the tile’s aspect ratio on the QoR.

  * •

Based on our results, we discuss a hierarchical implementation methodology
where clusters can be modeled as soft tiles to achieve optimal overall die
utilization.

##  II Architecture

The Snitch cluster [12] is an open-source RISC-V multicore cluster targeting
highly-efficient double-precision floating-point computing. It is the compute
unit of the Manticore architecture [13] where it is massively replicated.

Figure 1: Architecture of a Manticore Snitch cluster.

###  II-A Cluster Tile

Figure 1 shows the architecture of the Snitch cluster configuration used in
Manticore. It contains eight worker core complexes (CCs), each of which
combines a small integer core, a trace L0 instruction cache, and a large
double-precision Floating Point Unit (FPU) kept busy with custom architectural
extensions. An additional ninth CC without an FPU controls a cluster-level
Direct Memory Access (DMA) engine and can be used for cluster coordination.
All CCs share a tightly-coupled $128\text{\,}\mathrm{KiB}\text{/}$ L1 SPM
divided into 32 memory banks, each $64\text{\,}\mathrm{bit}\text{/}$ wide, via
a single-cycle SPM interconnect. Blocks of eight banks form _superbanks_ ,
which are accessed in parallel by the $512\text{\text{-}}\mathrm{bit}\text{/}$
DMA engine through a secondary wide interconnect. The CCs also share a two-way
$8\text{\,}\mathrm{KiB}\text{/}$ L1 I$ and an integer Multiply Divide Unit
(MulDiv). Finally, the DMA engine and L1 I$ share a duplex
$512\text{\text{-}}\mathrm{bit}\text{/}$ Advanced eXtensible Interface (AXI)
crossbar connection to the global memory system, which all CCs can access
through a $64\text{\text{-}}\mathrm{bit}\text{/}$ secondary AXI crossbar; both
crossbars are internally connected for convenience.

(a) Wide cluster tile.

(b) Square cluster tile.

(c) Tall cluster tile.

Figure 2: Three example top-level floorplans containing eight quadrants
$\mathcal{Q}$ with 4 clusters $\mathcal{C}$ each and one manager core
(purple). I/O ports are highlighted in yellow.

###  II-B System Integration

The cluster tile can be hierarchically replicated to form a manycore system
with thousands of cores [13]. For example, four clusters $\mathcal{C}$ can be
combined to form a quadrant $\mathcal{Q}$, an intermediate hierarchy level
with a shared read-only cache and connections to the memory system. Multiple
quadrants can then be combined to form the top-level manycore architecture,
which also includes application-grade manager cores, high-bandwidth die-to-die
interfaces, and additional peripherals.

Unlike the cluster tile with its tightly-coupled low-latency memories, the
global interconnect uses pipelineable and latency-tolerant links; it is,
therefore, less critical and can easily adapt to changing placement and
routing pressures. Thus, floorplanning efforts should be focused on the
massively replicated compute tile, which is not latency tolerant and highly
frequency-critical, therefore dictating the system’s performance, area, and
operating frequency. Nevertheless, the hierarchical scale-out strategy can
impose additional constraints onto the repeated tile, such as aspect ratio or
pin positioning. Figure 2 shows three examples of how different quadrant and
top-level organizations can require the cluster tile to take specific aspect
ratios, further motivating our experiments.

##  III Methodology

We use Synopsys Fusion Compiler 2020.09 to synthesize, place, and route the
Manticore Snitch cluster in GlobalFoundries’ $12\text{\,}\mathrm{nm}\text{/}$
advanced FinFET technology node for a set of memory placement styles and
aspect ratios ranging from very wide ($2.5$:$1$) to very tall ($1$:$2.5$). All
shown designs target a $1\text{\,}\mathrm{GHz}\text{/}$ clock frequency under
worst-case conditions (SS, $0.72\text{\,}\mathrm{V}\text{/}$,
$125\text{\,}\mathrm{\SIUnitSymbolCelsius}\text{/}$) with a core area of
$0.90\text{\,}{\mathrm{mm}}^{2}\text{/}$. The designs were taken to the route
optimization stage, with the tool trying to solve as many DRC violations as
possible but without running a sign-off phase. Finally, we evaluate the
physically implemented designs in terms of:

Effective frequency

Limits the compute throughput per tile.

Cell density

Limits the placeable logic (tiles) per unit die area.

#Buffers

Increases with expended timing and design rule fixing effort and impacts
leakage and switching power.

Routed wirelength (RtWL)

Worsens the transition times, crosstalk, and ohmic losses, requiring more
timing and design rule fixing and increasing leakage and switching power.

#DRC Violations

Indicates the routability of the design.

#Violating Paths/TNS

Capture the overall _severity_ and _ubiquity_ of timing violations and
increase with the optimization effort required to achieve a given effective
frequency.

##  IV Floorplanning

(a) FP${}_{\text{{1-sided}}}$.

(b) FP${}_{\text{{2-sided}}}$.

(c) FP${}_{\text{{u-shape}}}$.

Figure 3: Placed and routed designs for each base floorplan with main modules
highlighted.

To arrive at our evaluation floorplans, we will first observe the cluster
tile’s architecture by analyzing the interconnectivity of individual design
components since these properties will fundamentally impact our target
metrics.

Technology-dependent SRAM macros are used to implement all I$ and SPM banks.
With 32 of 36 memory macros and $20\text{\,}\mathrm{\char 37\relax}\text{/}$
of the design’s placeable area, the L1 SPM is the most challenging component
to place. Its fully-connected crossbar dominates all other interconnect logic
in complexity, providing an individual $64\text{\text{-}}\mathrm{bit}\text{/}$
low-latency link for all $26\text{\times}32$ master-slave pairs; we can
predict that it will be routing-dominated and, therefore, susceptible to low
cell density and routing congestion. In addition, this interconnect requires a
large unobstructed placement area and for all the SPM macros to be close to
it.

We propose the three representative floorplan styles to explore the QoR of the
Manticore Snitch cluster: FP${}_{\text{{1-sided}}}$,
FP${}_{\text{{2-sided}}}$, and FP${}_{\text{{u-shape}}}$. The I/O pins are
constrained to the left side, and the SPM macros are on the right side, in all
floorplan styles. The I$ macros are placed close to the left side not to
obstruct the SPM crossbar yet remain accessible to the CCs. Moreover, all
styles are vertically symmetric to keep the interconnect easily reachable by
all CCs and SPM macros.

FP${}_{\text{{1-sided}}}$, shown in Figure 3, places all $32$ SPM memory
macros on the right side in a block as compact as possible. It tries to keep
the interconnect as close as possible to the right side of the design. This
placement style might lead to narrow channels in the center of the design,
causing the “pinching” of the interconnect area. Moreover, the excessive macro
stacking causes some macros to be far from the standard cell area, which
challenges the routing to their pins.

FP${}_{\text{{2-sided}}}$, shown in Figure 3, tackles those limitations by
leaving a largely unconstrained placement area for the interconnect and
keeping all SPM macros easily accessible from the standard cell area. To do
so, it places the SPM macros in a block as wide as possible. However, this
placement style spreads the macros across the design, which might affect the
wire length and the timing.

FP${}_{\text{{u-shape}}}$, shown in Figure 3, is a midway point between the
two previous styles. This floorplan places the SPM macros in a vertically
symmetrical “U” shape enclosing the region where the SPM crossbar is placed;
this further minimizes their overall distance of the SPM crossbar and avoids
excessive stacking. The SPM macros are placed according to samples of the
generator function $f[n]=(n/\text{HH})^{-p}$, where $n\in\mathbb{Z}^{*}$
represents the $n$-th column of macros in the SPM region, $f[n]$ represents
the height of the corresponding column, HH is half of the height of the
cluster area, and the parameter $p$ is chosen so that the SPM macros occupy as
many rows as possible (i.e., closing the central channel).

##  V Implementation Results

This section compares the QoR for our three different memory macro placement
styles for a range of aspect ratios. We will then discuss how understanding
the aspect ratio scaling of different tile floorplans can help determine an
optimal hierarchical top-level floorplan.

###  V-A Base Floorplan Analysis

Figures 3, 3 and 3 highlight the placement of the main modules of the
Manticore Snitch cluster. The interconnect occupies a large central area in
the design, encircled by the CCs. The CCs are attracted by the L1 I$ and the
SPM crossbar, while the FPUs occupy the remaining area.

The Snitch cluster’s placeable cell area is approximately
$4.5\text{\,}\mathrm{MGE}\text{/}$. The uniform cluster area of
$0.9\text{\,}{\mathrm{mm}}^{2}\text{/}$ implies an average standard cell
utilization of $57.5\text{\,}\mathrm{\char 37\relax}\text{/}$ over the three
base floorplans. The standard cell density maps for all base floorplans are
shown in Figures 5, 5 and 5. The low cell density of the densely routed SPM
crossbar indicates that the density in the regions where the CCs and FPUs are
placed is, in fact, much higher than the average density suggests, peaking at
$80\text{\,}\mathrm{\char 37\relax}\text{/}$. This puts pressure on the rest
of the design, which ends up dense and having to avoid the routing-congested
interconnect region.

For Figures 3 and 3, the modules were placed in suitable locations. However,
in Figure 3, the lack of space between the interconnect and the L1 I$ macros
for the CCs and L0 I$ es results in an asymmetric module placement, where the
cores end up on the bottom of the floorplan. Furthermore, the module positions
indicate that this floorplan style leads to a cluster particularly sensitive
to changes in the aspect ratio in the direction of taller (narrower) clusters
since this will further reduce the distance between the crossbar and the L1 I$
macros.

Table I compares those three floorplan styles according to several QoR
metrics. FP${}_{\text{{u-shape}}}$ shows the best QoR overall with a square
$1$:$1$ aspect ratio, reaching the highest operating frequency, lowest TNS,
fewer violating paths, and shorter RtWL. FP${}_{\text{{1-sided}}}$ reaches
comparable results, the main difference being an operating frequency
$1.5\text{\,}\mathrm{\char 37\relax}\text{/}$ lower than the
FP${}_{\text{{u-shape}}}$ instance. This performance drop is due to pinching
of the interconnect by the central channel in the FP${}_{\text{{1-sided}}}$
instance.

Both the FP${}_{\text{{1-sided}}}$ and FP${}_{\text{{u-shape}}}$ instances
finished with 38 DRC violations. This low ($<100$) DRC violation count is
expected, since we only run our implementation flow to the route optimization
stage. Those violations are spread across the floorplan and can be solved
manually in a sign-off step. FP${}_{\text{{2-sided}}}$, on the other hand, had
227 violations, many of them shorts concentrated in the interconnect region.
Therefore, we consider this instance unfeasible. This high DRC count is due to
the interconnect placement between the FPUs and their CCs. There are not
enough resources to route those connections through the interconnect region,
which leads to a high amount of DRCs and an unfeasible design.

###  V-B Aspect Ratio Analysis

In this section, we analyze and compare the three floorplan styles as
discussed in Section IV for three aspect ratios (width:height)—a tall
($1$:$2.5$), a square ($1$:$1$), and a wide cluster tile ($2.5$:$1$)—with a
uniform cluster area of $0.9\text{\,}{\mathrm{mm}}^{2}\text{/}$. This aspect
ratio selection enables a variety of hierarchical scale-out floorplans, e.g.,
the ones shown in Figure 2. Our findings are summarized in Table I. In
addition, we plot four key metrics, effective frequency, RtWL, number of DRC
violations, and the number of inserted buffers in Figure 4. Overall, moving
away from a square floorplan causes a noticeable QoR degration.

FP${}_{\text{{1-sided}}}$FP${}_{\text{{2-sided}}}$FP${}_{\text{{u-shape}}}$

$860$$880$$900$$920$$940$$960$$1$:$2.5$$1$:$1$$2.5$:$1$Aspect RatioEff. Freq.
[$\mathrm{MHz}\text{/}$]

$15$$16$$17$$18$$1$:$2.5$$1$:$1$$2.5$:$1$Aspect RatioRouted Wirel.
[$\mathrm{m}\text{/}$]

$0$$1$$2$$3$$1$:$2.5$$1$:$1$$2.5$:$1$Aspect Ratio# DRCs [$\times 10^{3}$]

$120$$130$$140$$150$$1$:$2.5$$1$:$1$$2.5$:$1$Aspect Ratio#Buffers. [$\times
10^{3}$]

Figure 4: Effective frequency, RtWL, #DRCs, and number of inserted buffers for all considered floorplan styles as a function of the aspect ratio. TABLE I: Physical implementation results of the cluster soft tile instances with aspect ratios ranging from $1$:$2.5$ to $2.5$:$1$ and base floorplans FP${}_{\text{{1-sided}}}$, FP${}_{\text{{2-sided}}}$, and FP${}_{\text{{u-shape}}}$. Aspect Ratio |  $1$:$2.5$ |  |  $1$:$1$ |  |  $2.5$:$1$
---|---|---|---|---|---
Floorplan | FP${}_{\text{{1-sided}}}$ | FP${}_{\text{{2-sided}}}$ | FP${}_{\text{{u-shape}}}$ |  | FP${}_{\text{{1-sided}}}$ | FP${}_{\text{{2-sided}}}$ | FP${}_{\text{{u-shape}}}$ |  | FP${}_{\text{{1-sided}}}$ | FP${}_{\text{{2-sided}}}$ | FP${}_{\text{{u-shape}}}$
Eff. Freq. [$\mathrm{MHz}\text{/}$] | 888.8 | 886.5 | 875.7 |  | 927.6 | 939.8 | 940.7 |  | 921.7 | 909.1 | 925.1
TNS$\mathrm{ns}\text{/}$ | -33.8 | -48.2 | -103.3 |  | -25.5 | -30.2 | -24.7 |  | -37.5 | -40.2 | -78.2
#Violating Paths | 5352 | 5787 | 6819 |  | 4890 | 5372 | 4459 |  | 6163 | 5871 | 8271
RtWL$\mathrm{m}\text{/}$ | 17.1 | 17.6 | 17.0 |  | 15.8 | 15.9 | 15.6 |  | 16.9 | 16.9 | 16.6
#DRCs | 36 | 417 | 259 |  | 38 | 227 | 38 |  | 654 | 2943 | 86
#Buffers | $141.1103$ | $143.8103$ | $140.0103$ |  | $130.8103$ | $131.0103$ | $128.9103$ |  | $138.1103$ | $137.3103$ | $133.6103$
Cell Density | $59.5\text{\,}\mathrm{\char 37\relax}\text{/}$ | $60.7\text{\,}\mathrm{\char 37\relax}\text{/}$ | $59.7\text{\,}\mathrm{\char 37\relax}\text{/}$ |  | $57.3\text{\,}\mathrm{\char 37\relax}\text{/}$ | $57.9\text{\,}\mathrm{\char 37\relax}\text{/}$ | $57.4\text{\,}\mathrm{\char 37\relax}\text{/}$ |  | $58.7\text{\,}\mathrm{\char 37\relax}\text{/}$ | $58.9\text{\,}\mathrm{\char 37\relax}\text{/}$ | $58.5\text{\,}\mathrm{\char 37\relax}\text{/}$

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 5: Standard cell density map, from dark blue to red, for each
considered aspect ratio: $1$:$2.5$ (a), (b) (c), $1$:$1$ (d), (e), (f), and
$2.5$:$1$ (g), (h), (i). Within each aspect ratio, the images represent the
macro placement styles FP${}_{\text{{1-sided}}}$, FP${}_{\text{{2-sided}}}$,
and FP${}_{\text{{u-shape}}}$, respectively. The black regions contain the
SRAM macros. Images to scale.

For our three floorplan styles, pushing the floorplan to a tall aspect ratio
of $1$:$2.5$, shown in Figures 5, 5 and 5, degrades the overall QoR more than
pushing to a wide aspect ratio of $2.5$:$1$, shown in Figures 5, 5 and 5.
Similarly to the square floorplan FP${}_{\text{{2-sided}}}$ in Figure 5, the
three floorplans FP${}_{\text{{1-sided}}}$, FP${}_{\text{{2-sided}}}$, and
FP${}_{\text{{u-shape}}}$ for the tall aspect ratio of $1$:$2.5$ in Figures 5,
5 and 5 have an asymmetric module placement: the CCs cannot be placed in the
central location between the interconnect and the I$ banks and are squeezed to
one side. This placement impacts their QoR. Out of those three instances,
FP${}_{\text{{2-sided}}}$ and FP${}_{\text{{u-shape}}}$ are unfeasible due to
a high DRC violation count. Only FP${}_{\text{{1-sided}}}$ is feasible,
although at an operating frequency $5.5\text{\,}\mathrm{\char
37\relax}\text{/}$ lower than the fastest square-shaped instance.

Figures 5, 5 and 5 show the cell density maps for FP${}_{\text{{1-sided}}}$,
FP${}_{\text{{2-sided}}}$, and FP${}_{\text{{u-shape}}}$ for the wide aspect
ratio of $2.5$:$1$. The smaller height for these floorplans requires the SPM
banks to be arranged in more columns causing the fully-connected crossbar
(blue region) to be placed more towards the center of the floorplan. The
resulting free area on the right side gets filled with FPUs. For both
FP${}_{\text{{1-sided}}}$ and FP${}_{\text{{2-sided}}}$, the fully-connected
crossbar is squeezed and stretched by the wide SPM bank rows, respectively.
Thus, all connections between the CCs and the FPU must be routed through the
already congested crossbar area. The lack of routing resources in that area
increases the number of DRC violations for FP${}_{\text{{1-sided}}}$ and
FP${}_{\text{{2-sided}}}$. In contrast, the arrangement of the SPM banks in
the FP${}_{\text{{u-shape}}}$ instance leads to an undisturbed crossbar
placement. This allows the CCs to FPU connections to route around the high
routing congestion crossbar area. Nevertheless, the overall feasibility of
FP${}_{\text{{u-shape}}}$ with 86 DRC violations remains questionable.

###  V-C Hierarchical Design Flow Recommendations

Our analysis shows that the QoR of a soft tile is particularly sensitive to
aspect ratio variations and highly depends on the floorplan style, i.e., on
the memory macro placement. As the main compute unit, the latency-critical
cluster should strive to achieve the best possible QoR. The higher hierarchy
levels use latency-tolerant and pipelineable interconnects which can easily be
tuned to match the cluster’s performance. However, additional requirements
might propagate down the hierarchy when designing the top-level floorplan. For
example, the top-level floorplans from Figures 2 and 2 might require larger
channels between the bottom row of quadrants to allow the global crossbar to
route to a die-to-die interface IP or an HBM PHY, requiring a non-square
aspect ratio for the cluster tile. Similarly, the narrow channels between the
three rows of quadrants in Figure 2 might lead to congested regions, which
implies different aspect ratio requirements for the cluster tile.

Based on our findings, we propose to follow our hierarchical implementation
approach: in a first step, the designer should evaluate various cluster-level
floorplan styles to explore a set of aspect ratios which enable a variety of
top-level floorplans. The knowledge of the achievable cluster-level QoR and
optimal floorplans for each aspect ratio then allows the further exploration
of a set of top-level floorplans using only soft tile shapes that meet the
desired QoR.

##  VI Conclusions

Typically, modern high-performance accelerator architectures are based on
tightly-coupled clusters of PEs, physically implemented as rectangular soft
tiles. These tiles are often highly replicated and interconnected with a
latency-tolerant NoC to build scalable high-performance computing systems. The
tile’s size and aspect ratio strongly impact the achievable operating
frequency and energy efficiency. Nevertheless, they should be as flexible as
possible to enable a high density solution for the complete design. In this
paper, we focus on an open-source, high-performance multicore cluster as a
soft tile, which can be used to build a high-performance many-cluster system
by cluster replication. Based on architectural analysis, we proposed three
floorplan styles which can systematically be adapted to different aspect
ratios.

We then explored the QoR of placed-and-routed cluster implementations as a
soft tile in the GlobalFoundries $12\text{\,}\mathrm{nm}\text{/}$ advanced
FinFET technology node, considering wide ($2.5$:$1$), square, and tall
($1$:$2.5$) cluster aspect ratios with all proposed floorplan styles. No
single floorplan style leads to good QoR across all aspect ratios.
FP${}_{\text{{1-sided}}}$ is to be used when higher-level tiles require a tall
cluster floorplan, and FP${}_{\text{{u-shape}}}$ is better suited when a wide
cluster tile is required. Considering the overall QoR, the best $2.5$:$1$
instance achieves an effective frequency $1.7\text{\,}\mathrm{\char
37\relax}\text{/}$ lower than the best square instance, although the large TNS
and DRC violation count poses doubts on its feasibility. On the clusters with
an aspect ratio $1$:$2.5$, only the FP${}_{\text{{1-sided}}}$ instance was
feasible, albeit at an operating frequency $5.5\text{\,}\mathrm{\char
37\relax}\text{/}$ lower than the best square cluster instance.

Overall, the results suggest that pre-characterizing the building block’s QoR
for different aspect ratios and floorplan styles, combined with a preliminary
investigation of the top-level layout, can help the designer find top-level
hierarchical floorplans that are feasible and achieve high performance.

## References

  * [1] C.-S. Hoo, K. Jeevan, and H. Ramiah, “Cost reduction in bottom-up hierarchical-based VLSI floorplanning designs,” _International Journal of Circuit Theory and Applications_ , vol. 43, no. 3, pp. 286–306, 2015\.
  * [2] Y. Lin, S. Dhar, W. Li, H. Ren, B. Khailany, and D. Z. Pan, “Dreampiace: Deep learning toolkit-enabled GPU acceleration for modern VLSI placement,” in _2019 56th ACM/IEEE Design Automation Conference (DAC)_. Las Vegas, NV, USA: IEEE, 2019, pp. 1–6.
  * [3] B. Khailany, H. Ren, S. Dai, S. Godil, B. Keller, R. Kirby, A. Klinefelter, R. Venkatesan, Y. Zhang, B. Catanzaro, and W. J. Dally, “Accelerating chip design with machine learning,” _IEEE Micro_ , vol. 40, no. 6, pp. 23–32, 2020.
  * [4] S. Adya and I. Markov, “Fixed-outline floorplanning: enabling hierarchical design,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , vol. 11, no. 6, pp. 1120–1135, 2003.
  * [5] T.-C. Chen and Y.-W. Chang, “Floorplanning,” in _Electronic Design Automation_ , L.-T. Wang, Y.-W. Chang, and K.-T. T. Cheng, Eds. Boston: Morgan Kaufmann, 2009, pp. 575–634. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B9780123743640500175
  * [6] P. Ji, K. He, Y. Jin, H. Lan, and C. Li, “An iterative merging algorithm for soft rectangle packing and its extension for application of fixed-outline floorplanning of soft modules,” _Computers & Operations Research_, vol. 86, pp. 110–123, 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0305054817301211
  * [7] T. Yoshida, “Fujitsu high performance CPU for the Post-K computer,” in _Hot Chips_ , vol. 30. Stanford, CA, USA: IEEE Computer Society, 2018, pp. 1–22.
  * [8] J. Choquette, W. Gandhi, O. Giroux, N. Stam, and R. Krashinsky, “NVIDIA A100 tensor core GPU: Performance and innovation,” _IEEE Micro_ , vol. 41, no. 2, pp. 29–35, 2021.
  * [9] D. Ditzel, R. Espasa, N. Aymerich, A. Baum, T. Berg, J. Burr, E. Hao, J. Iyer, M. Izquierdo, S. Jayaratnam _et al._ , “Accelerating ml recommendation with over a thousand RISC-V/tensor processors on esperanto’s et-soc-1 chip,” in _2021 IEEE Hot Chips 33 Symposium (HCS)_ , IEEE. Palo Alto, CA, USA: IEEE, 2021, pp. 1–23.
  * [10] J. Cortadella, J. de San Pedro, N. Nikitin, and J. Petit, “Physical-aware system-level design for tiled hierarchical chip multiprocessors,” in _Proceedings of the 2013 ACM International symposium on Physical Design_. New York, NY, USA: Association for Computing Machinery, 2013, pp. 3–10.
  * [11] M. Cavalcante, S. Riedel, A. Pullini, and L. Benini, “MemPool: A shared-L1 memory many-core cluster with a low-latency interconnect,” in _2021 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. Grenoble, FR: IEEE, Mar. 2021, pp. 701–706.
  * [12] F. Zaruba, F. Schuiki, T. Hoefler, and L. Benini, “Snitch: A tiny pseudo dual-issue processor for area and energy efficient execution of floating-point intensive workloads,” _IEEE Transactions on Computers_ , vol. 70, no. 11, pp. 1845–1860, Nov. 2021.
  * [13] F. Zaruba, F. Schuiki, and L. Benini, “Manticore: A 4096-core RISC-V chiplet architecture for ultraefficient floating-point computing,” _IEEE Micro_ , vol. 41, no. 2, pp. 36–42, Mar. 2021.