*Send correspondence to Alexandre E. Adler, E-mail<EMAIL_ADDRESS>
# Modelling ground pickup for microwave telescopes

Alexandre E. Adler  The Oskar Klein Centre, Department of Physics, Stockholm
University, SE-106 91 Stockholm, Sweden  Adriaan J. Duivenvoorden  Joseph
Henry Laboratories of Physics, Jadwin Hall, Princeton University, Princeton,
NJ, USA 08544  Jon E. Gudmundsson  The Oskar Klein Centre, Department of
Physics, Stockholm University, SE-106 91 Stockholm, Sweden

###### Abstract

Microwave telescopes require an ever-increasing control of experimental
systematics in their quest to measure the Cosmic Microwave Background (CMB) to
exquisite levels of precision. One important systematic for ground and
balloon-borne experiments is ground pickup, where beam sidelobes detect the
thermal emission of the much warmer ground while the main beam is scanning the
sky. This generates scan-synchronous noise in experiment timestreams, which is
difficult to filter out without also deleting some of the signal from the sky.
Therefore, efficient modelling of pickup can help guide the design of
experiments and of analysis pipelines. In this work, we present an extension
to the beamconv algorithm that enables us to generate time-ordered data (TOD)
from beam-convolved sky and ground maps simultaneously. We simulate ground
pickup for both a ground-based experiment and a telescope attached to a
stratospheric balloon. Ground templates for the balloon experiment are
obtained by re-projecting satellite maps of the Earth’s microwave emission.

###### keywords:

Cosmic Microwave Background, Scan-synchronous noise, Beam convolution, CMB
Telescopes, Ground Pickup

##  1 INTRODUCTION

Studies of the microwave sky are crucial for cosmology. The Cosmic Microwave
Background (CMB) is the oldest light in the Universe: it has been travelling
towards Earth since roughly 380,000 years after the Big Bang. We observe the
CMB as a nearly isotropic field of blackbody radiation of
$\sim$2.7\text{\,}\mathrm{K}$$[1]. Small spatial fluctuations (anisotropies)
in the blackbody temperature, of the order of a few parts per $10^{4}$ have
been measured by several ground[2, 3, 4, 5], balloon-borne[6, 7, 8] and space-
based experiments[9, 10, 11]. These, in turn, lead to stringent constraints on
the history and evolution of the Universe, both before and after the CMB’s
emission. In fact, the CMB underpins the consensus cosmological model,
$\Lambda$CDM. Beyond the temperature anisotropies, the CMB is also partly
polarised [12]. This polarisation is weaker than the temperature anisotropies,
thus detecting it has proved more challenging. The polarisation state of
electromagnetic radiation can be expressed using the Stokes parameters: total
intensity $I$, the two linear polarisation directions $Q$ and $U$, and
circular polarisation $V$. Together, they form the Stokes vector $(I,Q,U,V)$.
So far no circular polarisation of cosmological origin has been observed[13,
14]. The $Q$ and $U$ polarisation can be combined into two rotationally
invariant (scalar) quantities: the parity-even $E$-mode field and the parity-
odd $B$-mode field.

$E$ and $B$-modes are produced by a variety of processes in the CMB. For the
sake of brevity and clarity, we refer the reader to 15, a recent review of the
topic. We will limit ourselves to stating that the detection of primordial CMB
$B$-mode polarisation would be strong evidence in favor of inflation[16], a
proposed period of extremely rapid expansion in the first instants of the
Universe. Finding solid evidence that inflation happened would have enormous
consequences for cosmology and high-energy physics. It is therefore no
surprise that many experiments are aiming to detect this polarisation[17, 18].

Advances in detector technology, cryogenics, and optical design have made
microwave detectors more compact and less noisy. The extremely challenging
nature of the measurement means these experiments need an exquisite control
over possible sources of systematic error. The optical design, the scanning
strategy, the electronics and readout chain, are all chosen with the need to
minimize systematics in mind. For that reason, simulations of possible
systematics are a key aspect of the design process.

A persistent systematic for ground-based and balloon-borne experiments is
scan-synchronous noise (SSN). Each detector in a CMB experiment generates a
timestream as an output. CMB telescopes usually scan back and forth in azimuth
at a fixed or near-fixed elevation. A contaminant fixed in azimuth will thus
produce spurious correlations in the noise over long periods of time as the
telescope comes back to the same azimuth repeatedly.

The leading source of SSN for ground and balloon experiments is ground pickup.
It results from the interaction between the telescope’s beam $B(\theta,\phi)$
and the ground surrounding the telescope. The total ground pickup is the
convolution between the beam and the ground. The beam for a telescope is
usually strongly directional. It is decomposed into the main beam, which
concentrates the quasi-totality of the power, and sidelobes, which are much
weaker but extend to large angles from the beam’s maximum. While the sidelobes
are typically very weak, the ground is extremely warm compared to the CMB
anisotropies. As a consequence, the amplitude of the coupling between
telescope sidelobes and the ground can generate noise comparable in intensity
to the sky signal. As the ground is fixed in the telescope’s frame of
reference, that noise falls under the umbrella of SSN.

Current ground based experiments typically suppress the amount of SSN that is
propagated to the science products (sky maps, angular power spectra,
cosmological parameter estimations) by effectively high-pass filtering the
timestreams[19]. If the filter frequency is higher than the frequency at which
the telescope comes back at the same (az, el) position, then the correlations
in the timestream associated with telescope motion will be suppressed.
However, SSN is not the only cause for long-period correlations in
timestreams. Indeed, large scale CMB anisotropies will lead to low-period
features in experimental timestreams. Then the high-pass filter will also
reduce what the experiment can learn about the large angular scales. The only
observational window on $B$-mode polarisation from primordial gravitational
waves is expected to be at multipoles of $\ell\sim 100$ (roughly
$1.8\text{\,}\mathrm{\SIUnitSymbolDegree}$) or smaller. It is clear that a CMB
experiment interested in probing these large angular scales will want to
minimize ground pickup so it can be less aggressive with its timestream
filtering.

As mentioned earlier, experiments rely on simulations to guide their design
choices. In this proceeding, we present an effort to simulate timestreams for
a sky signal contaminated by ground pickup. We present what is, to our
knowledge, the first open source library to generate timestreams for the beam-
convolved ground and sky simultaneously. In particular, we generate templates
of ground emission for two types of telescopes. The first is a telescope
situated in the heights of the Atacama Desert, near where the Atacama
Cosmology Telescope and the future Simons Observatory are located. The second
is a balloon-borne telescope whose position and altitude can change from day
to day. As the balloon finds itself flying over a wide variety of terrains, we
need to simulate ground templates for many different areas. To that end, we
re-project satellite maps of the Earth’s brightness temperature. The templates
are described in Section 2. In both cases, we model the instrument as a two-
lens refractor. Beams are simulated for 50 detectors. This constitutes Section
3. Beams and ground template are then convolved together using the fast
convolution algorithm beamconv[20] to generate timestreams following a set
scan strategy. This constitutes Section 4 We present the results in Section 5,
and discuss the next steps in Section 6.

##  2 GROUND MODELS

###  2.1 Modelling of the Atacama Desert

We place our telescope on the slopes of Cerro Toco, at the location of the
current Atacama Cosmology Telescope[3]. A panoramic view of the surrounding
terrain can be seen in Figure 1. In that picture, a number of mountains of
various heights rise over the horizon. We simplify that landscape down to a
number of triangles. The ground around the experiment up to the horizon, is
assumed to be a blackbody with a temperature of $300\text{\,}\mathrm{K}$,
which translates to a brightness temperature of
$$3.5\text{\times}{10}^{8}\text{\,}\mathrm{\SIUnitSymbolMicro
K}$_{\textrm{CMB}}$. The emission temperature of the mountains decreases with
altitude at a rate of $1\text{\,}\mathrm{\SIUnitSymbolCelsius}$ per
$100\text{\,}\mathrm{m}$.

Figure 1: Ground template created from site photographs and local temperature
estimates, converted to CMB brightness temperature units at
$95\text{\,}\mathrm{GHz}$. The temperature of the sky is set at
$0\text{\,}\mathrm{K}$. The highest peak, Cerro Toco, is
$400\text{\,}\mathrm{m}$ higher than the site and a few kilometers away.

###  2.2 Modelling the ground for a balloon-borne experiment

Temperature does not convert to brightness temperature uniformly for different
types of terrain. Since balloons will drift over enormous distances and can
take very different paths, we cannot reasonably expect to model the
temperature and the terrain analytically like in Section 2.1. Fortunately,
there exists actual data of the Earth’s brightness temperature in the
microwave band. Several of the US Defense Meteorological Satellite Program
(DMSP) satellites carry an instrument called SSMIS[21, 22], which stands for
Special Sensor Microwave Imager/Sounder. SSMIS measures the brightness
temperature of the ground in several bands[23]. We extract maps of emission in
the $91\text{\,}\mathrm{GHz}$ band. By combining the different scan patterns
and regions to generate near-complete daily maps of the Earth’s microwave
temperature. The maps are projected following the healpix[24] pixellization
convention, while assuming that the Earth is spherical.

We then need to transform these Earth maps into (Az-el) maps from the
telescope’s point of view. An observer at an altitude $z$ above the North Pole
of a sphere will see the surface up to a co-latitude of $\alpha_{fov}$:

$\alpha_{fov}=\cos^{-1}\left({\frac{R}{R+z}}\right)\>,$ (1)

where $R$ is the sphere’s radius. A point on the sphere with latitude
$\lambda$ and longitude $l$, within the spherical cap of angular radius
$\alpha_{fov}$, will get projected in a spherical coordinate system centered
on the observer:

$\displaystyle\theta=$
$\displaystyle\pi-\tan^{-1}\left(\frac{\sin(\pi/2-\lambda)}{(1-\cos(\pi/2-\lambda))+\frac{z}{R}}\right)\>,$
(2) $\displaystyle\phi=$ $\displaystyle\pi-l\>,$ (3)

with $(\theta,\phi)$ the co-latitude and longitude in that new coordinate
system. The choice of zero longitude is arbitrary, we impose that choice such
that if our observer was instead at a latitude $\lambda=\pi/2-\epsilon$,
$\epsilon\ll 1$ and longitude $0\text{\,}\mathrm{\SIUnitSymbolDegree}$,
$\phi=0$ would point towards the North Pole of the sphere.

Given a position $(\lambda,l)$ and an altitude $z$ for our telescope, we
rotate the Earth template such that the $(\lambda,l)$ corresponds to the pole,
and then project all pixels within $\alpha_{fov}$ using Equations 2 and 3. The
resulting ground template for the telescope is then infilled up to the horizon
line. A depiction of the process can be seen in Figure 2. We make the code
used to generate the projection available in a public Github repository.
111https://github.com/AERAdler/thefloorisdata

Figure 2: Top left: a map of microwave emission at $91\text{\,}\mathrm{GHz}$
from the Southern Hemisphere for a given day, obtained from SSMIS data. The
path of the balloon is overlaid in green. Top right, example of a balloon’s
field of view (black circle) as it passes over the coast of Chile at an
altitude of $35\text{\,}\mathrm{km}$. Bottom panel: projection of the ground
in the telescope’s frame of reference. The dashed line indicates zero
elevation, the horizon is $6\text{\,}\mathrm{\SIUnitSymbolDegree}$ below that.
Letters denote cardinal directions.

##  3 THE TELESCOPE

Our telescope is based on the two-lens silicon refractor design presented in
25, which we already worked with in a previous proceeding[26]. As we are
focused on the large angular scales, the design is optimized to have a wide
diffraction-limited field of view (DLFOV). The telescope should have
consistent performance from 30 to $350\text{\,}\mathrm{GHz}$, since multi-
frequency observations are needed to disentangle foregrounds. The two lenses
are roughly $310\text{\,}\mathrm{mm}$ in diameter, with a sag of
$16\text{\,}\mathrm{mm}$ on the primary. The axial distance from primary to
secondary is $450\text{\,}\mathrm{mm}$, and from secondary to the focal plane
$137\text{\,}\mathrm{mm}$. Accounting for lens thickness, this results in a
compact telescope roughly $625\text{\,}\mathrm{mm}$ in length. The DLFOV is
$30\text{\,}\mathrm{\SIUnitSymbolDegree}$ up to $350\text{\,}\mathrm{GHz}$.
The focal plane has a diameter of $250\text{\,}\mathrm{mm}$, therefore the
plate scale is
$\sim$0.12\text{\,}\mathrm{\SIUnitSymbolDegree}\text{\,}{\mathrm{mm}}^{-1}$$.
With a pitch size of $6\text{\,}\mathrm{mm}$, this means around $\sim 1300$
pixels could be fit on that focal plane.

The design is simulated with the optical modelling software TICRA-Tools[27],
formerly GRASP. Using physical optics (PO) and the Physical Theory of
Diffraction (PTD), we can simulate the propagation of radiation in the time-
reversed sense, that is, from the focal plane to the far-field. We generate
far-field beams for a given horn on the focal plane, modelling the radiation
propagating from the horn through the secondary and the primary lenses.
Scattering by a vacuum window in front of the primary lens and internal
reflection in the lenses are approximated as a Lambertian source placed just
in front of the primary. The total power of the Lambertian source is set to be
$1\text{\,}\mathrm{\char 37\relax}$ of the total beam power, in line with
theory[28]. The effect of an absorbing forebaffle outside of the optics is
approximated by an aperture in an infinite screen situated at the forebaffle’s
end. The Lambertian source and the beam exiting the primary lens are
propagated through the aperture and to the far-field. We do not attempt to
simulate the cryostat that encloses the lenses and the focal plane.

We select 50 pairs of (x,y) positions at random on the focal plane and conduct
PO simulations for each of them at $95\text{\,}\mathrm{GHz}$. The far-field
beams are sampled on two grids: a
$$3\text{\,}\mathrm{\SIUnitSymbolDegree}$\times$3\text{\,}\mathrm{\SIUnitSymbolDegree}$$
grid samples the main beam finely, while a coarser
$$60\text{\,}\mathrm{\SIUnitSymbolDegree}$\times$60\text{\,}\mathrm{\SIUnitSymbolDegree}$$
grid enables us to probe the sidelobes. The co- and cross-polar beams in the
TICRA-Tools output are converted to healpix maps of the Stokes parameters
$(I,Q,U)$. Figure 3 shows the azimuthally-averaged Stokes I beam profiles at
$95\text{\,}\mathrm{GHz}$. Beams that are further off-axis will have
comparatively more extended sidelobes as the main beam’s centre moves relative
to the centre of the scattering.

Figure 3: Azimuthally-averaged Stokes I far-field beam profile for all 50
detectors at $95\text{\,}\mathrm{GHz}$. A darker color indicates greater
distance from the center of the focal plane to the detector, up to
$14.5\text{\,}\mathrm{\SIUnitSymbolDegree}$ off-axis. The dashed black line
represents the far-field beam profile for an on-axis detector. We distinguish
a 46.5’ full-width half maximum main lobe and a much weaker sidelobe. The
orange outline shows parts of a beam that can intersect the ground for the
ground-based experiment, with the vertical dotted line marking when a beam on
the edge of the focal plane will start to see Cerro Toco. The pink outline
represents the same situation for the balloon-borne telescope. The dashed pink
line shows when a beam at the center of the focal plane will start to detect
the ground. By coincidence, it is the same value for both telescopes.

##  4 CONVOLUTION, TIMESTREAMS, MAPS

The signal measured by a detector pointing at celestial coordinates $(ra,dec)$
is the convolution of the sky signal and the beam over the full
$4\pi\text{\,}\mathrm{sr}$ of the sky. CMB experiments typically sample their
signal several times per second, and can observe for several weeks or month.
Therefore, convolution in real space for each of these measurements is
computationally very costly. However, convolution in real space is equivalent
to multiplication in harmonic space. The beam and sky maps can be decomposed
into the spherical harmonic base, giving us two set of coefficients ($b_{\ell
m}$ and $a_{\ell m}$ respectively). Multiplying these together is equivalent
to convolving the sky and the beam in real space[29, 30, 31].

This is the motivation behind beamconv, a python package that can generate
streams of time-ordered data (TOD) when given the spherical harmonic
coefficients and an observation schedule. Originally, beamconv was developed
to probe the effect of various beam systematics on our ability to reconstruct
accurate sky maps and CMB power spectra. beamconv uses spin-weighted spherical
harmonics (SWSH) to accurately capture the polarised nature of the beam and
the sky. At a given $(ra,dec)$, the data model simulated by beamconv is[32]:

$d=\\!\sum_{\ell,m,s}\Big{\\{}b^{\widetilde{I}}_{\ell s}\\!a^{I}_{\ell
m}\\!+b^{\widetilde{V}}_{\ell s}\\!a^{V}_{\ell
m}\\!+\frac{1}{2}\Big{[}{}_{-2}b^{\widetilde{P}}_{\ell s}\\!{}_{2}a^{P}_{\ell
m}+{}_{2}b^{\widetilde{P}}_{\ell s}\\!{}_{-2}a^{P}_{\ell
m}\Big{]}\Big{\\}}\times\sqrt{\frac{4\pi}{2\ell+1}}\mathrm{e}^{-\mathrm{i}s\psi}{}_{s}Y_{\ell
m}(ra,dec),$ (4)

with $d$ the measured time-ordered signal, $P=Q+iU$ the complex form of the
polarisation vector, ${}_{s}a^{X}_{\ell m}$ the SWSH coefficient for sky
component $X$ and $\psi$ the position angle of the detector. In this work, we
modified the data model, replacing the sky SWSH coefficients with those of the
ground and evaluating the SWSH in $(az,el)$ coordinates, thereby generating a
ground pickup signal that gets added to the normal beamconv timestreams.

To demonstrate the algorithm, we first perform a constant-elevation-scan with
our 50 detectors over all $360\text{\,}\mathrm{\SIUnitSymbolDegree}$ of
azimuth with and without the Atacama-like ground template. In Figure 4, we see
the difference between these two timestreams for the ’A’ detector of each
detector pair. The outline of the main elevation features are very visible for
the several of the timestreams, with detectors that point lower in elevation
picking up a larger ground signal. A representation of the focal plane is
visible as the right panel in Figure 4. It is also interesting to notice that
the azimuth of the main feature, which corresponds to pickup of the Cerro Toco
peak, varies from timestream to timestream. Complexities of the beam model
will therefore impact both the amplitude of pickup and its azimuthal
dependence.

Figure 4: Left: Difference between ground-contaminated and pure sky
timestreams for a full rotation in azimuth at a fixed boresight elevation of
$55\text{\,}\mathrm{\SIUnitSymbolDegree}$. The x-axis gives the azimuth at
which a particular detector is pointing. Different colors denote elevation
offsets in the range [$-14.5\text{\,}\mathrm{\SIUnitSymbolDegree}$,
$0\text{\,}\mathrm{\SIUnitSymbolDegree}$], with dark blue indicating the
timestream of a detector pointing $14.5\text{\,}\mathrm{\SIUnitSymbolDegree}$
lower than the boresight. All detectors pointing higher than the boresight
have had their timestream colored in yellow. Right: Pointing of the 50
detector pairs: their positions have been generated at random. The color of
each dot indicates the maximum difference between the two timestreams.

All ground-based CMB experiments filter their TOD in some way. Depending on
the expected noise, the best filtering procedure might differ. In simulations
and previous experiments, high-pass, pass-band[32], or polynomial[33] filters
have been applied to TOD. In our case, we implement a first-order Butterworth
high-pass filter, with the cutoff frequency set to $0.1\text{\,}\mathrm{Hz}$.
The TOD is then combined with detector pointing information and binned to
create $(I,Q,U)$ sky maps using a naive map-making algorithm. By
differentiating output maps that follow the same scan strategy, we can gauge
the effects of different modelling choices. In particular, we can look at the
difference maps between scans with and without the presence of a ground
template, or how well a particular filter reduces pickup.

##  5 RESULTS

Each of our two telescopes has its own scan strategy. The ground telescope’s
constant elevation scan probes a smaller area of the sky, but very deep, while
the balloon experiment’s scan strategy covers more than half of the celestial
sphere.

The ground-based telescope performs a constant-elevation scan, alternating
between two patches of the sky separated by the Galactic plane which we
typically call the ”Southern” and ”Northern” patches. The telescope slews in
azimuth between two limits at a fixed elevation at a rate of
$1\text{\,}\mathrm{\SIUnitSymbolDegree}\text{\,}{\mathrm{s}}^{-1}$, going back
and forth. Every ten minutes, the telescope moves in elevation to track the
motion of the original patch center. Boresight elevation is constrained
between 50 and $62\text{\,}\mathrm{\SIUnitSymbolDegree}$. The telescope
samples the sky at $50\text{\,}\mathrm{Hz}$ for 31 days. We show results for
the Southern Patch in Figure 5. There is a large amount of pickup due to the
ground, which the filter is partly successful at cutting out.

Figure 5: Binned Stokes I maps for the ground-based telescope in the region
$-50\leq\textrm{ra}\leq 110$, $-70\leq\textrm{dec}\leq-10$. Left panel: un-
filtered timestreams get binned into a map dominated by ground pickup. Towards
the top of the map, a region with less pickup shows the CMB signal. Right
panel: applying the high-pass filter on the timestreams gets rid of the
majority of the pickup, but also suppresses large-scale CMB temperature
anisotropies.

The balloon-borne telescope drifts beneath the balloon at an altitude of
$35\text{\,}\mathrm{km}$ above sea level, following the path depicted in the
top-left plot of Figure 2. The position is updated daily for a month. It
rotates continuously in azimuth at a speed of
$30\text{\,}\mathrm{\SIUnitSymbolDegree}\text{\,}{\mathrm{s}}^{-1}$ at a fixed
elevation of $35\text{\,}\mathrm{\SIUnitSymbolDegree}$. Every hour, the
direction of rotation is reversed. We make daily maps that are then co-added
to give the map for the whole simulated mission. The results can be seen in
Figure 6. The pickup is concentrated at positive declinations, reflecting that
these can only be accessed from positions lower on the horizon in the Southern
Hemisphere. It is much weaker than in the ground-based simulation, in line
with the observation in Figure 3 that a smaller fraction of each beam can
pickup the ground emission.

Figure 6: Difference in Stokes I maps between scans with and without ground
SSN for the balloon-borne telescope’s observations. The pickup is globally
quite weak, and concentrated towards the northern-most declinations sampled by
the telescope.

##  6 CONCLUSION

We have presented an extension to the beamconv code than enables the
generation of time-ordered data from beam-convolved ground maps. We
demonstrated that new extension by simulating two experimental setups, a
ground-based telescope picking up thermal emission from nearby terrain and an
instrument suspended from a balloon drifting through the stratosphere. In
order to generate the second setup’s ground templates, we also wrote a small
separate program to project daily maps of microwave emission from the SSMIS
satellite into ground templates in the telescope’s frame of reference. Our
ground templates make a number of simplifying assumptions, like the absence of
a ground screen for the ground-based telescope or ignoring solar reflections
and the balloon itself for the airborne instrument. While our beams were
generated from a fictional telescope design, our framework can easily be
adapted to simulate scan-synchronous noise for real current and future
experiments. In particular, we aim to use all of beamconv’s current
capabilities to study systematics effects for a future Ultra Long Duration
Balloon CMB experiment. On top of pickup effects, we will include half-wave
plate and beam non-idealities.

###### Acknowledgements.

The authors would like to thank Steven Benton, Kevin Crowley, Nadia Dachlythra
and Reijo Keskitalo for productive discussions. Alexandre Adler’s
participation to this SPIE conference is funded by the Birger and Gurli
Grundstroms research grant fund.

## References

  * [1] Fixsen, D. J., “The Temperature of the Cosmic Microwave Background,” ApJ 707, 916–920 (Dec. 2009).
  * [2] Kovac, J., Carlstrom, J. E., Dragovan, M., Holzapfel, W. L., and Halverson, N. W., “Imaging the Cosmic Microwave Background: the Degree Angular Scale Interferometer,” in [American Astronomical Society Meeting Abstracts #192 ], American Astronomical Society Meeting Abstracts 192, 52.13 (May 1998).
  * [3] Thornton, R. J. et al., “The Atacama Cosmology Telescope: The Polarization-sensitive ACTPol Instrument,” ApJS 227, 21 (Dec. 2016).
  * [4] The Polarbear Collaboration, Errard, J., Ade, P. A. R., Anthony, A., Arnold, K., Aubin, F., Boettger, D., Borrill, J., Cantalupo, C., Dobbs, M. A., Flanigan, D., Ghribi, A., Halverson, N., Hazumi, M., Holzapfel, W. L., Howard, J., Hyland, P., Jaffe, A., Keating, B., Kisner, T., Kermish, Z., Lee, A. T., Linder, E., Lungu, M., Matsumura, T., Miller, N., Meng, X., Myers, M., Nishino, H., O’Brient, R., O’Dea, D., Reichardt, C., Schanning, I., Shimizu, A., Shimmin, C., Shimon, M., Spieler, H., Steinbach, B., Stompor, R., Suzuki, A., Tomaru, T., Tran, H. T., Tucker, C., Quealy, E., Richards, P. L., and Zahn, O., “The new generation CMB B-mode polarization experiment: POLARBEAR,” in [Proceedings of the XLVth Rencontres de Moriond, Cosmology ], 1 (Nov. 2010).
  * [5] Ogburn, R. W., “The BICEP2 CMB polarization experiment,” in [Millimeter, Submillimeter, and Far-Infrared Detectors and Instrumentation for Astronomy V ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 7741, 77411G (July 2010).
  * [6] Crill, B. P. et al., “BOOMERANG: A Balloon-borne Millimeter-Wave Telescope and Total Power Receiver for Mapping Anisotropy in the Cosmic Microwave Background,” ApJS 148, 527–541 (Oct. 2003).
  * [7] Hanany, S. et al., “MAXIMA-1: A Measurement of the Cosmic Microwave Background Anisotropy on Angular Scales of $10^{-5}\>{}^{\circ}$,” ApJ 545, L5–L9 (Dec. 2000).
  * [8] Nagy, J., Optical Development for the SPIDER Balloon-Borne CMB Polarimeter, PhD thesis, Case Western Reserve University (2017).
  * [9] Smoot, G. F. et al., “Structure in the COBE Differential Microwave Radiometer First-Year Maps,” ApJ 396, L1 (Sept. 1992).
  * [10] Bennett, C. L., Larson, D., Weiland, J. L., Jarosik, N., Hinshaw, G., Odegard, N., Smith, K. M., Hill, R. S., Gold, B., Halpern, M., Komatsu, E., Nolta, M. R., Page, L., Spergel, D. N., Wollack, E., Dunkley, J., Kogut, A., Limon, M., Meyer, S. S., Tucker, G. S., and Wright, E. L., “Nine-year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Final Maps and Results,” ApJS 208, 20 (Oct 2013).
  * [11] Planck Collaboration, “Planck 2018 results. I. Overview and the cosmological legacy of Planck,” A&A 641, A1 (Sept. 2020).
  * [12] Kovac, J. M., Leitch, E. M., Pryke, C., Carlstrom, J. E., Halverson, N. W., and Holzapfel, W. L., “Detection of polarization in the cosmic microwave background using DASI,” Nature 420, 772–787 (Dec. 2002).
  * [13] Nagy, J. M., Ade, P. A. R., Amiri, M., Benton, S. J., Bergman, A. S., Bihary, R., Bock, J. J., Bond, J. R., Bryan, S. A., Chiang, H. C., Contaldi, C. R., Doré, O., Duivenvoorden, A. J., Eriksen, H. K., Farhang, M., Filippini, J. P., Fissel, L. M., Fraisse, A. A., Freese, K., Galloway, M., Gambrel, A. E., Gandilo, N. N., Ganga, K., Gudmundsson, J. E., Halpern, M., Hartley, J., Hasselfield, M., Hilton, G., Holmes, W., Hristov, V. V., Huang, Z., Irwin, K. D., Jones, W. C., Kuo, C. L., Kermish, Z. D., Li, S., Mason, P. V., Megerian, K., Moncelsi, L., Morford, T. A., Netterfield, C. B., Nolta, M., Padilla, I. L., Racine, B., Rahlin, A. S., Reintsema, C., Ruhl, J. E., Runyan, M. C., Ruud, T. M., Shariff, J. A., Soler, J. D., Song, X., Trangsrud, A., Tucker, C., Tucker, R. S., Turner, A. D., Van Der List, J. F., Weber, A. C., Wehus, I. K., Wiebe, D. V., and Young, E. Y., “A New Limit on CMB Circular Polarization from SPIDER,” ApJ 844, 151 (Aug. 2017).
  * [14] Padilla, I. L., Eimer, J. R., Li, Y., Addison, G. E., Ali, A., Appel, J. W., Bennett, C. L., Bustos, R., Brewer, M. K., Chan, M., Chuss, D. T., Cleary, J., Couto, J., Dahal, S., Denis, K., Dünner, R., Essinger-Hileman, T., Fluxá, P., Gothe, D., Haridas, S. K., Harrington, K., Iuliano, J., Karakla, J., Marriage, T. A., Miller, N. J., Núñez, C., Parker, L., Petroff, M. A., Reeves, R., Rostem, K., Stevens, R. W., Nunes Valle, D. A., Watts, D. J., Weiland, J. L., Wollack, E. J., and Xu, Z., “Two-year Cosmology Large Angular Scale Surveyor (CLASS) Observations: A Measurement of Circular Polarization at 40 GHz,” ApJ 889, 105 (Feb. 2020).
  * [15] Komatsu, E., “New physics from polarised light of the cosmic microwave background,” Nature Reviews Physics 21, 732 (May 2022).
  * [16] Crittenden, R., Davis, R. L., and Steinhardt, P. J., “Polarization of the Microwave Background Due to Primordial Gravitational Waves,” ApJ 417, L13 (Nov 1993).
  * [17] Abitbol, Maximilian H. et al., “CMB-S4 Technology Book, First Edition,” arXiv e-prints 1706, arXiv:1706.02464 (June 2017).
  * [18] Suzuki, A., Ade, P. A. R., Akiba, Y., Alonso, D., Arnold, K., Aumont, J., Baccigalupi, C., Barron, D., Basak, S., Beckman, S., Borrill, J., Boulanger, F., Bucher, M., Calabrese, E., Chinone, Y., Cho, S., Crill, B., Cukierman, A., Curtis, D. W., de Haan, T., Dobbs, M., Dominjon, A., Dotani, T., Duband, L., Ducout, A., Dunkley, J., Duval, J. M., Elleflot, T., Eriksen, H. K., Errard, J., Fischer, J., Fujino, T., Funaki, T., Fuskeland, U., Ganga, K., Goeckner-Wald, N., Grain, J., Halverson, N. W., Hamada, T., Hasebe, T., Hasegawa, M., Hattori, K., Hattori, M., Hayes, L., Hazumi, M., Hidehira, N., Hill, C. A., Hilton, G., Hubmayr, J., Ichiki, K., Iida, T., Imada, H., Inoue, M., Inoue, Y., Irwin, K. D., Ishino, H., Jeong, O., Kanai, H., Kaneko, D., Kashima, S., Katayama, N., Kawasaki, T., Kernasovskiy, S. A., Keskitalo, R., Kibayashi, A., Kida, Y., Kimura, K., Kisner, T., Kohri, K., Komatsu, E., Komatsu, K., Kuo, C. L., Kurinsky, N. A., Kusaka, A., Lazarian, A., Lee, A. T., Li, D., Linder, E., Maffei, B., Mangilli, A., Maki, M., Matsumura, T., Matsuura, S., Meilhan, D., Mima, S., Minami, Y., Mitsuda, K., Montier, L., Nagai, M., Nagasaki, T., Nagata, R., Nakajima, M., Nakamura, S., Namikawa, T., Naruse, M., Nishino, H., Nitta, T., Noguchi, T., Ogawa, H., Oguri, S., Okada, N., Okamoto, A., Okamura, T., Otani, C., Patanchon, G., Pisano, G., Rebeiz, G., Remazeilles, M., Richards, P. L., Sakai, S., Sakurai, Y., Sato, Y., Sato, N., Sawada, M., Segawa, Y., Sekimoto, Y., Seljak, U., Sherwin, B. D., Shimizu, T., Shinozaki, K., Stompor, R., Sugai, H., Sugita, H., Suzuki, J., Tajima, O., Takada, S., Takaku, R., Takakura, S., Takatori, S., Tanabe, D., Taylor, E., Thompson, K. L., Thorne, B., Tomaru, T., Tomida, T., Tomita, N., Tristram, M., Tucker, C., Turin, P., Tsujimoto, M., Uozumi, S., Utsunomiya, S., Uzawa, Y., Vansyngel, F., Wehus, I. K., Westbrook, B., Willer, M., Whitehorn, N., Yamada, Y., Yamamoto, R., Yamasaki, N., Yamashita, T., and Yoshida, M., “The LiteBIRD Satellite Mission: Sub-Kelvin Instrument,” Journal of Low Temperature Physics 193, 1048–1056 (Dec 2018).
  * [19] Poletti, D., Fabbian, G., Le Jeune, M., Peloton, J., Arnold, K., Baccigalupi, C., Barron, D., Beckman, S., Borrill, J., Chapman, S., Chinone, Y., Cukierman, A., Ducout, A., Elleflot, T., Errard, J., Feeney, S., Goeckner-Wald, N., Groh, J., Hall, G., Hasegawa, M., Hazumi, M., Hill, C., Howe, L., Inoue, Y., Jaffe, A. H., Jeong, O., Katayama, N., Keating, B., Keskitalo, R., Kisner, T., Kusaka, A., Lee, A. T., Leon, D., Linder, E., Lowry, L., Matsuda, F., Navaroli, M., Paar, H., Puglisi, G., Reichardt, C. L., Ross, C., Siritanasak, P., Stebor, N., Steinbach, B., Stompor, R., Suzuki, A., Tajima, O., Teply, G., and Whitehorn, N., “Making maps of cosmic microwave background polarization for B-mode studies: the POLARBEAR example,” A&A 600, A60 (Apr. 2017).
  * [20] Duivenvoorden, A. J., Gudmundsson, J. E., and Rahlin, A. S., “Full-sky beam convolution for cosmic microwave background applications,” MNRAS 486, 5448–5467 (Jul 2019).
  * [21] Bommarito, J. J., “DMSP special sensor microwave imager sounder (SSMIS),” in [Microwave Instrumentation for Remote Sensing of the Earth ], Shiue, J. C., ed., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 1935, 230–238 (Aug. 1993).
  * [22] Kunkee, D. B., Poe, G. A., Boucher, D. J., Swadley, S. D., Hong, Y., Wessel, J. E., and Uliana, E. A., “Design and Evaluation of the First Special Sensor Microwave Imager/Sounder,” IEEE Transactions on Geoscience and Remote Sensing 46, 863–883 (Apr. 2008).
  * [23] Brodzik, M. J., Long, D. G., Hardman, M. A., A. Paget, A., and Armstrong, R., “Measures calibrated enhanced-resolution passive microwave daily ease-grid 2.0 brightness temperature esdr,” (2016).
  * [24] Górski, K. M., Hivon, E., Banday, A. J., Wand elt, B. D., Hansen, F. K., Reinecke, M., and Bartelmann, M., “HEALPix: A Framework for High-Resolution Discretization and Fast Analysis of Data Distributed on the Sphere,” ApJ 622, 759–771 (Apr 2005).
  * [25] Gudmundsson, J. E., “Geometrical and physical optics analysis for mm-wavelength refractor telescopes designed to map the cosmic microwave background,” Appl. Opt. 59, 3324 (Apr. 2020).
  * [26] Adler, A. E. and Gudmundsson, J. E., “Modeling sidelobe response for ground-based mm-wavelength telescopes with the geometrical theory of diffraction,” in [Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 11453, 114534O (Dec. 2020).
  * [27] Ticra Tools User Manual, version 19.1.1, https://www.ticra.com/ (2020).
  * [28] Ruze, J., “Antenna tolerance theory - a review,” Proceedings of the IEEE 54 (Apr. 1966).
  * [29] Wandelt, B. D. and Górski, K. M., “Fast convolution on the sphere,” Phys. Rev. D 63, 123002 (June 2001).
  * [30] Challinor, A., Fosalba, P., Mortlock, D., Ashdown, M., Wandelt, B., and Górski, K., “All-sky convolution for polarimetry experiments,” Phys. Rev. D 62, 123002 (Dec. 2000).
  * [31] Prézeau, G. and Reinecke, M., “Algorithm for the Evaluation of Reduced Wigner Matrices,” ApJS 190, 267–274 (Oct. 2010).
  * [32] Duivenvoorden, A. J., Adler, A. E., Billi, M., Dachlythra, N., and Gudmundsson, J. E., “Probing frequency-dependent half-wave plate systematics for CMB experiments with full-sky beam convolution simulations,” MNRAS 502, 4526–4539 (Apr. 2021).
  * [33] Spider Collaboration, Ade, P. A. R., Amiri, M., Benton, S. J., Bergman, A. S., Bihary, R., Bock, J. J., Bond, J. R., Bonetti, J. A., Bryan, S. A., Chiang, H. C., Contaldi, C. R., Doré, O., Duivenvoorden, A. J., Eriksen, H. K., Farhang, M., Filippini, J. P., Fraisse, A. A., Freese, K., Galloway, M., Gambrel, A. E., Gandilo, N. N., Ganga, K., Gualtieri, R., Gudmundsson, J. E., Halpern, M., Hartley, J., Hasselfield, M., Hilton, G., Holmes, W., Hristov, V. V., Huang, Z., Irwin, K. D., Jones, W. C., Karakci, A., Kuo, C. L., Kermish, Z. D., Leung, J. S. Y., Li, S., Mak, D. S. Y., Mason, P. V., Megerian, K., Moncelsi, L., Morford, T. A., Nagy, J. M., Netterfield, C. B., Nolta, M., O’Brient, R., Osherson, B., Padilla, I. L., Racine, B., Rahlin, A. S., Reintsema, C., Ruhl, J. E., Runyan, M. C., Ruud, T. M., Shariff, J. A., Shaw, E. C., Shiu, C., Soler, J. D., Song, X., Trangsrud, A., Tucker, C., Tucker, R. S., Turner, A. D., van der List, J. F., Weber, A. C., Wehus, I. K., Wen, S., Wiebe, D. V., and Young, E. Y., “A Constraint on Primordial B-modes from the First Flight of the SPIDER Balloon-borne Telescope,” ApJ 927, 174 (Mar. 2022).


# PATRONoC: Parallel AXI Transport Reducing Overhead for Networks-on-Chip
targeting Multi-Accelerator DNN Platforms at the Edge

Vikram Jain†, Matheus Cavalcante⋆, Nazareno Bruschi⊕, Michael Rogenmoser⋆,
Thomas Benz⋆,
Andreas Kurth⋆, Davide Rossi⊕, Luca Benini⋆⊕ and Marian Verhelst†
†ESAT-MICAS, KU Leuven, Belgium, ⋆IIS, ETH Zurich, Switzerland, ⊕University of
Bologna, Italy
Email<EMAIL_ADDRESS>
###### Abstract

Emerging deep neural network (DNN) applications require high-performance
multi-core hardware acceleration with large data bursts. Classical network-on-
chips (NoCs) use serial packet-based protocols suffering from significant
protocol translation overheads towards the endpoints. This paper proposes
PATRONoC, an open-source fully AXI-compliant NoC fabric to better address the
specific needs of multi-core DNN computing platforms. Evaluation of PATRONoC
in a 2D-mesh topology shows 34 higher area efficiency compared to a state-of-
the-art classical NoC at 1 G . PATRONoC’s throughput outperforms a baseline
NoC by $2$-$8$$\times$ on uniform random traffic and provides a high
aggregated throughput of up to 350 Gi / on synthetic and DNN workload traffic.

###### Index Terms:

Networks-on-chip, multi-core DNN platforms, AXI, high-performance systems

##  I Introduction

Deep neural networks (DNNs) have become one of the primary workloads in
computing platforms of data centers and edge devices in the internet of things
(IoT). Given the high proliferation of DNN workloads, research into designing
and developing high-performance specialized hardware accelerators for DNN has
gained much interest, as evidenced by the several DNN accelerators presented
in the past decade [1]. In the quest to support the ever-growing requirements
of DNN workloads, hardware architectures have evolved from small single-core
implementations to homogeneous [2] and heterogeneous [3, 4, 5] multi-core
hardware implementations111In this paper, the terms core and accelerator are
used interchangeably.. The trend of going multi-core can bring performance
gains. However, it also brings new challenges, such as resource partitioning,
workload mapping, complex hardware implementations, memory hierarchy design,
and data communication bottlenecks between cores.

Multi-CPU-based general-purpose computing traditionally uses networks-on-chip
(NoCs) and their various optimizations for inter-CPU data communication. Many
topologies exist to balance the scalability of CPU cores, throughput, latency,
and area impact of the NoC. Moreover, NoC protocols are designed for
packetization and serialization over fairly narrow channels between cores
(e.g., 32 bits), which reduces the number of routing resources needed.
However, this implies additional hardware at the network’s edges for protocol
translation and serialization/deserialization (SERDES) from standard channel-
oriented protocols at the endpoints (e.g., AXI4 or AXI5) to the NoC protocol.
Moreover, due to their serialized nature, these NoCs need a high clock
frequency to meet the bandwidth requirements, thus needing clock domain
crossing hardware.

Such traditional narrow-channel NoCs work well for inter-CPU cache traffic.
However, the traffic of DNN workloads is mostly deterministic, with large
bursts of non-coherent data movements requiring high bandwidth interconnection
to achieve high performance and low latency. To achieve high bandwidth,
typical solutions either 1) use a narrow NoC and operate it at 2-8$\times$ the
core frequency [6] or 2) build a wide NoC with multiple channels [7]. The
latter solution gains traction as advanced technology scaling enables the
area-efficient integration of more and more on-chip interconnect resources [7,
8]. However, modern NoCs need more than just wide links to answer the needs of
DNN workloads, as packet-based serial NoC protocols are inadequate for
workloads that rely on burst-based traffic.

This paper proposes a template for burst-based homogeneous AXI-compliant NoCs
to address the requirements of emerging multi-core DNN platforms and to tackle
the challenges of packet-based serial NoCs. This work builds upon the open-
source elementary AXI building blocks of [9], which focuses on crossbar-based
topologies, towards a fully-configurable open source AXI-based NoC framework,
PATRONoC. PATRONoC is subsequently used in a mesh topology and extensively
benchmarked to demonstrate the benefits of having AXI-based NoCs. As such,
this work makes the following contributions:

  * •

We present an open-source parameterizable AXI-compliant NoC designed for
providing high bandwidth links for multi-core DNN computing platforms. The NoC
is available at https://github.com/pulp-platform/axi.

  * •

We demonstrate that using an AXI protocol for the NoC creates a fully
homogeneous network interface to avoid high cost of protocol translation and
provides a standard plug-and-play support for ease of integration.

  * •

We show that using the AXI protocol end-to-end, a multi-channel, wide NoC with
burst support and high bandwidth between cores as well as to-and-from memory
can be supported, thereby improving performance of DNN applications on multi-
core platforms.

The rest of the paper is organized as follows. Section II discusses the
architectural overview of the proposed NoC, followed by details of the NoC’s
physical implementation with GlobalFoundries’ modern 22FDX technology in
Section III. We evaluate our NoC with synthetic and real traffic patterns
extracted from DNN workloads in Section IV. Subsequently, we compare our work
against other modern NoC solutions in Section V before presenting our
conclusions in Section VI.

##  II Interconnect Architecture of PATRONoC

Figure 1: PATRONoC instances as a 2x2 mesh (left) and a 4x4 mesh (right). The
AXI masters and slaves are not shown in the 4x4 mesh for ease of readability.
Elementary blocks used for the NoC are also shown: XP (bottom-left) and XBAR
(bottom-right). Red XP is 3-master and 3-slave, light blue XP is 4-master and
4-slave, and, dark blue XP is 5-master and 5-slave.

This section provides architectural and physical implementation details of
PATRONoC for a mesh topology. NoCs are built with many elementary routing
elements, each forwarding data from the ingress ports to the egress ports
according to their topology-specific routing table. In this work, we extend
the AXI crosspoint (XP) from [9], shown in Fig. 1 (bottom), allowing it to be
used as PATRONoC’s routing element. The XP consists of a configurable crossbar
(XBAR) switch and ID remappers to ensure isomorphic XP ports. It is fully AXI-
compliant and supports bursts, multiple outstanding transactions, and
transaction ordering. We used the XP as the building block for a homogeneous,
2D mesh topology NoC with widely configurable dimensions, as shown in Fig. 1.
Although this work uses the 2D mesh as a proof-of-concept, any regular
topology, such as a torus, butterfly, or ring, can also be modularly built
using our building blocks. We focused on the mesh due to its popularity in
research and its remarkable simplicity, scalability, and efficiency [7]. Fig.
1 shows the two mesh topologies, 2x2 (top-left) and 4x4 (top-right), used to
evaluate PATRONoC.

The meshes are built by instantiating the XPs in a 2D grid and connecting the
NESW-bound links to neighboring XPs. AXI masters and slaves can be connected
as NoC endpoints at each XP. A common AXI master is a core or a DNN
accelerator, and AXI slaves can be memory or I/O tiles. Each XBAR is
configured with a static routing table used for deterministic dimension-
ordered routing in the mesh. Specifically, PATRONoC uses a source-based YX
routing scheme, as shown with the green arrows in Fig. 1, to reduce the
complexity of the route calculation step of the crosspoints. In this
algorithm, a transaction is first passed forward in the same column until it
reaches the same row as the destination XP and then passed forward in the same
row until it reaches the destination XP. An automated script generates the
address-based routing table for each XP which is used for routing the AXI
transactions based on their destination address.

TABLE I: Main parameters of the PATRONoC 2D mesh. Parameter | Values
---|---
Mesh Dimension |  N$\times$M
Number of AXI Masters |  1 to N$\times$M (default)
Number of AXI Slaves |  1 to N$\times$M (default)
Data Width |  81024
Address Width |  Arch. dependent (32 or 64)
ID Width |  1 to 16
Max #Outstanding Trans. | 1 to 128
XBAR Connectivity | Partial (default) or Fully connected
Register Slice | Single channel or all channels (default)

PATRONoC is highly parameterizable, taking advantage of the flexibility of the
AXI protocol. The parameters that can be tuned at design time are shown in
Table I. The number of AXI masters and slaves indicate the number of connected
cores and memory/IO tiles in the design. Both ranges for possible number of
AXI masters and slaves are valid for the N$\times$M 2D mesh and are topology-
dependent. For example, in a concentrated mesh, multiple masters and slaves
can connect to the same XP. Furthermore, the data width (DW) can be tuned to
meet the system’s bandwidth requirements, while the address width (AW) can be
tuned to support a larger global address space.

The AXI protocol identifies transactions with IDs used by the master endpoints
to distinguish independent transactions. The number of unique IDs can be
configured using the ID width (IW) and increases with the number of masters.
All transactions from the same master with the same ID must remain ordered,
but there is no ordering requirement between transactions with different IDs.
Multiple outstanding transactions enable the master to hide the memory
latency. A higher max. number of outstanding transactions (MOT) improves
performance, as all AXI building blocks can support multiple in-flight
transactions, preventing bandwidth degradation when the NoC is saturated.

The XBAR connectivity parameter configures the XP to either connect all slave
ports to all master ports in the case of a fully-connected network or
partially connect slaves and masters in the case of a mesh or other non-point-
to-point topologies. The last parameter is the register slice (cut), shown in
Fig. 1, that can be optionally inserted at design time on some or all AXI
channels, improving the timing of the design at the cost of increased latency.
The rest of the paper evaluates PATRONoC in 2x2 and 4x4 mesh topologies with
multiple configurations based on the DW, AW, IW, and MOT parameters.

##  III Implementation Results

This section provides implementation results in terms of complexity and
scalability of the NoC and its parameters. The implementation is done in
GlobalFoundries’ 22FDX technology node using a ten-layer metal stack. We used
eight-track standard cells of SLVT/LVT type, characterized at worst-case
scenario (SS/0.72/125) for timing analysis. The designs from Section II are
synthesized using Synopsys’ Design Compiler 2022.03 in topographical mode,
taking physical endpoint placement constraints into account. All designs
achieve a clock frequency of 1 G at the worst-case condition corner with a
register slice on every AXI channel.

Figure 2: Implementation results showing area versus bisection bandwidth of
PATRONoC and ESP-NoC [10] in 2x2 mesh configurations. PATRONoC’s
configurations are represented as AXI_AW_DW_IW.

The 2x2 PATRONoC mesh, shown in Fig. 1 (top-left), is first synthesized with
different AW and DW parameters, keeping $\text{IW}{}=\qty{2}{}$,
$\text{MOT}{}=1$, and other parameters at default values. Fig. 2 shows the
area versus bisection bandwidth (DW-dependent) of the mesh for the different
configurations. As expected, the design area scales up with increasing AW and
DW, taking up mere 174 k for the smallest configuration of
$\text{AW}{}=\qty{32}{}$ and $\text{DW}{}=\qty{32}{}$. The biggest design
shown in Fig. 2, with $\text{DW}{}=\qty{512}{}$, takes an on-chip area of 830
k .

The benefit of having a homogeneous NoC is evident when the design is compared
to classic NoC solutions. This work uses ESP-NoC [10] as our baseline NoC.
ESP-NoC is a state-of-the-art open-source packet-based NoC including six
planes for coherent and non-coherent traffic for multi-core heterogeneous
systems. Synthesis results showing the area of the 2x2 ESP-NoC mesh in its
$32$\-- and $64$\--flit configurations are presented in Fig. 2. Compared to
PATRONoC’s configuration with $\text{AW}{}=\qty{32}{}$ and
$\text{DW}{}=\qty{64}{}$, ESP-NoC takes up 68 more area to provide only 25
more throughput (five $32$\- wide planes providing 160 G /). The area overhead
can be attributed to ESP-NoC’s multiple planes with large protocol translation
interfaces at each endpoint. The advantage of PATRONoC is much more evident in
Fig. 2 when comparing its area efficiency (slope) with ESP-NoC. We define area
efficiency as the bisection bandwidth normalized to the standard cell area,
providing a measure of NoC performance at a given complexity. Fig. 2 shows
that PATRONoC is closer to the Pareto front providing better area efficiency
compared to the ESP-NoC in $32$\- and $64$\- configurations.

Figure 3: Implementation results showing area vs. bisection bandwidth of
PATRONoC in 4x4 mesh configurations (left). Configurations are represented as
AXI_AW_DW_IW. Area vs. MOT tradeoff for $\text{DW}{}=\qty{64}{}$ (right).

We implement the 4x4 mesh shown in Fig. 1 (top-right) to show the scalability
of PATRONoC. For building the 4x4 mesh, the IW of the AXI blocks is increased
to 4 to support 16 unique IDs required for 16 masters. The results of the area
and bisection bandwidth of this mesh are summarized in Fig. 3 (left). As the
mesh dimensions change, the area overhead of the NoC becomes approximately 32
compared to the 2x2 mesh in similar AW and DW configurations, leading to a
drop in area efficiency by 25. Increasing the MOT improves the performance of
the NoC at the cost of larger complexity in terms of area. Fig. 3 (right)
shows the tradeoff between MOT and the area of the 4x4 PATRONoC with
$\text{DW}{}=\qty{64}{}$. While this work focuses more on performance and area
aspects of the NoC, the power consumption at 1 G for the 4x4 PATRONoC is 45 m
(for $\text{DW}{}=\qty{32}{}$) and 171 m (for $\text{DW}{}=\qty{512}{}$) on
uniform random traffic. This accounts for less than 10 of the projected power
consumption of a complete platform, assuming that a typical DNN accelerator
connected to one NoC node uses 100200 m .

##  IV Performance Evaluation

PATRONoC’s performance is characterized in terms of throughput versus injected
load through a cycle-accurate register-transfer level (RTL) simulation. This
section evaluates the performance of the 4x4 PATRONoC mesh in two
configurations: 1) as a slim NoC with $\text{DW}{}=\qty{32}{}$ and 2) as a
wide NoC with $\text{DW}{}=\qty{512}{}$, both with $\text{AW}{}=\qty{32}{}$,
$\text{IW}{}=\qty{4}{}$, and $\text{MOT}{}=8$. Each master is a DMA engine,
and the slaves are AXI-capable memories that cater to the DMA requests. The
configurable and workload-specific maximum burst length is used by the RTL
model of the DMA engine to create AXI-compliant bursts (adhering to address
boundaries and max number of beats) for the NoC. In our evaluation framework,
the workload-specific burst length is randomized within a user-defined range
to emulate a random burst length with a random source and destination address,
while the bursts in the NoC are subject to AXI compliance. All analyses assume
a clock frequency of 1 G for the endpoints and the NoCs.

###  IV-A Uniform Random Traffic

Figure 4: Uniform Random Traffic with Poisson distribution using Noxim
simulator for a 4x4 2D mesh and uniform random traffic on the slim PATRONoC
with increasing DMA burst length.

The Noxim simulator [11] is used to set the baseline NoC performance, taking a
4x4 mesh with the default XY routing, $32$\- flits, and eight flits per packet
to closely match the slim PATRONoC configurations. Fig. 4 shows the non-
exhaustive characterization of the Noxim NoC on this traffic in two
configurations: 1) a standard single virtual channel with $4$ flits per router
buffer for a compact implementation, and 2) four virtual channels with $32$
flits per router buffer for high performance. The saturation throughput of the
Noxim NoCs are 1.6 Gi / and 2.25 Gi /, respectively. Increasing the number of
virtual channels (VCs) [12] and flits per buffer improves the NoC’s
performance, but also increases router complexity.

Fig. 4 also shows the NoC throughput for the uniform random traffic running on
the 4x4 slim PATRONoC mesh. It is clear that PATRONoC is beneficial for burst
traffic. At small transfer lengths of less than 4, similar to normal CPU
traffic, PATRONoC performs equivalently to the Noxim NoC with 1.5 Gi /
throughput. However, when using longer bursts, PATRONoC’s performance improves
and reaches up to 19 Gi / aggregated throughput at DMA burst lengths up to 10
Ki and 64 Ki . This provides an improvement of 8.4$\times$ over the saturation
throughput achieved by the best Noxim NoC configuration (4 VCs, buffers
32-flit deep), showing that PATRONoC largely outperforms it by using bursts.

###  IV-B Synthetic Traffic

Fig. 5 shows the three synthetic traffic patterns considered: 1) all global
access, 2) max two-hop access, and, 3) max single-hop access. We characterize
the 4x4 PATRONoC mesh in both slim and wide configurations with the synthetic
patterns.

a.) All global access: In this traffic pattern, all the AXI master and DMA
endpoints communicate with a single slave endpoint leading to predominately
global accesses. Fig. 5a) shows this traffic pattern on the 4x4 mesh, where
the endpoint $(2,1)$ acts as the AXI slave. b.) Max two-hop access: In this
use case, the AXI slave accesses are distributed to four endpoints $(1,1)$,
$(1,2)$, $(2,1)$, and $(2,2)$. This considers architectures that have a
distributed shared L2/L1 memory, either uniform or non-uniform. The $16$ AXI
masters can communicate to any of the four endpoints, but in this case, the
masters are restricted to only communicate to slaves which are a maximum of
two hops away. c.) Max single-hop access: In this traffic pattern, the AXI
slaves are further distributed across eight endpoints along the edges except
for the corners. The $16$ masters are restricted to access only slaves which
are at most one hop away. The last two cases are considered because in traffic
from many DNN workloads, data scheduling can be done on nearby cores to
prevent long latency and low-performance data communication.

Figure 5: Synthetic traffic patterns for the performance evaluation.

The slim NoC can be used in architectures that are area-constrained but
require more throughput than what most traditional NoCs can provide. Fig. 6
(left) shows the NoC utilization, with respect to bisection bandwidth, of the
slim NoC on the three synthetic patterns at different burst sizes. Starting
with traffic pattern a.), the slim NoC provides a minimum of 1.5 Gi / of
throughput with short bursts. This is approximately 4.7 NoC utilization
considering the slim NoC has a 32 Gi / bisection bandwidth. The access pattern
limits the traffic to a few links of the NoC and, thus, a low utilization is
expected. The throughput improves considerably with increasing burst length
and reaches a maximum of 6 Gi / for burst lengths up to 64 Ki , providing a
NoC utilization of around 18.75. For pattern b.), the NoC performs similarly
to pattern a.) for short burst lengths. However, the aggregated throughput
improves considerably with larger bursts and saturates at 17.2 Gi / for burst
lengths up to 10;64 Ki . This leads to a higher NoC utilization of about
53.75, showing that all mesh links can be utilized more efficiently. Similar
to pattern b.), the pattern c.) under-performs at small bursts but the
aggregated saturation throughput at larger bursts improves to 22.5 Gi / for
bursts up to 64 Ki with a NoC utilization of 70.3.

Figure 6: NoC utilization at maximum injected load for the synthetic random
traffic running on the slim and wide PATRONoC using all global access, max 2
hop, and max 1 hop traffic patterns with different DMA burst sizes.

The wide NoC is geared towards high-bandwidth large-burst multi-core DNN-
workload traffic. A significant performance gain can be achieved with such
wide NoC, but being parameterizable means that also alternative DWs between
32;512 can be considered by designers to find an optimal size for given system
requirements. Fig. 6 (right) shows the NoC utilization characteristic of the
wide NoC on the synthetic access patterns with different burst sizes. For the
traffic pattern a.), the wide NoC can only achieve a utilization of 0.29 at
small bursts up to 4 large, providing a maximum throughput of 1.5 Gi /
(bisection bandwidth of 512 Gi /). As seen with the slim NoC, this is an
expected performance degradation with this access pattern. The degradation in
NoC utilization is further exacerbated by the wide DWs but short burst
lengths. The throughput improves, however, with larger burst sizes of up to 64
Ki and reaches saturation at 95 Gi / with 18.55 NoC utilization. Both patterns
b.) and c.) result in low throughput and utilization with small burst sizes.
The aggregated throughput improves at larger bursts with length up to 10;64 Ki
reaching 255 Gi / (49.8 utilization) and 345 Gi / (67.4 utilization) for the
patterns b.) and c.), respectively.

###  IV-C DNN Workload Traffic

Figure 7: Overview of the DNN workloads used for PATRONoC evaluation. FWD and
BWD in (a) represents the forward and backward propagation workloads,
respectively, used in DNN training.

Synthetic traffic does not capture the full scope of access patterns in real
multi-core hardware architectures running DNN workloads. In order to
characterize the NoC in more realistic use cases, this section evaluates three
emulated CNN-based workloads: a.) distributed training, b.) parallelized
convolutions, and c.) pipelined convolutions, shown in Fig. 7. We use GVSoC
[13] to generate real traffic patterns for the RTL simulation. GVSoC is an
open-source, highly configurable, and event-driven simulator for heterogeneous
RISC-V-based SoC platforms used for full-system software development and
performance evaluation.

a.) Distributed training: For this workload, we replicate and deploy a
ResNet-34 (90 channel shrink factor) distributed training model for the
ImageNet dataset on 16 cores. In terms of data communication, a mix of L2 to
L1 (core), L1 (core) to L2, and L1 (core) to L1 (core) transfers are needed.
b.) Parallelized convolution [14]: This is a CNN-based inference workload in
which the layers of the network and inputs are tiled and deployed on separate
cores. This is a pure L2 to L1 (core) and L1 (core) to L2 memory traffic
pattern and has no inter-core communication. c.) Pipelined convolution [14]:
Depth-first or pipeline dataflow is used in many new DNN platforms to
efficiently run CNN-based inference. In this scheme, layers are executed in
parallel, in a pipelined way across the different cores to reduce the data
traffic to higher memory levels [15]. This workload has mostly L1 (core) to L1
(core) traffic and only cores $0$ and $15$ do L1 (core) to/from L2 transfers.

Fig. 8 shows the evaluation results of the 4x4 slim and wide NoCs running the
three DNN workloads. For the slim NoC, the parallelized convolution—which
consists of mostly core to/from shared memory transfers—reaches a throughput
of 4.27 Gi /. For the training workload, the throughput is better than the
parallelized convolution workload as it involves a mix of core to/from shared
memory and core-to-core transfers. On the pipelined convolution, which
consists of predominantly core-to-core traffic, the NoC achieves a high 19.17
Gi / throughput. Similar trends are reported for the 4x4 PATRONoC wide NoC
shown in Fig. 8 (right), but at much higher throughput, with pipeline
convolution reaching a peak throughput of 310 Gi /.

Slim NoCWide NoC$0$$100$$200$$300$$5.18$$83.1$$4.27$$68.5$$19.17$$310.7$Throughput ( Gi /)Conv TrainPar ConvPipe Conv Figure 8: Throughput analysis for DNN workload traffic on the PATRONoC. TABLE II: Comparison of PATRONoC with state-of-the-art NoCs in SoCs | Metric
---|---
Work | Open | Full | Burst- | Config- | NoC-BW
| Source | AXI | support | urable | (Gbps)∗
SpiNNaker [16] | $\times$ | $\times$ | $\times$ | $\times$ | 5 (async)
Reza et al [17] | $\times$ | $\times$ | $\times$ | $\times$ | 4000
MCM [18] | $\times$ | $\times$ | $\times$ | $\times$ | 35
MC-NoC [6] | $\times$ | $\times$ | $\times$ | $\times$ | 2368
NeuNoC [19] | $\times$ | $\times$ | $\times$ | $\times$ | -
TETRIS [20] | $\times$ | $\times$ | $\times$ | $\times$ | -
PUMA [21] | $\times$ | $\times$ | $\times$ | $\times$ | -
OpenSoC [22] | ✓ | $\times$ | $\times$ | ✓ | -
ESP-SoC [4] | ✓ | $\times$ | $\times$ | Limited | 351
Celerity [23] | ✓ | $\times$ | $\times$ | Limited | 80
FlexNoC [24] | $\times$ | $\times$ | $\times$ | - | -
Constellation [25] | ✓ | $\times$ | $\times$ | ✓ | -
Andreas et al. [9] | ✓ | ✓ | ✓ | ✓ | 2146
PATRONoC | ✓ | ✓ | ✓ | ✓ | 2700
∗Normalized to 1 GHz for fair comparison. |  |

##  V Related Work

NoCs are an active area of research, and much effort has gone into optimizing
topologies, routing algorithms, flow control schemes, and the
microarchitecture of routers [12, 26, 27]. Multi-core (CPU) architectures have
been exploiting these optimizations of NoCs for many decades. However, NoCs
for multi-accelerator DNN platforms are still in nascent stage. Table II
provides a brief overview of state-of-the-art NoCs used in multi-core DNN
platforms compared to PATRONoC. PATRONoC is the only design that provides
open-source AXI-compliant homogeneous burst-based configurable NoC for multi-
core DNN platforms. Moreover, PATRONoC outperforms most of the designs in
terms of throughput, with the exception of [17], which uses a bigger 8x8
concentrated mesh (CMesh) topology with primarily local access patterns.
Moreover, its results are taken from the gem5 simulator [28], and the RTL of
the design is not openly available. Using a CMesh topology for PATRONoC would
similarly improve its performance.

OpenSoC Fabric [22] is among the few open-source NoCs with a custom non-
coherent NoC protocol. It provides a socket to plug AXI-Lite-based endpoints.
Unfortunately, AXI-Lite does not support bursts needed by high-performance
systems. The ESP framework [10, 4] also provides an open-source implementation
of its multi-plane NoC, supporting coherent and non-coherent endpoints. The
NoC is a 2D mesh topology and uses a custom packet-based protocol. We used
ESP-NoC as a baseline for comparison with PATRONoC. Section III shows that
PATRONoC is more area efficient and provides higher bandwidth owing to its
homogeneous network. BaseJump Manycore is an open-source non-coherent NoC
based on a 2D mesh used in the Celerity chip [23]. Those NoCs are generally
limited to meshes and use classical packet-based NoC protocols, which lead to
high area overhead and low bandwidth. In comparison, PATRONoC can be used to
design any topology, while providing a highly parameterizable NoC.

##  VI Conclusion

This work presented the first homogeneous AXI-compliant network-on-chip
architecture, building a complete open-source infrastructure for generating
various NoC topologies. Using the benefits of a burst-based AXI protocol,
PATRONoC targets the emerging field of multi-core DNN platforms requiring
high-bandwidth burst-based traffic. The NoC provides high-performance gain
compared to state-of-the-art NoCs by using its burst capability and achieves
up to a maximum of 310 Gi / aggregated throughput on DNN workloads. The work
provides insight into the exploration of different design parameters which
affect the performance and complexity of the NoC. It also enables future work
to explore different NoC topologies which might be suited for emerging DNN
platforms.

## References

  * [1] A. Reuther _et al._ , “AI Accelerator Survey and Trends,” in _HPEC_ , 2021.
  * [2] V. Jain _et al._ , “TinyVers: A 0.8-17 TOPS/W, 1.7 $\mu$W-20 mW, Tiny Versatile System-on-chip with State-Retentive eMRAM for Machine Learning Inference at the Extreme Edge,” in _VLSI_ , 2022.
  * [3] K. Ueyoshi _et al._ , “DIANA: An End-to-End Energy-Efficient Digital and ANAlog Hybrid Neural Network SoC,” in _ISSCC_ , 2022.
  * [4] T. Jia _et al._ , “A 12nm Agile-Designed SoC for Swarm-Based Perception with Heterogeneous IP Blocks, a Reconfigurable Memory Hierarchy, and an 800MHz Multi-Plane NoC,” in _ESSCIRC_ , 2022.
  * [5] A. Di Mauro _et al._ , “Kraken: A Direct Event/Frame-Based Multi-sensor Fusion SoC for Ultra-Efficient Visual Processing in Nano-UAVs,” in _HCS_ , 2022.
  * [6] J.-Y. Kim _et al._ , “A 118.4 GB/s Multi-Casting Network-on-Chip With Hierarchical Star-Ring Combined Topology for Real-Time Object Recognition,” _JSSC_ , 2010.
  * [7] J. Balfour _et al._ , “Design Tradeoffs for Tiled CMP On-Chip Networks,” in _ICS_ , 2006.
  * [8] D. Petrisko _et al._ , “NoC Symbiosis (Special Session Paper),” in _NOCS_ , 2020.
  * [9] A. Kurth _et al._ , “An Open-Source Platform for High-Performance Non-Coherent On-Chip Communication,” _IEEE TComp_ , 2022.
  * [10] D. Giri _et al._ , “NoC-Based Support of Heterogeneous Cache-Coherence Models for Accelerators,” in _NOCS_ , 2018.
  * [11] V. Catania _et al._ , “Noxim: An open, extensible and cycle-accurate network on chip simulator,” in _ASAP_ , 2015.
  * [12] W. J. Dally _et al._ , _Principles and practices of interconnection networks_. Elsevier, 2004.
  * [13] N. Bruschi _et al._ , “GVSoC: A Highly Configurable, Fast and Accurate Full-Platform Simulator for RISC-V based IoT Processors,” in _ICCD_ , 2021\.
  * [14] N. Bruschi _et al._ , “Scale up your In-Memory Accelerator: Leveraging Wireless-on-Chip Communication for AIMC-based CNN Inference,” in _AICAS_ , 2022.
  * [15] K. Goetschalckx _et al._ , “DepFiN: A 12nm, 3.8TOPs depth-first CNN processor for high res. image processing,” in _VLSI_ , 2021.
  * [16] E. Painkras _et al._ , “SpiNNaker: A 1-W 18-Core System-on-Chip for Massively-Parallel Neural Network Simulation,” _IEEE JSSC_ , 2013.
  * [17] M. F. Reza _et al._ , “Energy-Efficient and High-Performance NoC Architecture and Mapping Solution for Deep Neural Networks,” in _NOCS_ , 2019.
  * [18] B. Zimmer _et al._ , “A 0.32–128 TOPS, Scalable Multi-Chip-Module-Based Deep Neural Network Inference Accelerator With Ground-Referenced Signaling in 16 nm,” _JSSC_ , 2020.
  * [19] X. Liu _et al._ , “Neu-NoC: A high-efficient interconnection network for accelerated neuromorphic systems,” in _ASP-DAC_ , 2018.
  * [20] M. Gao _et al._ , “TETRIS: Scalable and Efficient Neural Network Acceleration with 3D Memory,” _SIGARCH Comput. Archit. News_ , 2017.
  * [21] A. Ankit _et al._ , “PUMA: A programmable ultra-efficient memristor-based accelerator for machine learning inference,” in _ASPLOS_ , 2019.
  * [22] F. Fatollahi-Fard _et al._ , “OpenSoC Fabric: On-chip network generator,” in _ISPASS_ , 2016.
  * [23] S. Davidson _et al._ , “The Celerity Open-Source 511-Core RISC-V Tiered Accelerator Fabric: Fast Architectures and Design Methodologies for Fast Chips,” _MICRO_ , 2018.
  * [24] J.-J. Lecler _et al._ , “Application driven network-on-chip architecture exploration & refinement for a complex SoC,” _DAES_ , 2011.
  * [25] J. Zhao _et al._ , “Constellation: An Open-Source SoC-Capable NoC Generator,” in _NoCArc_ , 2022.
  * [26] N. E. Jerger _et al._ , “On-chip networks,” _Synthesis Lectures on Computer Architecture_ , 2017.
  * [27] G. De Micheli _et al._ , “Networks on chips: From research to products,” in _Proceedings of the 47th Design Automation Conference_ , 2010.
  * [28] J. Lowe-Power _et al._ , “The gem5 Simulator: Version 20.0+,” _CoRR_ , 2020. [Online]. Available: https://arxiv.org/abs/2007.03152