# Radar Fields: An Extension of Radiance Fields to SAR

Thibaud Ehret1 Roger Marí1 Dawa Derksen2 Nicolas Gasnier2 Gabriele Facciolo1
1Université Paris-Saclay, CNRS, ENS Paris-Saclay, Centre Borelli, France
2Centre National d’Études Spatiales, France
<EMAIL_ADDRESS>
###### Abstract

Radiance fields have been a major breakthrough in the field of inverse
rendering, novel view synthesis and 3D modeling of complex scenes from multi-
view image collections. Since their introduction, it was shown that they could
be extended to other modalities such as LiDAR, radio frequencies, X-ray or
ultrasound. In this paper, we show that, despite the important difference
between optical and synthetic aperture radar (SAR) image formation models, it
is possible to extend radiance fields to radar images thus presenting the
first “radar fields”. This allows us to learn surface models using only
collections of radar images, similar to how regular radiance fields are
learned and with the same computational complexity on average. Thanks to
similarities in how both fields are defined, this work also shows a potential
for hybrid methods combining both optical and SAR images.

##  1 Introduction

(a) Regular radiance fields

(b) Proposed radar fields

Figure 1: The difference between the integration process of a classic optical
radiance field and the proposed radar field. While the radiance field
integrates along each rays (in black), radar fields integrates across rays (in
red) in a given azimuth plane. Similarly to radiance fields, radar fields can
be optimized from multiple SAR images to recover the underlying surface of the
scene.

Many of the latest breakthroughs in the domain of 3D reconstruction have been
inspired by Neural Radiance Fields (NeRF) [37]. By definition, NeRFs are a
variant of Neural Fields that learn a 3D representation of the scene based on
the measurement of radiance, given a set of optical images. In the past years,
NeRFs have been reshaped, reworked and adapted to various use cases. In
particular, their application domain has been extended far beyond computer
vision [20, 55].

This diversity of application domains is largely due to the flexibility of the
Neural Fields paradigm. Neural Fields represent a path forward for including
physics into the Neural Network learning process, a longstanding aim in AI
research. The common goal of all of the Neural Field research works cited in
the following section is to learn a high-dimensional representation (e.g. a 3D
shape) from a set of low-dimensional data (e.g. 2D images). From a physical
point of view, the low-dimensional measurements can be seen as boundary
conditions for a set of partial differential equations (PDE) that describe the
formation process of the observations. In this setting learning amounts to an
iterative optimization process to find a high-dimensional representation that
simultaneously satisfies both the PDE and the boundary conditions. The
learning process generally involves starting with a randomly (or arbitrarily)
initialized scene. A set of low-dimensional observations are then computed
using a forward “rendering” process, based on a discretized form of the PDE.
These observations are confronted to the real measurements, yielding a loss.
The loss is back-propagated onto the representation to improve its coherence
with the observations. This process is repeated until convergence is reached.

In this work, we focus on using Synthetic Aperture Radar (SAR) data instead of
optical images to achieve 3D reconstruction. We introduce Radar Fields, a new
framework for learning a volumetric model based on multiple SAR measurements
of a scene. Since the wavelengths used in SAR experience minimal interactions
with atmospheric gases or aerosols, the propagation medium is virtually
transparent and non-absorbing. Thus, unlike optical sensors, SAR has day-night
cloud-penetrating imaging capacities. Unlike interferometric SAR (InSAR), the
proposed methodology does not prescribe the geometric configuration of the
acquisitions. It could therefore be used with opportunistic acquisitions. We
believe that the main impact of this research will be for global 3D
reconstruction, due to the wide-scale availability and high quality of the
latest SAR Earth Observation data from missions such as TerraSAR-X [38] or
Capella [17, 49]. The methodology that we present is general and could equally
be applied to aerial SAR or to other radar-based measurement systems.

To summarize, our contributions are the following:

  1. 1.

The definition of radar fields, a differentiable rendering framework adapted
to radar image synthesis.

  2. 2.

A new surface model model definition, inspired by digital surface models, that
can easily be used in frameworks like radar fields for remote sensing
applications.

  3. 3.

Adapting key training techniques, like NeRF random depth sampling or being
able to train despite noisy data, to radar fields.

##  2 Related works

Our work is not the first to exploit SAR images to produce 3D models. In fact,
space-borne SAR sensors are currently used to produce large scale Digital
Surface Models (DSM) such as Copernicus DEM [1]. The TanDEM-X mission [61]
uses the principle of radar interferometry to estimate the Earth’s surface
topography. The derived Copernicus DEM product describes the elevation of the
terrain and built-up land elements with an average error of 2-4 meters. The
spatial resolution of the product ranges from 10-30 meters.

An alternative technique to interferometry, known as SAR tomography [44, 48],
consists in using stacks of 2D Synthetic Aperture Radar (SAR) images to
estimate the surface height with a finer spatial resolution (1-2m). This
technique involves first estimating the covariance matrix in each radar cell.
In a second step, spectral analysis is used to invert this covariance matrix,
resulting in an approximation of the surface height distribution. The
resulting DSM achieves a mean altitude error ranging from 1-2 meters in urban
areas. However, SAR Tomography commonly suffers from missing parts in urban
surfaces, particularly in flat areas such as ground or rooftops. Moreover, the
spatial heterogeneity of the height distribution near abrupt altitude
transitions can result in a blurry reconstruction [40].

Other works propose deep learning-based single-image height estimation methods
using a Convolutional Neural Network encoder-decoder architecture [41, 42],
taking a SAR image as input and directly predicting depth maps. These methods
provide fast inference times, but the training requires large amounts of
ground-truth 3D data synchronized with multi-view SAR acquisitions to
generalize to different viewing angles or landscapes.

Neural fields have seen a number of applications in optical-based satellite
imagery, starting with the seminal work S-NeRF [15] followed by Sat-NeRF [34],
and recently, EO-NeRF [35]. S-NeRF initially tackled the problem of varying
lighting conditions; Sat-NeRF introduced a latent transient vector to
disentangle temporary objects such as cars from the static scene, and EO-NeRF
achieved state-of-the-art 3D reconstruction quality by explicitly modeling the
shadow effects and learning per-image color corrections. Another recent work,
SparseSat-NeRF [58], reduced the need for a large number of input images using
depth priors based on stereo-photogrammetry. Season-NeRF [19] models seasonal
variations to enable temporal interpolation between different acquisition
dates. Sat-Mesh [39] replaced the NeRF model with an implicit surface
representation, yielding impressive visual DSM quality, but not disentangling
the transient objects or generalizing to unseen solar angles. RS-NeRF [54] and
SatensoRF [59] have successfully reduced the optimization time, but like Sat-
Mesh do not generalize to unseen solar angles. In the domain of aerial images,
issues such as large-scale data [56, 51] and image co-registration [8] have
also been addressed with NeRF approaches.

Since the arrival of NeRFs, researchers in various fields have also
successfully applied the Neural Fields paradigm to entirely different data
than optical images. For example, in the domain of radiative transfer and
light modeling, these include applications to LiDAR [23, 50], Time-of-flight
cameras [2] and non-line of sight cameras [47]. Impressive results have also
been achieved using event based cameras, which detect a difference in pixel
intensity rather than the absolute pixel intensity [46, 33, 24, 27]. In other
regions of the electromagnetic spectrum, applications have risen for 3D
reconstruction based on X-rays [9, 10] and Radio Frequency [60]. The equations
of light propagation being similar in nature to the equations of sound
propagation, the same paradigm can be adapted to the acoustic setting [7],
with applications in ultra-sound [31, 53] and sonar [43]. Finally, the
principle of implicit learning has also yielded exploratory works in geodesy
[25] and gravitational lensing [30].

##  3 Introduction to radiance fields

NeRF [37] represents a static scene as a continuous volumetric function
$\mathcal{F}$. This function can, for example, be encoded by a fully-connected
neural network (MLP) [37], a voxel decomposition [26] or a tensor
interpolation [6]. $\mathcal{F}$ predicts the emitted RGB color
$\mathbf{c}=(r,g,b)$ and a non-negative scalar volume density $\sigma$ at a 3D
point $\mathbf{x}=(x,y,z)$ of the scene seen from a viewing direction
$\mathbf{d}=(d_{x},d_{y},d_{z})$, _i.e_.

$\mathcal{F}:(\mathbf{x},\mathbf{d})\mapsto(\mathbf{c},\sigma).$ (1)

Multi-view consistency is encouraged by restricting the network to predict the
volume density $\sigma$ based only on the spatial coordinates $\mathbf{x}$,
while allowing the color $\mathbf{c}$ to be predicted as a function of both
$\mathbf{x}$ and and other view-dependent features, such as the viewing
direction $\mathbf{d}$.

Given a set of input views and their camera poses, $\mathcal{F}$ is optimized
by rendering the color of individual rays traced across the scene and
intersecting the known pixels. Each ray r is defined by a point of origin
$\mathbf{o}$ and a direction vector $\mathbf{d}$. The color
$\mathbf{c}(\mathbf{r})$ of a ray $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ is
computed as

$\mathbf{c}(\mathbf{r})=\sum_{i=1}^{N}T_{i}\alpha_{i}\mathbf{c}_{i}.$ (2)

The rendered color $\mathbf{c}(\mathbf{r})$ results from the weighted
integration of the colors $\mathbf{c}_{i}$ predicted at $N$ different points
sampled along the ray $\mathbf{r}$. Each point $\mathbf{x}_{i}$ in
$\mathbf{r}$ is obtained as $\mathbf{x}_{i}=\mathbf{o}+t_{i}\mathbf{d}$, where
$t_{i}$ is the depth step.

The contribution of each point to the rendering equation (2) follows a light
transmittance model based on the opacity $\alpha_{i}$ and transmittance
$T_{i}$ values, which follow the geometry defined by the density $\sigma$

$\alpha_{i}=1-\exp(-\sigma_{i}(t_{i+1}-t_{i}));\quad
T_{i}=\prod_{j=1}^{i-1}\left(1-\alpha_{j}\right).$ (3)

The opacity $\alpha_{i}$ increases with $\sigma_{i}$ and is the probability
that a point belongs to a non-transparent surface. The transmittance $T_{i}$
is the probability that light arrives without hitting previous opaque points
in the ray $\mathbf{r}$.

Given (3), the depth $d(\mathbf{r})$ observed in the direction of a ray
$\mathbf{r}$ can be rendered in a similar manner to (2) [14, 45] as

$d(\mathbf{r})=\sum_{i=1}^{N}T_{i}\alpha_{i}t_{i}.$ (4)

The radiance field is optimized by minimizing the mean squared error (MSE)
between the rendered color and the actual color of the input views:

$\sum_{\mathbf{r}\in\mathcal{R}}\|\mathbf{c}(\mathbf{r})-\mathbf{c}_{\text{GT}}(\mathbf{r})\|_{2}^{2},$
(5)

where $\mathbf{c}_{\text{GT}}(\mathbf{r})$ is the observed color of the pixel
intersected by the ray $\mathbf{r}$, and $\mathbf{c}(\mathbf{r})$ is the color
predicted by the radiance field using (2). $\mathcal{R}$ is the set of rays in
each input batch.

### Surface models.

The concept of radiance fields has been extended to surface models [57, 52,
13, 18, 32]. Instead of considering the scene as continuous volumetric
function $\mathcal{F}$, the scene is represented as a surface $\mathcal{S}$.
This surface is characterized by an implicit function $\mathcal{F}$ such that

$\mathcal{S}=\left\\{\bm{x}\in\mathbb{R}^{3}~{}|~{}\mathcal{F}(\bm{x})=0\right\\}.$
(6)

Denoting $d$ the minimum distance between $\bm{x}$ and $\mathcal{S}$,
$\mathcal{F}$ is defined as a signed distance function (SDF) with

$\mathcal{F}(\bm{x})=\begin{cases}d(\bm{x},\mathcal{S})&\mbox{if
}\,\bm{x}\text{ is outside }\mathcal{S},\\\ -d(\bm{x},\mathcal{S})&\mbox{if
}\,\bm{x}\text{ is inside }\mathcal{S}.\end{cases}$ (7)

This property is enforced using an additional loss term to enforce the Eikonal
property of the SDF

$\mathcal{L}_{Eikonal}=\sum_{x\in\Omega}\left(\|\nabla\mathcal{F}(\bm{x})\|_{2}^{2}-1\right)^{2}$
(8)

with $\Omega$ containing points sampled near the surface as well as uniformly
in the entire volume.

The optimization of a surface based model is similar to that of a regular
radiance field. The surface model is transformed into a volumetric model using
a function that produces a pseudo-density from the SDF value at a given point.
For example, in VolSDF [57], the transformation function $\Psi$ is defined
with the cumulative distribution function of the Laplace function

$\Psi(d)=\begin{cases}\frac{1}{2}\exp\left(-\frac{d}{\beta}\right)&\text{for
}d\geqslant 0\\\ 1-\frac{1}{2}\exp\left(\frac{d}{\beta}\right)&\text{for
}d<0,\end{cases}$ (9)

where $\alpha$ and $\beta$ are optimized with the rest of the model. The
rendering operation remains unchanged, as (2) and (3) use the pseudo-density

$\sigma_{i}=\Psi\left(\mathcal{F}\left(\bm{x}_{i}\right)\right).$ (10)

Representing the geometry with an implicit function $\mathcal{F}$ opens the
door to exploit additional information, such as the normal of the surface,
_i.e_. $n(\bm{x})=\nabla\mathcal{F}(\bm{x})$ when
$\nabla\mathcal{F}(\bm{x})\neq\textbf{0}$, which is prevented by the Eikonal
constraint.

##  4 Radar fields

The models presented in Section 3 have been proposed for optical images. We
propose here to extend the concept of radiance fields to SAR images.

### Introduction to the SAR image formation model.

A SAR imaging device is an active system. It is usually composed of an antenna
mounted on a moving spaceborne or airborne platform that emits electromagnetic
pulses in a side-looking geometry and then receives echoes that are
backscattered by the ground surface. The satellite flight direction is called
along-track or azimuth, while the direction of the beam is called across-track
or range. For each azimuth pulse emitted, the satellite receives the echoes,
demodulates and samples them, orders them by range, and stores them as complex
numbers separating the amplitude and phase components of the signal. The
amplitude is related to the ground reflectivity and angle, while the phase
contains information about the satellite-target path delay (modulo the
wavelength). The resulting image is said to be the raw format. Because of the
large footprint of the electromagnetic beam and the length of the pulse, in
this format, the response from a single scatterer on the ground is spread over
many pixels and the spatial resolution is very low. The resolution is largely
improved through a process known as focusing [11], which yields a complex-
valued image equivalent to employing short pulses (for range) and a narrow
beam perpendicular to the motion of the satellite known as zero-doppler plane
(corresponding to azimuth). Unlike optical images, SAR images represent the
distance to reflectors in the range direction. If two reflectors are at the
same distance from the satellite (in the same azimuth plane), the response
will be the sum of both, resulting in foreshortening, layover and shadow
effects. Luckily, this range image can be analogously interpreted as a
projected view of the scene from a direction perpendicular to the incidence
angle and illuminated by the radar pulse, where objects appear as if they are
transparent. The SAR acquisition process is illustrated in Figure 2.

Figure 2: Acquisition process of a SAR image: An electromagnetic pulse is sent
from the antenna and Points A and C, at the same distance from the sensor, are
projected to the same pixel while B is projected on a different pixel.

### Definition of radar fields.

Using the notations from Section 3, we want to learn the function
$\mathcal{F}$ corresponding to the surface model of the scene using a
collection of SAR images. In this section, we derive the model for a given
azimuth plane. This is not a limitation since the image is equivalent to a
zero-doppler acquisition for each azimuth plane as mentioned in the previous
paragraph, except in the case of double or multiple bounces, which we discuss
in Section 6.

Consider a set of rays $(\bm{r}_{j})$, with respective origin and direction
$(\bm{o}_{j},\bm{v}_{j})$, in the given azimuth plane and
$(d_{i})_{i\in\llbracket 1,N\rrbracket}$ the distance sampling associated to
these rays. Let $s_{\bm{r}_{j}}(d_{i})$ be the sensed signal associated to
$\bm{r}_{j}$ at distance $d_{i}$. Following Section 3, this yields

$s_{\bm{r}_{j}}(d_{i})=-T_{i,j}\alpha_{i,j}\left\langle\bm{v}_{j}|\bm{n}_{i,j}\right\rangle^{\theta},$
(11)

where the definition of $T_{i,j}$ and $\alpha_{i,j}$, for a fixed $j$, is the
same as in (3), $\bm{n}_{ij}=\nabla\mathcal{F}(\bm{o}_{j}+d_{i}\bm{v}_{j})$,
and where the coefficient $\theta$ is used to model the specularity of the
reflection [4, 3]. Indeed, in the case of SAR images, there is no color: the
intensity of the reflected signal is characterized by the surface.
Traditionally in SAR simulation [4, 3], the reflected signal $s$ is given by

$s=-\left\langle\bm{v}|\bm{n}\right\rangle^{\theta}$ (12)

with $\bm{v}$ the incidence direction and $\bm{n}$ the normal to the surface.
Often a purely Lambertian model, corresponding to $\theta=1$, is assumed.
Larger values of $\theta$ are used to represent specular surfaces. The
parameter $\theta$ can be learned at the same time as the location of the
surface.

As described in the previous paragraph and represented in Figure 1, the
integration is performed across rays for SAR images and not along rays as in
optical images: for a given azimuth, reflectors that are at the same distance
are projected into the same pixel. To satisfy this condition, the rendering
operation (2) must be replaced by

$s(d_{i})=\sum_{\bm{r}_{j}}s_{\bm{r}_{j}}(d_{i})=-\sum_{\bm{r}_{j}}T_{i,j}\sigma_{i,j}\left\langle\bm{v}_{j}|\bm{n}_{i,j}\right\rangle^{\theta}.$
(13)

Note how the role of indices $i$ and $j$ are inverted between Eq. (2) and Eq.
(13) and how $\mathbf{c}$ depends on the ray while $s$ depends only the
distance. While the position of rays is well defined by the sensor
configuration in the optical case (see Sec. 3), this is not the case for SAR.
Conversely, the sampling of the $\bm{x}_{i,j}$ is perfectly defined in the SAR
case – following the range sampling – but is flexible in the optical case.
Finally, computing the results for all the azimuth planes gives the complete
SAR image.

### Differences between radiance and radar fields.

While radiance and radar fields are conceptually similar, we list some
important differences here. They are also summarized in Table 1. The first
difference is that to generate a single SAR pixel, it is necessary to compute
all rays corresponding to its azimuth plane. Thankfully, computing a single
pixel provides the result for the entire range line at the same time. This
means that while rendering a single pixel requires more computations, the
overall amount of computations necessary to learn a radar field remains the
same, on average, as the one needed to learn a radiance field. In practice the
only difference is how rays are batched. While there is no requirement in the
radiance field case, it is necessary to batch all rays of a given azimuth
plane together during computation for radar fields. Note that with additional
assumptions on the scene, it is possible to slightly relax this requirement.
Indeed, if an approximate position of the surface is known a priori, then it
would be possible to split the distance sampling into multiple smaller batches
that can be computed separately. Note that no such assumption is made for the
experiments presented in Section 5.

A key technique to improve the performance in NeRF [37] is to introduce some
noise when sampling along rays instead of using a fixed uniform sampling. This
is not possible with a radar field because the sampling step along rays is
defined by the range sampling. It is however possible to transpose this idea
to ray sampling since, as shown in Eq. (13), the final signal does not require
a specific ray position. as long as $\bm{o}_{j}$ stays in its corresponding
azimuth plane and that it does not impact the range sampling. If we denote by
$\bm{w}_{j}$ the vector orthogonal to $\bm{v}_{j}$ inside the azimuth plane,
then the perturbed ray origin is defined by

$\tilde{\bm{o}}_{j}=\bm{o}_{j}+n\bm{w}_{j}\text{ with }n\sim\mathcal{N}(0,1).$
(14)

While optical images have a good SNR in normal illumination conditions, SAR
images are fundamentally noisy. They exhibit speckle noise, which has been
studied and shown to follow a complex Wishart distribution [21, 22]. The
distribution $p$ of the noisy sample covariance matrix $\bm{C}$ of dimension
$d$ corresponding to the noise-free covariance $\bm{\Sigma}$ in a $L$ looks
configuration is

$p(\bm{C})=\frac{L^{Ld}|\bm{C}|^{L-d}}{\Gamma_{d}(L)|\bm{\Sigma}|^{L}}\exp\left(-L\operatorname{Tr}\left(\bm{\Sigma}^{-1}\bm{C}\right)\right),$
(15)

with $\Gamma_{d}(L)=\pi^{d(d-1)/2}\prod_{k=1}^{d}\Gamma(L-k+1)$. When working
with single-channel single-look intensity images, the sample covariance matrix
reduces to an intensity $I$ and this model can be simplified into a
multiplicative speckle noise with regard to the noise-free reflectance $R$,
such that

$I=n\times R\quad\quad\text{with}\quad\quad n\sim\Gamma(1,1).$ (16)

Learning the surface despite the noise is not a problem in practice. Indeed,
comparing the noiseless generated view to the noisy samples is sufficient.
This is similar to Noise2Noise [29] that shows that it is possible to learn a
denoising network despite having only noisy samples. SAR2SAR [12] showed that
this idea can also be applied to SAR data and [16] that it is possible to
learn the noiseless images of a scene using a single burst of images, even
when the noise has a bias.

| Radiance field | Radar Field
---|---|---
Complexity |  height $\times$ width $\times$ samples |  azimuth $\times$ range $\times$ samples (on average)
Integration | Along the ray | Across the ray
Distance sampling |  Variable (along a ray) |  Fixed (defined by data)
Ray sampling |  Fixed (defined by data) |  Variable (in a given azimuth plane)
Noise | Noise-free |  Follows Eq. 16
Table 1: Main differences between regular radiance fields and the proposed
radar fields.

Figure 3: Results of surface learning using five SAR images on two toy
examples. The first represents a pyramid (first row is the ground truth and
second row the learned model) and the second a round pile (ground truth in the
third row and learned model in the last row). From left to right:
visualization of the surface model, DSM corresponding to the surface model,
and two images corresponding to two views used during training.

Figure 4: Results of surface learning using five SAR images on simulated data
from two real DSMs. The first represents mount Fuji (first row is the ground
truth and second row the learned model) and the second the Piton de la
Fournaise (ground truth in the third row and learned model in the last row).
From left to right: visualization of the surface model, DSM corresponding to
the surface model, and two images corresponding to two views used during
training.

### Surface model implementation.

In remote sensing, ground surfaces are usually represented by a DSM, _i.e_. a
2D map of the altitude of the ground at each position. For our experiments we
propose to use the same formalism to define the surface in the proposed radar
field. From $dsm:[0,1]^{2}\rightarrow[0,1]$, we define

$\mathcal{F}(\bm{x})=z-dsm(x,y)$ (17)

with $\bm{x}=(x,y,z)\in[0,1]^{3}$. While this function is not exactly an SDF
since its value might not necessarily correspond to the exact distance to the
surface, it nonetheless defines an implicit surface as in Eq. 6. The results
shown in Sec. 5 were computed using a rasterized DSM that can be assimilated
to an image whose pixel represent the height value of the surface. It is the
value of each pixel that is then directly optimized. Subpixel heights are
estimated using a bilinear interpolation of the nearest values and normals to
the defined surface are computed accordingly. One of the advantage is that the
Eikonal loss (8) is not necessary and it is possible define simple surface
regulation terms like

$\sum_{\bm{x}}\sum_{\bm{y}\in\mathcal{N}(\bm{x})}\|dsm(\bm{x})-dsm(\bm{y})\|_{2}^{2},$
(18)

with $\mathcal{N}(\bm{x})$ the set of neighbors of $\bm{x}$, that we use for
the experiments. Note that the presented concept should remain compatible with
the more classic definitions of radiance fields.

##  5 Experiments

We apply the proposed radar field model to synthetic examples generated from
toy surfaces and real DSMs to demonstrate its potential.

### Synthetic data generation.

We generate the synthetic SAR acquisitions, using the same image formation
model described in Section 4. Indeed, given that the ground truth is also
represented by a DSM, the radar field synthesis equations can be used to
simulate the images corresponding to the query views. The main difference is
that noise needs to be added at the end of the process. For this, following
Eq. (16), we use instead $\tilde{s}(d)=n\times s(d)$ with $n\sim\Gamma(1,1)$
and $s(d)$ defined by Eq. (13). The optimization process requires only the
generated noisy images and the geometric parameters associated to each view.

|
---|---
|
Figure 5: Example showing that it is possible to learn the $\theta$
coefficient (see Eq.(12)) characterizing the surface property (bottom) at the
same time as the surface model (top). Left is the ground truth while right is
the learned model. The learned model was initialized with $\theta=1$
everywhere.

### Parameters.

All the experiments presented in this section were done using the same
parameters and the images generated corresponds to the same five camera
positions. The models were trained during 10000 steps with an initial learning
rate of 1. We reduced the learning rate to 0.1 at epoch 5000 and to $10^{-2}$
at step 8000. Azimuth planes were sampled uniformly from all images and
grouped into batches of size 64, and we sampled 256 rays per azimuth plane.
Except when specified otherwise, we consider Lambertian surfaces with
$\theta=1$.

### Results.

Figures 3 and 4 show results both with toy examples (a pyramid and a circular
pile) as well as simulations using real DSMs (of the mount Fuji and the Piton
de la Fournaise). These results show that it is indeed possible to learn the
ground surface from a few opportunistic SAR acquisitions using the proposed
radar fields. By quantifying the altitude reconstruction errors relative to
the ground pixel resolution, we observe errors in the order of $10^{-2}$
pixels. Lastly, Figure 5 shows an example of reconstruction of a non
Lambertian surface, where the specularity map and geometry are both estimated
during the optimization. In all these examples, the learned surfaces are very
accurate despite the presence of a strong noise in the training data.
Additional results are available in the supplementary material.

##  6 Discussion and conclusion

In this section, we discuss the limitations and possible improvements of the
model presented in Section 4.

Figure 6: Example of multiple bounces for SAR signal. The signal is partially
reflected in $A$ but also bounces and is reflected a second time in $B$. This
second bounce is sensed as a virtual reflection in $C$.

### Multi bounce.

Depending on the surface configuration, the SAR signal may be subject to
multiple bounces. This phenomena occurs when the signal received by the sensor
is not the one directly reflected by an ideal Lambertian surface, but one
coming from a second or further reflection. Corners are a common source of
double bounces. We distinguish between two cases: the first is when bounces
occur in a given azimuth plane, while the second is a more generic case where
bounces are not restricted to that plane.

Figure 6 illustrates the case of double bounce in a given azimuth plane. Part
of the signal is reflected at the first contact with the surface – as modeled
so far – but also bounces to another part of the surface before being
reflected back to the sensor. This process could be simulated by continuing
rays after the first surface intersection, similar to how additional rays are
computed in [35] to predict shadows. This increases the complexity of the
model by a factor equal to the maximum number of bounces considered. It is
important to note that the apparent position of the reflection associated to
the double bounce is not the same as the true reflection position. Indeed, the
acquisition system measures the traveling time of the impulse signal from
emission to reception and as such assumes that the reflection happened at half
the traveling time. For example, in Figure 6, the signal that bounced from $A$
to $B$ and back to the receiver appears at the same position as if it were
reflected by point $C$.

The generic case is more complex than the latter because the zero-doppler
assumption is not true anymore. This means that a ray corresponding to a given
azimuth plane can impact the signal of another azimuth plane. Such a generic
multi-bounce scenario would require batching multiple azimuth planes together
to render each pixel, making the method highly unfeasible unless strong priors
or other constraints are exploited in future work.

### Phase and polarimetric information.

One additional attribute of SAR data that could be incorporated is the phase
information. Indeed, SAR images are made of complex numbers containing the
intensity and the phase of the signal. SAR Interferometry (InSAR) exploits the
phase of multiple images with adequate acquisition configurations.
Multitemporal InSAR images can be used to measure small terrain deformation
[36], and across-track InSAR [5] can be used to build digital surface models
(DSM) using a pair of images acquired from two antennas (e.g., SRTM), as
mentioned in Section 2. Therefore, we see potential in the phase information
to improve altitude accuracy in future work. Similarly, polarimetric
information could also be modeled. Polarimetric information is widely used to
determine the scattering mechanism [28] and as such can be useful when
estimating the surface model. It can also be useful when estimating additional
scene properties, such as semantic classification.

### Conclusion.

We presented radar fields, an extension of radiance fields to SAR imagery.
While the model presented in this paper is a proof of concept tested on
simulated synthetic data, we believe that it opens up new exciting
opportunities, especially in remote sensing where high-quality SAR images are
becoming increasingly common and the generation of surface models from
satellite images is a major research topic. Moreover, it also shows a
potential to inspire future hybrid models combining both opportunistic optical
and SAR acquisitions in a joint framework.

## References

  * [1] Copernicus DEM handbook, last consulted 10/10/23.
  * Attal et al. [2021] Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O’Toole.  Törf: Time-of-flight radiance fields for dynamic scene view synthesis.  _Advances in neural information processing systems_ , 34:26289–26301, 2021.
  * Auer [2011] Stefan Josef Auer.  _3D synthetic aperture radar simulation for interpreting complex urban reflection scenarios_.  PhD thesis, Technische Universität München, 2011.
  * Balz and Stilla [2009] Timo Balz and Uwe Stilla.  Hybrid GPU-based single-and double-bounce SAR simulation.  _IEEE Transactions on Geoscience and Remote Sensing_ , 47(10):3519–3529, 2009.
  * Budillon et al. [2020] Alessandra Budillon, Christoph H. Gierull, Vito Pascazio, and Gilda Schirinzi.  Along-track interferometric SAR systems for ground-moving target indication: Achievements, potentials, and outlook.  _IEEE Geoscience and Remote Sensing Magazine_ , 8(2):46–63, 2020.
  * Chen et al. [2022] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  TensoRF: Tensorial radiance fields.  In _Computer Vision – ECCV 2022_ , pages 333–350, 2022.
  * Chen et al. [2023a] Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, and Andrea Vedaldi.  Novel-view acoustic synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6409–6419, 2023a.
  * Chen and Lee [2023] Yu Chen and Gim Hee Lee.  DReg-NeRF: Deep registration for neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 22703–22713, 2023.
  * Chen et al. [2023b] Zixuan Chen, Jianhuang Lai, Lingxiao Yang, and Xiaohua Xie.  Cunerf: Cube-based neural radiance field for zero-shot medical image arbitrary-scale super resolution.  _arXiv preprint arXiv:2303.16242_ , 2023b.
  * Corona-Figueroa et al. [2022] Abril Corona-Figueroa, Jonathan Frawley, Sam Bond-Taylor, Sarath Bethapudi, Hubert PH Shum, and Chris G Willcocks.  Mednerf: Medical neural radiance fields for reconstructing 3D-aware ct-projections from a single x-ray.  In _2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_, pages 3843–3848. IEEE, 2022\.
  * Cumming and Wong [2005] Ian G Cumming and Frank H Wong.  _Digital signal processing of synthetic aperture radar data_.  Artech House, 2005.
  * Dalsasso et al. [2021] Emanuele Dalsasso, Loic Denis, and Florence Tupin.  SAR2SAR: A semi-supervised despeckling algorithm for SAR images.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 14:4321–4329, 2021.
  * Darmon et al. [2022] François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry.  Improving neural implicit surfaces geometry with patch warping.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6260–6269, 2022.
  * Deng et al. [2022] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised NeRF: Fewer views and faster training for free.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Derksen and Izzo [2021] Dawa Derksen and Dario Izzo.  Shadow neural radiance fields for multi-view satellite photogrammetry.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 1152–1161, 2021.
  * Ehret et al. [2019] Thibaud Ehret, Axel Davy, Pablo Arias, and Gabriele Facciolo.  Joint demosaicking and denoising by fine-tuning of bursts of raw images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 8868–8877, 2019.
  * Farquharson et al. [2018] Gordon Farquharson, William Woods, Craig Stringham, Navneet Sankarambadi, and Lucas Riggi.  The capella synthetic aperture radar constellation.  In _EUSAR 2018; 12th European Conference on Synthetic Aperture Radar_ , pages 1–5. VDE, 2018.
  * Fu et al. [2022] Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao.  Geo-NeuS: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction.  _Advances in Neural Information Processing Systems_ , 35:3403–3416, 2022.
  * Gableman and Kak [2023] Michael Gableman and Avinash Kak.  Incorporating season and solar specificity into renderings made by a NeRF architecture using satellite images.  _arXiv preprint arXiv:2308.01262_ , 2023.
  * Gao et al. [2022] Kyle Gao, Yina Gao, Hongjie He, Denning Lu, Linlin Xu, and Jonathan Li.  NeRF: Neural radiance field in 3D vision, a comprehensive review.  _arXiv e-prints_ , pages arXiv–2210, 2022.
  * Goodman [1976] J. W. Goodman.  Some fundamental properties of speckle$\ast$.  _J. Opt. Soc. Am._ , 66(11):1145–1150, 1976.
  * Goodman [1963] N. R. Goodman.  Statistical analysis based on a certain multivariate complex Gaussian distribution (an introduction).  _The Annals of Mathematical Statistics_ , 34(1):152 – 177, 1963.
  * Huang et al. [2023] Shengyu Huang, Zan Gojcic, Zian Wang, Francis Williams, Yoni Kasten, Sanja Fidler, Konrad Schindler, and Or Litany.  Neural LiDAR fields for novel view synthesis.  _arXiv preprint arXiv:2305.01643_ , 2023.
  * Hwang et al. [2023] Inwoo Hwang, Junho Kim, and Young Min Kim.  Ev-NeRF: Event based neural radiance field.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 837–847, 2023.
  * Izzo and Gómez [2021] Dario Izzo and Pablo Gómez.  Geodesy of irregular small bodies via neural density fields: geodesynets.  _arXiv preprint arXiv:2105.13031_ , 2021.
  * Karnewar et al. [2022] Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra.  Relu fields: The little non-linearity that could.  In _ACM SIGGRAPH 2022 Conference Proceedings_ , pages 1–9, 2022.
  * Klenk et al. [2023] Simon Klenk, Lukas Koestler, Davide Scaramuzza, and Daniel Cremers.  E-NeRF: Neural radiance fields from a moving event camera.  _IEEE Robotics and Automation Letters_ , 8(3):1587–1594, 2023.
  * Lee and Pottier [2009] J.S. Lee and Eric Pottier.  _Polarimetric Radar Imaging : From basics to applications._ CRC Press, Taylor & Francis Editor,, 2009.
  * Lehtinen et al. [2018] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila.  Noise2noise: Learning image restoration without clean data.  In _International Conference on Machine Learning_ , pages 2965–2974. PMLR, 2018.
  * Levis et al. [2022] Aviad Levis, Pratul P Srinivasan, Andrew A Chael, Ren Ng, and Katherine L Bouman.  Gravitationally lensed black hole emission tomography.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19841–19850, 2022.
  * Li et al. [2021] Honggen Li, Hongbo Chen, Wenke Jing, Yuwei Li, and Rui Zheng.  3D ultrasound spine imaging with application of neural radiance field method.  In _2021 IEEE International Ultrasonics Symposium (IUS)_ , pages 1–4. IEEE, 2021.
  * Li et al. [2023] Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, and Chen-Hsuan Lin.  Neuralangelo: High-fidelity neural surface reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8456–8465, 2023.
  * Low and Lee [2023] Weng Fei Low and Gim Hee Lee.  Robust e-NeRF: NeRF from sparse & noisy events under non-uniform motion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 18335–18346, 2023.
  * Marí et al. [2022] Roger Marí, Gabriele Facciolo, and Thibaud Ehret.  Sat-NeRF: Learning multi-view satellite photogrammetry with transient objects and shadow modeling using RPC cameras.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 1310–1320, 2022.
  * Marí et al. [2023] Roger Marí, Gabriele Facciolo, and Thibaud Ehret.  Multi-date earth observation NeRF: The detail is in the shadows.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2034–2044, 2023.
  * Massonnet et al. [1993] Didier Massonnet, Marc Rossi, César Carmona, Frédéric Adragna, Gilles Peltzer, Kurt Feigl, and Thierry Rabaute.  The displacement field of the Landers earthquake mapped by radar interferometry.  _Nature_ , 364(6433):138–142, 1993.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing scenes as neural radiance fields for view synthesis.  In _Computer Vision – ECCV 2020_ , pages 405–421, 2020.
  * Pitz and Miller [2010] Wolfgang Pitz and David Miller.  The TerraSAR-X satellite.  _IEEE Transactions on Geoscience and Remote Sensing_ , 48(2):615–622, 2010.
  * Qu and Deng [2023] Yingjie Qu and Fei Deng.  Sat-mesh: Learning neural implicit surfaces for multi-view satellite reconstruction.  _Remote Sensing_ , 15(17):4297, 2023.
  * Rambour et al. [2019] Clément Rambour, Loïc Denis, Florence Tupin, Hélène Oriot, Yue Huang, and Laurent Ferro-Famil.  Urban surface reconstruction in SAR tomography by graph-cuts.  _Computer Vision and Image Understanding_ , 188:102791, 2019\.
  * Recla and Schmitt [2022] Michael Recla and Michael Schmitt.  Deep-learning-based single-image height reconstruction from very-high-resolution SAR intensity data.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 183:496–509, 2022.
  * Recla and Schmitt [2023] Michael Recla and Michael Schmitt.  From relative to absolute heights in SAR-based single-image height prediction.  In _2023 Joint Urban Remote Sensing Event (JURSE)_ , pages 1–4. IEEE, 2023.
  * Reed et al. [2021] Albert Reed, Thomas Blanford, Daniel C Brown, and Suren Jayasuriya.  Implicit neural representations for deconvolving sas images.  In _OCEANS 2021: San Diego–Porto_ , pages 1–7. IEEE, 2021.
  * Reigber and Moreira [2000] A. Reigber and A. Moreira.  First demonstration of airborne SAR tomography using multibaseline l-band data.  _IEEE Transactions on Geoscience and Remote Sensing_ , 38(5):2142–2152, 2000.
  * Roessle et al. [2022] Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner.  Dense depth priors for neural radiance fields from sparse input views.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Rudnev et al. [2023] Viktor Rudnev, Mohamed Elgharib, Christian Theobalt, and Vladislav Golyanik.  EventNeRF: Neural radiance fields from a single colour event camera.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4992–5002, 2023.
  * Shen et al. [2021] Siyuan Shen, Zi Wang, Ping Liu, Zhengqing Pan, Ruiqian Li, Tian Gao, Shiying Li, and Jingyi Yu.  Non-line-of-sight imaging via neural transient fields.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43(7):2257–2268, 2021.
  * Shi et al. [2020] Yilei Shi, Richard Bamler, Yuanyuan Wang, and Xiao Xiang Zhu.  SAR tomography at the limit: Building height reconstruction using only 3–5 TanDEM-X bistatic interferograms.  _IEEE Transactions on Geoscience and Remote Sensing_ , 58(11):8026–8037, 2020.
  * Stringham et al. [2019] Craig Stringham, Gordon Farquharson, Davide Castelletti, Eric Quist, Lucas Riggi, Duncan Eddy, and Scott Soenen.  The capella x-band sar constellation for rapid imaging.  In _IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium_ , pages 9248–9251. IEEE, 2019.
  * Tao et al. [2023] Tang Tao, Longfei Gao, Guangrun Wang, Peng Chen, Dayang Hao, Xiaodan Liang, Mathieu Salzmann, and Kaicheng Yu.  LiDAR-: Novel LiDAR view synthesis via neural radiance fields.  _arXiv preprint arXiv:2304.10406_ , 2023.
  * Turki et al. [2022] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan.  Mega-NeRF: Scalable construction of large-scale NeRFs for virtual fly-throughs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12922–12931, 2022.
  * Wang et al. [2021] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction.  In _Advances in Neural Information Processing Systems_ , pages 27171–27183. Curran Associates, Inc., 2021.
  * Wysocki et al. [2023] Magdalena Wysocki, Mohammad Farid Azampour, Christine Eilers, Benjamin Busam, Mehrdad Salehi, and Nassir Navab.  Ultra-nerf: Neural radiance fields for ultrasound imaging.  _arXiv preprint arXiv:2301.10520_ , 2023.
  * Xie et al. [2023] Songlin Xie, Lei Zhang, Gwanggil Jeon, and Xiaomin Yang.  Remote sensing neural radiance fields for multi-view satellite photogrammetry.  _Remote Sensing_ , 15(15):3808, 2023.
  * Xie et al. [2022] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar.  Neural fields in visual computing and beyond.  In _Computer Graphics Forum_ , pages 641–676. Wiley Online Library, 2022.
  * Xu et al. [2023] Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, and Dahua Lin.  Grid-guided neural radiance fields for large urban scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8296–8306, 2023.
  * Yariv et al. [2021] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume Rendering of Neural Implicit Surfaces.  In _Advances in Neural Information Processing Systems_ , pages 4805–4815. Curran Associates, Inc., 2021.
  * Zhang and Rupnik [2023] Lulin Zhang and Ewelina Rupnik.  Sparsesat-NeRF: Dense depth supervised neural radiance fields for sparse satellite images.  In _ISPRS Annals 2023_ , 2023.
  * Zhang and Li [2023] Tongtong Zhang and Yuanxiang Li.  Fast satellite tensorial radiance field for multi-date satellite imagery of large size.  _arXiv preprint arXiv:2309.11767_ , 2023.
  * Zhao et al. [2023] Xiaopeng Zhao, Zhenlin An, Qingrui Pan, and Lei Yang.  Nerf2 - neural radio-frequency radiance fields.  _arXiv preprint arXiv:2305.06118_ , 2023.
  * Zink et al. [2014] Manfred Zink, Markus Bachmann, Benjamin Brautigam, Thomas Fritz, Irena Hajnsek, Alberto Moreira, Birgit Wessel, and Gerhard Krieger.  TanDEM-X : The new global DEM takes shape.  _IEEE Geoscience and Remote Sensing Magazine_ , 2(2):8–23, 2014.


# On Memory Codelets: Prefetching, Recoding, Moving and Streaming Data

Dawson Fox<EMAIL_ADDRESS>/<EMAIL_ADDRESS>Argonne National
LaboratoryLemontIllinoisUSA University of DelawareNewarkDelawareUSA ,  Jose
Monsalve Diaz<EMAIL_ADDRESS>Argonne National
LaboratoryLemontIllinoisUSA and  Xiaoming Li<EMAIL_ADDRESS>University of
DelawareNewarkDelawareUSA

(2018; 22 December 2022)

###### Abstract.

For decades, memory capabilities have scaled up much slower than compute
capabilities, leaving memory utilization as a major bottleneck. Prefetching
and cache hierarchies mitigate this in applications with easily predictable
memory accesses or those with high locality. In other applications like sparse
linear algebra or graph-based applications, these strategies do not achieve
effective utilization of memory. This is the case for the von Neumann model of
computation, but other program execution models (PXM) provide different
opportunities. Furthermore, the problem is complicated by increasing levels of
heterogeneity and devices’ varying memory subsystems. The Codelet PXM
presented in this paper provides a program structure that allows for well-
defined prefetching, streaming, and recoding operations to improve memory
utilization and efficiently coordinate data movement with respect to
computation. We propose the Memory Codelet, an extension to the original
Codelet Model, to provide users these functionalities in a well-defined manner
within the Codelet PXM.

Codelets, Program Execution Models, Sequential Codelet Model, Heterogeneity,
Memory Recode, Near Memory Compute

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; February 25 – March 01, 2023; Montreal,
Canada††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Compute capabilities on a single chip increased rapidly for decades as a
result of Moore’s Law and Dennard Scaling. Memory performance, however, is
increasingly becoming the first order constraint. There are two main causes
behind the so-called “memory wall”. Firstly, memory latency and bandwidth have
increased much slower than their logic circuit counterparts (Schor and Larson,
2022). Secondly, cache hierarchies and prefetching mechanisms, two of the
industry’s most prevalent answers to this problem, naturally lost their
benefits when the memory accesses became more concurrent and less predictable.
As a result, modern day applications with less common memory access patterns
are often memory bound, and the memory wall remains a major obstacle on the
systems of today. Graph-based and sparse linear algebra applications,
particularly their parallel implementations, are examples that can have poor
performance on conventional systems due to concurrent pointer chasing and
irregular memory accesses.

Recently, computer architects have been exploring innovative ways of
overcoming the memory wall. Smart prefetching (Kallurkar and Sarangi, 2016;
Talati et al., 2021; Jouppi, 1990; Yu et al., 2015; Zhang et al., 2018), near-
memory and in-memory computing (Kwon et al., 2019; Ke et al., 2020; Ahn et
al., 2015), recoding engines (Fang et al., 2015, 2017; Schwedock et al., 2022;
Rawal et al., 2019), and other techniques have been proposed to accelerate
applications past the memory wall. An approach similar to ours has been
attempted (Ham et al., 2019) but not involved with or benefiting from the
Codelet Model PXM, building off of (Smith, 1982) instead. As the architectures
become more heterogeneous, the problem of orchestration of computation and
data becomes more challenging. Furthermore, memory-focused accelerators bring
additional problems that would be difficult to tackle with current execution
models. However, the orchestration of computation across a highly
heterogeneous system that features these innovations is typically left to the
software developer. The status-quo software-handmaid solution is simply not
productive nor scalable. Our key insight is that a Program Execution Model
(PXM) (Qu et al., 2017; Dennis, 2017) for extremely heterogeneous systems,
supporting both compute and memory semantics, will greatly relieve the
developer’s burden and likely lead to higher productivity and efficiency. This
paper presents the concept of Memory Codelets, a definition that extends from
previous versions of the Codelet Model (Gao et al., 2011; Diaz et al., 2022a;
Monsalve et al., 2019; Suetterlein et al., 2013) with a focus on smarter
memory orchestration in the presence of extreme heterogeneous architectures.

###  1.1. Motivation

Currently, the most common orchestration method for accelerators is the
offloading model, similar to the organization of execution commonly seen in
heterogeneous systems that make use of GPUs and other accelerators. This model
distinguishes a host, usually the CPU where an operating system runs, and
multiple devices called accelerators. Work is divided up into kernels that the
host deploys into accelerators. Data is transferred to/from the accelerator’s
memory and the host memory. While most accelerators provide manual memory
management capabilities, some modern systems are also implementing unified
shared memory to maintain automatic coherence between accelerator and host
memories.

Orchestration of programs in the offloading model is a manual process. The
user must coordinate scheduling of data and kernels into the different
accelerators. From the perspective of the host, a kernel is just a device
function initiated through the driver API, and the user or system must
guarantee that data needed for its execution is appropriately allocated and
initialized prior to the execution of the kernel. A kernel itself does not
provide any information that explicitly mentions the relationship about the
memory it requires to perform its computation. Therefore, the user relies on
manual memory management and system responses to page faults or cache faults
to coordinate memory needed for the execution of a kernel. As a consequence,
the scheduler does not have any knowledge of memory operations, or how a
kernel depends on specific data, instead, memory and compute must be organized
in a meticulous way that respects such dependencies. As data movement grows
more important for achieving high performance, the burden on the programmer
grows greater.

Existing alternatives to the offloading model allow kernels/tasks to have
unrestricted behavior and lack clear definitions of the data consumed and
produced, which does not aid in predicting memory accesses or utilizing the
memory systems better. The common alternative is heterogeneous tasking models
(Ayguade et al., 2009; de Supinski et al., 2018; Aguilar Mena et al., 2022; Li
et al., 2021; Fernández et al., 2014; NVIDIA, 2019). Kernels are represented
as tasks that are connected by dependencies forming a graph. However, tasks
are not side-effect free, as they are part of a unified shared memory system,
allowing any pointer within the task to reference any part of the memory.
Furthermore, models often use dependencies only to represent control flow
dependencies, not data dependencies. Thus, although memory locations (e.g.
variables or pointers) are used to name dependencies, these are not
necessarily used to make decisions about memory orchestration, nor do they
represent all the set of memory locations that can be accessed by the task.
Take for example an OpenMP target region that uses the depend clause to define
dependencies, and the map clause to define data movement. The depend clause
only determines the producer-consumer relationship of tasks, while the map
clause only determines memory movement operations between host and device.
However, a task may contain pointers not defined in the depend clause, for
example when dealing with global variables. This is worse when a unified
shared memory system spans across host and device, since pointers in the tasks
may interact with any part of the subsystem (Herrera Guaitero et al., 2022).
Such freedom makes it hard to predict task latency and side effects. As a
result, performing operations such as streaming, data recoding, or using
heterogeneous systems with memory accelerators, are not well differentiated in
the program description and execution, and must be managed by the user. Saying
it differently, the scheduler is not in charge of orchestrating memory and its
relationship to compute. Restricting tasks to always define dependencies
helps. Such an approach is used in the OpenMP Cluster model (Yviquel et al.,
2018, 2022), allowing the runtime to perform smart memory management. However,
this does not yet resolve issues with complex memory hierarchies found in
heterogeneous systems.

Previous work has demonstrated that the Codelet Model can be an effective
programming model for heterogeneous systems (Geng et al., 2020; Fox et al.,
2022; Diaz et al., 2022b). However, Codelets being non-preemptive can be both
a strength and a weakness. Without the ability to interrupt Codelets, data
locality becomes essential when designing a high performance and efficient
program. If required data is not local to a Compute Unit (CU) when a Codelet
is executed, the processor effectively stalls while the data is fetched. With
proper management, on the other hand, Codelets’ atomic nature can ensure that
computation is performed while the necessary data is local and that the data
need only be local for as short a time as possible. This missing capability is
the key to realize the Codelet model’s full potential on extreme scale
heterogeneous systems. Prior work has suggested percolation as an important
way to improve performance through the memory wall (Tan et al., 2008). Our
proposal of the Memory Codelet provides exactly such an explicit mechanism to
orchestrate memory for compute Codelets. Memory Codelets will be executed on a
near-memory architecture dedicated to Memory Codelets only, and would be able
to prefetch data, stream data, and perform recode operations. More specific
operations like pointer swizzling might also be handled by Memory Codelets. On
systems with less conventional memory hierarchies or with multiple devices,
Memory Codelets would also be able to explicitly move data throughout the
memory hierarchy (for example, to local scratchpad memory) and coordinate
reads, writes, streams, and recode operations to the various devices.

###  1.2. Contributions

The contributions of this paper are described as follows:

  * •

Conceptualization of Memory Codelets and their integration into the Codelet
Model

  * •

Integration of Memory Codelets into the Codelet Model, the Sequential Codelet
Model, and their Abstract Machines

  * •

Simple examples that demonstrate the use of Memory Codelets through Sequential
Codelet Model semantics

##  2\. Background

Program Execution Models (PXMs) as a concept are effectively a holistic system
view that offers clear and well-defined behavior at all levels of the system
from the hardware to the software. As this is often difficult to achieve
without significant resources and end-to-end design, PXMs are often relegated
to the domain of software runtimes in practice; however, they still offer the
user/developer an organization of execution that can be relied upon and used
to build effective programs for systems. This is especially important as the
industry trends towards extreme heterogeneity, and various programming models
and APIs might be used to craft a single high performing program. A well
defined program execution model is a fundamental step towards
hardware/software co-design (Ang et al., 2022). A more precise definition of
PXMs can be found in (Qu et al., 2017; Dennis et al., 2011; Dennis, 1997).

###  2.1. Codelet Model

The Codelet Model is a dataflow-inspired PXM to organize computation with an
accompanying abstract machine (Gao et al., 2011; Suetterlein et al., 2013). It
is both fine-grained and event driven, breaking programs into Codelets, non-
preemptive portions of sequential computation with defined inputs and outputs.
As such, Codelets are the quantum unit of scheduling of a Codelet-based
program. Programs in the Codelet Model are described by Direct Acyclic Graphs,
with nodes in the graph representing Codelets and directed arcs representing
data and control dependencies between them. This allows the program to clearly
define the ordering of Codelet execution only where necessary while permitting
flexibility in Codelet scheduling otherwise. Codelets are also event driven,
allowing dependencies to be seen as split phase transactions, or as
activations based on external events. To benefit more from locality, Codelets
are grouped into Threaded Procedures (TPs). The Codelet Abstract Machine
defines the components of a system that executes Codelet Programs. It
designates the roles of Compute Unit (CU) and Scheduling Unit (SU): the SU is
responsible for creation of TPs and scheduling Codelets during runtime, while
the CU is responsible for ”firing” (executing) Codelets once their
dependencies are fulfilled. For a more fleshed out description of the basic
Codelet PXM, view prior publications (Suettlerlein et al., 2013; Fox et al.,
2022).

Since CUs are general in the abstract machine, almost any computational
architecture can be mapped to them, which enables the Codelet Model PXM to
gracefully handle extreme heterogeneity. As long as Codelets contain
computation that can feasibly be performed on at least one CU of the system or
has a version that can be executed on that CU, heterogeneous scheduling in the
Codelet Model implementation can enable dynamic heterogeneity as in (Geng et
al., 2020). The event-driven dependency-based scheduling can aid users to
reason about synchronization between the Codelets of the program, whereas
typical heterogeneous execution on conventional systems leaves synchronization
and explicit data movement to the user. Such a burden can slow the development
process and lead to hours of debugging. Furthermore, with systems increasingly
having various specialized architectures, computation of individual tasks (or
in this case, Codelets) may be drastically sped up. In some applications this
may lead to increased strain on the memory architecture, which indicates that
better memory management and data movement throughout the memory hierarchy is
paramount to continue improving performance.

###  2.2. Sequential Codelet Model

The Sequential Codelet Model (SCM) (Diaz et al., 2022a; Monsalve et al., 2019;
Monsalve Diaz, 2021) is an extension of the original Codelet Model (Gao et
al., 2011; Suetterlein et al., 2013). SCM is heavily inspired by instruction
level parallelism (ILP) techniques in sequential computing architectures. In
ILP, dataflow is used to discover dependencies in the instruction stream and
allow parallel execution of independent instructions. In particular, SCM
heavily draws from out-of-order execution that uses register names to respect
true dependencies while removing anti- and output dependencies.

In SCM, Codelet graphs are defined as a sequential stream of instructions
where dependencies are registers. However, compared to traditional registers
in sequential architectures, the dependencies between Codelets are larger in
size. This is possible because the pipeline in charge of executing the Codelet
program (i.e. Fetch, decode, execute, memory and write back) sits on top of
the compute units. Therefore, registers are assigned to a location equivalent
to the last level of cache (LLC). The execution stage of the pipeline is
comprised of these CUs, which can be implemented as any architecture such as
Streaming Multiprocessors (in GPGPUs), a single or multi core CPU system, an
FPGA, or any other exotic/specialized architecture.

Control flow instructions are also supported in SCM. These instructions allow
dynamically defining more complex Codelet Graphs by using conditional and
unconditional jumps, loops, and a subset of basic arithmetic operations.
Additionally, memory operations are necessary to move data back and forth
between the upper level memory (e.g. DRAM) and the register file. In this work
we formalize this concept as Memory Codelets, which is equivalent to memory
operations in sequential ISAs, yet more powerful thanks to the possibility of
Memory Codelets to be user defined. A complete description of the Sequential
Codelet Model and its realization in the SuperCodelet architecture can be
found in (Monsalve Diaz, 2021; Diaz et al., 2022a; Monsalve et al., 2019).

##  3\. Memory Codelets and the Codelet Abstract Machine

Traditional instruction set architectures (ISA) group instructions according
to their functionality. Such distinction also has an effect in the functional
unit used in the pipeline of the architecture that implements the ISA. Among
the different groups, memory instructions (e.g. load, and store operations)
focus on the interaction and movement of data in and out of the compute
pipeline. In addition to ISA instructions, current system architectures
feature caching and prefetching mechanisms with the purpose of managing memory
in the system, such that it increases performance while respecting a set of
rules enforced by memory consistency and coherency models. Prefetching, for
example, is an event driven mechanism that is triggered based on multiple
access patterns across multiple memory requests. In addition to the ISA,
conventional systems typically have multilevel cache hierarchies that are
completely invisible from the software perspective and do not provide
reconfigurability nor programmability in the memory hierarchy. The rigidity of
the coordination of memory in the hierarchy does not allow enough flexibility
to execute programs with uncommon memory access patterns at high performance.

In comparison, the organization of programs in the Codelet Model gives a
definite and specific plan of what data is consumed/produced by each Codelet,
which provides the information necessary for prefetching (Talati et al.,
2021), recoding (Fang et al., 2017; Schwedock et al., 2022), and streaming
strategies that are well integrated into the abstract machine and provides
customization of the memory management. By virtue of the Codelet Model PXM,
these strategies would benefit most from being implemented at multiple levels
of the system stack and employing hardware/software co-design. More benefits
of the Codelet Model PXM in future extremely heterogeneous architectures are
summarized in (Fox et al., 2022).

Memory Codelets in the Codelet Abstract Machine have a similar role for memory
management as the prefetching mechanism does in conventional execution. Memory
Codelets allow for data to be moved, manipulated, reorganized or streamed
through different components in the system, aiming to serve compute Codelets
such that their execution time is driven by arithmetic operations rather than
memory accesses. Memory Codelets make memory management explicit, and they are
particularly useful when memory access patterns cannot be recognized in the
memory subsystem. This is the case for applications that cannot easily take
advantage of caches and prefetching. Like traditional Codelets, Memory
Codelets are event- and data-driven, and form independent nodes in the Codelet
graph. However, instead of mapping to compute units that are heavily
specialized for arithmetic operations, memory Codelets map to specialized
units that emphasize throughput and low latency (Fang et al., 2017, 2015;
Schwedock et al., 2022; Shah, 2022).

###  3.1. Memory Codelet Abstract Machine

Let us begin with a definition of an abstract machine that extends from the
Sequential Codelet Model, and the original Codelet Model. Figure 1 shows the
different components of the Memory Codelet Abstract Machine. We focus on a
single level of machine hierarchy, while this can be extended to other levels
of the machine abstraction. As in traditional Codelet Models, the architecture
is made out of Compute Units (CUs) and Scheduling Units (SUs). Additionally, a
Memory Codelet Unit (MCU) is included, that is in charge of execution of
memory codelets. There are two aspects that make the Memory Codelet Unit
different to any other compute unit. First, its compute capabilities are
tailored for fast data transformation (e.g. (Fang et al., 2017; Schwedock et
al., 2022; Chien, 2022)) and movement. Second, it can directly interact and
communicate with the different memory storage components of the system,
including local memory, external memory, and specialized memory structures
(e.g. FIFO queues).

Because the Codelet Model clearly defines Codelets’ inputs and outputs, Memory
Codelets can be leveraged to benefit from the static Codelet graph of the
program being executed. The Codelet Graph defines a partial execution order of
the Codelets based on their data dependencies. The actual execution order of
Codelets depends heavily on the scheduling mechanism employed in the Codelet
Model implementation; for example, DARTS (Suettlerlein et al., 2013), a
software runtime implementation of the Codelet Model, employs multiple
scheduling mechanisms such as round robin and work stealing that the developer
can choose for their program. These scheduling mechanisms always respect data
dependencies to ensure correct program execution.

###  3.2. Prefetching, Streaming, and Recoding with Memory Codelets

When a Memory Codelet is executed, data does not need to always pass through
the MCU silicon fabric. The data can be moved directly between different
physical locations in the system (e.g. CU to CU or DRAM to a CU directly).
Thus, the MCU does not have to be a bottle neck for memory transactions, but a
coordinator for memory across the Codelet Abstract Machine. Since Memory
Compute Units can be seen as traditional near memory compute architectures,
data can be fetched to the MCU, transformed, and delivered to other physical
locations. In keeping with the goals of a PXM, the MCU is designed to
understand Codelet semantics for scheduling and dependencies management.

This mechanism can be used to ensure data locality prior to the scheduling and
execution compute Codelets. This can be an effective prefetching mechanism,
especially given that memory Codelets are written by the developer who has
knowledge of the access patterns of the program. Thus, Memory Codelets can
perform the necessary preprocessing to determine the data needed for the
compute Codelet, even if these operations are complex. For example, data does
not need to be contiguous or respect a simple stride pattern. A Memory Codelet
can perform pointer chasing across the graph to obtain the necessary
properties from different nodes.

Figure 1. Memory Codelet Abstract Machine with possible heterogeneous CU
architectures mentioned.

With the Memory Codelet Unit being implemented as a near-memory processor,
latency for memory accesses will be reduced compared to if a traditional
compute core (CU) were to perform it. An example of this behavior can be seen
below, where data is able to be reliably prefetched in a timely manner due to
the dependencies clearly dictated by the program graph. The necessary input
data for Codelet Comp1 can be prefetched from memory through Memory Codelet
LoadData1_2048L while Codelet Comp0 is being executed. Hence, overlapping
compute of Comp0, prefetching of data into R2048L_3, followed by no DRAM
memory access during execution of Comp1. This example is general and can be
extended to various situations. Furthermore, we envision that with the help of
DMA-like hardware, the Memory Codelet Unit should be able to reliably feed the
necessary data to the Compute Units in the system.

⬇

1// Load data for Comp0, prefetch data for Comp1

2MEMCOD LoadData0_2048L R2048L_2, R64B_6, R64B_22;

3MEMCOD LoadData1_2048L R2048L_3, R64B_7, R64B_23;

4// Comp0 depends on Data0 (R_2)

5// Comp1 depends on Comp0 (R_1) and Data1 (R_3)

6COD Comp0_2048L R2048L_1, R2048L_2;

7COD Comp1_2048L R2048L_3, R2048L_1, R2048L_3;

8// Store computation result

9MEMCOD StoreData_2048L R2048L_3, R64B_7, R64B_23;

Listing 1: Sparse GEMM outer product partial prefetching example

In programs where the access pattern within the register is known by producer
and consumer, streaming can be fully overlapped by computation in a timely
fashion. To further support this organization of execution, hardware-based
FIFO queues could be added to the system as mentioned in (Fox et al., 2022;
Raskar, 2021). With hardware FIFOs available, the streaming could occur as
early as allowed by the algorithm and the Codelet Model semantics without
using on-CU resources (like local memory) that might be needed by other
Codelets’ execution. The FIFO queue can be used as a buffer between the memory
loads performed by the Memory Codelet Unit and the consumer Codelet performing
main computation. Hence, traditional streaming can be implemented as a
pipeline from a Memory Codelet to a compute Codelet via FIFO queue.

In the example below, a smart streaming based outer product sparse GEMM
application is shown. The Memory Codelets walk the compressed CSR and CSC
formats and stream elements to the compute Codelet to perform the outer-
product multiplications, producing partial matrices. As the partial result
matrices are created, we can imagine that the register that connects
spOuterMatMult and PartialsSum, R2048L_4, acts as a FIFO, such that they are
streamed to the PartialsSum Codelet to form the complete result matrix.

⬇

1// Stream chunks of matrix to spOuterMatMult

2MEMCOD StreamCSCBlock_2048L R2048L_2, R64B_6, R64B_22;

3MEMCOD StreamCSRBlock_2048L R2048L_3, R64B_7, R64B_23;

4// Perform outer product mult

5// Stream partial result mats. out

6COD spOuterMatMult_2048L R2048L_4, R2048L_2, R2048L_3;

7// Stream in partial matrices and sum

8COD PartialsSum_2048L R64B_8, R2048L_4

Listing 2: Outer product sparse GEMM streaming

Beyond this, Memory Codelets being used for prefetching and streaming can be
extended into performing recode operations, similar to (Schwedock et al.,
2022; Fang et al., 2017, 2015). A recode operation could easily be crafted by
pipelining both prefetching/streaming and preprocessing in the Memory Codelet
Unit itself. This can be viewed in terms of batches of data: the CU will be
performing main computation on the earliest batch of data while the Memory
Codelet Unit is performing preprocessing on the next batch and the third batch
is in flight. Though these applications of Memory Codelets are somewhat
dependent on the implementation and the specific memory hierarchy structure of
the architecture in use, their concepts are applicable in many cases and the
extended Codelet Model provides a cohesive model of program execution. An
example can be seen below where the same outer product computation is
performed, but both matrices are in CSC form, so a Memory Codelet performs
recoding to change the second matrix to CSR for ease of computation. This
example maintains the same streaming format as the earlier example.

⬇

1// Fetch block of B; recode block of C into CSR format;

2// stream both to CU

3MEMCOD FetchCSCBlock_2048L R2048L_2, R64B_6, R64B_22;

4MEMCOD ConvertCSCBlock_2048L R2048L_3, R64B_7, R64B_23;

5// Do sp outer product mult; stream partial mat. out

6COD spOuterMatMult_2048L R2048L_4, R2048L_2, R2048L_3;

7// Sum streamed-in matrices, store result

8COD PartialsSum_2048L R64B_8, R64B2

Listing 3: Outer product sparse GEMM streaming with recode operation

###  3.3. Modification and Use of Conventional Memory Systems

Modern memory subsystems are complex with a very large design space, and can
be difficult to effectively utilize in non conventional applications with
irregular memory behavior. Though Memory Codelets can certainly aid effective
memory use strategies in Codelet programs, their effects are largely dependent
on the memory hierarchy of the system and its protocols. Two non-exclusive
paths can be targeted to improve performance of programs based on utilization
of a system’s memory hierarchy: modification of memory systems to include
less-conventional components/strategies and tuning of programs to better their
use of conventional components.

####  3.3.1. Tuned Use of Conventional Memory Systems

If we assume that the program is executing on a conventional processing system
with a 3-level cache hierarchy between the processor and DRAM, timeliness of
prefetching becomes paramount due to the danger of data being evicted from the
cache early. We speculate that this issue could be mitigated thanks to Memory
Codelets and the static information present in the Codelet Graph; L1 and L2
use would be predictable based on the properties of the Codelets. Furthermore,
increasing performance of the system based on prefetching can benefit from
more targeted approaches, as in (Lu et al., 2020). Publications such as (Liu
and Sun, 2015)(Liu and Sun, 2017) can illuminate the importance of eliminating
pure miss cycles in the cache hierarchy. With this information improved
scheduling mechanisms can be applied and Memory Codelets can be organized to
improve bandwidth utilization. This is one of the major benefits of a
software-programmable unit that can perform prefetching, combined with
scheduling that is memory-aware. Though the memory hierarchy in use is
detached from the Codelet Model semantics, in a conventional system the LLC
can be thought of as Threaded Procedure memory. This indicates that ideally,
before a Codelet’s firing, the data it requires should be resident on that
CU’s L1 or L2 cache (depending on size). The wisdom in the citations above
could then be applied conceptually as balancing the flow of data between TP
memory and CU memory, with prediction made easier by the Codelet Program
graph.

####  3.3.2. Modified Memory Systems

Despite improving effective utilization on conventional memory systems, each
memory hierarchy has its own drawbacks. In this case, consider the earlier
distinction between LLC as TP memory and L1/L2 caches as CU local memory.
Codelets have well defined input and output and we expect the input data to be
local to the CU at the time of firing. This means in practice, a Codelet’s
“size” would be limited based on the size of local memory storage. However,
this can be avoided through the use of streaming Codelets, especially with the
aid of Memory Codelets and FIFO queues. This would be implemented best with
FIFO queues in hardware, but this may require modification of the memory
hierarchy and how data is moved through it. Codelet Model semantics and the
configurability provided by Memory Codelets would allow hardware-based FIFO
queues to coexist with typical memory hierarchies.

In addition to hardware FIFO queues, a hierarchy of scratchpad memory units
could replace the typical cache hierarchy, or more practically, the upper
levels of it. This would allow the software to have more control over how data
is prefetched in anticipation of firing Codelets and at what granularity data
is moved between DRAM and the CUs. A scratchpad memory system would
particularly aid in programs that typically achieve low performance on
conventional systems due to low use of data loaded because of cache line size.
In addition, it would avoid slowdowns in programs that stall often due to
cache thrashing and cache line invalidations. It is a possibility that both to
maintain coherence and improve performance throughout the memory hierarchy,
Memory Codelets could be inserted into the Codelet Graph by the compiler based
on analysis of the graph and the Codelets’ dependencies, somewhat analogous to
the strategy of (Talati et al., 2021) in a different PXM.

Fig. 2 shows an illustration of strategies to use Memory Codelets to improve a
program that is hindered by typical cache protocols. In this Figure T5
represents the runtime of a program in a traditional architecture. T4 benefits
from prefetching data. T3 is lowered by prefetching data and recoding it, such
that the computation itself is faster. T2 uses streaming from a Memory Codelet
to the computation Codelet via FIFO. Finally T1 combines all the approaches:
Prefetching, recoding, and streaming.

Figure 2. A representation of how Memory Codelets could be used to accelerate
programs with poor cache behavior. Blue and yellow boxes include loading
memory in a way that bypasses cache (e.g. loading from scratchpad memory).
Yellow boxes include recoding operations done by the MCU such that main
computation is more regular.

####  3.3.3. Memory Codelets and Coherency

Because Memory Codelets can allow for fine-grained control over data movement
throughout the memory hierarchy of the system, they can help the Codelet Model
to fulfill the role of providing coherency to the user at a software level.
Most conventional systems with multiple compute units on a chip (such as
multiple cores in a multiprocessor) have mechanisms to provide memory
coherency throughout the memory hierarchy and between the compute units.
Coherency at the hardware level allows the software to have a flat, one
dimensional view of the memory without having to manage multiple copies of
data. While this is certainly useful for programmers developing multithreaded
applications, the hardware mechanisms that provide the coherency can use up
precious chip real estate and the cost can scale poorly as the number of
compute units and physical memory units increase on a chip. Beyond this,
enforcing the coherency model through cache line invalidation can reduce
effective use of memory resources. The need for hardware coherency can be
thought of as a crutch for programmers using the threading model for parallel
programming; in other words, the threading model of computation is so ill-
defined and prone to non-determinism that programmers would have great
difficulty developing on a multithreaded system without hardware coherency
(Lee, 2006). However, the situation depends entirely on the semantics of the
Program Execution Model employed in the system.

In the threading model of parallel programming, various threads can
concurrently execute and access the entire memory space of the program. This
places the entire burden of synchronization and avoiding data races on the
programmer’s shoulders, which is notoriously difficult (Lee, 2006).
Furthermore, on even the most modestly heterogeneous systems, coherency
generally becomes the responsibility of the programmer through explicit data
movements between host and device.

In the Codelet Model PXM, computation is broken down into pieces of
computation that are partially ordered by data dependencies between them as
expressed in the Codelet Graph. Through this mechanism, data races are avoided
and the necessary synchronization is achieved. Moreover, because the program
is broken down into Codelets and their specific input and output data, two or
more Codelets that access the same data (with at least one of the accesses
being a write/store) will only execute when no data dependency has been
declared between them, which in a correctly written program indicates the
program is intended to have non-deterministic behavior in that program
section. In other words, Codelets are partially ordered based on their data
dependencies.

Readers may question the difference in difficulty between writing
synchronization into a multithreaded program and correctly writing the
dependencies in a Codelet Model program; the main difference lies in the
finite behavior of Codelets and the data they access. Synchronization and
orchestration of threads that can access any data in the address space is
unwieldy and unmanageable, whereas finer-grained Codelets with well defined
data access make the process more straightforward. As systems trend towards
extreme heterogeneity, hardware coherency may not be feasible. The CXL 3.0
specification includes coherency mechanisms (Compute Express Link, 2022) but
no implementations have been released yet. While this is generally considered
manageable in a CPU-GPU only system, with various accelerators or specialized
architectures, each possibly having their own memory hierarchy it rapidly
becomes less manageable. The burden is eased with a well defined PXM and
Memory Codelets to provide clear functionality. This issue is discussed
further with respect to chiplet-based systems in (Fox et al., 2022).

##  4\. Conclusion

In this paper we introduce the concept of Memory Codelet into the Codelet
Model. We define an extended abstract machine with a Memory Codelet Unit that
supports coordination of prefetching, streaming and memory scheduling
operations, as well as data recoding operations. We demonstrate through the
use of the Sequential Codelet Model and how Memory Codelets can be used in the
context of an application. Future work includes implementation and testing of
this work on heterogeneous architectures.

## References

  * (1)
  * Aguilar Mena et al. (2022) Jimmy Aguilar Mena, Omar Shaaban, Vicenç Beltran, Paul Carpenter, Eduard Ayguade, and Jesus Labarta Mancho. 2022.  OmpSs-2@Cluster: Distributed Memory Execution of Nested OpenMP-style Tasks. In _Euro-Par 2022: Parallel Processing_ , José Cano and Phil Trinder (Eds.). Springer International Publishing, Cham, 319–334.
  * Ahn et al. (2015) Junwhan Ahn, Sungpack Hong, Sungjoo Yoo, Onur Mutlu, and Kiyoung Choi. 2015.  A Scalable Processing-in-Memory Accelerator for Parallel Graph Processing. In _Proceedings of the 42nd Annual International Symposium on Computer Architecture_ (Portland, Oregon) _(ISCA ’15)_. Association for Computing Machinery, New York, NY, USA, 105–117.  https://doi.org/10.1145/2749469.2750386
  * Ang et al. (2022) James Ang, Andrew A Chien, Simon David Hammond, Adolfy Hoisie, Ian Karlin, Scott Pakin, John Shalf, and Jeffrey S Vetter. 2022\.  _Reimagining Codesign for Advanced Scientific Computing: Report for the ASCR Workshop on Reimagining Codesign_.  Technical Report. USDOE Office of Science (SC)(United States).
  * Ayguade et al. (2009) Eduard Ayguade, Nawal Copty, Alejandro Duran, Jay Hoeflinger, Yuan Lin, Federico Massaioli, Xavier Teruel, Priya Unnikrishnan, and Guansong Zhang. 2009.  The Design of OpenMP Tasks.  _IEEE Transactions on Parallel and Distributed Systems_ 20, 3 (2009), 404–418.  https://doi.org/10.1109/TPDS.2008.105
  * Chien (2022) Andrew Chien. 2022\.  Recode/UDP - architecture for efficient data representation and transformation.  http://people.cs.uchicago.edu/~aachien/lssg/research/10x10/
  * Compute Express Link (2022) Compute Express Link 2022.  _CXL 3.0 Specification_.  Compute Express Link.
  * de Supinski et al. (2018) Bronis R. de Supinski, Thomas R. W. Scogland, Alejandro Duran, Michael Klemm, Sergi Mateo Bellido, Stephen L. Olivier, Christian Terboven, and Timothy G. Mattson. 2018.  The Ongoing Evolution of OpenMP.  _Proc. IEEE_ 106, 11 (2018), 2004–2019.  https://doi.org/10.1109/JPROC.2018.2853600
  * Dennis (1997) J. Dennis. 1997\.  A Parallel Program Execution Model Supporting Modular Software Construction. In _MPPM ’97: Proceedings of the Conference on Massively Parallel Programming Models_. IEEE Computer Society, Washington, DC, USA, 50.
  * Dennis (2017) Jack B. Dennis. 2017\.  Principles to Support Modular Software Construction.  _Journal of Computer Science and Technology_ 32, 1 (jan 2017), 3–10.  https://doi.org/10.1007/s11390-017-1702-6
  * Dennis et al. (2011) Jack B. Dennis, Robert Pavel, and Guang R. Gao. 2011\.  Comparative Evaluation of Alternative Program Execution Models. (September 2011).  Technical Memo.
  * Diaz et al. (2022a) Jose M Monsalve Diaz, Kevin Harms, Rafael A. Herrera Guaitero, Diego A. Roa Perdomo, Kalyan Kumaran, and Guang R. Gao. 2022a.  The SuperCodelet Architecture. In _Proceedings of the 1st International Workshop on Extreme Heterogeneity Solutions_ (Seoul, Republic of Korea) _(ExHET ’22)_. Association for Computing Machinery, New York, NY, USA, Article 2, 6 pages.  https://doi.org/10.1145/3529336.3530823
  * Diaz et al. (2022b) Jose M Monsalve Diaz, Kevin Harms, Rafael A. Herrera Guaitero, Diego A. Roa Perdomo, Kalyan Kumaran, and Guang R. Gao. 2022b.  The SuperCodelet Architecture. In _Proceedings of the 1st International Workshop on Extreme Heterogeneity Solutions_ (Seoul, Republic of Korea) _(ExHET ’22)_. Association for Computing Machinery, New York, NY, USA, Article 2, 6 pages.  https://doi.org/10.1145/3529336.3530823
  * Fang et al. (2015) Yuanwei Fang, Tung T. Hoang, Michela Becchi, and Andrew A. Chien. 2015.  Fast Support for Unstructured Data Processing: The Unified Automata Processor. In _Proceedings of the 48th International Symposium on Microarchitecture_ (Waikiki, Hawaii) _(MICRO-48)_. Association for Computing Machinery, New York, NY, USA, 533–545.  https://doi.org/10.1145/2830772.2830809
  * Fang et al. (2017) Yuanwei Fang, Chen Zou, Aaron J. Elmore, and Andrew A. Chien. 2017\.  UDP: A Programmable Accelerator for Extract-Transform-Load Workloads and More. In _2017 50th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. 55–68.
  * Fernández et al. (2014) Alejandro Fernández, Vicenç Beltran, Xavier Martorell, Rosa M. Badia, Eduard Ayguadé, and Jesus Labarta. 2014.  Task-Based Programming with OmpSs and Its Application. In _Euro-Par 2014: Parallel Processing Workshops_ , Luís Lopes, Julius Žilinskas, Alexandru Costan, Roberto G. Cascella, Gabor Kecskemeti, Emmanuel Jeannot, Mario Cannataro, Laura Ricci, Siegfried Benkner, Salvador Petit, Vittorio Scarano, José Gracia, Sascha Hunold, Stephen L. Scott, Stefan Lankes, Christian Lengauer, Jesús Carretero, Jens Breitbart, and Michael Alexander (Eds.). Springer International Publishing, Cham, 601–612.
  * Fox et al. (2022) Dawson Fox, Jose M Monsalve Diaz, and Xiaoming Li. 2022\.  Chiplets and the Codelet Model.  _arXiv preprint arXiv:2209.06083_ (2022).
  * Gao et al. (2011) Guang Gao, Joshua Suetterlein, and Stephane Zuckerman. 2011\.  Toward an Execution Model for Extreme-Scale Systems \- Runnemede and Beyond. (April 2011).  Technical Memo.
  * Geng et al. (2020) Tongsheng Geng, Marcos Amaris, Stéphane Zuckerman, Alfredo Goldman, Guang R. Gao, and Jean-Luc Gaudiot. 2020.  PDAWL: Profile-Based Iterative Dynamic Adaptive WorkLoad Balance on Heterogeneous Architectures. In _Job Scheduling Strategies for Parallel Processing: 23rd International Workshop, JSSPP 2020, New Orleans, LA, USA, May 22, 2020, Revised Selected Papers_ (New Orleans, LA, USA). Springer-Verlag, Berlin, Heidelberg, 145–162.  https://doi.org/10.1007/978-3-030-63171-0_8
  * Ham et al. (2019) Tae Jun Ham, Juan L. Aragón, and Margaret Martonosi. 2019\.  Efficient Data Supply for Parallel Heterogeneous Architectures.  _ACM Trans. Archit. Code Optim._ 16, 2, Article 9 (apr 2019), 23 pages.  https://doi.org/10.1145/3310332
  * Herrera Guaitero et al. (2022) Rafael Andres Herrera Guaitero, Jose M Monsalve Diaz, Thomas Applencourt, Xiaoming Li, and Johannes Doerfert. 2022\.  Automatic Asynchronous Execution of Synchronously Offloaded OpenMP Target Regions. In _LLVM-HPC Workshop at the 2022 International Conference for High Performance Computing, Networking, Storage and Analysis (SC22)_. Springer International Publishing.
  * Jouppi (1990) Norman P. Jouppi. 1990\.  Improving Direct-Mapped Cache Performance by the Addition of a Small Fully-Associative Cache and Prefetch Buffers.  _SIGARCH Comput. Archit. News_ 18, 2SI (may 1990), 364–373.  https://doi.org/10.1145/325096.325162
  * Kallurkar and Sarangi (2016) Prathmesh Kallurkar and Smruti R. Sarangi. 2016.  pTask: A smart prefetching scheme for OS intensive applications. In _2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. 1–12.  https://doi.org/10.1109/MICRO.2016.7783706
  * Ke et al. (2020) Liu Ke, Udit Gupta, Benjamin Youngjae Cho, David Brooks, Vikas Chandra, Utku Diril, Amin Firoozshahian, Kim Hazelwood, Bill Jia, Hsien-Hsin S. Lee, Meng Li, Bert Maher, Dheevatsa Mudigere, Maxim Naumov, Martin Schatz, Mikhail Smelyanskiy, Xiaodong Wang, Brandon Reagen, Carole-Jean Wu, Mark Hempstead, and Xuan Zhang. 2020.  RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing. In _2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA)_. 790–803.  https://doi.org/10.1109/ISCA45697.2020.00070
  * Kwon et al. (2019) Youngeun Kwon, Yunjae Lee, and Minsoo Rhu. 2019.  TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learning. In _Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture_ (Columbus, OH, USA) _(MICRO ’52)_. Association for Computing Machinery, New York, NY, USA, 740–753.  https://doi.org/10.1145/3352460.3358284
  * Lee (2006) E.A. Lee. 2006.  The problem with threads.  _Computer_ 39, 5 (2006), 33–42.  https://doi.org/10.1109/MC.2006.180
  * Li et al. (2021) Zexin Li, Yuqun Zhang, Ao Ding, Husheng Zhou, and Cong Liu. 2021.  Efficient algorithms for task mapping on heterogeneous CPU/GPU platforms for fast completion time.  _Journal of Systems Architecture_ 114 (2021), 101936.  https://doi.org/10.1016/j.sysarc.2020.101936
  * Liu and Sun (2017) Yuhang Liu and Xian-He Sun. 2017.  CaL: Extending data locality to consider concurrency for performance optimization.  _IEEE Transactions on Big Data_ 4, 2 (2017), 273–288.
  * Liu and Sun (2015) Yu-Hang Liu and Xian-He Sun. 2015.  LPM: concurrency-driven layered performance matching. In _2015 44th International Conference on Parallel Processing_. IEEE, 879–888.
  * Lu et al. (2020) Xiaoyang Lu, Rujia Wang, and Xian-He Sun. 2020.  Apac: An accurate and adaptive prefetch framework with concurrent memory access analysis. In _2020 IEEE 38th International Conference on Computer Design (ICCD)_. IEEE, 222–229.
  * Monsalve et al. (2019) Jose Monsalve, Kevin Harms, Kumaran Kalyan, and Guang Gao. 2019\.  Sequential Codelet Model of Program Execution. A Super-Codelet model based on the Hierarchical Turing Machine. In _2019 IEEE/ACM Third Annual Workshop on Emerging Parallel and Distributed Runtime Systems and Middleware (IPDRM)_. 1–8.  https://doi.org/10.1109/IPDRM49579.2019.00005
  * Monsalve Diaz (2021) Jose Monsalve Diaz. 2021\.  _Sequential codelet model: A SuperCodelet program execution model and architecture_.  Ph. D. Dissertation. University of Delaware.
  * NVIDIA (2019) NVIDIA. 2019.  Getting Started with CUDA Graphs (Accessed Dec 2022).  _NVIDIA_ (2019).  {https://developer.nvidia.com/blog/cuda-graphs/}
  * Qu et al. (2017) Peng Qu, Jin Yan, You-Hui Zhang, and Guang R. Gao. 2017\.  Parallel Turing Machine, a Proposal.  _Journal of Computer Science and Technology_ 32, 2 (mar 2017), 269–285.  https://doi.org/10.1007/s11390-017-1721-3
  * Raskar (2021) Siddhisanket Raskar. 2021\.  _Dataflow software pipelining for codelet model using hardware-software co-design_.  Ph. D. Dissertation. University of Delaware.
  * Rawal et al. (2019) Arjun Rawal, Yuanwei Fang, and Andrew Chien. 2019\.  Programmable Acceleration for Sparse Matrices in a Data-Movement Limited World. In _2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)_. 47–56.  https://doi.org/10.1109/IPDPSW.2019.00016
  * Schor and Larson (2022) David Schor and Matt Larson. 2022.  IEDM 2022: Did we just witness the death of Sram?  https://fuse.wikichip.org/news/7343/iedm-2022-did-we-just-witness-the-death-of-sram/
  * Schwedock et al. (2022) Brian C. Schwedock, Piratach Yoovidhya, Jennifer Seibert, and Nathan Beckmann. 2022.  Täkō: A Polymorphic Cache Hierarchy for General-Purpose Optimization of Data Movement. In _Proceedings of the 49th Annual International Symposium on Computer Architecture_ (New York, New York) _(ISCA ’22)_. Association for Computing Machinery, New York, NY, USA, 42–58.  https://doi.org/10.1145/3470496.3527379
  * Shah (2022) Agam Shah. 2022\.  Chipmakers looking at New Architecture to drive Computing ahead.  https://www.hpcwire.com/2022/11/23/chipmakers-looking-at-new-architecture-to-drive-computing-ahead/
  * Smith (1982) James E. Smith. 1982\.  Decoupled Access/Execute Computer Architectures. In _Proceedings of the 9th Annual Symposium on Computer Architecture_ (Austin, Texas, USA) _(ISCA ’82)_. IEEE Computer Society Press, Washington, DC, USA, 112–119.
  * Suetterlein et al. (2013) Joshua Suetterlein, Stéphane Zuckerman, and Guang R. Gao. 2013.  An Implementation of the Codelet Model.  In _Euro-Par 2013 Parallel Processing_ , Felix Wolf, Bernd Mohr, and Dieter an Mey (Eds.). Springer Berlin Heidelberg.
  * Suettlerlein et al. (2013) Joshua Suettlerlein, Stéphane Zuckerman, and Guang R. Gao. 2013.  An Implementation of the Codelet Model. In _Euro-Par 2013 Parallel Processing_ , Felix Wolf, Bernd Mohr, and Dieter an Mey (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 633–644.
  * Talati et al. (2021) Nishil Talati, Kyle May, Armand Behroozi, Yichen Yang, Kuba Kaszyk, Christos Vasiladiotis, Tarunesh Verma, Lu Li, Brandon Nguyen, Jiawen Sun, John Magnus Morton, Agreen Ahmadi, Todd Austin, Michael O’Boyle, Scott Mahlke, Trevor Mudge, and Ronald Dreslinski. 2021.  Prodigy: Improving the Memory Latency of Data-Indirect Irregular Workloads Using Hardware-Software Co-Design. In _2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_. 654–667.  https://doi.org/10.1109/HPCA51647.2021.00061
  * Tan et al. (2008) Guangming Tan, Vugranam C Sreedhar, and Guang R Gao. 2008\.  Just-in-time locality and percolation for optimizing irregular applications on a manycore architecture. In _International Workshop on Languages and Compilers for Parallel Computing_. Springer, 331–342.
  * Yu et al. (2015) Xiangyao Yu, Christopher J. Hughes, Nadathur Satish, and Srinivas Devadas. 2015.  IMP: Indirect Memory Prefetcher. In _Proceedings of the 48th International Symposium on Microarchitecture_ (Waikiki, Hawaii) _(MICRO-48)_. Association for Computing Machinery, New York, NY, USA, 178–190.  https://doi.org/10.1145/2830772.2830807
  * Yviquel et al. (2018) Hervé Yviquel, Lauro Cruz, and Guido Araujo. 2018\.  Cluster Programming Using the OpenMP Accelerator Model.  _ACM Trans. Archit. Code Optim._ 15, 3, Article 35 (aug 2018), 23 pages.  https://doi.org/10.1145/3226112
  * Yviquel et al. (2022) Hervé Yviquel, Marcio Pereira, Emílio Francesquini, Guilherme Valarini, Gustavo Leite, Pedro Rosso, Rodrigo Ceccato, Carla Cusihualpa, Vitoria Dias, Sandro Rigo, Alan Souza, and Guido Araujo. 2022\.  The OpenMP Cluster Programming Model.  (2022).  https://doi.org/10.1145/3547276.3548444 arXiv:arXiv:2207.05677
  * Zhang et al. (2018) Dan Zhang, Xiaoyu Ma, Michael Thomson, and Derek Chiou. 2018\.  Minnow: Lightweight Offload Engines for Worklist Management and Worklist-Directed Prefetching. In _Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems_ (Williamsburg, VA, USA) _(ASPLOS ’18)_. Association for Computing Machinery, New York, NY, USA, 593–607.  https://doi.org/10.1145/3173162.3173197