# Fluid energy cascade rate and kinetic damping: new insight from 3D Landau-
fluid simulations

R. Ferrand11affiliationmark:  , F. Sahraoui11affiliationmark:  , D.
Laveder22affiliationmark:  , T. Passot22affiliationmark:  , P.L.
Sulem22affiliationmark:  and S. Galtier11affiliationmark:  33affiliationmark:
<EMAIL_ADDRESS>1 Laboratoire de Physique des Plasmas,
CNRS, École polytechnique, Université Paris-Saclay, Sorbonne Université,
Observatoire de Paris-Meudon, F-91128 Palaiseau Cedex, France  2 Université
Côte d’Azur, Observatoire de la Côte d’Azur, CNRS, Laboratoire J.L. Lagrange,
Boulevard de l’Observatoire, CS 34229, 06304 Nice Cedex 4, France  3 Institut
Universitaire de France (IUF)

###### Abstract

Using an exact law for incompressible Hall magnetohydrodynamics (HMHD)
turbulence, the energy cascade rate is computed from three-dimensional HMHD-
CGL (bi-adiabatic ions and isothermal electrons) and Landau fluid (LF)
numerical simulations that feature different intensities of Landau damping
over a broad range of wavenumbers, typically $0.05\lesssim
k_{\perp}d_{i}\lesssim 100$. Using three sets of cross-scale simulations where
turbulence is initiated at large, medium and small scales, the ability of the
fluid energy cascade to “sense” the kinetic Landau damping at different scales
is tested. The cascade rate estimated from the exact law and the dissipation
calculated directly from the simulation are shown to reflect the role of
Landau damping in dissipating energy at all scales, with an emphasis on the
kinetic ones. This result provides new prospects on using exact laws for
simplified fluid models to analyze dissipation in kinetic simulations and
spacecraft observations, and new insights into theoretical description of
collisionless magnetized plasmas.

##  1 Introduction

The understanding of turbulent astrophysical plasmas remains to date a
challenging problem: their chaotic nature and the complexity of the mechanisms
at work in such media impose limitations to the methods one can use to study
them efficiently. Yet, enhancing our understanding of turbulent plasmas would
provide the keys to solve a variety of problems related to energy dissipation,
particle heating and acceleration. Examples of systems where these processes
are crucial include the solar wind (SW) and planetary magnetospheres (Bruno &
Carbone, 2005; Matthaeus & Velli, 2011; Goldstein et al., 1995; Sahraoui et
al., 2020), accretion flows around compact objects (Balbus & Hawley, 1998;
Quataert & Gruzinov, 1999) and fusion devices (Diamond et al., 2005; Garbet,
2006; Fujisawa, 2021). In the SW, the heating problem is reflected by the slow
decline of ion temperature (as function of the radial distance from the sun)
in comparison with the prediction from the adiabatic expansion model of the
wind (Richardson et al., 1995). Turbulence has long been proposed as a way to
explain this behavior (Matthaeus et al., 1999), through scale-by-scale
transfer of energy (i.e., cascade) toward small (kinetic) scales where
dissipation is more effective (Schekochihin et al., 2009). A common tool used
to estimate this energy dissipation is the formalism of exact law for fully
developed turbulence first introduced by Kolmogorov (1941) to study
incompressible neutral fluids. In this formalism, energy is assumed to be
injected at large scales at a constant rate per unit volume $\varepsilon$,
which is assumed to be equal to the rate of cascade to smaller scales and to
the rate of dissipation at those scales. Assuming statistical homogeneity and
stationarity of the turbulent fields, and the existence of an inertial range
in which both forcing and dissipation mechanisms are negligible, the cascade
rate $\varepsilon$ must remain constant in the inertial range (Kolmogorov,
1941; Monin, 1959; Antonia et al., 1997). The formalism of exact law has been
extended to (in)compressible magnetized plasmas within various approximations
(Politano & Pouquet, 1998; Galtier, 2008; Banerjee & Galtier, 2013; Andrés &
Sahraoui, 2017; Hellinger et al., 2018; Andrés et al., 2018; Ferrand et al.,
2019).

Exact laws have been used successfully to measure the energy cascade rate in
the SW (Smith et al., 2006; Podesta et al., 2007; Sorriso-Valvo et al., 2007;
MacBride et al., 2008; Marino et al., 2008; Carbone et al., 2009; Smith et
al., 2009; Stawarz et al., 2009; Osman et al., 2011; Coburn et al., 2015;
Banerjee et al., 2016; Hadid et al., 2017) and terrestrial magnetosheath
(Hadid et al., 2018; Andrés et al., 2019). In those studies, the estimated
cascade rate was interpreted as the turbulence energy dissipation rate, and
hence used to quantify the amount of plasma heating due to turbulence
(Sorriso-Valvo et al., 2007; Carbone et al., 2009; Banerjee et al., 2016).
However, as explained above, such an equivalence between injection, cascade
and dissipation rates stems only from the hypothesis underlying exact laws
derivation and cannot be demonstrated in spacecraft observations. Indeed,
while in numerical simulations the injection, cascade and dissipation rates
can generally be estimated separately and compared to each other as done in
this paper, estimating (irreversible) dissipation from spacescraft observation
is a challenge and, generally, only the cascade rate, which is directly linked
to measurable quantities through the exact law, is accessible (Sorriso-Valvo
et al., 2007; Hadid et al., 2017). Thus, in spacecraft data, interpreting the
energy cascade rate as the actual dissipation rate is not straightforward.
This is particularly true because of the weakly collisional nature of the SW:
in such plasmas classical viscous and/or resistive effects are absent, and
dissipation is expected to occur via kinetic effects (e.g., Landau and
cyclotron resonances) (Leamon et al., 1998; Sahraoui et al., 2009, 2010; He et
al., 2015; Chen et al., 2019) that are not captured by usual fluid
descriptions of plasmas. A fundamental question arises here: is the fluid
turbulent cascade rate estimated in simulations and spacecraft observations of
space plasmas representative of the actual kinetic dissipation in those media?
It is the main goal of this paper to address this question, which impacts the
use of fluid models to interpret part of in-situ spacecraft observations in
the near-Earth space and the theoretical (fluid vs. kinetic) modeling of
weakly collisional plasmas. In contrast with previous studies based on 2D
hybrid particle-in-cell simulations (Hellinger et al., 2018; Bandyopadhyay et
al., 2020), the use of 3D LF models give the possibility to isolate the
influence of electron and ion Landau damping, neglecting all the other kinetic
effects, and is therefore very suited to address the question of interest
here.

##  2 Theoretical model

Although we are dealing with weakly compressible regimes we chose, for the
sake of simplicity, to use here the exact law derived by Ferrand et al. (2019)
for incompressible HMHD (see below about the use of more general compressible
models). Starting from the incompressible HMHD equations, and under the usual
assumptions of time stationarity, space homogeneity and infinite (kinetic and
magnetic) Reynolds numbers, one can derive for the energy cascade rate in the
inertial range the expression
$\varepsilon=\varepsilon^{MHD}+\varepsilon^{Hall}$, with

$\displaystyle\varepsilon^{MHD}=$
$\displaystyle-\frac{1}{4}\bm{\nabla}_{\bm{\ell}}\cdot\left\langle(|\delta\textbf{v}|^{2}+|\delta\textbf{b}|^{2})\delta\textbf{v}-2(\delta\textbf{v}\cdot\delta\textbf{b})\delta\textbf{b}\right\rangle,$
(1) $\displaystyle\varepsilon^{Hall}=$
$\displaystyle-\frac{1}{8}d_{i}\bm{\nabla}_{\bm{\ell}}\cdot\left\langle
2(\delta\textbf{b}\cdot\delta\textbf{j})\delta\textbf{b}-|\delta\textbf{b}|^{2}\delta\textbf{j}\right\rangle,$
(2)

where v, $\textbf{b}={\bf B}/\sqrt{\mu_{0}\rho_{0}}$ and
$\textbf{j}=\bm{\nabla}\times\textbf{b}$ are the velocity, magnetic field and
electric current in Alfvén units ($\rho_{0}$ is the constant mass density) and
$d_{i}$ is the ion inertial length. Fields are taken at points x and
$\textbf{x}^{\prime}$ separated by a spatial increment
$\bm{\ell}=\textbf{x}^{\prime}-\textbf{x}$, and the notations ${\bf
v}\equiv{\bf v}(\textbf{x})$ and ${\bf v}^{\prime}\equiv{\bf
v}(\textbf{x}^{\prime})$ are adopted. We then define the increment operator
$\delta$ as $\delta{\bf v}={\bf v}^{\prime}-{\bf v}$, and
$\bm{\nabla}_{\bm{\ell}}$ as the derivative operator with respect to the
increment $\bm{\ell}$.

##  3 Simulation data

###  3.1 Presentation of the data

In this study, HMHD-CGL refers to a fluid model with anisotropic ion pressure
whose gyrotropic components parallel and perpendicular to the local magnetic
field obey nonlinear dynamical equations where the heat fluxes are neglected
(bi-adiabatic approximation introduced by Chew, Goldenberg and Low (Chew et
al., 1956), thus the acronym). The electrons are assumed isothermal.
Differently, the LF model retains the nonlinear dynamics of the parallel and
perpendicular pressures and heat fluxes for both the ions and electrons, and
involves a closure at the level of the fourth-order moments, consistent with
the low-frequency linear kinetic theory (Snyder et al., 1997; Passot & Sulem,
2007). The main assumption for modeling Landau damping consists in retaining
the imaginary contribution of the plasma response function in the closure
relation which expresses the last retained fluid moment of the hierarchy in
terms of the lower ones. In Fourier space, this procedure generates factors of
the form “$i\,{\rm sgn}(k_{z})$” which, in physical space, identifies with the
Hilbert transform along the ambient magnetic field (Hammett & Perkins, 1990;
Hunana et al., 2019). It is then possible to generalize this formulation to
take into account magnetic field line distortion, using the convolution form
of the Hilbert transform (Snyder et al., 1997). Its approximation in the
numerical code is discussed in Passot et al. (2014). In both models, finite
ion and electron Larmor radius corrections are neglected, thus reducing the
kinetic effects to Landau damping. The Ohm’s law includes the Hall term and
the electron pressure contribution. Turbulence is forced with counter-
propagating kinetic Alfvén waves (KAWs) making an angle $\theta$ with the
ambient magnetic field, at the largest scales of the simulation domain. This
corresponds to transverse wavenumbers $k_{\perp,f}$, whose values are
summarized in Table 1. The amplitudes obey a Langevin equation, with an
oscillation frequency given by the KAW linear dispersion relation (TenBarge et
al., 2014). We also introduce two thresholds in order to constrain the sum of
perpendicular kinetic and magnetic energies to stay within a certain range.
Small-scale dissipation is ensured by the hyperviscosity and hyperdiffusivity
terms in the velocity and induction equations, of the form
$\textbf{d}_{\nu}=\nu(\Delta_{\perp}+\alpha\partial_{z}^{2})^{4}{\bf v}$ and
$\textbf{d}_{\eta}=\eta(\Delta_{\perp}+\alpha\partial_{z}^{2})^{4}\textbf{b}$,
with $\alpha$ being an anisotropy coefficient.

In all the simulations, $\beta_{i}=1$ and the ion and electron pressures are
taken isotropic and equal initially. The other parameters are reported in
Table 1. The simulations are performed using a desaliased spectral code (at
2/3 of the maximum wavenumber) with a third-order Runge-Kutta scheme for time
stepping.

Run |  $k_{\perp,f}d_{i}$ |  Resolution |  $\theta$ |  $\nu=\eta$ |  $\alpha$
---|---|---|---|---|---
CGL1 |  $0.045$ |  $512^{3}$ |  $83^{\circ}$ |  $7.35\times 10^{-8}$ |  80
CGL2 |  $0.045$ |  $512^{3}$ |  $75^{\circ}$ |  $7.35\times 10^{-8}$ |  10
CGL3 |  $0.5$ |  $512^{2}\times 1024$ |  $75^{\circ}$ |  $10^{-14}$ |  2.5
CGL4 |  $0.011$ |  $1024^{3}$ |  $75^{\circ}$ |  $3\times 10^{-3}$ |  5
LF1 |  $0.045$ |  $512^{3}$ |  $83^{\circ}$ |  $7.35\times 10^{-8}$ |  1
LF2 |  $0.045$ |  $512^{3}$ |  $75^{\circ}$ |  $7.35\times 10^{-8}$ |  1
LF3 |  $0.5$ |  $432^{3}$ |  $75^{\circ}$ |  $7\times 10^{-14}$ |  1.5
LF4 |  $0.011$ |  $512^{3}$ |  $75^{\circ}$ |  $3\times 10^{-3}$ |  2
Table 1: List of runs and their relevant parameters, where CGLx and LFx refer
to HMHD-CGL and LF simulations, respectively. The ratio of the longitudinal to
transverse box sizes is given by $\tan(\theta)$.

Two different propagation angles for the KAWs driven at the largest scale of
each LF simulation were chosen, hence tuning Landau damping to two different
levels (Kobayashi et al., 2017). This can be seen in Fig. 1 which compares the
linear dispersion relation and damping rate of the KAWs: the higher the
propagation angle, the lower the damping rate at a given scale. Note that,
while changing the angle, we do not change the amplitude of the fluctuations
at the driving scale (i.e., $\omega_{NL}$ remains constant), and so the
nonlinear parameter $\chi=\omega_{NL}/\omega_{L}$ also varies: when the angle
decreases, $k_{\parallel}$ increases and so does $\omega_{L}$, thus $\chi$ is
reduced. As the ratio $\gamma/\omega_{L}$ is approximately constant for high
oblique angles $\theta$ (e.g., Fig. 7 in Sahraoui et al. (2012)), the strength
of the Landau damping relative to the cascade rate
$\gamma/\omega_{NL}=(1/\chi)(\gamma/\omega_{L})$ thus increases as the angle
decreases.

Figure 1: Frequency $\omega$ and damping rate $\gamma$ (normalized by the ion
gyrofrequency $\omega_{ci}$) of KAWs versus the normalized transverse
wavenumber $k_{\perp}d_{i}$ (where $d_{i}$ refers to the ion inertial length)
for the LF model at the two propagation angles $\theta=83^{\circ}$ and
$75^{\circ}$ used in driving the simulations ($\beta_{i}=1$, $T_{i}=T_{e}$).

###  3.2 Energy balance and time stationarity

Let ${\mathcal{E}}_{{\rm tot}}(t)$ be the total energy of the system at time
$t$, ${\mathcal{I}}_{\rm t}(t)$ the injection rate due to the external forcing
on the perpendicular velocity components, and ${\mathcal{D}}_{\rm h}(t)$ the
total dissipation rate due to the hyperviscous and hyperdiffusive terms. Since
Landau damping does not affect the total energy balance, total energy
conservation implies

$\frac{d}{dt}{\mathcal{E}}_{{\rm tot}}(t)={\mathcal{I}}_{\rm
t}(t)-{\mathcal{D}}_{\rm h}(t).$ (3)

Denoting by ${\mathcal{E}}_{{\rm int}}$ and ${\mathcal{E}}_{\|}$ the parts of
the total energy associated with the pressure components (internal energy) and
the parallel velocity and magnetic field components entering the kinetic and
magnetic parts, we can write

$\frac{d}{dt}{\mathcal{E}}_{{\rm tot}}(t)-\frac{d}{dt}{\mathcal{E}}_{{\rm
int}}(t)-\frac{d}{dt}{\mathcal{E}}_{\|}(t)\equiv\frac{d}{dt}{\mathcal{E}}_{\perp}(t)\approx
0,$ (4)

where ${\mathcal{E}}_{\perp}$ is the sum of the perpendicular kinetic and
magnetic energies, a quantity bound to remain nearly constant by the forcing
procedure.

Because of computational constraints, the time evolution of the different
energy components is computed for low resolution (LR) simulations analogs of
runs CGL3 and LF3 and shown in Fig. 2 (injection and dissipation rates needed
to perform this extra study were not output at a high-enough frequency in the
large-resolution simulations). From this figure it is conspicuous that the
time evolution of total energy, injection and hyperdissipation is consistent
with the energy conservation (3). Moreover, one can see the driving procedure
at play in keeping the perpendicular energy ${\mathcal{E}}_{\perp}$ roughly
constant. Its time stationarity is in practice established when the
hyperdissipation rate has reached a constant value. When comparing CGL3-LR
with LF3-LR, one notices that run LF3-LR requires a larger injection rate to
maintain the same level of turbulence on the magnetic and perpendicular
velocity than in run CGL3-LR, since Landau damping efficiently converts a part
of the injected energy into internal energy. This is evidenced by the dashed
green curve in Fig. 2 (bottom), which shows that the increase of the internal
energy is consistent with the heating by heat fluxes. Moreover, the
hyperdissipation rate is lower on run LF3-LR, suggesting that part of the
cascading energy is taken by Landau damping, as will be evidenced in next
section.

Figure 2: Low resolution runs of CGL3 (CGL3-LR, top) and LF3 (LF3-LR, bottom):
Time evolution of the total energy injected in the system
$\int_{0}^{t}{\mathcal{I}}_{\rm t}dt$ (solid red line), total energy
${\mathcal{E}}_{{\rm tot}}$ (solid blue), internal energy ${\mathcal{E}}_{{\rm
int}}$ (solid green), perpendicular energy ${\mathcal{E}}_{\perp}$ (solid
magenta, roughly constant) and the time integrated hyperdissipation
$\int_{0}^{t}{\mathcal{D}}_{\rm h}dt$ (dashed red). The piece of dashed green
curve (right) starting at $t=250$, whose vertical position is arbitrary,
displays the heating due to heat fluxes, which is consistent with the increase
of internal energy.

##  4 Calculation of the energy cascade rate

Using Eqs. (1)-(2), we compute the transverse energy cascade rate for each
simulation as a function of the perpendicular increment by averaging over all
spatial positions in the simulation box and different increment vectors, at a
time for which the simulations reached a stationary state. Increment vectors
$\bm{\ell}=(\ell_{\perp},\ell_{\parallel})$ are selected following the angle
averaging method of Taylor et al. (2003), and only the increments forming an
angle of at least $45^{\circ}$ with the parallel direction are retained. As
already mentioned, the exact law used is the one for incompressible HMHD,
whereas the simulations are weakly compressible. A comparison (not shown) with
a full compressible HMHD exact law (Andrés et al., 2018) only showed slight
change ($\lesssim 10\%$ in the inertial range) of the cascade rate with
respect to the current estimate from the incompressible model. The transverse
cascade rate is then averaged over all increments of equal value of
$\ell_{\perp}$. The transverse hyperdissipation is computed in Fourier space
as

$\varepsilon^{diss}(\ell_{\perp})=\int_{0}^{k_{\perp}}dk^{\prime}_{\perp}\int
k^{\prime
8}_{\perp}(\eta|\textbf{b}({\bm{k}}^{\prime})|^{2}+\nu|\textbf{v}({\bm{k}}^{\prime})|^{2})k^{\prime}_{\perp}d\theta^{\prime}dk^{\prime}_{z}$
(5)

where we use $\ell_{\perp}=\pi/k_{\perp}$.

For simulations forced at intermediate scales, whose results are reported in
Fig. 3, the behavior of the MHD and Hall contributions to the energy cascade
rate are similar, with the latter rising up at sub-ion scales, then dominating
the former at about the ion inertial length. The total energy cascade rate is
roughly constant on more than one decade of scales in the simulations with
$\theta=83^{\circ}$, in particular in CGL1, which demonstrates the existence
of an inertial range. To highlight the effect of Landau damping on the cascade
rate, we compare in Fig. 4 the cascade rates from the LF simulations with
$\theta=75^{\circ}$ and $\theta=83^{\circ}$, normalized to the corresponding
ones from the CGL simulations. We observe a stronger decrease (by up to a
factor 5) in the normalized cascade rate at small scales for LF2
($\theta=75^{\circ}$), i.e. for the simulation with the strongest Landau
damping, than for LF1 ($\theta=83^{\circ}$) for which the normalized cascade
rate remains nearly constant at all scales. This result clearly relates the
enhancement of Landau damping at kinetic scales to the decline of the energy
cascade rate at these scales. We note also the consistency between the
(transverse) hyperdissipation and the cascade rate at the smallest scale of
the simulation box (Fig. 3).

Figure 3: Energy cascade rate $\varepsilon$, its ideal MHD and Hall components
together with the transverse hyperdissipation computed for runs CGL1 (top) and
LF1 (bottom). Plain lines represent positive values and dashed lines negative
values. Figure 4: Ratios of the energy cascade rate computed for LF
simulations over the one for CGL simulations for a driving wave angle
$\theta=83^{\circ}$ (black) and $\theta=75^{\circ}$ (red).

We complement our study with the cascade rates estimated from simulations
forced at even smaller scales (LF3 and CGL3) with $\theta=75^{\circ}$ and
reported in Fig. 5. Simulation LF3 exhibits a strong decrease in
$\varepsilon^{MHD}$ partially compensated by a quick rising of the Hall
component, giving no clear inertial range, in contrast to CGL3 which still
behaves similarly to the simulations forced at intermediate scales. As shown
below, this effect may be attributed to the fact that Landau dissipation
reaches high levels at the sub-ion scales of LF3, whereas CGL3 contains no
dissipation mechanisms other than hyper viscosity and diffusivity, which are
bound to act only at the smallest scales. Note that the sudden changes of sign
observed at large scales in some components of the cascade rates in Figs. 3
and 5 are likely to be due to the proximity of the forcing. Those observed at
small scales for the MHD component of run CGL3 would result from numerical
errors in the calculation of $\varepsilon^{MHD}$ given its very small
magnitude at those scales.

Figure 5: Same as in Fig. 3 for runs CGL3 (top) and LF3 (bottom).

To obtain a full picture as to how Landau damping affects the energy cascade
rate, we performed simulations forced at large scales (LF4 and CGL4).
Combining the runs CGL2-3-4 and LF2-3-4 we construct a multi-scale energy
cascade rate over nearly three decades of scales that highlights the effect of
Landau damping on it. As the simulations were run at different scales, the
amplitude of the forcing was changed to ensure that each simulation reaches a
fully turbulent state. Therefore, we renormalized the cascade rate
$\varepsilon$ obtained from the different simulations to match the one of
intermediate runs CGL2 and LF2, while taking care to discard the smallest
scales of intermediate and large-scale forcing cascade rates to ensure that
hyperviscosity is not acting at intermediate scales of the reconstructed
energy cascade. Fig. 6 shows the full energy cascade rate for CGL and LF runs
for the driving wave angle $\theta=75^{\circ}$. CGL runs exhibit an almost
constant energy cascade rate over two and a half decades of scales, whereas
$\varepsilon_{LF}$ decreases steadily over scales and reaches its minimum
value at the smallest ones, confirming that the behavior already observed in
Fig. 4 remains valid over a broader range of scales.

Figure 6: Energy cascade rates reconstructed with CGL2-3-4 runs and LF2-3-4
runs. The ranges spanned by each simulation are delimited by the black dotted
lines. A slight irregularity is observed on the green curve at the transition
between CGL2 and CGL3, which is caused by an insufficient overlap of the
cascade rates at these scales.

##  5 Influence of Landau dissipation

###  5.1 Heating due to heat fluxes

In the wake of the previous results an important question arises : can the
drop in the energy cascade rate for LF runs be directly connected to Landau
damping? For this purpose, we calculate the heating due to heat fluxes in
presence of Landau damping. For each species, the pressure equations with the
Hall term and the gyrotropic heat fluxes read

$\displaystyle\frac{d}{dt}\ln\left(\frac{p_{\|}|B|^{2}}{\rho^{3}}\right)$
$\displaystyle=-\frac{2c}{|B|}\widehat{\textbf{b}}\cdot\bm{\nabla}\times\bm{E_{H}}$
$\displaystyle-\frac{1}{p_{\|}}\left(-2q_{\perp}\bm{\nabla}\cdot\widehat{\textbf{b}}+\bm{\nabla}\cdot(q_{\|}\widehat{\textbf{b}})\right),$
(6) $\displaystyle\frac{d}{dt}\ln\left(\frac{p_{\perp}}{\rho|B|}\right)$
$\displaystyle=\frac{c}{|B|}\widehat{\textbf{b}}\cdot\bm{\nabla}\times\bm{E_{H}}$
$\displaystyle-\frac{1}{p_{\perp}}\left(q_{\perp}\bm{\nabla}\cdot\widehat{\textbf{b}}+\bm{\nabla}\cdot(q_{\perp}\widehat{\textbf{b}})\right).$
(7)

We define the parallel, perpendicular and total entropies per unit mass

$s_{\|}=\frac{c_{V}}{3}\ln(\frac{p_{\|}|B|^{2}}{\rho^{3}}),~{}s_{\perp}=\frac{2c_{V}}{3}\ln(\frac{p_{\perp}}{\rho|B|}),$
(8)
$s=s_{\|}+s_{\perp}=\frac{c_{V}}{3}\ln(\frac{p_{\|}p_{\perp}^{2}}{\rho^{5}}),$
(9)

where $c_{V}$ is the specific heat at constant volume. Denoting by $e$ the
internal energy per unit mass, the internal energy per unit volume reads
$E\equiv\rho e=p_{\perp}+\frac{1}{2}p_{\|}=\frac{3}{2}nT$ where
$T=\frac{1}{3}(2T_{\perp}+T_{\|})$. From $e=c_{V}T$, one gets
$c_{V}=\frac{3}{2m}$ (the Boltzmann constant is included in the definition of
temperature). The total entropy then obeys

$\partial_{t}(\rho s)+\bm{\nabla}\cdot\left(\rho
s\bm{u}+(\frac{q_{\perp}}{T_{\perp}}+\frac{q_{\|}}{2T_{\|}})\widehat{\textbf{b}}\right)=(\frac{1}{T_{\|}}-\frac{1}{T_{\perp}})q_{\perp}\bm{\nabla}\cdot\widehat{\textbf{b}}-\left(\frac{q_{\perp}}{T_{\perp}}(\widehat{\textbf{b}}\cdot\bm{\nabla})\ln
T_{\perp}+\frac{q_{\|}}{2T_{\|}}(\widehat{\textbf{b}}\cdot\bm{\nabla})\ln
T_{\|}\right).$ (10)

From the form of the right hand side of Eqs. (6)-(7), we can conclude that the
rates of change of the parallel and perpendicular entropies per unit mass
($s^{p}_{\|}$ and $s^{p}_{\perp}$ respectively) associated with a production
(or destruction) and excluding transport or exchanges between the parallel and
perpendicular directions (see e.g. Hazeltine et al. (2013)), are given by

$\displaystyle\frac{d}{dt}s_{\|}^{p}=\frac{1}{\rho
T_{\|}}q_{\perp}\bm{\nabla}\cdot\widehat{\textbf{b}}-\frac{q_{\|}}{2\rho
T_{\|}}(\widehat{\textbf{b}}\cdot\bm{\nabla})\ln T_{\|}$ (11)
$\displaystyle\frac{d}{dt}s_{\perp}^{p}=-\frac{1}{\rho
T_{\perp}}q_{\perp}\bm{\nabla}\cdot\widehat{\textbf{b}}-\frac{q_{\perp}}{\rho
T_{\perp}}(\widehat{\textbf{b}}\cdot\bm{\nabla})\ln T_{\perp}.$ (12)

The associated rates of heat production per unit mass are related by
$dQ_{\|}/dt=T_{\|}ds_{\|}^{p}/dt$ and
$dQ_{\perp}/dt=T_{\perp}ds_{\perp}^{p}/dt$. We thus get, for the total heat
production $Q=Q_{\|}+Q_{\perp}$

$\partial_{t}(\rho Q)+\bm{\nabla}\cdot\left(\rho
Q\bm{u}\right)=-\frac{q_{\|}}{2}(\widehat{\textbf{b}}\cdot\bm{\nabla})\ln
T_{\|}-q_{\perp}(\widehat{\textbf{b}}\cdot\bm{\nabla})\ln T_{\perp}.$ (13)

The global heating is thus given by

$H=-\int\left(\frac{q_{\|}}{2}(\widehat{\textbf{b}}\cdot\bm{\nabla})\ln
T_{\|}+q_{\perp}(\widehat{\textbf{b}}\cdot\bm{\nabla})\ln
T_{\perp}\right)d^{3}x,$ (14)

where $q_{\|}$ and $q_{\perp}$ are the heat fluxes obtained from the
integration of the model closed at the level of the fourth-rank moments.

We can define a spectral density for the heating rate $H$ (also referred to as
co-spectrum) in the form

$H({\bm{k}})=-\frac{1}{2}\left(\frac{1}{2}{\cal F}\\{q_{\|}\\}(-{\bm{k}}){\cal
F}\left\\{(\widehat{\textbf{b}}\cdot\bm{\nabla})\ln
T_{\|}\right\\}({\bm{k}})+{\cal F}\\{q_{\perp}\\}(-{\bm{k}}){\cal
F}\left\\{(\widehat{\textbf{b}}\cdot\bm{\nabla})\ln
T_{\perp}\right\\}({\bm{k}})+c.c.\right)$ (15)

where ${\cal F}$ denotes the Fourier transform.

A few remarks can be made here:
1\. In all the simulations we have performed, the volume integrated heat
production is observed to be positive but its pointwise value can be negative
in relatively small regions of space. This contrasts with the
(semi-)collisional regime where the heat fluxes obey Fourier laws of the form
$q=-\kappa(\widehat{\textbf{b}}\cdot\bm{\nabla})T$, making the heat production
positive everywhere in space.
2\. Inserting in Eq. (14) the quantities $q_{\perp}$ and $q_{\|}$ obtained by
the integration of the dynamical equation for the heat fluxes results in
taking into account in the heating rate contributions originating from the
heat flux present when a quasi-normal closure is implemented (i.e. where the
fourth-rank cumulants are taken equal to zero, thus making the Landau damping
disappear). In the present simulations, this contribution does not exceed
$15\%$ of the total heating rate. In order to only deal with the heat flux
originating from the Landau damping, it would be necessary to define a
conserved entropy for the quasi-normal closure and evaluate its rate of change
due to the introduction of Landau damping. This is left for future work as it
is not straightforward.
3\. More importantly, this heating rate takes into account the Landau damping
on all the waves present in the simulations, including the magnetosonic waves.
At this level, it appears difficult to separate the contributions of the KAW
and to evaluate their dissipation by Landau damping. Nevertheless, these
magnetosonic waves get dissipated at large scales, thus at small enough scales
the estimated heating rate mostly results from Landau damping of KAWs and it
becomes possible to compare it to the cascading energy. This particularity is
also the reason why Landau damping appears to be acting at all scales in all
the results presented above, even in simulations forced at large scales.

Figure 7: Spectral densities of the heating rate $D_{\rm L}(k_{\perp})$ (red)
and of the magnetic (blue) and kinetic (green) hyper-dissipation as functions
of the transverse wavenumber $k_{\perp}$ for run LF3. A straight line of slope
5.2 is supplemented, for comparison with the scale-variation of the magnetic
hyper-dissipation.

The fact that Landau damping is present at all scales in the simulation can be
seen by estimating the spectral density of total heating rate at a given
wavenumber $k_{\perp}$, $D_{\rm L}(k_{\perp})=\int
k_{\perp}H({\bm{k}})dk_{z}d\theta$, where $H({\bm{k}})$ is the sum of the
spectral densities given by equation (15) for both the ions and the electrons.
This spectral density is represented in Fig. 7 along with the densities of
hyperdissipation and hyperdiffusivity. One clearly sees that the heating rate
due to the presence of heat fluxes dominates hyper-dissipation over a broad
range of scales due to the dissipation of KAWs and magnetosonic modes, the two
becoming comparable only at the smallest scales (note that the magnetic hyper-
dissipation dominates at small scales over the kinetic one).

###  5.2 Dissipation due to Landau damping

The energy ${\mathcal{E}}_{\perp}(t)$ of the (quasi-incompressible) KAWs that
cascade towards small scales, and which is the subject of our study, obeys

$\frac{d}{dt}{\mathcal{E}}_{\perp}(t)={\mathcal{I}}^{C}(t)-{\mathcal{D}}^{C}_{\rm
L}(t)-{\mathcal{D}}^{C}_{\rm h}(t),$ (16)

where ${\mathcal{I}}^{C}$ is the part of the injection rate that contributes
to the KAW cascade (the other part is transferred to magnetosonic modes which
are dominantly dissipated at large scales), while ${\mathcal{D}}^{C}_{\rm L}$
and ${\mathcal{D}}^{C}_{\rm h}$ are the parts of the Landau and hyperviscous
(and hyperdiffusive) dissipation that affect the cascading modes. Using
cylindrical coordinates and assuming time stationarity, one can write the
integrated energy balance at each Fourier mode as (adopting roman scripts for
spectral densities):

$\epsilon(k_{\perp})=\int_{0}^{k_{\perp}}\left\\{I^{C}(k^{\prime}_{\perp})-D^{C}_{\rm
L}(k^{\prime}_{\perp})-D^{C}_{\rm
h}(k^{\prime}_{\perp})\right\\}dk^{\prime}_{\perp}.$ (17)

Considering two wavenumbers $k_{\perp 1}$ and $k_{\perp 2}$ large enough so
that the forcing (which is concentrated at large scales) leads to
$\int_{0}^{k_{\perp
1}}I^{C}(k^{\prime}_{\perp})dk^{\prime}_{\perp}=\int_{0}^{k_{\perp
2}}I^{C}(k^{\prime}_{\perp})dk^{\prime}_{\perp}={\mathcal{I}}^{C}$, yet small
enough for hyperviscous dissipation to be negligible, one obtains:

$\epsilon(k_{\perp 1})-\epsilon(k_{\perp 2})=\int_{k_{\perp 1}}^{k_{\perp
2}}D^{C}_{\rm L}(k^{\prime}_{\perp})dk^{\prime}_{\perp}\lesssim\int_{k_{\perp
1}}^{k_{\perp 2}}D_{\rm L}(k^{\prime}_{\perp})dk^{\prime}_{\perp}.$ (18)

The inequality draws closer to an equality for values of $k_{\perp}$ large
enough so that all magnetosonic modes have been dissipated.

Equation (18) can be used to estimate a correction to the energy cascade rate
which would take into account the energy lost due to Landau damping. We do so
for run LF3: using this equation we add to the transfer rate the cumulative
Landau dissipation between an arbitrary scale (chosen however to be not too
large nor too small) and the running (smaller) scale $l_{\perp}$. Two of these
resulting corrected rates $\varepsilon^{corr}$ are shown in Fig. 8. They
appear to be almost constant, and as such they behave very similarly to the
transfer rate of run CGL3 (Fig. 5). The slight increase of
$\varepsilon^{corr}$ towards small scales probably reflects the (weak)
contribution of some remaining magnetosonic waves to the calculated Landau
damping. This clearly demonstrates that the energy lost along the cascade due
to Landau damping is well captured by the decline of the (fluid) cascade rate
at the corresponding scales.

Figure 8: Energy cascade rate $\varepsilon$ (red) and transverse
hyperdissipation (violet) for run LF3. The orange and brown curves show the
same $\varepsilon$ corrected by Landau damping integrated between
$\ell_{\perp}$ and a reference scale $\ell_{\perp}=0.1\pi d_{i}$ and
$\ell_{\perp}=0.2\pi d_{i}$ respectively.

A complementary estimate of energy dissipation can be done in Fourier space by
also taking into account hyperdissipation. Indeed, assuming stationarity, one
can also derive that

$\epsilon(k_{\perp})={\mathcal{I}}^{C}-{\mathcal{D}}^{C}_{\rm
L}+\int_{k_{\perp}}^{\infty}D^{C}_{\rm
L}(k^{\prime}_{\perp})dk^{\prime}_{\perp}-\int_{0}^{k_{\perp}}D^{C}_{\rm
h}(k^{\prime}_{\perp})dk^{\prime}_{\perp}=\int_{k_{\perp}}^{\infty}\left\\{D^{\rm
C}_{\rm h}(k^{\prime}_{\perp})+D^{C}_{\rm
L}(k^{\prime}_{\perp})\right\\}dk^{\prime}_{\perp}.$ (19)

Equation (19) indicates that, as expected, the rate of energy transfer at the
wavenumber $k_{\perp}$ identifies with the sum of the rates of Landau and
hyperdissipation beyond this wavenumber. One can compare the second right-
hand-side term of this equation to the energy cascade rate
$\epsilon(k_{\perp})$ obtained from the IHMHD exact law, as displayed in Fig.
9. The difference between the two curves, which is especially significant at
large scales, is due to the fact that the estimation of the dissipation
includes the Landau damping of magnetosonic modes, whereas the cascade rate
considers only incompressible modes. At smaller scales however, where
magnetosonic modes have already been dissipated, the dissipation and cascade
rates decreases parallel to each other: this indicates that, at scales not yet
affected by hyperdissipation, the decay of $\epsilon(k_{\perp})$ in a spectral
interval identifies with Landau dissipation within this interval.

Figure 9: Energy cascade rate $\epsilon(k_{\perp})$ (black line) together with
Landau and hyper-dissipation (red line) computed with equation (19) for run
LF3.

Figs. 8 and 9 clearly demonstrates that, through the cascade, the energy lost
due to Landau damping is well captured by the decline of the (fluid) cascade
rate at the corresponding scales. Note that a similar decline of the fluid
cascade rate at kinetic scales was reported in 2D hybrid PIC simulations and
spacecraft observations in the SW and magnetosheath (Hellinger et al., 2018;
Bandyopadhyay et al., 2020). Also, Sorriso-Valvo et al. (2019) found a
correlation between enhancement of a proxy of the local cascade rate and the
signatures of wave-particle interactions in MMS data.

##  6 Conclusion

In this study, we tackle a fundamental question about the ability of fluid
exact laws to reflect the presence of kinetic (Landau) damping. By
constructing multi-scale energy cascade and dissipation rates using the HMHD
model on a variety of turbulence simulations bearing different intensities of
Landau damping, we showed that the presence of Landau damping at small
(kinetic) scales is reflected by the steady decline of the energy cascade rate
at the same scales, which was found to be comparable to the effective Landau
dissipation at those scales. By demonstrating the ability of a fluid exact law
to provide a correct estimate of kinetic dissipation in the sub-ion range of
numerical simulations, this work provides a means to evaluate the amount of
energy that is dissipated into particle heating in spacecraft data: the
decline of the cascade rate allows one to evaluate the kinetic dissipation as
a function of scale. This should help investigating (at least partially) a
longstanding problem in astrophysical plasmas about energy partition between
ions and electrons (Kawazura et al., 2019), which are generally heated at
different scales.

The study presented in this paper only makes use of Landau damping. It would
be interesting in future works to extend these conclusions to a broad variety
of kinetic effects and to test them on more general simulations of the SW,
featuring a plasma turbulence driven by other types of waves than slightly
perturbed KAWs. It is also important to stress that, even if the
oversimplified (yet fully nonlinear) fluid models of turbulence can provide
good estimates of the amount of energy that is dissipated into particle
heating, they do not specify how this dissipation occurs. The answer to this
question and those related to the fate of energy when handed to the plasma
particles requires a kinetic treatment.

##  7 Acknowledgements

This work was granted access to the HPC resources of CINES/IDRIS under the
allocation A0060407042. Part of the computations have also been done on the
“Mesocentre SIGAMM” machine, hosted by Observatoire de la Côte d’Azur.

## References

  * Andrés et al. (2018) Andrés, N., Galtier, S., & Sahraoui, F. 2018, Phys. Rev. E, 97, 013204
  * Andrés & Sahraoui (2017) Andrés, N., & Sahraoui, F. 2017, Physical Review E, 96, 053205
  * Andrés et al. (2019) Andrés, N., Sahraoui, F., Galtier, S., et al. 2019, Phys. Rev. Lett., 123, 245101
  * Antonia et al. (1997) Antonia, R., Ould-Rouis, M., Anselmet, F., & Zhu, Y. 1997, J. Fluid Mech., 332, 395
  * Balbus & Hawley (1998) Balbus, S. A., & Hawley, J. F. 1998, Reviews of Modern Physics, 70
  * Bandyopadhyay et al. (2020) Bandyopadhyay, R., Sorriso-Valvo, L., Chasapis, A., et al. 2020, Phys. Rev. Lett., 124, 225101
  * Banerjee & Galtier (2013) Banerjee, S., & Galtier, S. 2013, Phys. Rev. E, 87, 013019
  * Banerjee et al. (2016) Banerjee, S., Hadid, L., Sahraoui, F., & Galtier, S. 2016, Astrophys. J. Lett., 829, L27
  * Bruno & Carbone (2005) Bruno, R., & Carbone, V. 2005, Living Rev. Solar Phys., 2, 4
  * Carbone et al. (2009) Carbone, V., Marino, R., Sorriso-Valvo, L., Noullez, A., & Bruno, R. 2009, Phys. Rev. Lett., 103, 061102
  * Chen et al. (2019) Chen, C. H. K., Klein, K. G., & Howes, G. G. 2019, Nat. Commun., 10, 740
  * Chew et al. (1956) Chew, G. F., Goldberger, M. L., & Low, F. E. 1956, Proc. R. Soc. London. A, 236, 112
  * Coburn et al. (2015) Coburn, J., Forman, M., Smith, C., Vasquez, B., & Stawarz, J. 2015, Phil. Trans. R. Soc. A, 373, 20140150
  * Diamond et al. (2005) Diamond, P. H., Itoh, S. I., & Hahm, T. S. 2005, Plasma Phys. Control. Fusion, 47, R35
  * Ferrand et al. (2019) Ferrand, R., Galtier, S., Sahraoui, F., et al. 2019, Astrophys. J., 881, 50
  * Fujisawa (2021) Fujisawa, A. 2021, Proceedings of the Japan Academy, Series B, 97, 103
  * Galtier (2008) Galtier, S. 2008, Phys. Rev. E, 77, 015302
  * Garbet (2006) Garbet, X. 2006, Comptes Rendus Physique, 7, 573, turbulent transport in fusion magnetised plasmas
  * Goldstein et al. (1995) Goldstein, M. L., Roberts, D. A., & Matthaeus, W. 1995, Annu. Rev. Astron. Astrophys., 33, 283
  * Hadid et al. (2017) Hadid, L., Sahraoui, F., & Galtier, S. 2017, Astrophys. J., 838, 11
  * Hadid et al. (2018) Hadid, L. Z., Sahraoui, F., Galtier, S., & Huang, S. Y. 2018, Phys. Rev. Lett., 120, 055102
  * Hammett & Perkins (1990) Hammett, G. W., & Perkins, F. W. 1990, Phys. Rev. Lett., 64, 3019
  * Hazeltine et al. (2013) Hazeltine, R. D., Mahajan, S. M., & Morrison, P. J. 2013, Phys. Plasmas, 20, 022506
  * He et al. (2015) He, J., Wang, L., Tu, C., Marsch, E., & Zong, Q. 2015, Astrophys. J. Lett., 800, L31
  * Hellinger et al. (2018) Hellinger, P., Verdini, A., Landi, S., Franci, L., & Matteini, L. 2018, Astrophys. J., 857, L19
  * Hunana et al. (2019) Hunana, P., Tenerani, A., Zank, G. P., et al. 2019, Journal of Plasma Physics, 85, 205850603
  * Kawazura et al. (2019) Kawazura, Y., Barnes, M., & Schekochihin, A. 2019, PNAS, 116, 771
  * Kobayashi et al. (2017) Kobayashi, S., Sahraoui, F., Passot, T., et al. 2017, Astrophys. J., 839, 122
  * Kolmogorov (1941) Kolmogorov, A. 1941, Dokl. Akad. Nauk SSSR, 32, 16
  * Leamon et al. (1998) Leamon, R. J., Smith, C. W., Ness, N. F., Matthaeus, W. H., & Wong, H. K. 1998, J. Geophys. Res., 103, 4775
  * MacBride et al. (2008) MacBride, B., Smith, C., & Forman, M. 2008, Astrophys. J., 679, 1644
  * Marino et al. (2008) Marino, R., Sorriso-Valvo, L., Carbone, V., et al. 2008, Astrophys. J., 677, L71
  * Matthaeus & Velli (2011) Matthaeus, W. H., & Velli, M. 2011, Space Sci. Rev., 160, 145
  * Matthaeus et al. (1999) Matthaeus, W. H., Zank, G. P., Smith, C. W., & Oughton, S. 1999, Phys. Rev. Lett., 82, 3444
  * Monin (1959) Monin, A. S. 1959, Dokl. Akad. Nauk SSSR, 125, 515
  * Osman et al. (2011) Osman, K. T., Wan, M., Matthaeus, W. H., Weygand, J. M., & Dasso, S. 2011, Phys. Rev. Lett., 107, 165001
  * Passot et al. (2014) Passot, T., Henri, P., Laveder, D., & Sulem, P.-L. 2014, The European Physical Journal D, 68, doi:10.1140/epjd/e2014-50160-1
  * Passot & Sulem (2007) Passot, T., & Sulem, P. L. 2007, Phys. Plasmas, 14, 082502
  * Podesta et al. (2007) Podesta, J. J., Forman, M. A., & Smith, C. W. 2007, Phys. Plasmas, 14, 092305
  * Politano & Pouquet (1998) Politano, H., & Pouquet, A. 1998, Phys. Rev. E, 57, 21
  * Quataert & Gruzinov (1999) Quataert, E., & Gruzinov, A. 1999, Astrophys. J., 520, 248
  * Richardson et al. (1995) Richardson, J. D., Paularena, K. I., Lazarus, A. J., & Belcher, J. W. 1995, Geophys. Res. Lett., 22, 325
  * Sahraoui et al. (2012) Sahraoui, F., Belmont, G., & Goldstein, M. L. 2012, Astrophys. J., 748, 100
  * Sahraoui et al. (2010) Sahraoui, F., Goldstein, M. L., Belmont, G., Canu, P., & Rezeau, L. 2010, Phys. Rev. Lett., 105, 131101
  * Sahraoui et al. (2009) Sahraoui, F., Goldstein, M. L., Robert, P., & Khotyaintsev, Y. V. 2009, Phys. Rev. Lett., 102, 231102
  * Sahraoui et al. (2020) Sahraoui, F., Hadid, L., & Huang, S. 2020, Rev. Mod. Plasmas Phys., 4, 4
  * Schekochihin et al. (2009) Schekochihin, A., Cowley, S. C., Dorland, W., et al. 2009, Astrophys. J. Supp., 182, 310
  * Smith et al. (2006) Smith, C. W., Hamilton, K., Vasquez, B. J., & Leamon, R. J. 2006, Astrophys. J. Lett., 645, L85
  * Smith et al. (2009) Smith, C. W., Stawarz, J. E., Vasquez, B. J., Forman, M. A., & MacBride, B. T. 2009, Phys. Rev. Lett., 103, 201101
  * Snyder et al. (1997) Snyder, P. B., Hammett, G. W., & Dorland, W. 1997, Phys. Plasmas, 4, 3974
  * Sorriso-Valvo et al. (2007) Sorriso-Valvo, L., Marino, R., Carbone, V., et al. 2007, Phys. Rev. Lett., 99, 115001
  * Sorriso-Valvo et al. (2019) Sorriso-Valvo, L., Catapano, F., Retinò, A., et al. 2019, Phys. Rev. Lett., 122, 035102
  * Stawarz et al. (2009) Stawarz, J. E., Smith, C. W., Vasquez, B. J., Forman, M. A., & MacBride, B. T. 2009, Astrophys. J., 697, 1119
  * Taylor et al. (2003) Taylor, M. A., Kurien, S., & Eyink, G. L. 2003, Phys. Rev. E, 68, 026310
  * TenBarge et al. (2014) TenBarge, J., Howes, G., Dorland, W., & Hammett, G. 2014, Comp. Phys. Comm., 185, 578


# DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme
Long Sequence Transformer Models

Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang Shuaiwen Leon
Song, Samyam Rajbhandari, Yuxiong He
Microsoft Inc
<EMAIL_ADDRESS>
###### Abstract

Computation in a typical Transformer-based large language model (LLM) can be
characterized by batch size, hidden dimension, number of layers, and sequence
length. Until now, system works for accelerating LLM training have focused on
the first three dimensions: data parallelism for batch size, tensor
parallelism for hidden size and pipeline parallelism for model depth or
layers. These widely studied forms of parallelism are not targeted or
optimized for long sequence Transformer models. Given practical application
needs for long sequence LLM, renewed attentions are being drawn to sequence
parallelism. However, existing works in sequence parallelism are constrained
by memory-communication inefficiency, limiting their scalability to long
sequence large models. In this work, we introduce DeepSpeed-Ulysses, a novel,
portable and effective methodology for enabling highly efficient and scalable
LLM training with extremely long sequence length. DeepSpeed-Ulysses at its
core partitions input data along the sequence dimension and employs an
efficient all-to-all collective communication for attention computation.
Theoretical communication analysis shows that whereas other methods incur
communication overhead as sequence length increases, DeepSpeed-Ulysses
maintains constant communication volume when sequence length and compute
devices are increased proportionally. Furthermore, experimental evaluations
show that DeepSpeed-Ulysses trains 2.5x faster with 4x longer sequence length
than the existing method SOTA baseline.

##  1 Introduction

Training large models with long sequences is becoming very important across
the board from generative AI to models for scientific discovery. On generative
AI side, conversational AI, knowledge-rich long document summarization and
video generation require reasoning over long contexts in spatial and temporal
domains. For example, multimodal foundation models such as ones that process
speech, images and waveforms concurrently require long context reasoning over
high dimensional inputs with long sequences. Similarly, chapter and book level
summarization (estimated at tens and hundreds of thousands of words) are of
great importance in conversational AI and abstractive summarization tasks
(Beltagy et al., 2020; Kryściński et al., 2022; MosaicML, 2023) and have shown
to benefit from long sequence training (Xiong et al., 2023; Peng et al., 2023;
Touvron et al., 2023). The debut of ChatGPT (and subsequent similar open
source and "product" LLM brands) has pushed chat application to the forefront
of modern AI, making chat applications to be more relevant than ever before.
Processing long sequence is crucial for supporting longer histories in chat
applications (Touvron et al., 2023).

Long sequence length is equally critical for AI for science opening doors for
better understanding of structure biology, health care, climate and weather
forecasting (Nguyen et al., 2023) and large molecular simulation (Zvyagin et
al., 2022). For instance, by adapting large language models with gene
sequences, we can create language models that can learn the evolutionary
patterns of genomes using simple alphabets and extremely long sequences (the
human genome has 6.4 billion letters) (Zvyagin et al., 2022). In health care,
diagnostic predictive model conditioned on entire patient care record requires
context of long sequences (Li et al., 2022a; Gao et al., 2021).

Despite the emerging importance of long sequence length for both generative AI
and AI for science, existing large model training systems and the underlying
parallelism technologies (data, tensor, pipeline, sequence parallelism) are
limited in their ability to support the efficient long sequence training. Two
challenges with existing parallelism approach come to the fore. First,
existing parallelism approach such as data, tensor and pipeline parallelism
cannot address the scaling along sequence dimension. Second, existing sequence
parallelism approaches are not effective because of memory-communication
inefficiencies. Furthermore, existing approaches have limited usability
requiring intrusive and error prone code refactoring.

In this paper, we introduce DeepSpeed-Ulysses (or Ulysses, a very long novel),
a simple, portable, and effective methodology for enabling highly efficient
and scalable LLM training with extremely long sequence lengths. DeepSpeed-
Ulysses partitions individual samples along the sequence dimension among
participating GPUs. Then right before the attention computation, it employs
all-to-all communication collective on the partitioned queries, keys and
values such that each GPU receives the full sequence but only for a non-
overlapping subset of the attention heads. This allows the participating GPUs
to compute attention for different attention heads in parallel. Finally,
DeepSpeed-Ulysses employs another all-to-all to gather the results along the
attention heads while re-partitioning along the sequence dimension.

In this work, we put forward the following contributions of DeepSpeed-Ulysses
to advance state of the art in long sequence parallelism:

  * •

DeepSpeed-Ulysses trains Transformer models 4x larger sequence lengths than
existing systems, while enabling training with sequences with over a million
tokens.

  * •

Communication reduction of over 10x compared to existing systems, resulting in
throughput improvements of up to 2.5x, and sustained throughput of over 175
TFlops/GPU (over 54% of hardware peak).

  * •

Fully general and implementation agnostic attention: DeepSpeed sequence
parallelism (Ulysses) supports dense as well as sparse attention, and it works
with efficient attention implementations such as FlashAttention v2 (Dao,
2023).

  * •

Support for massive model training: DeepSpeed sequence parallelism works
together with ZeRO-3 to not only support large sequence lengths but also
massive model sizes.

  * •

Easy-to-use and portable, requiring minimal code changes to the existing
training frameworks.

In subsequent sections, we provide background and related work, a detailed
discussion of DeepSpeed sequence parallelism core design, communication
complexity analysis, experimental evaluation and comparison with existing
work.

##  2 Background and Related Work

In this section, we present a brief overview of Transformer architecture, mode
of parallelism to accelerate Transformer training and a discussion on closely
related work to our approach

###  2.1 Background

This section briefly introduces Transformer architecture and highlights
different mode of parallelism of deep neural network in general and
Transformer model in particular. This brief discussion is followed by specific
focus on closely related work.

####  2.1.1 Transformer Architecture

Figure 1: Multi-head attention Transformer

Shown in Figure 1 is a sketch of building blocks of a typical multihead
attention Transformer architecture (Vaswani et al., 2017). It consists of
input sequences which are projected into queries (_Q_),keys (_K_) and values
(_V_) embeddings. _QKV_ are typically a 3D tensor of size $N,b,d$ where $N$ is
sequence length, $b$ is micro batch size and $d$ is hidden dimension. The
$QKV$ tensors are fed to the attention block, a central component of
Transformer model. Outputs of attentions are inputs to the multilayer
perceptron (MLP) or position-wise feed-forward block of Transformer
architecture.

The attention block followed by MLP block are replicated multiple times to
form an encoder, a decoder or an encoder-decoder Transformer network.

####  2.1.2 Mode of Parallelism

Data parallelism (Dean et al., 2012) is de facto method of accelerating neural
network training and has been applied widely with different neural network
architectures and applications. Data parallelism in its simplest form
partitions input data across sample or batch dimension while replicating model
parameters across compute devices. Data parallelism is effective when the
batch size is sufficiently large to hide communication cost in compute.
However, it is limited when model is large and model parameter replication
across devices is practically infeasible. ZeRO (Rajbhandari et al., 2020,
2021) optimization addresses this problem by partitioning model parameters
across available compute devices. Moreso, large batch is known to have impacts
on model quality (Keskar et al., 2016).

It is worth to note that our proposed approach is orthogonal to both data
parallelism and ZeRO. Our proposed approach can be used with both methods.
Also, by leveraging sequence parallelism to keep global batch size at
reasonable size on large systems, we effectively ameliorate the impact of
large batch size on model convergence. Sequence parallelism serves two
purposes in this regard. First, sequence parallelism can accelerate time to
solution for same (already explored) long sequence length; in other words,
sequence parallelism reduces the iteration time proportional to additional
compute resources. Second, sequence parallelism enables longer sequence
training or continual pretraining where training context length gradually
increase over time (Xiong et al., 2023). Consider a real world scenario of
large scale training on 1024 GPUs. The initial exploratory or pretraining set
up of a (proxy) LLM has a sequence length of 8192 (8K), a micro batch size of
1 (thus, 8 million token global size) per GPU. A simple change to improve the
quality of the pretrained model requires a change of sequence length from 8K
to 32K, which would result in approximately 32 million global batch size.
However, increasing the global batch size is not an option due to the negative
impact on model quality. Therefore, sequence parallelism comes in handy as a
system optimization technique with no requirement for laborious hyperparameter
search. In this scenario, sequence parallelism allows for large batch sizes to
be split across multiple GPUs without increasing the global batch size,
regardless of the sequence length.

Tensor (Shoeybi et al., 2019) and pipeline parallelism (Narayanan et al.,
2019; Huang et al., 2018; Narayanan et al., 2021) are two other popular
methods for large scale training. Collectively, tensor and pipeline
parallelism are called model parallelism, and are targeted at compute
operators in large models. In contrast to data parallelism, model parallelism
are used when models are too large (as it is in many LLMs) and can not be
fully replicated across data parallel ranks. Tensor parallelism splits compute
operators (i.e., attention and MLPs) within a layer and pipeline parallelism
splits model in a depth-wise (layer-wise) fashion. 3D parallelism (Team and
Majumder, 2020; Smith et al., 2022) combines data parallelism, tensor
parallelism and pipeline parallelism to achieve higher throughput in
comparison to the 3 constituents components at a cost of extensive code
rewrite and productivity overhead (Wang et al., 2023).

###  2.2 Related Work

For a broad overview and survey of distributed training methods for deep
neural networks please see (Ben-Nun and Hoefler, 2019). These methods are
broadly categorized into data and model parallelism as described above.
However, all of existing parallel methods are limited in dealing with
intermediate activation memory overhead associated with extremely long
sequence.

While recent works in sequence parallelism address the memory overhead, they
are lacking in communication efficiency, thus limited in scaling capability.
Similar to our work, all existing works in sequence parallelism partition the
input data along sequence dimension but differ in what input projections are
partitioned and how partitions are aggregated and communicated for attention
computation.

The authors in (Li et al., 2022b) (henceforward called _ColAI-SP_) introduce
ring self attention, a ring-like communication collective in which query
projections are local whereas key and values projections are transmitted in a
ring-style to compute global attention, resulting in communication complexity
linear in message size, $M$. Megatron-LM sequence parallelism (Korthikanti et
al., 2022) approach is tightly integrated with Megatron tensor parallelism.
Megatron LM partitions sequence along sequence dimensions and applies
allgather and reduce scatter collective to aggregate _QKV_ projections for
attention computation. Communication complexity analysis shows that unlike our
approach, Megatron-LM sequence parallelism communication volume increase
linearly with message size ($M$) regardless of number of compute devices.
DeepSpeed-Ulysses on the other hand keeps communication volume consistent by
increasing GPUs proportional to message size or sequence length see 3.2 for
more details.

Table 1 summarizes how DeepSpeed-Ulysses differs from other existing methods.
DeepSpeed-Ulysses has communication efficiency advantage over the other two
methods. It also benefits from leveraging ZeRO (Rajbhandari et al., 2020,
2021) optimization for model parameter partitioning across both sequence and
data parallel groups. DeepSpeed-Ulysses supports different kinds of attention
and it is easy to use. Megatron-LM sequence parallelism is tightly integrated
with Megatron-LM tensor parallelism limiting both its memory efficiency and
easy of use. _ColAI-SP_ requires a different (specific) kind of attention and
is not easy to use. It is not clear how well _ColAI-SP_ ring self-attention
generalizes to other attention types and mechanisms.

Method | Comm | Activation | Parameter | Attention | Ease
---|---|---|---|---|---
complexity | memory efficiency | memory efficiency | agnostic | of use
ColAI-SP (Li et al., 2022b) | $O(M)$ |  TextRenderingMode=FillStroke, LineWidth=.75pt, ✓ | x | x | x
Megatron-SP (Korthikanti et al., 2022) | $O(M)$ |  TextRenderingMode=FillStroke, LineWidth=.75pt, ✓ | x |  TextRenderingMode=FillStroke, LineWidth=.75pt, ✓ | x
DS-Ulysses | $O(M/P)$ |  TextRenderingMode=FillStroke, LineWidth=.75pt, ✓ |  TextRenderingMode=FillStroke, LineWidth=.75pt, ✓ |  TextRenderingMode=FillStroke, LineWidth=.75pt, ✓ |  TextRenderingMode=FillStroke, LineWidth=.75pt, ✓
Table 1: Comparison of our work (DS-Ulysses) to other sequence parallelism
methods.

There are related works in sparse Transformer particularly focusing on full-
attention approximation such as sparse attention (Child et al., 2019;
Choromanski et al., 2020; Zaheer et al., 2021; Beltagy et al., 2020). There
are also recent works on single GPU memory and compute efficient attention. A
popular example in this category is Flash attention (Dao et al., 2022; Dao,
2023), which leverages known techniques such as tiling and recomputation for
compute and memory efficiency. These works are orthogonal to our work and were
leveraged accordingly.

##  3 DeepSpeed-Ulysses Core Design

###  3.1 System Design

Figure 2: DeepSpeed sequence parallelism (DeepSpeed-Ulysses) design

Figure 2 shows the core design of DeepSpeed-Ulysses. As with the known
transformer architecture, the design consists of input sequences _N_
partitioned across _P_ available devices. Each local _N/P_ partition is
projected into queries (_Q_), keys (_K_) and values (_V_) embeddings. Next,
(_QKV_) embeddings are gathered into global _QKV_ through highly optimized
all-to-all collectives between participating compute devices. Sequel to all-
to-all collective is the attention computation per head in the form:

$Outputcontext=Softmax((QK^{T})/\sqrt{(}d))V$ (1)

After the attention computation, another all-to-all collective transforms
output context tensor of attention computation to sequence (_N/P_) parallel
for subsequent operators (MLP MatMul, layer norm etc) in the remaining modules
of transformer layer block.

###  3.2 Communication Analysis

What distinguishes DeepSpeed-Ulysses from the other existing long-sequence
approaches is our much smaller aggregate communication volume and overall
better scalability with increasing degree of sequence parallelism compared to
existing solutions, as demonstrated by the communication volume analysis
below:

On modern clusters with intra-node NVSwitch interconnect and inter-node fat
tree IB topology, the communication volume transmitted per link for an all-to-
all for aggregate message of size _M_ over _P_ GPUs is _M/P_. For a
transformer model with hidden size h, sequence length of N, and parallelism
degree of P, DS-Sequence performs all-to-all for the _QKV_ projections with an
aggregate message size of _3Nh_ before the attention computation, and another
all-to-all for output context projection with a size _Nh_ for each transformer
layer. Therefore, DeepSpeed sequence parallelism incurs an aggregate
communication volume per link of _4Nh/P_ (or with the complexity of _O(N/P)_.
Note that this communication volume is constant when both _N_ and _P_ are
increased proportionally.

In contrast, the existing approaches like Megatron-LM incur communication
volume that increases linearly with N regardless of P, resulting in the
communication complexity of _O(N)_. For instance, Megatron-LM performs two
all-gather with the message volume of _Nh_ and two reduce-scatter with the
volume of _Nh_ for each transformer layer. However, the cost of each all-
gather and reduce-scatter of size _M_ remains _M_ when _P >> 1_, instead of
_M/P_. Therefore, Megatron-LM sequence parallelism incurs a communication
volume per link of _4Nh_ which is _P_ times larger than that for DeepSpeed
sequence parallelism. This allows DeepSpeed sequence parallelism to enable
training with extremely long sequences while achieving significantly higher
training efficiency compared to the existing approaches. Our evaluation
results match this analysis.

###  3.3 Memory Efficiency

While DeepSpeed sequence parallelism reduces the activation memory when
training with longer sequences, it does not impact the memory consumed by the
model states. Therefore, to support large sequence length training with a
large language model, DeepSpeed sequence parallelism is integrated with
ZeRO-3. ZeRO Redundancy Optimizer Stage 3 (ZeRO-3) (Rajbhandari et al., 2020,
2021) is a memory optimization technique for training large models. Unlike the
classic data parallel training of neural networks where model states are
replicated across data parallel ranks, ZeRO-3 optimizes memory usage by
partitioning model states across data parallel ranks. However, with sequence
parallelism, training data can be considered in both batch (sample) and
sequence dimensions and the associated parallel groups combined to form a
larger group for ZeRO parallelism. Therefore, we extend ZeRO-3 partitioning to
combination of data parallel and sequence parallel ranks. In other words, in
DeepSpeed sequence parallelism, ZeRO partitions model states across both
sequence and data parallel group and collects per rank partitions (allgather)
when they are needed. Similarly, gradients are reduced across both data and
sequence parallel ranks for parameter update. ZeRO support allows for huge
memory savings in both sequence and data dimensions and enables scaling not
just to large sequence lengths but also to large models.

###  3.4 General and Attention Agnostic Solution

DeepSpeed implementation of distributed attention module is general enough to
support any attention: e.g., self-attention, cross-attention, causal attention
in both their dense and sparse counterparts, and their various optimized
kernels that support long-sequence at local attention level such as different
versions of FlashAttention. The general property of DeepSpeed-Ulysses stems
from the modular nature of its core design: an attention-centric sequence
parallelism design. Prior to attention computation is sequence parallelism of
N/P partition, attention computation is head parallelism with full attention
per head but just with fewer heads, thus attention computation can be replaced
with any type of attention mechanisms, e.g., dense attention and various forms
of sparse attention.

##  4 Evaluation

We evaluate DeepSpeed-Ulysses (DeepSpeed Sequence) on GPT (Radford et al.,
2019), a foundation model for many NLP tasks on up to 256 A100 GPUs. Our
evaluations are five-fold: i) sequence length scalability, ii) throughput for
dense attention and comparison with existing system, and iii) throughput with
sparse attention and comparison with existing system, iv) parallel scaling
study and v) convergence study of Deep sequence parallelism. We discuss and
present evaluations from each of these categories next.

###  4.1 Sequence Length Scalability

The first set of experiments is strong scaling of sequence length up to 1
million tokens on 1.2 billion parameter GPT model. Results of this evaluation
are shown in Figure 3. DeepSpeed sequence parallelism allows increasing
sequence length linearly with the number of GPUs and sequence length scales
linearly relative to and maintains similar computation throughput across
different sequence length at appropriate GPU count.

Figure 3: DeepSpeed sequence parallelism strong scalability evaluation at
different sequence length and GPU counts

###  4.2 Dense Attention Evaluation

Next, we evaluate DeepSpeed sequence parallelism on 7 billion (7B) and 30
billion (30B) parameter GPT dense attention models and compare against
Megatron-LM’s sequence parallelism on 32 and 64 A100 GPUs respectively. The
results of these evaluations are shown in Figures 4 and 5.

We compare DeepSpeed sequence parallelism with Megatron-LM for 7B and 30B
models running various sequence lengths. For our evaluation we chose the
sequence parallelism degree and micro-batch size that produced the best
performance (measured as throughput or TFLOPs) for both DeepSpeed sequence
parallelism and Megatron-LM, this we call optimal (batch size-sequence length)
configurations. For DeepSpeed sequence parallelism, we always use a ZeRO
parallelism degrees of 32 and 64 for 7B and 30B models respectively.

Figures 4 and 5 show that DeepSpeed sequence parallelism consistently
outperforms Megatron-LM for the sequence length that can be run with both. In
addition, DeepSpeed sequence parallelism can run longer sequence than
Megatron-LM. DeepSpeed sequence parallelism performance advantages are two
folds: (1) DeepSpeed sequence parallelism in combination with ZeRO-3 fits more
samples than Megatron-LM because of the memory optimization leading to higher
throughput (2) DeepSpeed sequence parallelism benefits from efficient all-to-
all communication relative to all-gather communication as applied in Megatron-
LM sequence parallelism.

Figure 4: Evaluation of DeepSpeed-Ulysses and Megatron LM on 7B parameter
model with dense attention (32 GPUs) Figure 5: Evaluation of DeepSpeed-Ulysses
and Megatron LM on 30B parameter model with dense attention (64 GPUs)

###  4.3 Sparse Attention Evaluation

Similarly, we evaluate DeepSpeed sequence parallelism on 7 billion and 30
billion parameter sparse attention models and benchmark against Megatron-LM
sequence parallelism. Results of our evaluation are shown in Figures 6 and 7.
We observe similar trends with sparse attention as dense attention
experiments. We observe more than 2x throughput performance of DeepSpeed
sequence parallelism compared to Megatron-LM. For memory saving, DeepSpeed
sequence parallelism leveraging ZeRO-3 scales to 4x longer sequence lengths
than Megatron-LM.

DeepSpeed sequence parallelism outperforms Megatron-LM for sequence length
that can be run with both. In fact, the current DeepSpeed throughput is
bottlenecked by the local sparse attention implementation, and as a result
DeepSpeed throughput decreases as the sequence length increases. We expect
this gap in performance between DeepSpeed and Megatron-LM to increase further
for larger sequence lengths as we improve the performance of the local sparse
attention implementation in future.

Figure 6: Evaluation of DeepSpeed-Ulysses and Megatron LM on 7B parameter
model with blocked sparse attention (32 GPUs) Figure 7: Evaluation of
DeepSpeed-Ulysses and Megatron LM on 30B parameter model with blocked sparse
attention (64 GPUs)

###  4.4 Parallel Scaling Study

Table 2: Parallel scaling study with fixed sequence length Seqlen | GPUs | Time (ms) | TFLOPs
---|---|---|---
131072 | 64 | 32432.1333 | 165.526667
131072 | 128 | 17052.5143 | 157.41
131072 | 256 | 9886.7 | 136.09
Table 3: Parallel scaling study with varying sequence length Seqlen | GPUs | Time (ms) | TFLOPs
---|---|---|---
65536 | 64 | 9676.76 | 161.3626667
131072 | 128 | 17052.5143 | 157.41
262144 | 256 | 33486.5 | 147.4

Furthermore, we conduct parallel scaling studies of DeepSpeed-Ulysses along
two axes. First, we fix sequence length at 131,072 tokens and increase GPU
count from 64 to 256. Second, we increase the GPU count proportionally to the
increase in sequence length. The results of these experiments are shown in
Tables 2 and 3 respectively. For both evaluations, we used GPT-7B dense model
at global batch size of 8. The tables show iteration time in microseconds as
well as the achieved throughput measured in per GPU TFLOPs. Table 2 can be
interpreted as strong scaling and shows that execution time decreases almost
linearly as we increase the GPU count. Table 3 on the other hand, is a form of
weak scaling (not in the traditional sense) with caveat that attention
computation, a function of sequence length, is quadratic in complexity. In
other words, as we increase sequence length, the work increases quadratically.

Communication overhead can be attributed to slight decrease in throughput as
we increase communication workload (that is, sequence length or GPU count).
This overhead notwithstanding, we observe good scaling at high percentages of
theoretical peak GPU performance across the two studies. These good scaling
results indicate good parallel efficiency of DeepSpeed-Ulysses.

###  4.5 Convergence Study

Lastly, Figure 8 shows convergence of a 1.3 billion GPT model at 32K sequence
length on 8 A100 GPUs with sequence parallelism degree set at 4 for both
DeepSpeed-Ulysses and Megatron-LM sequence parallelism. For DeepSpeed sequence
parallelism, we evaluate convergence with different ZeRO stages. DeepSpeed
sequence parallelism is a purely system optimization technique that enables
training of long sequence Transformer model, thus there is no (negative) on
quality of trained models, this assertion is validated through experiments and
is shown in Figure 8.

Figure 8: Convergence evaluation of DeepSpeed-Ulysses with different ZeRO
memory optimization stages

##  5 Conclusion

In conclusion, we present a memory and communication efficient DeepSpeed
Sequence as enabling technology for long sequence large Transformer training.
DeepSpeed Sequence enables sequence parallelism across GPUs (by extension
other AI accelerators), parallelizing sequence across all components of the
Transformer model, including streamline support for SOTA Flash (dense and
sparse) attention. Training with DeepSpeed Sequence allows both model size and
sequence length to scale near indefinitely unbounded by single GPU memory
limitation and at a high fraction of peak compute performance.

## References

  * Beltagy et al. [2020] Iz Beltagy, Matthew E. Peters, and Arman Cohan.  Longformer: The long-document transformer, 2020.
  * Kryściński et al. [2022] Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev.  Booksum: A collection of datasets for long-form narrative summarization, 2022.
  * MosaicML [2023] MosaicML.  Introducing mpt-7b: A new standard for open-source, commercially usable llms.  https://https://www.mosaicml.com/blog/mpt-7b, 2023.
  * Xiong et al. [2023] Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma.  Effective long-context scaling of foundation models, 2023.
  * Peng et al. [2023] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.  Yarn: Efficient context window extension of large language models, 2023.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Nguyen et al. [2023] Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K. Gupta, and Aditya Grover.  Climax: A foundation model for weather and climate, 2023.
  * Zvyagin et al. [2022] Maxim Zvyagin, Alexander Brace, Kyle Hippe, Yuntian Deng, Bin Zhang, Cindy Orozco Bohorquez, Austin Clyde, Bharat Kale, Danilo Perez-Rivera, Heng Ma, et al.  Genslms: Genome-scale language models reveal sars-cov-2 evolutionary dynamics.  _bioRxiv_ , pages 2022–10, 2022.
  * Li et al. [2022a] Yikuan Li, Ramsey M. Wehbe, Faraz S. Ahmad, Hanyin Wang, and Yuan Luo.  Clinical-longformer and clinical-bigbird: Transformers for long clinical sequences.  _CoRR_ , abs/2201.11838, 2022a.  URL https://arxiv.org/abs/2201.11838.
  * Gao et al. [2021] Shang Gao, Mohammed Alawad, M. Todd Young, John Gounley, Noah Schaefferkoetter, Hong Jun Yoon, Xiao-Cheng Wu, Eric B. Durbin, Jennifer Doherty, Antoinette Stroup, Linda Coyle, and Georgia Tourassi.  Limitations of transformers on clinical text classification.  _IEEE Journal of Biomedical and Health Informatics_ , 25(9):3596–3607, 2021.  doi:10.1109/JBHI.2021.3062322.
  * Dao [2023] Tri Dao.  Flashattention-2: Faster attention with better parallelism and work partitioning, 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008, 2017.
  * Dean et al. [2012] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al.  Large scale distributed deep networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Rajbhandari et al. [2020] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  Zero: Memory optimizations toward training trillion parameter models.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–16. IEEE, 2020.
  * Rajbhandari et al. [2021] Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He.  Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , SC ’21, 2021.
  * Keskar et al. [2016] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  _arXiv preprint arXiv:1609.04836_ , 2016.
  * Shoeybi et al. [2019] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.  Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019.
  * Narayanan et al. [2019] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Granger, Phil Gibbons, and Matei Zaharia.  Pipedream: Generalized pipeline parallelism for dnn training.  In _ACM Symposium on Operating Systems Principles (SOSP 2019)_ , October 2019.
  * Huang et al. [2018] Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Zhifeng Chen.  Gpipe: Efficient training of giant neural networks using pipeline parallelism.  _ArXiv_ , abs/1811.06965, 2018.
  * Narayanan et al. [2021] Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia.  Memory-efficient pipeline-parallel dnn training.  In _International Conference on Machine Learning_ , pages 7937–7947. PMLR, 2021.
  * Team and Majumder [2020] DeepSpeed Team and Rangan Majumder.  DeepSpeed: Extreme-scale model training for everyone.  https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/, 2020.
  * Smith et al. [2022] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al.  Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model.  _arXiv preprint arXiv:2201.11990_ , 2022.
  * Wang et al. [2023] Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, and Yuxiong He.  Zero++: Extremely efficient collective communication for giant model training, 2023.
  * Ben-Nun and Hoefler [2019] Tal Ben-Nun and Torsten Hoefler.  Demystifying parallel and distributed deep learning: An in-depth concurrency analysis.  _ACM Computing Surveys (CSUR)_ , 52(4):1–43, 2019.
  * Li et al. [2022b] Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You.  Sequence parallelism: Long sequence training from system perspective, 2022b.
  * Korthikanti et al. [2022] Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro.  Reducing activation recomputation in large transformer models, 2022.
  * Child et al. [2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.  Generating long sequences with sparse transformers.  _CoRR_ , abs/1904.10509, 2019.
  * Choromanski et al. [2020] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy J. Colwell, and Adrian Weller.  Rethinking attention with performers.  _CoRR_ , abs/2009.14794, 2020.  URL https://arxiv.org/abs/2009.14794.
  * Zaheer et al. [2021] Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed.  Big bird: Transformers for longer sequences, 2021.
  * Dao et al. [2022] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.


* [27] C. G. Mayhew and A. R. Teel, “Global stabilization of spherical orientation by synergistic hybrid feedback with application to reduced-attitude tracking for rigid bodies,” _Automatica_ , vol. 49, no. 7, pp. 1945–1957, 2013.
  * [28] V. J. Lumelsky and A. Stepanov, “Dynamic path planning for a mobile automaton with limited information on the environment,” _IEEE Trans. Automat. Contr._ , vol. 31, no. 11, pp. 1058–1063, 1986.
  * [29] L. Wang, A. D. Ames, and M. Egerstedt, “Safety barrier certificates for collisions-free multirobot systems,” _IEEE Trans. on Robotics_ , vol. 33, no. 3, pp. 661–674, 2017.
  * [30] J. Chai and R. G. Sanfelice, “Forward invariance of sets for hybrid dynamical systems (Part I),” _IEEE Trans. Automat. Contr._ , 2019.
  * [31] R. T. Rockafellar, “Clarke’s tangent cones and the boundaries of closed sets in $\mathbf{R}^{n}$,” _Nonlinear analysis: Theory, Methods & Applications_, vol. 3, pp. 145–154, 1979.

Soulaimane Berkane received his Engineering and M.Sc. degrees in Automatic
Control from Ecole Nationale Polytechnique, Algeria, in 2013, and his PhD in
Electrical Engineering from the University of Western Ontario, Canada, in
2017. He held postdoctoral positions at the University of Western Ontario,
Canada, and at KTH Royal Institute of Technology, Sweden, between 2018 and
2019. He is currently an assistant professor at the Department of Computer
Science and Engineering, University of Quebec in Outaouais, Canada. His
research interests are in the area of nonlinear control theory with
applications to robotic and autonomous systems.
---
Andrea Bisoffi received the M.Sc. degree in Automatic Control Engineering from
Politecnico di Milano, Italy, in 2013 and the Ph.D. degree in Mechatronics
from the University of Trento, Italy, in 2017. In 2015–-2016 he was a visiting
scholar in the Control Group at the University of Cambridge, UK. He was a
postdoctoral researcher at KTH Royal Institute of Technology, Sweden in
2017-2019 and is currently one at the University of Groningen, The
Netherlands. His current research interests include hybrid and nonlinear
control systems, with applications to mechanical, and robotic systems.
---
Dimos V. Dimarogonas received the Diploma in Electrical and Computer
Engineering in 2001 and the Ph.D. in Mechanical Engineering in 2007, both from
the National Technical University of Athens (NTUA), Greece. From 2007 to 2010,
he held postdoctoral positions at KTH Royal Institute of Technology and
Massachusetts Institute of Technology. He is currently a Professor in
Automatic Control, School of Electrical Engineering and Computer Science, KTH
Royal Institute of Technology. His current research interests include multi-
agent and hybrid systems with applications to autonomous systems. He serves on
the Editorial Board of Automatica and the IEEE Transactions on Control of
Network Systems. Dr. Dimarogonas is a recipient of an ERC Starting Grant in
2014, an ERC Consolidator Grant in 2019, and a Wallenberg Academy Fellowship
in 2015. He is a member of the Technical Chamber of Greece and a Senior Member
of the IEEE.
---