# Zeros of Random Polynomial Mappings in Several Complex Variables

T. Bayraktar, T. Bloom and N. Levenberg

###### Abstract

We generalize and strengthen some previous results on asymptotics of
normalized zero measures and currents associated to random polynomials and
random polynomial mappings in several complex variables.

##  1 Introduction

We will consider random polynomials in $\mathbb{C}^{d}$ of degree at most $n$
of the form

$H_{n}(z):=\sum_{j=1}^{m_{n}}a_{j}p_{j}(z)$ (1.1)

where the $a_{j}$ are i.i.d. complex random variables;
$\\{p_{1},...,p_{m_{n}}\\}$ form a basis for $\mathcal{P}_{n}$, the
holomorphic polynomials of degree at most $n$ in $\mathbb{C}^{d}$; and
$m_{n}=$dim$(\mathcal{P}_{n}$). The random variables $a_{j}$ are defined on a
probability space $(\Omega,\mathcal{H},\mathbb{P})$. In the situation where
one starts with $K\subset\mathbb{C}^{d}$ compact and $\mu$ a probability
measure on $K$ which is nondegenerate, i.e., $||p||_{L^{2}(\mu)}=0$ for a
polynomial $p$ implies $p\equiv 0$, a well-studied situation is to let
$\\{p_{j}\\}$ be an orthonormal basis in $L^{2}(\mu)$ for $\mathcal{P}_{n}$
constructed via Gram-Schmidt applied to a monomial basis
$\\{z^{\alpha}=z_{1}^{\alpha_{1}}\cdots z_{d}^{\alpha_{d}}\\}$ of
$\mathcal{P}_{n}$. In this setting, convergence of the sequence of
(deterministic) plurisubharmonic (psh) functions $\\{\frac{1}{2n}\log
B_{n}(z)\\}$, where $B_{n}(z):=\sum_{j=1}^{m_{n}}|p_{j}(z)|^{2}$, to the
extremal psh function $V_{K}$ associated to $K$ plays a major role.

We aim to generalize and strengthen some of the results occurring in [1], [2]
and [7] on asymptotics of normalized zero measures and currents associated to
$H_{n}$ in two ways. First, we allow $\\{p_{j}\\}$ to be an asymptotically
Chebyshev family which need not come from orthonormal polynomials; and second,
we strive for the weakest conditions on the i.i.d. complex random variables
$a_{j}$ in order to have the appropriate convergence as, e.g., in [9] and [6].

For the classical Kac ensemble
$H_{n}(z)=\sum_{j=0}^{n}a_{j}z^{j}=a_{n}\prod_{k=1}^{n}(z-z_{nk})$ in one
variable, Ibragimov and Zaporozhets [12] showed that the finite logarithmic
moment condition $\mathbb{E}(\log(1+|a_{j}))<\infty$ is a necessary and
sufficient condition for the sequence
$\mu_{H_{n}}:=\frac{1}{n}\sum_{k=1}^{n}\delta_{z_{kn}}$ to converge weakly
almost surely to normalized arclength measure $\frac{1}{2\pi}d\theta$ on the
unit circle $T$. This is the potential-theoretic equilibrium measure $\mu_{T}$
of $T$; and the monomials $\\{z^{j}\\}$ are an orthonormal basis for $L^{2}$
of the same measure. Kabluchko and Zaporozhets [13] proved some results for
convergence in probability of $\mu_{H_{n}}\to\mu_{T}$ weakly. Pritsker and
Ramachandran [14] gave some almost sure convergence results for special
polynomial bases on certain compacta in $\mathbb{C}$. Bloom and Dauvergne [6]
showed that for a nonpolar compact set $K\subset\mathbb{C}$ with equilibrium
measure $\mu_{K}$, given a Bernstein-Markov measure $\mu$ on $K$ and an
orthonormal basis of polynomials $\\{p_{j}\\}$ for $L^{2}(\mu)$, if
$\mathbb{P}(|a_{j}|>e^{|z|})=o(1/|z|)$, then $\mu_{H_{n}}\to\mu_{K}$ weakly in
probability. Dauvergne generalized these univariate results in [9]: for his
class of asymptotically minimal polynomials $\\{p_{j}\\}$ for $K$, which
includes orthonormal polynomials for a Bernstein-Markov measure $\mu$ on $K$
and essentially all “classical” sets of polynomials (Chebyshev, Fekete, Fejer,
etc.), the condition $\mathbb{E}(\log(1+|a_{j}))<\infty$ is a necessary and
sufficient condition for the sequence $\mu_{H_{n}}\to\mu_{K}$ weakly almost
surely while the condition $\mathbb{P}(|a_{j}|>e^{|z|})=o(1/|z|)$ is a
necessary and sufficient condition for the sequence $\mu_{H_{n}}\to\mu_{K}$
weakly in probability.

We work in $\mathbb{C}^{d}$ with $d>1$. Bloom and Shiffman [8], building upon
earlier work of Shiffman and Zelditch [15], proved some results for Gaussian
random polynomials where the basis polynomials are orthonormal polynomials for
a Bernstein-Markov measure $\mu$ on $K\subset\mathbb{C}^{d}$. In this setting,
the authors ([1], [2] and [7]) have given sufficient conditions on
$\\{a_{j}\\}$ in order to get certain almost sure convergence results for
normalized zero currents associated to random $\\{H_{n}\\}$. In the next
section, we define a notion of asymptotically Chebyshev polynomials for $K$
from [5] which generalizes the $L^{\infty}-$version of Dauvergne’s
asymptotically minimal polynomials. With such a basis, we prove a determinstic
result on the corresponding sequence of deterministic psh functions
$\\{\frac{1}{2n}\log B_{n}(z)\\}$. This is used in section 3 to verify a
convergence in probability result for our random $\\{H_{n}\\}$ under the
weakest possible condition: $\mathbb{P}(|a_{j}|>e^{|z|})=o(1/|z|^{d})$. Here
we utilize some probabilistic arguments from [6]. Then in section 4, we
strengthen the results of [1] and [2] in the higher codimension case, i.e., we
consider normalized zero currents associated to random polynomial mappings
$F^{k}_{n}=(H_{n}^{1},H_{n}^{2},\dots,H_{n}^{k}):\mathbb{C}^{d}\to\mathbb{C}^{k}$.
Here we can allow different bases for each random polynomial $H_{n}^{i}$. The
proofs offered here are much simpler than in the first author’s papers [1] and
[2] and our hypotheses on the random coefficients are weaker. We mention,
however, that the first author’s main result in [2] (Theorem 1.2 there) holds
in the more general setting of random Laurent polynomial mappings associated
to convex polytopes. The key deterministic ingredient we use is a lemma on
convergence of closed, positive currents of Fornaess and Sibony [10] which we
state as our Proposition 4.4.

##  2 Deterministic results on asymptotically Chebyshev polynomials

As in the introduction we let $m_{n}=\ $dim$(\mathcal{P}_{n}$). Let

$L(\mathbb{C}^{d}):=\\{u\in PSH(\mathbb{C}^{d}):u(z)-\log|z|=0(1),\
|z|\to\infty\\},$

the Lelong class of psh functions of minimal growth. Given a nonpluripolar
compact subset $K\subset\mathbb{C}^{d}$, let

$V_{K}(z):=\sup\\{u(z):u\in L(\mathbb{C}^{d}),\ u\leq 0\ \hbox{on}\ K\\}$

and $V_{K}^{*}(z):=\limsup_{\zeta\to z}V_{K}(\zeta)$. Then $V_{K}^{*}\in
L(\mathbb{C}^{d})$ and it is known that

$V_{K}(z)=\sup\\{\frac{1}{deg(p)}\log|p(z)|:p\in\cup_{n}\mathcal{P}_{n},\
||p||_{K}\leq 1\\}.$

Moreover, given a Bernstein-Markov measure $\mu$ on $K$, i.e., i.e., if for
each $\epsilon>0$ there exists a positive constant $C=C(\epsilon)$ such that
$||p||_{K}\leq C(1+\epsilon)^{deg(p)}||p||_{L^{r}(\mu)}$ for
$p\in\cup_{n}\mathcal{P}_{n}$ for some (and hence any) $r\geq 1$, it is
straightforward to see that for the sequence of Bergman functions
$B_{n}(z):=\sum_{j=1}^{m_{n}}|p_{j}(z)|^{2}$, where $\\{p_{j}\\}$ is an
orthonormal basis of polynomials for $L^{2}(\mu)$, we have $\frac{1}{2n}\log
B_{n}(z)\to V_{K}$ pointwise on $\mathbb{C}^{d}$; moreover the convergence is
locally uniform if $K$ is regular; i.e., if $V_{K}$ is continuous (cf.
Proposition 3.1, [7]). For simplicity, throughout this paper we assume our
compact sets are regular. This deterministic result is crucial in proving
asymptotic results on random polynomials as in (1.1).

We fix an ordering $\prec$ on $\mathbb{N}^{d}$ so that $\alpha\prec\beta$ if
$|\alpha|\leq|\beta|$. Writing

$e_{i}(z)=z^{\alpha}=z^{\alpha(i)}:=z_{1}^{\alpha_{1}}\cdots
z_{d}^{\alpha_{d}}$

and $|\alpha|:=\alpha_{1}+\cdots+\alpha_{d}$, let

$P(\alpha)=P(\alpha(i)):=\\{e_{i}(z)+\sum_{j<i}c_{j}e_{j}(z):c_{j}\in\mathbb{C}\\}=\\{z^{\alpha(i)}+\sum_{\beta\prec\alpha(i)}c_{\beta}z^{\beta}\\}.$

Given a compact set $K\subset\mathbb{C}^{d}$ we define the Chebyshev constants

$\tau_{\alpha(i)}(K):=\inf\\{||p||_{K}:p\in P(\alpha(i))\\}.$

Following [16], we use these to define directional Chebyshev constants. We
have $\frac{\alpha(i)}{|\alpha(i)|}\in\Sigma$, the standard $(d-1)-$simplex in
$\mathbb{R}^{d}$:

$\Sigma=\\{\theta=(\theta_{1},...,\theta_{d})\in\mathbb{R}^{d}:\sum_{j=1}^{d}\theta_{j}=1,\
\theta_{j}\geq 0,\ j=1,...,d\\}.$

Let $\Sigma^{o}:=\\{\theta\in\Sigma:\ \theta_{j}>0,\ j=1,...,d\\}$. Zaharjuta
[16] showed that for $\theta\in\Sigma^{o}$, the limit

$\tau(K,\theta):=\lim_{|\alpha(i)|\to\infty,\
\frac{\alpha(i)}{|\alpha(i)|}\to\theta}\tau_{\alpha(i)}(K)^{1/|\alpha(i)}\
\hbox{exists};$

and we call $\tau(K,\theta)$ the directional Chebyshev constant of $K$ in the
direction $\theta$.

Following [5], given $\theta\in\Sigma^{o}$, we call a sequence of polynomials
$\\{q_{k}\\}_{k\in Y_{\theta}}$, $Y_{\theta}\subset\mathbb{N}$, where
$q_{k}(z)=z^{\beta(k)}+\sum_{\beta\prec\beta(k)}c_{\beta}z^{\beta}$
$\theta-$asymptotically Chebyshev for $K$ if

$\lim_{k\to\infty,\ k\in Y_{\theta}}deg(q_{k})=\infty,\ \lim_{k\to\infty,\
k\in Y_{\theta}}\beta(k)/|\beta(k)|=\theta,$

and

$\lim_{k\to\infty,\ k\in
Y_{\theta}}||q_{k}||_{K}^{1/|\beta(k)|}=\tau(K,\theta).$

If we need to specify the multiindex and degree we will write $q_{j}$ as
$q_{n,\alpha}$ where $|\alpha|=n=deg(q_{n,\alpha})$. Here the ordering $\prec$
then gives us the ordering of the sequence $\\{q_{j}\\}_{j}$. We call the
sequence $\\{q_{j}\\}_{j\in\mathbb{N}^{d}}$ asymptotically Chebyshev for $K$
if for each $\theta\in\Sigma^{o}$, we can find a subsequence
$Y_{\theta}\subset\mathbb{N}$ so that $\\{q_{k}\\}_{k\in Y_{\theta}}$ is
$\theta-$asymptotically Chebyshev for $K$. We can assume that there is one
$q_{j}$ for each multiindex $\alpha$. Bloom (Theorem 4.2 of [5]) showed that
for such a sequence

$[\limsup_{j\to\infty}\frac{1}{deg(q_{j})}\log\frac{|q_{j}(z)|}{||q_{j}||_{K}}]^{*}=V_{K}(z)\
\hbox{for}\ z\not\in\widehat{K}$ (2.1)

where $\widehat{K}:=\\{z\in\mathbb{C}^{d}:|p(z)|\leq||p||_{K},\
p\in\cup_{n}\mathcal{P}_{n}\\}$.

######  Remark 2.1.

It is clear that if $\\{q_{j}\\}=\\{q_{n,\alpha}\\}$ is asymptotically
Chebyshev for $K$ and $\\{c_{n,\alpha}\\}$ are positive constants such that
for all $\epsilon>0$ there exists $n_{0}=n_{0}(\epsilon)$ so that for $n\geq
n_{0}$

$\exp(-\epsilon n)\leq c_{n,\alpha}\leq\exp(\epsilon n),$

then $\\{\widetilde{q}_{j}\\}=\\{\widetilde{q}_{n,\alpha}\\}$ is
asymptotically Chebyshev for $K$ where
$\widetilde{q}_{n,\alpha}:=c_{n,\alpha}q_{n,\alpha}$. In particular, given a
(BM) measure $\mu$ for $K$ and any asymptotically Chebyshev sequence
$\\{q_{j}\\}$ for $K$ where $q_{j}\in P(\alpha(j))$, for any $r\geq 1$ the
sequence $\\{\widetilde{q}_{j}\\}$, where
$\widetilde{q}_{j}:=q_{j}/||q_{j}||_{L^{r}(\mu)}$, is asymptotically Chebyshev
for $K$.

Given an asymptotically Chebyshev sequence $\\{q_{j}\\}$ for $K$, let
$p_{j}(z):=|q_{j}(z)|/||q_{j}||_{K}$. We will also use the notation
$p_{n,\alpha}$ for $p_{j}$ if need be. We will always assume that there is one
$q_{j}$ for each multiindex $\alpha$ and that $q_{1}(z)=p_{1}(z)\equiv 1$. In
analogy with the case of an orthonormal basis associated to a Bernstein-Markov
measure, we define, for $n=1,2,...$, the function

$B_{n}(z):=\sum_{j=1}^{m_{n}}|p_{j}(z)|^{2}.$ (2.2)

It is easy to see that

$\limsup_{n\to\infty}\frac{1}{2n}\log B_{n}(z)\leq V_{K}(z)\ \hbox{for}\
z\in\mathbb{C}^{d}$ (2.3)

and

$\lim_{n\to\infty}\frac{1}{2n}\log B_{n}(z)=0\ \hbox{for}\ z\in\widehat{K}.$
(2.4)

Indeed, since $||p_{j}||_{K}=1$, by definition of $V_{K}$, $|p_{j}(z)|\leq
e^{deg(p_{j})V_{K}(z)}$. Thus

$B_{n}(z)\leq\sum_{j=1}^{m_{n}}e^{2deg(p_{j})V_{K}(z)}\leq
m_{n}e^{2nV_{K}(z)}$

which gives (2.3) since $m_{n}=0(n^{d})$. Note that this estimate also shows
that the family $\\{\frac{1}{2n}\log B_{n}\\}$ is locally uniformly bounded
above on $\mathbb{C}^{d}$. Now since $q_{1}(z)\equiv 1$ we have $B_{n}(z)\geq
1$ so

$\liminf_{n\to\infty}\frac{1}{2n}\log B_{n}(z)\geq 0\ \hbox{for}\
z\in\mathbb{C}^{d}$

which, together with (2.3), proves (2.4). We use these estimates to prove our
next result.

######  Proposition 2.2.

Let $\\{q_{j}\\}_{j}$ be asymptotically Chebyshev for $K$. Then for the
sequence $\\{B_{n}\\}$ in (2.2) we have the following: given any subsequence
$Y\subset\mathbb{N}$, there is a further subsequence $Y_{0}\subset Y$ and a
countable dense set of points $\\{w_{r}\\}$ in $\mathbb{C}^{d}$ such that

$\lim_{n\to\infty,\ n\in Y_{0}}\frac{1}{2n}\log B_{n}(w_{r})=V_{K}(w_{r}),\
r=1,2,...$

###### Proof.

Fix a subsequence $Y\subset\mathbb{N}$. Equation (2.4) shows the pointwise
convergence of the full sequence $\\{\frac{1}{2n}\log B_{n}\\}$ to $0$ on
$\widehat{K}$ so we only need to find a subsequence $Y_{0}\subset Y$ and an
appropriate countable dense set of points $\\{w_{r}\\}$ in
$\mathbb{C}^{d}\setminus\widehat{K}$. For the subsequence $Y$, the collection
of polynomials $\\{q_{n,\alpha}\\}_{n\in Y,\ |\alpha|=n}$ satisfies the
hypotheses of Theorem 4.2 of [5] – note for each $n\in Y$ we keep all of the
polynomials $q_{n,\alpha}$ with $|\alpha|=n$. Thus for
$z\in\mathbb{C}^{d}\setminus\widehat{K}$ we have

$[\limsup_{n\to\infty,\ n\in Y,\
|\alpha|=n}\frac{1}{n}\log\frac{|q_{n,\alpha}(z)|}{||q_{n,\alpha}||_{K}}]^{*}=V_{K}(z)$

so that

$\limsup_{n\to\infty,\ n\in Y,\
|\alpha|=n}\frac{1}{n}\log\frac{|q_{n,\alpha}(z)|}{||q_{n,\alpha}||_{K}}=\limsup_{n\to\infty,\
n\in Y,\ |\alpha|=n}\frac{1}{n}\log|p_{n,\alpha}(z)|=V_{K}(z)$ (2.5)

on $\mathbb{C}^{d}\setminus\widehat{K}$ except perhaps a pluripolar set.

Let $\\{D_{r}\\}$ be a countable collection of open balls in
$\mathbb{C}^{d}\setminus\widehat{K}$ such that choosing one point in each
$D_{r}$ yields a countable dense set of points in
$\mathbb{C}^{d}\setminus\widehat{K}$. Note that each $D_{r}$ is nonpluripolar.
Starting with $D_{1}$, choose $w_{1}\in D_{1}$ so that (2.5) holds at $w_{1}$.
Then for some subsequence of pairs $(n_{1},\alpha_{1}),(n_{2},\alpha_{2}),...$
with $n_{j}\in Y$ we have

$\lim_{s\to\infty}\frac{1}{n_{s}}\log|p_{n_{s},\alpha_{s}}(w_{1})|=V_{K}(w_{1}).$

Let $Y_{1}=\\{n_{1},n_{2},...\\}\subset Y$. We claim that

$\lim_{n\to\infty,\ n\in Y_{1}}\frac{1}{2n}\log B_{n}(w_{1})=V_{K}(w_{1}).$
(2.6)

Here (2.6) follows from (2.3), the previous equality, and the elementary fact
that for any $n\in\mathbb{N}$ and any $z\in\mathbb{C}^{d}$,

$\frac{1}{2n}\log B_{n}(z)\geq\frac{1}{n}\log|p_{n,\alpha}(z)|.$

We now repeat this argument with a point $w_{2}\in D_{2}$ to get a subsequence
$Y_{2}\subset Y_{1}$ with

$\lim_{n\to\infty,\ n\in Y_{2}}\frac{1}{2n}\log B_{n}(w_{2})=V_{K}(w_{2}).$

Continuing, we get a countable family of nested subsequences $Y\supset
Y_{1}\supset Y_{2}...$ and points $w_{r}\in D_{r}$ with

$\lim_{n\to\infty,\ n\in Y_{r}}\frac{1}{2n}\log B_{n}(w_{r})=V_{K}(w_{r}).$

Then using a diagonalization argument, we get a sequence $Y_{0}$ which is a
subsequence of each $Y_{r}$ (except for the first $r$ terms) so that
$Y_{0}\subset Y$ and for each $r=1,2,...$ we have

$\lim_{n\to\infty,\ n\in Y_{0}}\frac{1}{2n}\log B_{n}(w_{r})=V_{K}(w_{r}).$

∎

######  Remark 2.3.

Suppose $d=1$. The condition that $\\{q_{j}\\}_{j}$ be asymptotically
Chebyshev for $K$ is the same as the notion in [9] of asymptotically minimal
for the case $p=\infty$. Moreover, it is straightforward to see that
$\\{q_{j}\\}_{j}$ asymptotically Chebyshev for $K$ is equivalent to

$\lim_{n\to\infty}\frac{1}{2n}\log B_{n}(z)=V_{K}(z),\ z\in\mathbb{C}$

where here we take the full sequence $n\in\mathbb{N}$.

As a corollary of Proposition 2.2, we will get the conclusion that the full
sequence $\\{\frac{1}{2n}\log B_{n}\\}$ converges in
$L^{1}_{loc}(\mathbb{C}^{d})$ to $V_{K}$. We utilize the following result on
subharmonic functions, stated for $\mathbb{C}=\mathbb{R}^{2}$ as Theorem 2.4
in [6].

######  Proposition 2.4.

Let $\\{u_{n}\\}$ be a sequence of subharmonic functions on
$D\subset\mathbb{R}^{m},\ m\geq 2$ which are locally uniformly bounded above
on $D$ and let $u$ be subharmonic and continuous in $D$. Suppose that

$[\limsup_{n\to\infty}u_{n}(w)]^{*}\leq u(w),\ w\in D$

and there is a countable dense set of points $\\{w_{r}\\}$ in $D$ such that

$\lim_{n\to\infty}u_{n}(w_{r})=u(w_{r}),\ r=1,2,...$

Then $u_{n}\to u$ in $L^{1}_{loc}(D)$.

######  Corollary 2.5.

Let $\\{q_{j}\\}_{j}$ be asymptotically Chebyshev for $K$. Then
$\frac{1}{2n}\log B_{n}\to V_{K}$ in $L^{1}_{loc}(\mathbb{C}^{d})$.

###### Proof.

We show that for any subsequence $Y_{1}\subset\mathbb{N}$, there exists a
subsequence $Y_{2}\subset Y_{1}$ such that

$\lim_{n\to\infty,\ n\in Y_{2}}\frac{1}{2n}\log B_{n}=V_{K}\ \hbox{in}\
L^{1}_{loc}(\mathbb{C}^{d}).$ (2.7)

For this it suffices to observe that for any subsequence $Y\subset\mathbb{N}$,
we have

$[\limsup_{n\to\infty,\ n\in Y}\frac{1}{2n}\log B_{n}(z)]^{*}\leq V_{K}(z),\
z\in\mathbb{C}^{d}$ (2.8)

which follows immediately from (2.3). For then, by Proposition 2.2, given a
subsequence $Y_{1}\subset\mathbb{N}$, we get a subsequence $Y_{2}\subset
Y_{1}$ and a countable dense set $\\{w_{r}\\}$ in $\mathbb{C}^{d}$ such that

$\lim_{n\to\infty,\ n\in Y_{2}}\frac{1}{2n}\log B_{n}(w_{r})=V_{K}(w_{r}),\
r=1,2,...$

Using (2.8) for $Y=Y_{2}$, we can apply Proposition 2.4 to conclude (2.7)
(recall that $\\{\frac{1}{2n}\log B_{n}\\}$ is locally uniformly bounded above
on $\mathbb{C}^{d}$).

∎

We conclude this section with examples of asymptotically Chebyshev sequences
$\\{q_{j}\\}=\\{q_{n,\alpha}\\}$ for a regular compact set
$K\subset\mathbb{C}^{d}$.

  1. 1.

$\\{q_{j}\\}=\\{q_{n,\alpha}\\}$ where $q_{j}\in P(\alpha(j))$ with
$||q_{j}||_{K}=\tau_{\alpha(j)}(K):=\inf\\{||p||_{K}:p\in P(\alpha(j))\\}$.

  2. 2.

$\\{q_{j}\\}=\\{q_{n,\alpha}\\}$ where, for $\mu$ a (BM) measure for $K$,
$q_{j}\in P(\alpha(j))$ is the $L^{2}(\mu)-$minimal (orthogonal) polynomial in
$P(\alpha(j))$ for $K$; i.e.,
$||q_{j}||_{L^{2}(\mu)}=\inf\\{||p||_{L^{2}(\mu)}:p\in P(\alpha(j))\\}$.

  3. 3.

$\\{q_{j}\\}=\\{q_{n,\alpha}\\}$ are Fekete polynomials associated to an array
of Fekete points for $K$ (see [5], p. 1562). For a generalization of this
example, see Corollary 4.4 of [5].

  4. 4.

$\\{q_{j}\\}=\\{q_{n,\alpha}\\}$ are Leja polynomials associated to a sequence
of Leja points for $K$ (see [5], p. 1562).

##  3 Convergence in probability

In the following sections, unless explicitly stated otherwise, we consider
random polynomials of degree at most $n$ of the form (1.1); i.e.,
$H_{n}(z):=\sum_{j=1}^{m_{n}}a_{j}p_{j}(z)$, where the $a_{j}$ are i.i.d.
complex random variables and $\\{p_{j}:=q_{j}/||q_{j}||_{K}\\}$ where
$\\{q_{j}\\}$ are asymptotically Chebyshev polynomials for $K$. Here we fix
the ordering $\prec$ on $\mathbb{N}^{d}$ from last section so that
$\\{q_{j}\\}=\\{q_{n,\alpha}\\}$. We work in the corresponding probability
space $(\Omega,\mathcal{H},{\mathbb{P}})$. We assume the $a_{j}$ are non-
degenerate; i.e., their distribution is not supported at a single point.
Sometimes to emphasize the random nature of our objects, we will include the
variable $\omega\in\Omega$; e.g., we may write $a_{j}(\omega),\
H_{n}(z,\omega)$, etc. Our aim in this section is to to prove the following
generalization of Theorem 7.6 in [6].

######  Theorem 3.1.

Let $a_{j}$ be i.i.d. complex random variables satisfying
$\mathbb{P}(|a_{j}|>e^{|z|})=o(1/|z|^{d})$, let
$\\{p_{j}:=q_{j}/||q_{j}||_{K}\\}$ where $\\{q_{j}\\}$ are asymptotically
Chebyshev for $K$, and let $H_{n}(z):=\sum_{j=1}^{m_{n}}a_{j}p_{j}(z)$ be
random polynomials. Then, in probability,

$\frac{1}{n}\log|H_{n}|\to V_{K}\ \hbox{in}\ L^{1}_{loc}(\mathbb{C}^{d}).$

######  Remark 3.2.

We conclude that every subsequence $Y\subset\mathbb{N}$ has a further
subsequence $Y_{0}\subset Y$ such that almost surely

$\lim_{n\to\infty,\ n\in Y_{0}}dd^{c}\frac{1}{n}\log|H_{n}|\to dd^{c}V_{K}\
\hbox{as currents}.$

Here we use the normalization
$dd^{c}=\frac{i}{\pi}\partial\overline{\partial}$.

We utilize some results in [6] in a multivariate setting. The first is the
multivariate version of Lemma 3.1 of [6]. We observe that $m_{n}=0(n^{d})$;
hence the condition $\mathbb{P}(|a_{j}|>e^{|z|})=o(1/|z|^{d})$ (instead of
$o(1/|z|)$).

######  Lemma 3.3.

Let $a_{j}$ be i.i.d. complex random variables satisfying
$\mathbb{P}(|a_{j}|>e^{|z|})=o(1/|z|^{d})$, let
$\\{p_{j}:=q_{j}/||q_{j}||_{K}\\}$ where $\\{q_{j}\\}$ are asymptotically
Chebyshev for $K$, and let $H_{n}(z):=\sum_{j=1}^{m_{n}}a_{j}p_{j}(z)$ be
random polynomials. For any sequence $Y\subset\mathbb{N}$ there is a further
subsequence $Y_{0}\subset Y$ such that almost surely, the family
$\\{\frac{1}{n}\log|H_{n}|\\}_{n\in Y_{0}}$ is locally uniformly bounded above
and for all $z\in C^{d}$

$\limsup_{n\to\infty,\ n\in Y_{0}}\frac{1}{n}\log|H_{n}(z)|\leq V_{K}(z).$
(3.1)

###### Proof.

There are only two modifications to the proof of Lemma 3.1 in [6]. First, the
condition $\mathbb{P}(|a_{j}|>e^{|z|})=o(1/|z|^{d})$ yields that for
$\epsilon>0$ we have $\mathbb{P}(|a_{j}|>e^{\epsilon|z|})=o(1/|z|^{d})$ and
hence letting

$\Omega_{n,\epsilon}:=\\{\omega\in\Omega:|a_{j}(\omega)|\leq e^{\epsilon n},\
j=1,...,m_{n}\\}$

we have

$\mathbb{P}(\Omega^{c}_{n,\epsilon})\leq\sum_{j=1}^{m_{n}}\mathbb{P}(|a_{j}(\omega)|>e^{\epsilon
n})\leq m_{n}o(1/n^{d})\to 0$

as $n\to\infty$ since $m_{n}=0(n^{d})$. Thus for any sequence
$Y\subset\mathbb{N}$ there is a further subsequence
$Y_{0}=\\{n_{s}\\}_{s}\subset Y$ with
$\sum_{s=1}^{\infty}\mathbb{P}(\Omega^{c}_{n_{s},\epsilon})<\infty$. By Borel-
Cantelli, almost surely there exists $s_{0}=s_{0}(\epsilon)$ so that for
$s\geq s_{0}$

$|a_{j}(\omega)|\leq e^{\epsilon n_{s}},\ j=1,2,...,n_{s}.$

Next, for $j=1,...,m_{n}$, since $||p_{j}||_{K}=1$ we have directly that

$|p_{j}(z)|\leq e^{deg(p_{j})V_{K}(z)}\leq e^{nV_{K}(z)}\ \hbox{so that}\
\frac{1}{n}\log|p_{j}(z)|\leq V_{K}(z),\ z\in\mathbb{C}^{d};$

and hence using this (in place of local uniform convergence of
$\\{\frac{1}{2n}\log|B_{n}|\\}$, which we don’t necessarily have) yields

$\frac{1}{n_{s}}\log|H_{n_{s}}(z)|\leq\frac{1}{n_{s}}\log\bigl{(}\sum_{j=1}^{m_{n_{s}}}|a_{j}p_{j}(z)|\bigr{)}\leq\epsilon+V_{K}(z),\
z\in\mathbb{C}^{d}$

almost surely. This yields the local uniform boundedness of the family
$\\{\frac{1}{n_{s}}\log|H_{n_{s}}|\\}_{n_{s}\in Y_{0}}$, as well as (3.1) upon
letting $\epsilon$ go to $0$.

∎

The following is a multivariate version of Theorem 5.2 of [6]; it is a
restatement of Theorem 7.5 of [6]. The key ingredient is a version of the
Kolmogorov-Rogozin inequality stated as Theorem 5.1 of [6].

######  Theorem 3.4.

Let $\\{b_{j}\\}$ be a sequence of nonzero complex numbers satisfying

$\lim_{n\to\infty}\frac{1}{2n}\log\sum_{j=1}^{m_{n}}|b_{j}|^{2}=:B\geq 0.$

Let $\\{a_{j}\\}$ be i.i.d. complex random variables satisfying
$\mathbb{P}(|a_{j}|>e^{|z|})=o(1/|z|^{d})$. Then

$\lim_{n\to\infty}\frac{1}{n}\log\sum_{j=1}^{m_{n}}|a_{j}b_{j}|=B$

in probability.

We now proceed with the proof of Theorem 3.1.

###### Proof.

To show that

$\frac{1}{n}\log|H_{n}|\to V_{K}\ \hbox{in}\ L^{1}_{loc}(\mathbb{C}^{d})$

in probability, we must show that for any sequence $Y\subset\mathbb{N}$ there
is a further subsequence $Y_{2}\subset Y$ such that

$\lim_{n\to\infty,\ n\in Y_{2}}\frac{1}{n}\log|H_{n}|=V_{K}\ \hbox{in}\
L^{1}_{loc}(\mathbb{C}^{d})$

almost surely. From Lemma 3.3, for any sequence $Y\subset\mathbb{N}$ there is
a further subsequence $Y_{0}\subset Y$ such that almost surely, the family
$\\{\frac{1}{n}\log|H_{n}|\\}_{n\in Y_{0}}$ is locally uniformly bounded above
and for all $z\in\mathbb{C}^{d}$

$\limsup_{n\to\infty,\ n\in Y_{0}}\frac{1}{n}\log|H_{n}(z)|\leq V_{K}(z).$

Next, using Proposition 2.2, there is a further subsequence $Y_{1}\subset Y$
and a countable dense set of points $\\{w_{r}\\}$ in $\mathbb{C}^{d}$ such
that

$\lim_{n\to\infty,\ n\in Y_{1}}\frac{1}{2n}\log B_{n}(w_{r})=V_{K}(w_{r}),\
r=1,2,...$

Thus the hypotheses of Theorem 3.4 hold with $b_{j}=p_{j}(w_{r})$ and
$B=V_{K}(w_{r})$; note we may assume $\\{w_{r}\\}$ are chosen so that
$p_{j}(w_{r})\not=0$ for all $j,r$. Hence for each $w_{r}$ we have

$\lim_{n\to\infty,\ n\in Y_{1}}\frac{1}{n}\log|H_{n}(w_{r})|=V_{K}(w_{r})$

in probability. By passing to a further subsequence $Y_{2}\subset Y_{1}$, for
each $w_{r}$ we have

$\lim_{n\to\infty,\ n\in Y_{2}}\frac{1}{n}\log|H_{n}(w_{r})|=V_{K}(w_{r})$

almost surely.

Then by Proposition 2.4 applied to the sequence $\\{H_{n}\\}_{n\in Y_{2}}$,
almost surely

$\lim_{n\to\infty,\ n\in Y_{2}}\frac{1}{n}\log|H_{n}|=V_{K}\ \hbox{in}\
L^{1}_{loc}(\mathbb{C}^{d}).$

∎

##  4 Distribution of zeros in higher codimensions

In this section, we focus on the distribution of simultaneous zeros of systems
of random polynomials. Our bases will come from asymptotically Chebyshev
polynomials, but we allow different compact sets $K$. To be precise, for
$k\in\\{1,2,...,d\\}$, let $K_{1},...,K_{k}\subset\mathbb{C}^{d}$ be regular
compact sets and let $\\{p_{n}^{j}=q_{n}^{j}/||q_{n}^{j}||_{K_{j}}\\}_{n}$,
$j=1,...,k$ be polynomial bases where $\\{q_{n}^{j}\\}$ are asymptotically
Chebyshev for $K_{j},\ j=1,...,k$. We consider random polynomial mappings
$F^{k}_{n}=(H_{n}^{1},H_{n}^{2},\dots,H_{n}^{k}):\mathbb{C}^{d}\to\mathbb{C}^{k}$
of degree $n$ where

$H_{n}^{j}(z):=\sum_{l=1}^{m_{n}}a_{l}^{j}p_{n}^{j}(z)\in\mathcal{P}_{n}$

and $\\{a_{l}^{j}\\}_{l=1}^{m_{n}},\ j=1,...,k$ are i.i.d. random
coefficients. We denote the set of all such degree $n$ random maps by
$\mathcal{P}_{n}^{k}:=\prod_{j=1}^{k}\mathcal{P}_{n}$ To distinguish the
cases, we write that the random variables $a_{l}^{j}$ are defined on a
probability space $(\Omega^{k},{\mathcal{H}}^{k},{\mathbb{P}}^{k})$ so that
for $k=1$ we recover $(\Omega,\mathcal{H},\mathbb{P})$.

We are interested in the asymptotic distribution of the simultaneous zero
varieties given by

$V_{F^{k}_{n}}:=\\{z\in\mathbb{C}^{d}:H_{n}^{1}(z)=\dots=H_{n}^{k}(z)=0\\}=V_{H_{n}^{1}}\cap\cdots\cap
V_{H_{n}^{k}}.$

By Bertini’s theorem [11, pp 137], under appropriate hypotheses on the random
coefficients, with probability one the zero sets $V_{H^{j}_{n}}$ are smooth
and intersect transversally. In particular, for almost every
$F^{k}_{n}\in\mathcal{P}_{n}^{k}$ the simultaneous zero set $V_{F^{k}_{n}}$ is
a complex submanifold of codimension $k$. If $k=d$, $V_{F^{d}_{n}}$ is
(generically) a set of $n^{d}$ points. Moreover, almost surely the current of
integration $Z_{F^{k}_{n}}$ along the set $V_{F^{k}_{n}}$ is given by

$Z_{F^{k}_{n}}=Z_{H_{n}^{1}}\wedge\dots\wedge Z_{H_{n}^{k}}\ \hbox{where}\
Z_{H^{j}_{n}}:=dd^{c}\log|H^{j}_{n}|.$

The main result of this section is the following.

######  Theorem 4.1.

Let $\\{a_{l}^{j}\\}$ be a sequence of i.i.d. random variables. We assume that

  * (i)

$\mathbb{E}[\big{(}\log(1+|a_{l}^{j}|\big{)}^{d}]<\infty$.

  * (ii)

The law of the random coefficients $a_{l}^{j}$ is of the form $\phi(z)dm(z)$
where $\phi:\mathbb{C}\to\mathbb{R}$ is a bounded, measurable function and
$dm$ is Lebesgue measure on $\mathbb{C}$.

Then for each $k\in\\{1,2,...,d\\}$ almost surely in $\mathcal{H}^{k}$ we have

$\frac{1}{n^{k}}Z_{F_{n}^{k}}\to dd^{c}V_{K_{1}}\wedge\cdots\wedge
dd^{c}V_{K_{k}}\ \text{as}\ n\to\infty$

in the sense of currents. In particular, for $k=d$,

$\frac{1}{n^{d}}\sum_{\zeta\in V_{F^{d}_{n}}}\delta_{\zeta}\to
dd^{c}V_{K_{1}}\wedge\cdots\wedge dd^{c}V_{K_{d}}.$

We will need the following lemmas (cf. [6], Lemma 3.3; [9], Lemma 4.1; [2,
3]).

######  Lemma 4.2.

Let $\\{a_{j}\\}$ be a sequence of i.i.d. random variables on a probability
space $(\Omega,\mathcal{F},\mathbb{P})$ which satisfy
$\mathbb{E}[\big{(}\log(1+|a_{j}|\big{)}^{d}]<\infty$. For each $\epsilon>0$
there exists a random constant $C=C(\omega)$ such that almost surely

$|a_{j}|\leq C\exp(\sqrt[d]{j\epsilon})\ \hbox{for all}\ j.$

###### Proof.

For a nonnegative random variable $X$ on $(\Omega,\mathcal{F},\mathbb{P})$ we
have

$\sum_{j=1}^{\infty}\mathbb{P}[X\geq j]\leq\mathbb{E}[X]\leq
1+\sum_{j=1}^{\infty}\mathbb{P}[X\geq j].$

Letting $X=\frac{1}{\epsilon}(\log(1+|a_{1}|))^{d}$ and using the assumption
that $a_{j}$ are identically distributed, we obtain

$\sum_{j=1}^{\infty}\mathbb{P}[|a_{j}|\geq e^{\sqrt[d]{j\epsilon}}]<\infty.$

Hence, by the Borel-Cantelli lemma there exists a $j_{0}=j_{0}(\epsilon)$ so
that

$|a_{j}|<e^{\sqrt[d]{j\epsilon}}\ \hbox{for all}\ j\geq j_{0}.$

Then we can find a a random constant $C$ such that almost surely

$|a_{j}|<Ce^{\sqrt[d]{j\epsilon}}\ \hbox{for all}\ j.$

∎

For notational simplicity, we consider the coefficients

$a^{(n)}:=(\\{a^{1}_{l}\\}_{l=1,...,m_{n}},...,\\{a^{k}_{l}\\}_{l=1,...,m_{n}})$

of $F^{k}_{n}=(H_{n}^{1},H_{n}^{2},\dots,H_{n}^{k})$ as points in
$(\mathbb{C}^{m_{n}})^{k}$. For
$a,b\in(\mathbb{C}^{m_{n}})^{k}=\mathbb{C}^{km_{n}}$, we write $\langle
a,b\rangle:=\sum_{j=1}^{km_{n}}a_{j}b_{j}$. The following lemma will be
useful.

######  Lemma 4.3.

Let $\\{a_{l}^{j}\\}$ be a sequence of i.i.d. random variables satisfying (ii)
of Theorem 4.1: $a_{l}^{j}$ is of the form $\phi(z)dm(z)$ where
$\phi:\mathbb{C}\to\mathbb{R}$ is a bounded, measurable function. Let
$u\in(\mathbb{C}^{m_{n}})^{k}$ be a unit vector. Then for each $\epsilon>0$

${\mathbb{P}}^{k}(a^{(n)}\in(\mathbb{C}^{m_{n}})^{k}:|\langle
a^{(n)},u\rangle|<e^{-n\epsilon})=O(m_{n}e^{-2n\epsilon}).$

Here it is important to observe that the estimate $O(m_{n}e^{-2n\epsilon})$ is
independent of the unit vector $u\in(\mathbb{C}^{m_{n}})^{k}$.

###### Proof.

Since $u$ is a unit vector without loss of generality we may assume that
$|u_{1}|\geq\frac{1}{\sqrt{km_{n}}}$ where $u=(u_{1},u_{2},\dots,u_{km_{n}}).$
Let

$L_{n}:={\mathbb{P}}^{k}(a^{(n)}\in(\mathbb{C}^{m_{n}})^{k}:|\langle
a^{(n)},u\rangle|<e^{-n\epsilon}).$

Then applying the change of variables
$\alpha_{1}=\sum_{i=1}^{km_{n}}a_{i}u_{i},\alpha_{2}=a_{2},\dots,\alpha_{km_{n}}=a_{km_{n}}$,
we obtain

$\displaystyle L_{n}$ $\displaystyle=$
$\displaystyle\int_{\mathbb{C}^{km_{n}-1}}\int_{|\alpha_{1}|<e^{-n\epsilon}}\frac{1}{|u_{1}|^{2}}\phi(\frac{\alpha_{1}-\sum_{i=2}^{km_{n}}\alpha_{i}u_{i}}{u_{1}})\phi(\alpha_{2})\dots\phi(\alpha_{km_{n}})dm(\alpha_{1})\dots
dm(\alpha_{km_{n}})$ $\displaystyle\leq$ $\displaystyle Ckm_{n}\pi
e^{-2n\epsilon}$

where $|\phi(z)|\leq C$. ∎

We will also need the following deterministic result on currents (cf. [10,
Proposition 3.2]). For the reader’s convenience, we include the elementary
proof.

######  Proposition 4.4.

Let $\\{u_{j}\\}$ and $u$ be psh functions on a domain
$\Omega\subset\mathbb{C}^{d}$. We also let $\\{T_{j}\\}$ and $T$ be positive
closed currents of bidegree $(p,p)$ on $\Omega$. Assume that

  1. 1.

$u_{j}$ and $u$ are locally uniformly bounded above;

  2. 2.

$u_{j}\to u$ in $L^{1}_{loc}(\Omega)$;

  3. 3.

$T_{j}\to T$ in the sense of currents;

  4. 4.

$u_{j}T_{j}$ has locally uniformly bounded mass on $\Omega$.

Then every weak limit $S$ of $u_{j}T_{j}$ satisfies $S\leq uT$, i.e., $uT-S$
is a positive current of bidegree $(p,p)$.

###### Proof.

Given a smooth, positive form $\chi$ of compact support $X\subset\Omega$ and
bidegree $(d-p,d-p)$, by slight abuse of notation we must show that
$S\wedge\chi\leq uT\wedge\chi$. Let $f$ be a continuous function on $X$ with
$f\geq u$ on $X$. By Hartogs’ lemma,

$\limsup_{j\to\infty}[\sup_{X}(u_{j}-f)]\leq\sup_{X}(u-f).$

Thus for $\epsilon>0$, for $j\geq j_{0}(\epsilon)$, $u_{j}\leq f+\epsilon$ on
$X$. Since $\chi$ and $T_{j}$ are positive,

$u_{j}T_{j}\wedge\chi\leq(f+\epsilon)T_{j}\wedge\chi$

so that $S\wedge\chi\leq(f+\epsilon)T_{j}\wedge\chi$. Since $f\geq u$ and
$\epsilon>0$ are arbitrary, the result follows.

∎

We now give the proof of Theorem 4.1.

###### Proof.

We first make a general observation about sequences
$\\{\frac{1}{n}\log|H_{n}|\\}$ where $\\{H_{n}\\}$ are random polynomials
associated to a regular compact set $K$ in our setting. Noting that for any
$w\in\mathbb{C}^{d}$, the vector

$u_{w}:=(\frac{p_{1}(w)}{\sqrt{B_{n}(w)}},...,\frac{p_{m_{n}}(w)}{\sqrt{B_{n}(w)}})$

is a unit vector in $\mathbb{C}^{m_{n}}$ where
$B_{n}(w):=\sum_{j=1}^{m_{n}}|p_{j}(w)|^{2}$ as in (2.2), and that

$\frac{1}{n}\log|H_{n}(w)|=\frac{1}{n}\log|\langle
a^{(n)},u_{w}\rangle|+\frac{1}{2n}\log B_{n}(w)$ (4.1)

where $a^{(n)}=\\{a_{j}\\}_{j=1,...,m_{n}}$, since the sequence
$\\{\frac{1}{2n}\log B_{n}\\}$ is locally uniformly bounded above (cf., (2.3)
in the asymptotically Chebyshev case), from Lemma 4.2, almost surely the
sequences $\\{\frac{1}{n}\log|H_{n}|\\}$ are locally uniformly bounded above.

The case $k=1$: It is enough to show that almost surely
$\frac{1}{n}\log|H_{n}(z)|\to V_{K}$ in $L^{1}_{loc}(\mathbb{C}^{d}).$ Note
that by Lemma 4.2 for each $\epsilon>0$ there exists a random constant $C$
such that almost surely

$\sum_{j=1}^{m_{n}}|a_{j}|^{2}\leq C^{2}m_{n}e^{2\sqrt[d]{\epsilon m_{n}}}.$

Using $m_{n}=O(n^{d})$ and Cauchy-Schwarz we deduce that there exists a set
$\mathcal{A}\subset\mathcal{H}^{1}=\mathcal{H}$ of probability one such that
for every sequence $\\{H_{n}\\}\in\mathcal{A}$ we have

$\displaystyle\limsup_{n\in I}\frac{1}{n}\log|H_{n}(z)|$ $\displaystyle=$
$\displaystyle\limsup_{n\in
I}\big{(}\frac{1}{n}\log\big{(}\frac{|H_{n}(z)|}{\sqrt{B_{n}(z)}}\big{)}+\frac{1}{2n}\log
B_{n}(z)\big{)}$ $\displaystyle\leq$
$\displaystyle\limsup_{n\to\infty}\big{(}\frac{1}{2n}\log(\sum_{j=1}^{m_{n}}|a_{j}|^{2})+\frac{1}{2n}\log
B_{n}(z)\big{)}$ $\displaystyle\leq$ $\displaystyle\epsilon^{1/d}+V_{K}(z)$

on $\mathbb{C}^{d}$ (recall $m_{n}=0(n^{d})$). Hence
$(\displaystyle\limsup_{n\in I}\frac{1}{n}\log|H_{n}(z)|)^{*}\in
L(\mathbb{C}^{d})$ and

$(\limsup_{n\in I}\frac{1}{n}\log|H_{n}(z)|)^{*}\leq V_{K}(z)\ \text{for}\
z\in\mathbb{C}^{d}.$ (4.2)

Now by Proposition 2.2 given any subsequence $Y\subset\mathbb{N}$, there is a
further subsequence $Y_{0}\subset Y$ and a countable dense set of points
$\\{w_{r}\\}$ in $\mathbb{C}^{d}$ such that

$\lim_{n\to\infty,\ n\in Y_{0}}\frac{1}{2n}\log B_{n}(w_{r})=V_{K}(w_{r}),\
r=1,2,...$

Applying (4.1) to $w_{r}\in\mathbb{C}^{d}$,

$\frac{1}{n}\log|H_{n}(w_{r})|=\frac{1}{n}\log|\langle
a^{(n)},u_{w_{r}}\rangle|+\frac{1}{2n}\log B_{n}(w_{r})$

where $a^{(n)}=\\{a_{j}\\}_{j=1,...,m_{n}}$. By the $m=1$ case of Lemma 4.3
and Borel-Cantelli, with probability one in $\mathcal{H}$,

$\liminf_{n\to\infty,\ n\in Y_{0}}\frac{1}{n}\log|H_{n}(w_{r})|\geq
V_{K}(w_{r}),\ r=1,2,...$ (4.3)

The assertion follows from Proposition 2.4.

The case $k>1$: In order to prove the equidistribution result in this higher
codimension case we apply induction on the codimension $k$. We assume that the
assertion holds for $k-1$. Thus there exists a set
$\mathcal{A}^{k-1}\subset\mathcal{H}^{k-1}$ of probability one such that for
each
$\\{F_{n}^{k-1}=(H_{n}^{1},H_{n}^{2},\dots,H_{n}^{k-1})\\}_{n=1}^{\infty}\in\mathcal{A}^{k-1}$
we have

$T_{n}:=\frac{1}{n^{k-1}}Z_{F_{n}^{k-1}}\to dd^{c}V_{K_{1}}\wedge\cdots\wedge
dd^{c}V_{K_{k-1}}\ \text{as}\ n\to\infty$

in the sense of currents. Now for each
$\\{H_{n}^{k}\\}_{n=1}^{\infty}\in\mathcal{A}^{1}$ let us denote by
$u_{n}:=\frac{1}{n}\log|H_{n}^{k}|$. We claim that the sequence of currents
$u_{n}T_{n}$ has locally bounded mass on $\mathbb{C}^{d}$. To this end, let
$\beta:=dd^{c}|z|^{2}$ and
$\beta_{p}:=\frac{1}{p!}\beta\wedge\dots\wedge\beta$ ($p$ times) for
$p\in\\{1,...,d\\}$. We need to show that for each compact set
$C\subset\mathbb{C}^{d}$ the sequence

$\int_{C}u_{n}T_{n}\wedge\beta_{d-k+1}$

is almost surely uniformly bounded. Indeed,

$\displaystyle\int_{C}u_{n}T_{n}\wedge\beta_{d-k+1}$ $\displaystyle=$
$\displaystyle\frac{1}{n^{k-1}}\int_{C\cap V_{F_{n}^{k-1}}}u_{n}\beta_{d-k+1}$
(4.4) $\displaystyle\leq$ $\displaystyle\frac{1}{n^{k-1}}\int_{C\cap
V_{F_{n}^{k-1}}}M_{C}\beta_{d-k+1}$ (4.5) $\displaystyle\leq$ $\displaystyle
M_{C}\frac{1}{n^{k-1}}Vol_{2k-2}(C\cap V_{F_{n}^{k-1}})$ (4.6)

where $M_{C}$ comes from the (almost surely) uniform upper bound on
$\\{u_{n}\\}$ on $C$. On the other hand, since $\frac{1}{2n}\log B_{n}(z)\geq
0$, by Lemma 4.3 and the Borel-Cantelli Lemma we have

$\displaystyle\int_{C}u_{n}T_{n}\wedge\beta_{d-k+1}$ $\displaystyle=$
$\displaystyle\frac{1}{n^{k-1}}\int_{C\cap V_{F_{n}^{k-1}}}u_{n}\beta_{d-k+1}$
(4.7) $\displaystyle\geq$
$\displaystyle-\epsilon\frac{1}{n^{k-1}}Vol_{2k-2}(C\cap V_{F_{n}^{k-1}}).$
(4.8)

Thus, by Proposition 4.4 and induction hypothesis we conclude that every weak
limit $S$ of $u_{n}T_{n}$ satisfies $S\leq
V_{K_{k}}dd^{c}V_{K_{1}}\wedge\cdots\wedge dd^{c}V_{K_{k-1}}$ on
$\mathbb{C}^{d}$. But if $u_{n_{j}}T_{n_{j}}\to S$ as currents, then
$dd^{c}u_{n_{j}}\wedge T_{n_{j}}\to dd^{c}S$. Since the mass of
$dd^{c}u_{n_{j}}\wedge T_{n_{j}}$ is (almost surely) $1$, this in turn implies
that $dd^{c}S$ has mass equal to $1$. Since
$V_{K_{k}}dd^{c}V_{K_{1}}\wedge\cdots\wedge dd^{c}V_{K_{k-1}}-S$ is a positive
closed current and

$dd^{c}\bigl{(}V_{K_{k}}\wedge dd^{c}V_{K_{1}}\wedge\cdots\wedge
dd^{c}V_{K_{k-1}}\bigr{)}-dd^{c}S=dd^{c}V_{K_{1}}\wedge\cdots\wedge
dd^{c}V_{K_{k}}-dd^{c}S$

has mass zero, we deduce that $dd^{c}S=dd^{c}V_{K_{1}}\wedge\cdots\wedge
dd^{c}V_{K_{k}}$. Hence, the assertion follows. ∎

We end with some remarks on the hypotheses of Theorem 4.1.

  1. 1.

We made the assumption that the law of the random coefficients $a_{l}^{j}$ is
of the form $\phi(z)dm(z)$ where $\phi:\mathbb{C}\to\mathbb{R}$ is a bounded,
measurable function. This was used in Lemma 4.3 and also so that almost surely
the zero sets of $Z_{H^{j}_{n}}$ are smooth and intersect transversally. A
weaker hypotheses on $a_{l}^{j}$ would suffice for this latter property.

  2. 2.

Theorem 4.1 remains true in certain cases for independent random variables
which are not necessarily identically distributed. Some natural tail
probability and small ball assumptions suffice; we refer the reader to [4] and
the conditions (B1) and (B2) therein.

  3. 3.

We conjecture that in the codimension one case ($k=1$), the single hypothesis
(i), $\mathbb{E}[\big{(}\log(1+|a_{j}|\big{)}^{d}]<\infty$, is sufficient for
the conclusion that almost surely in $\mathcal{H}$ we have
$\frac{1}{n}dd^{c}\log|H_{n}|\to dd^{c}V_{K}\ \text{as}\ n\to\infty$. For
$d=2$ and the case of the bivariate Kac ensemble, this was proved in [6],
Theorem 7.7.

  4. 4.

A strong converse to Theorem 4.1 in $\mathbb{C}$ was given by Dauvergne in
[9]: for univariate random polynomials $H_{n}(z)=\sum_{j=0}^{n}a_{j}p_{j}(z)$
where $\\{p_{j}\\}$ are asymptotically minimal for $K\subset\mathbb{C}$ (see
Remark 2.3) and $\\{a_{j}\\}$ are nondegenerate i.i.d. random variables, if
$\mathbb{E}\big{(}\log(1+|a_{j}|\big{)}=\infty$, then the sequence
$\mu_{H_{n}}$ has no almost sure limit (in the space of probability measures
on $\mathbb{C}$). In $\mathbb{C}^{d}$ for $d\geq 3$ and $k=1$, some special
cases of a converse to Theorem 4.1 are known; see [3].

## References

  * [1] T. Bayraktar, Equidistribution of zeros of random holomorphic sections, _Indiana U. Math. J._ , 65, (2016), no. 5, 1759-1793.
  * [2] T. Bayraktar, Zero distribution of random sparse polynomials, _Mich. Math. J._ , 66, (2017), no. 2, 389-419.
  * [3] T. Bayraktar.  On global universality for zeros of random polynomials.  Hacet. J. Math. Stat., 48(2):384–398, 2019.
  * [4] T. Bayraktar, D. Coman, and G. Marinescu.  Universality results for zeros of random holomorphic sections.  Trans. Amer. Math. Soc., 373 (6): 3765–3791, 2020.
  * [5] T. Bloom, On families of polynomials which approximate the pluricomplex Green function, _Indiana U. Math. J._ , 50, (2001), no. 4, 1545-1566.
  * [6] T. Bloom and D. Dauvergne, Asymptotic zero distribution of random orthogonal polynomials, arXiv:1801.10125v2.
  * [7] T. Bloom and N. Levenberg, Random polynomials and pluripotential-theoretic extremal functions, _Potential Analysis_ , 42 (2015) no. 2, 311-334.
  * [8] T. Bloom and B. Shiffman, Zeros of random polynomials on $\mathbb{C}^{m}$, _Math. Res. Lett._ , 14 (2007), no. 3, 469-479.
  * [9] D. Dauvergne, A necessary and sufficient condition for convergence of the zeros of random polynomials, _Advances in Mathematics_ , 384 (2021), 1-33.
  * [10] J. E. Fornæss and N. Sibony, Oka’s inequality for currents and applications, emphMath. Ann., 301 (1995), no. 3, 399-419.
  * [11] P. Griffiths and J. Harris, _Principles of Algebraic Geometry_ , Wiley Classics Library, John Wiley & Sons, Inc., New York, 1994, reprint of the 1978 original.
  * [12] I. Ibragimov, and D. Zaporozhets, On distribution of zeros of random polynomials in complex plane, _Prokhorov and contemporary probability theory_ , Springer, 2013, pp. 303-323.
  * [13] Z. Kabluchko and D. Zaporozhets, Asymptotic distribution of complex zeros of random analytic functions, _Annals of Probability_ , 42, 4 (2014), 1374-1395.
  * [14] I. Pritsker and K. Ramachandran, Equidistribution of zeros of random polynomials, _J. Approx. Theory_ , 215, (2017), 106-117.
  * [15] B. Shiffman and S. Zelditch, Distribution of zeros of random and quantum chaotic sections of positive line bundles, _Comm. Math. Phys._ , 200, 3 (1999), 661-683.
  * [16] V. P. Zaharjuta, Transfinite diameter, Chebyshev constants, and capacity for compacta in $\mathbb{C}^{n}$, _Math. USSR Sbornik_ , 25 (1975), no. 3, 350-364.


# 2BP: 2-Stage Backpropagation

Christopher Rae
<EMAIL_ADDRESS>
EPCC, The University of Edinburgh
Joseph Lee
<EMAIL_ADDRESS>
EPCC, The University of Edinburgh
James Richings
<EMAIL_ADDRESS>
EPCC, The University of Edinburgh

###### Abstract

As Deep Neural Networks (DNNs) grow in size and complexity, they often exceed
the memory capacity of a single accelerator, necessitating the sharding of
model parameters across multiple accelerators. Pipeline parallelism is a
commonly used sharding strategy for training large DNNs. However, current
implementations of pipeline parallelism are being unintentionally bottlenecked
by the automatic differentiation tools provided by ML frameworks. This paper
introduces 2-stage backpropagation (2BP). By splitting the backward
propagation step into two separate stages, we can reduce idle compute time. We
tested 2BP on various model architectures and pipelining schedules, achieving
increases in throughput in all cases. Using 2BP, we were able to achieve a
1.70x increase in throughput compared to traditional methods when training a
LLaMa-like transformer with 7 billion parameters across 4 GPUs.

##  1 Introduction

In the era of big data and artificial intelligence, deep neural networks
(DNNs) have emerged as powerful tools for a wide range of applications, from
natural language processing to genomic sequencing. As these models grow in
size and complexity they can no longer fit within the memory of a single
accelerator, leading to a variety of novel techniques being introduced,
including 3D parallelism[29, 26, 17], Zero Redundancy Optimizer(ZeRO) [25] and
parameter offloading to host memory or NVMe storage [9, 13, 17, 27]. 3D
parallelism refers to the three main strategies for parallelising the
computation of DNNs: firstly, data parallelism, which refers to the technique
of dividing a dataset across multiple accelerators each with their own copies
of the model; pipeline parallelism, which distributes model layers across
multiple accelerators; and lastly tensor parallelism, which splits individual
model layers across multiple accelerators. While data parallelism does not
directly address the issue of increasing model sizes, it enhances processing
speed and can be used to reduce the memory required for activations by
reducing the local batch size for each accelerator. The last two forms of
parallelism fall under the umbrella of model parallelism, and are often
necessitated by models with sizes exceeding the capacity of a single
accelerator. In particular, tensor parallelism is particularly effective only
for extremely large models due to large amount communication overhead, which
results in inefficient resource utilization when applied to smaller models.

Focusing on pipeline parallelism, naive implementations often result in a long
proportion of idle compute time, which is caused by computational
dependencies. The ratio of idle compute to compute is often referred to as the
bubble ratio. To reduce the bubble ratio, a range of pipeline parallelism
algorithms have been developed [14, 8, 18, 11, 34, 35, 19]. Pipeline parallel
algorithms can be split into 2 categories: synchronous and asynchronous.
Synchronous algorithms require a flush at the end of each training step, in
order to keep all of the accelerators training on the same version of the
model weights. Asynchronous algorithms do not preform this flush, often
resulting in a smaller bubble ratio at the cost of not converging as well as
synchronous algorithms [11, 34, 35].

Deep neural networks are generally composed of a sequence of layers. The
output of each layer $l$ is a function of the parameters (e.g. weights and
biases) within that layer $w_{l}$ and the output of the previous layer
$z_{l-1}$, such that $z_{l}=f_{l}(z_{l-1},w_{l})$. The basic training of DNNs
involves three primary steps: forward propagation, used to calculate the
intermediate activations, backward propagation, where we calculate the
gradients of the model’s parameters and finally, the optimizer step which
updates the model’s parameters using the calculated gradient. In practice, the
backward propagation step is typically abstracted away by the machine learning
framework, and not directly programmed by the user. ML frameworks such as
PyTorch [2] and TensorFlow [1] use reverse mode automatic differentiation to
calculate the gradients, which often involves tracing the computation graph
produced by the forward pass at runtime and calculating the derivatives by
starting at the end of the graph and propagating to the beginning. The
intuition behind reverse mode automatic differentiation stems from the chain
rule. As we backward propagate through each layer in our neural network, two
calculations take place: the partial derivative of the loss with respect of
the parameters $\frac{\partial L}{\partial w_{l}}$ and the partial derivative
of the loss with respect to the input of the layer $\frac{\partial L}{\partial
z_{l-1}}$. Both operations depend on a combination of the activations
calculated in the forward propagation and the derivative of the loss with
respect to the output $\frac{\partial L}{\partial z_{l}}$. Both of these
calculations are handled by the ML framework under a single predefined
backward function. This approach works well in a single accelerator setting as
it is more memory efficient not having to store the intermediate derivatives
$\frac{\partial L}{\partial z_{l}}$ for later computation, but as we scale our
model and pipeline parallelism becomes necessary, the calculation of
$\frac{\partial L}{\partial w_{l}}$ is not immediately required, and can be
reordered such that it is calculated only after the backward propagation has
started on the preceding accelerator. The objective of this work is to apply
this splitting and reordering of the backward propagation step, and measure
the performance gained for a variety of model architectures.

##  2 Background & related work

A wide range of pipeline parallelism schedules have been introduced and
studied. The concept of a pipelining schedule was first introduced with GPipe
[14], a pipeline parallelism library developed by Google. GPipe introduced the
innovative idea of splitting mini-batches further into micro-batches, allowing
different accelerators to train on multiple micro-batches in parallel. This
approach effectively decreases the bubble ratio by allowing for overlapped
compute between accelerators, thereby enhancing the efficiency of the training
process.

The next prominent synchronous pipelining schedule to emerge was 1F1B (1
forward, 1 backward), sometimes referred to as PipeDream-Flush. This schedule
was originally proposed by PipeDream [11] as an asynchronous pipeline
schedule, but was later adapted for use in Megatron-LM [29] and DAPPLE [8] as
a synchronous pipelining schedule. Megatron-LM is a research-oriented
framework designed for large language model (LLM) training at scale, while
DAPPLE is a synchronous training framework that combines data parallelism and
pipeline parallelism for training large DNNs. It is also important to note
that DeepSpeed [26] independently developed a pipeline scheduling algorithm
that is equivalent to 1F1B. To further reduce the bubble ratio when using
1F1B, an interleaved pipelining schedule [29] can be used. We can use this
interleaved schedule to decrease the idle compute at the cost of an increase
in communication.

Chimera [18] is another noteworthy bidirectional pipeline scheduling
algorithm. This method involves storing two copies of the model across the
accelerators, which results in doubling the memory consumed by the model’s
weights. Despite this increase in memory usage and the need for a more complex
communication scheme, Chimera significantly reduces the bubble ratio by up to
50%, making it a highly efficient scheduling strategy.

There are a handful of libraries that have implemented tools to utilise
pipeline parallelism at runtime. FairScale [7] provides an implementation of
GPipe for PyTorch sequential modules. DeepSpeed offers their own
implementation of 1F1B also for PyTorch sequential modules. Megatron-LM allows
users to use 1F1B or 1F1B interleaved to train LLMs. Colossal-AI [17] (much
like Megatron-LM) offers 1F1B or 1F1B interleaved support for a wide range of
transformers.

At a similar time as this work is produced, Zero Bubble Pipeline Parallelism
[23] was introduced, which also separates the backward propagation step into 2
separate stages. The authors adds support for 2BP (described in later Section
3) on top of the linear layers in Megatron-LM as well as implementing a custom
pipelining schedules that uses 2BP to further reduce bubble time to
theoretically zero. Our work evaluates the performance of 2BP on a wide range
of model architectures and evaluates the scaling performance of the strategy.

##  3 2-Stage Backpropagation

###  3.1 Definitions

For each layer during back propagation, we define the computation of the
gradient w.r.t to the output of the preceding layer $\frac{\partial
L}{\partial z_{l-1}}$ as ‘backward-p1’ and computation of the gradient w.r.t
to the layer parameters $\frac{\partial L}{\partial w_{l}}$ as ‘backward-p2’.
We describe 2BP as the process of splitting up the backward pass into
backward-p1 and backward-p2, and delaying the computation of backward-p2 in
order to maximise accelerator utilization. 2BP can be applied on top of any
pipelining schedule, including Gpipe and 1F1B. Figure 1 displays the effects
of different pipelining schedules, with and without the use of 2BP.

###  3.2 Implementation

Our implementation is built on top of PyTorch. However, we do not use
PyTorch’s automatic differentiation engine (torch.autograd). This granted us
greater flexibility when implementing 2BP. We replaced torch.autograd by
manually implementing the backward pass operation for all of the modules
employed within the models we chose to benchmark. Each module has a forward
and a backward-p1 function; if that module contains parameters (equivalent to
torch.nn.Parameter) then it also has a backward-p2 function. We can simulate
the behaviour of torch.autograd by calling backward-p2 directly after a
backward-p1 call as we backward propagate through the compute graph, instead
of delaying the computation.

Figure 1: Pipelining schedules (Naive, Gpipe, 1F1B-1, 1F1B-2) with and without
2BP. This figure assumes that the time taken to compute the forward,
backward-p1 and backward-p2 passes are equal.

We test the effects of 2BP on 4 model architectures across 4 pipelining
schedules. The first model chosen was a transformer architecture [33] with 7
billion parameters (Transformer-7b). This model is based on the LLaMa [32] and
PaLM [3] architectures (steps including rotary embedding [30], SwiGLU MLP
[28], RMSNorm [36], no linear bias). Transformer-7b has a context length of
1024 and a model dimension of 4096. The second chosen model is another well-
known bidirectional transformer model, BERT-Large [5]. To expand the test set
beyond transformer models, a relatively new architecture, Mamba-1.4B [10] was
chosen. Lastly, to evaluate the performance on an architecture with a non-
uniform computational graph, we have chosen ResNet-152 [12], a convolutional
neural network (CNN) model. A model with a non-uniform compute graph refers to
a model who’s activations do not share a constant shape throughout the model,
transformers are an example of a model with a uniform compute graph as their
activations keep a constant shape at every block. By selecting these four
diverse models, we ensure a comprehensive assessment of 2BP across various
architectural designs and computational demands.

The 4 pipelining schedules we test are Gpipe, 1F1B-1, 1F1B-2 and a naive
approach with no scheduling algorithm and maximum bubble time. 1F1B is split
into 2 categories: 1F1B-1 which indicates that the mini-batch is split into
$N$ micro-batches where $N$ is the number of pipeline processes (also equal to
the number of accelerators), and 1F1B-2 which indicates that the mini-batch is
split into $2N$ micro-batches.

For our GPipe implementation, we delay the computation of backward-p2 until
all micro-batches have finished both forward and backward-p1 steps. We then
concatenate the activations and intermediate derivatives of all micro-batches
over the batch dimensions, meaning we only have to call backward-p2 once
rather than $N$ times as shown in Figure 2. For both 1F1B implementations
(without 2BP), every accelerator (except the last) has some idle time after
their backward-p1 calls. After the backward-p1 operation has been called for a
micro-batch, all the necessary intermediates have been calculated and stored
in memory for use in the computation of backward-p2 for that micro-batch. This
allows us to fill that idle time between backward-p1 calls with backward-p2
calls (see Figure 1 for a visual representation). Once all the backward-p1
steps have been called, we concatenate the remaining forward and backward-p1
intermediates over the batch dimension and compute backward-p2 the same way as
was done with GPipe. When working with models with non-uniform compute graphs
(such as CNNs), executing backward-p2 steps during idle time may not be
optimal, since the backward-p2 step may take longer than the original idle
time, resulting in excess idle time on a different accelerator.

Figure 2: Combining each microbatch’s backward-p2 into a single operation.

Table 1 shows the theoretical bubble ratios as functions of $N$ where $N$ is
the number of pipeline processes. Bubble ratio is defined as the fraction of
total runtime which is idle, i.e. idle time divided by the total runtime.
These functions assume that the forward, backward-p1 and backward-p2 functions
take equal lengths of time, which does not hold true in practice but acts as a
good estimate for comparing the performance scaling for the different
schedules as the number of processes varies. The throughput gain is the
speedup of the total runtime from using and not using 2BP, and can be obtained
from $\frac{1-b}{1-a}$, where $b$ is the bubble ratio with 2BP and $a$ is the
bubble ratio without.

Table 1: Bubble ratios and throughput gains for each pipelining schedule. Equal time for forward, backward-p1, and backward-p2 assumed. Pipelining Schedule | Bubble Ratio | 2BP Bubble Ratio | Throughput Gain
---|---|---|---
Naive | $\frac{N-1}{N}$ | $\frac{2(N-1)}{2N+1}$ | $\frac{3N}{2N+1}$
GPipe | $\frac{N-1}{2N-1}$ | $\frac{2(N-1)}{2(N-1)+3N}$ | $\frac{3(2N-1)}{2(N-1)+3N}$
1F1B-1 | $\frac{N-1}{2N-1}$ | $\frac{N-1}{N-1+3N}$ | $\frac{3(2N-1)}{N-1+3N}$
1F1B-2 | $\frac{N-1}{3N-1}$ | $\frac{N-1}{N-1+6N}$ | $\frac{3(3N-1)}{N-1+6N}$

We also used torch.jit.script to compile the backward-p1 operations for both
softmax and RMSNorm, resulting in significant speed up for each. We do not
include this pre-training compilation step in our timing measurements, since
this is required whether 2BP is in place or not. The data collected came from
training on randomly generated data. This is done instead of using an actual
dataset since from our experience with the system, dataloading can be a
significant bottleneck and optimising dataloading is beyond the scope of this
paper. All of the throughput results were collected from training on randomly
generated data samples of size 16384 and averaged over 4 epochs.

##  4 Evaluation

Our experiments were performed on two systems at EPCC: the Edinburgh
International Data Facility (EIDF) GPU-Service [6, 22] and Cirrus [4]. Most of
the results were collected from the GPU nodes on the EIDF GPU-service
containing 4 40GB Nvidia A100 connected via SXM4 interconnect, and the Mamba
throughput results were collected on nodes with 4 80GB Nvidia A100s connected
via SXM4 interconnect. Cirrus supports multi-node GPU jobs, which is
particularly useful for scaling tests. Cirrus GPU nodes contains 4 16GB Nvidia
V100 GPUs connected via an SXM2 interconnect. For our PyTorch implementation,
since autograd is not used, we are able to perform our runs with PyTorch’s
inference mode activated. PyTorch’s distributed library is used for p2p
communication with a NCCL [31] backend.

Table 2 summarises the model architecture used for our experiments. Although
the optimizer [16, 20] chosen for each benchmark is not relevant for this
paper, the optimizer calculations are taken into account during the throughput
measurements. The naive implementation does not use micro-batches but uses a
mini-batch of $4\times$micro-batch size. When training ResNet152, in order to
keep the batch normalization [15] computation equal between pipeline schedule,
the naive implementation uses a mini-batch size of 8 and 4 gradient
accumulation steps. All models with the exception of ResNet152 distributed the
number of blocks equally amongst the 4 GPUs (excluding the embedding blocks
and prediction heads where appropriate). ResNet152 contains 50 ResNet
bottlenecks which were split in the ratio [10, 14, 14, 12] across the 4 GPUs
respectively. ResNet152 also contains some initial convolutions computed by
GPU 0 and a classification head computed by GPU 3. The loss is always handled
by GPU 3 as it computes the model’s output.

Table 2: Model hyperparameters used for benchmarking. Model | Data type | Micro-Batch size | Optimizer
---|---|---|---
Mamba-1.4b | fp16 | 2 | AdamW
LLaMa-7b | fp16 | 1 | Adam
ResNet152 | fp32 | 8 | SGD
BERT-Large | fp16 | 2 | Adam

###  4.1 2BP throughput

Figure 3: Sample throughput for each model with different pipeline schedules.
Light blue bars represent schedule without 2BP, and dark blue represents with
2BP. Numbers above bars represent throughtput gain from using 2BP.

Figure 3 summarises the throughput results for different models with different
pipeline schedules, with and without 2BP. Speedup can be observed for every
case with the use of 2BP, ranging from 1.10x for 1F1B-1 on ResNet152 to up to
1.70x for 1F1B-1 on Transformer-7b. The observed difference in performance
gain for different models may be caused by multiple factors, including the
non-uniformity of the ResNet152 model results in unwanted idle compute time,
as well as the smaller problem size compared to models like Transformer-7b,
this would also explain the smaller performance gain of BERT-Large at 1.26x.

The time taken for the backward-p1 step compared to backward-p2 step is
usually uneven, and depends heavily on the model architecture. For example,
for 2D batch normalization, the backward-p2 operation is significantly simpler
than the backward-p1 operation. Furthermore, some operations, e.g. the scalar
dot-product attention, do not require a backward-p2 operation but have a
significant backward-p1 operation. This difference in computational complexity
between backward-p1 and backward-p2 may cause the variation of performance
gain observed between model architectures.

###  4.2 2BP memory consumption

Figure 4: Maximum memory usage across the 4 GPUs. The "peak memory (GB)" is
measured by obtaining the peak reserved memory on each GPU and taking then the
maximum. Light blue bars represent schedule without 2BP, and dark blue
represents with 2BP. Numbers above bars represent the increase in memory from
using 2BP.

While 2BP is able to reduce idle computation time, it comes at the cost an
increase in peak memory usage. Figure 4 summarises the effect on peak GPU
memory usage (i.e. the GPU which reserved the maximum memory) for each model
and pipeline schedule, with and without 2BP. It can be seen that the increase
in memory consumption varies significantly between the model architectures but
also the pipeline schedules. In Mamba with the 1F1B-2 pipelining schedule, we
see an increase in memory of 2.67x; on the other hand, for training
Transformer-7b with the 1F1B-1 pipelining schedule, the memory consumption
only increases by 1.02x.

The increase in memory is a result of 2 different characteristics of 2BP:
first the intermediate derivatives $\frac{\partial L}{\partial z_{l}}$ have to
be stored in memory after backward-p1 for reuse in backward-p2. Second, the
activations that are needed for the delayed backward-p2 calls are held for
longer. The GPU with the largest memory consumption is dependent on the
pipelining schedule being used and the uniformity of the models compute graph.

The specific operations used in a model can result in a significant increase
in peak memory; for Linear and Convolution layers, both the input activations
and output derivatives need to be stored in memory for backward-p2. On the
other hand, operations that are purely functional such as ReLU and Scalar Dot-
Product Attention release their activations during the backward-p1 calls.

Comparing the memory footprint for different pipelining schedules, Gpipe
requires all process to save the activations of all the micro-batches for the
backward pass, resulting in a large memory footprint for each device. For
1F1B-1, without the use of 2BP, GPU 0 will always have the largest activation
memory since it has to store the activations for $N$ micro-batches; with 2BP,
the GPU with the greatest activation memory is now dependent on the model
architecture. Although GPU 0 has the activations of all micro-batches saved in
memory, it only ever has to store $1$ micro-batch worth of intermediate
derivatives. GPU $N-1$ has to store $N$ micro-batches worth of intermediate
derivatives. The peak memory required by activations is dependent on how many
activations are released after backward-p1.

The reason for the large increase in peak memory usage for 1F1B-2 as seen by
the is the fact that the majority of the activations and intermediate
derivatives need to be held in memory until backward-p2 is called. Although we
did not implement this, by calling backward-p2 on some of the micro-batches
halfway through the 1F1B-2 training step, as visualised in Figure 5, it may be
possible to recover the peak memory usage closer to that of 1F1B-1. As further
work to this study, we would like to investigate how the frequency at which
backward-p2 is called during a training step affects performance and memory
consumption, especially as we scale to versions of 1F1B with greater numbers
of micro-batches e.g. $8N$ micro-batches.

Figure 5: Alternative memory efficient schedule for 1F1B-2 with 2BP

###  4.3 Scaling

In this section we look into the effects of scaling on 2BP. Table 1 shows that
as the number of accelerators $N$ increases, the bubble ratio increases. At
the same time the performance gain between training with and without 2BP
should increase since the rate at which the bubble ratio increases is greater
when not using 2BP. Using a BERT-like model, we test the effect of scaling
when using 1F1B-1 and 1F1B-2 as they give the greatest throughput. Both
implementations use a micro-batch size of 2.

####  4.3.1 Fixed model size

Figure 6: Scaling the number of GPUs with a fixed model size. Light blue bars
represent schedule without 2BP, and dark blue represents with 2BP. Numbers
above bars represent throughtput gain from using 2BP.

We use a BERT-like model with 32 blocks to perform a scaling test with fixed
model size. Figure 6 summarises the effect of 2BP with varying number of
accelerators. For 1F1B-1, the performance increase from 2BP goes from 1.21x
for 4 GPUs, to 1.20x for 8 GPUs, and 1.18x for 16 GPUs. For 1F1B-2, the gain
is 1.15x, 1.14x and 1.11x respectively. Even though the theoretical prediction
from Table 1 suggests that the throughput gain should increase as the number
of accelerators increase, the formula did not take into account the increase
in communication required, especially when going above 4 GPUs (on our system)
requires inter-node communication.

####  4.3.2 Variable model size

Figure 7: Scaling the number of GPUs with a variable global model size. Light
blue bars represent schedule without 2BP, and dark blue represents with 2BP.
Numbers above bars represent throughtput gain from using 2BP.

Here we test the performance as we scale the model size with the number of
processors; each accelerator computes 8 BERT-like blocks. From Figure 7, we
see that for 1F1B-1, the performance gains are 1.28x, 1.24x, and 1.23x for 4,
8, and 16 GPUs. For 1F1B-2 they are 1.13x and 1.10x for 4 and 8 GPUs, whereas
the 16 GPU run resulted in an out of memory error. This error is caused by
storing the activations and intermediate derivatives of 16 micro-batches on
GPU $N-1$. As stated in the previous section, the performance gain should in
theory increase with the number of GPUs; however we observe a similar
degradation in performance gain, which is most likely due to the increase in
communication required.

###  4.4 GPU compute occupancy

As mentioned in section 3, micro-batches are concatenated (copied into
contiguous memory location) across their batch dimensions during the compute
of backward-p2. In theory this should allow for a greater utilization of
compute resources when working with GPUs due to their SIMT architecture. We
tested the effect of this concatenation, and the results are summarised in
Table 3. In practice, we did not observe a significant difference in whether
the concatenation was performed, or if the $N$ backward-p2 steps were simply
performed in a loop (see Figure 2). It is possible that concatenation does
provide speed up, but the concatenation step itself is time consuming and
neutralises the benefits.

As the necessary inputs for all backward-p2 operations across the entire model
exist in memory at the time of calling the first backward-p2 operation, in
theory all backward-p2 operations can be called in parallel. The easiest way
to perform this on Nvidia GPUs is via CUDA streams. PyTorch provides an API to
use CUDA streams, but we observed that it drastically increased the compute
time of the backward-p2 operations, to the point at which it was faster to
call each operation in serial. CUDA graphs may be a alternative method to
achieve further parallelisation during backward-p2 calls, which we intend to
explore in the future.

Table 3: Average throughput using the 1F1B-1 pipelining schedule and 2BP, with and without concatenating micro-batches during the backward-p2 step. Model | Avg Throughput w/ | Avg Throughput w/o
---|---|---
Transformer-7b | 7120.88 | 7100.69
Bert-Large | 40427.41 | 40387.41
Mamba-1.4b | 12437.91 | 12431.13
ResNet152 | 194.93 | 193.10

##  5 Further work

As stated previously, 2BP can cause an increase in activation memory. Here we
propose potential methods to decrease the effects of the increase in
activation memory, which we will investigate in the future. Firstly, by
performing intermediate derivative checkpointing, which would work similarly
to activation checkpointing [21] in which some activations are not stored in
memory but recomputed during the backward pass, but applied to the
intermediate derivates. Activation checkpointing is a widely used practice for
training DNNs. The intermediate derivative recalculations could potentially be
overlapped with idle compute to result in minimal performance decreases
compared to running 2BP without intermediate derivative checkpointing.

Another potential way to decrease memory consumption caused by storing the
intermediate derivatives would be to temporarily store the intermediates from
the first few micro-batches into either host memory or NVMe storage [24, 27],
and the number of micro-batches to offload would depend on pipelining schedule
and GPU rank.

When training large DNNs, pipeline parallelism is usually not the sole
distributed parallelism paradigm employed. Data parallelism is very often used
along side pipeline parallelism, and data parallelism can be optimised by
overlapping communication of the gradients with computation during the
backward propagation through the network. As the gradients are not calculated
until the delayed computation during backward-p2, we expect it will be
significantly harder to fully overlap communication with computation,
especially when working on systems with slower communication infrastructure.
We aim to explore this in the future.

##  6 Conclusion and discussion

For this work, we have applied 2BP, which splits the backward propagation step
into 2 stages, to training 4 different DNN model architectures. We have
demonstrated that this is able to reduce the idle compute time, resulting in
significant increase in throughput on top of the SOTA pipelining schedules.

As part of our implementation of 2BP, the backward pass operations had to be
manually implemented; these operations have been implemented by PyTorch but
are not exposed to be used functionally though PyTorch’s Python API. The
authors would like to see this functionality exposed to users, allowing for
easier and more granular control over the implementation of custom backward
propagation methods such as the one demonstrated in this work.

## Acknowledgments and Disclosure of Funding

This work used the Cirrus UK National Tier-2 HPC Service at EPCC
(http://www.cirrus.ac.uk) funded by the University of Edinburgh and EPSRC
(EP/P020267/1). This work was supported by the Edinburgh International Data
Facility (EIDF) and the Data-Driven Innovation Programme at the University of
Edinburgh. For the purpose of open access, the author has applied a Creative
Commons Attribution (CC BY) licence to any Author Accepted Manuscript version
arising from this submission.

## References

## References

  * [1] Martín Abadi et al. “TensorFlow, Large-scale machine learning on heterogeneous systems”, 2015 DOI: 10.5281/zenodo.4724125
  * [2] Jason Ansel et al. “PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation” In _29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24)_ ACM, 2024 DOI: 10.1145/3620665.3640366
  * [3] Aakanksha Chowdhery et al. “PaLM: Scaling Language Modeling with Pathways”, 2022 arXiv:2204.02311 [cs.CL]
  * [4] “Cirrus Hardware — cirrus.ac.uk” [Accessed 26-03-2024], https://www.cirrus.ac.uk/about/hardware.html, 2022
  * [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2019 arXiv:1810.04805 [cs.CL]
  * [6] “Edinburgh International Data Facility — edinburgh-international-data-facility.ed.ac.uk” [Accessed 22-03-2024], https://edinburgh-international-data-facility.ed.ac.uk/, 2022
  * [7] FairScale authors “FairScale: A general purpose modular PyTorch library for high performance and large scale training”, https://github.com/facebookresearch/fairscale, 2021
  * [8] Shiqing Fan et al. “DAPPLE: A Pipelined Data Parallel Approach for Training Large Models”, 2020 arXiv:2007.01045 [cs.DC]
  * [9] Jiarui Fang et al. “Parallel Training of Pre-Trained Models via Chunk-Based Dynamic Memory Management” In _IEEE Transactions on Parallel and Distributed Systems_ 34.1 Institute of ElectricalElectronics Engineers (IEEE), 2023, pp. 304–315 DOI: 10.1109/tpds.2022.3219819
  * [10] Albert Gu and Tri Dao “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”, 2023 arXiv:2312.00752 [cs.LG]
  * [11] Aaron Harlap et al. “PipeDream: Fast and Efficient Pipeline Parallel DNN Training”, 2018 arXiv:1806.03377 [cs.DC]
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun “Deep Residual Learning for Image Recognition”, 2015 arXiv:1512.03385 [cs.CV]
  * [13] Haichen Huang et al. “Elixir: Train a Large Language Model on a Small GPU Cluster”, 2023 arXiv:2212.05339 [cs.DC]
  * [14] Yanping Huang et al. “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism”, 2019 arXiv:1811.06965 [cs.CV]
  * [15] Sergey Ioffe and Christian Szegedy “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, 2015 arXiv:1502.03167 [cs.LG]
  * [16] Diederik P. Kingma and Jimmy Ba “Adam: A Method for Stochastic Optimization”, 2017 arXiv:1412.6980 [cs.LG]
  * [17] Shenggui Li et al. “Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training”, 2023 arXiv:2110.14883 [cs.LG]
  * [18] Shigang Li and Torsten Hoefler “Chimera: efficiently training large-scale neural networks with bidirectional pipelines” In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , SC ’21 ACM, 2021 DOI: 10.1145/3458817.3476145
  * [19] Ziming Liu, Shenggan Cheng, Haotian Zhou and Yang You “Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency” In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , SC ’23 ACM, 2023 DOI: 10.1145/3581784.3607073
  * [20] Ilya Loshchilov and Frank Hutter “Decoupled Weight Decay Regularization”, 2019 arXiv:1711.05101 [cs.LG]
  * [21] E. Naveen and P. Kumar “Checkpointing in Practice for Memory-Efficient Training on the Edge” In _2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS)_ , 2019, pp. 2759–2766 DOI: 10.1109/HPCC/SmartCity/DSS.2019.00387
  * [22] “Overview - EIDF User Documentation — epcced.github.io” [Accessed 26-03-2024], https://epcced.github.io/eidf-docs/services/gpuservice/, 2022
  * [23] Penghui Qi, Xinyi Wan, Guangxing Huang and Min Lin “Zero Bubble Pipeline Parallelism”, 2023 arXiv:2401.10241 [cs.DC]
  * [24] Samyam Rajbhandari et al. “ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning”, 2021 arXiv:2104.07857 [cs.DC]
  * [25] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase and Yuxiong He “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models”, 2020 arXiv:1910.02054 [cs.LG]
  * [26] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase and Yuxiong He “DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters” In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’20 Virtual Event, CA, USA: Association for Computing Machinery, 2020, pp. 3505–3506 DOI: 10.1145/3394486.3406703
  * [27] Jie Ren et al. “ZeRO-Offload: Democratizing Billion-Scale Model Training”, 2021 arXiv:2101.06840 [cs.DC]
  * [28] Noam Shazeer “GLU Variants Improve Transformer”, 2020 arXiv:2002.05202 [cs.LG]
  * [29] Mohammad Shoeybi et al. “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism”, 2020 arXiv:1909.08053 [cs.CL]
  * [30] Jianlin Su et al. “RoFormer: Enhanced Transformer with Rotary Position Embedding”, 2023 arXiv:2104.09864 [cs.CL]
  * [31] “The NVIDIA Collective Communication Library (NCCL)” [Accessed 27-03-2024], https://github.com/NVIDIA/nccl, 2022
  * [32] Hugo Touvron et al. “LLaMA: Open and Efficient Foundation Language Models”, 2023 arXiv:2302.13971 [cs.CL]
  * [33] Ashish Vaswani et al. “Attention Is All You Need”, 2023 arXiv:1706.03762 [cs.CL]
  * [34] Bowen Yang et al. “PipeMare: Asynchronous Pipeline Parallel DNN Training”, 2020 arXiv:1910.05124 [cs.DC]
  * [35] PengCheng Yang et al. “Group-based Interleaved Pipeline Parallelism for Large-scale DNN Training” In _International Conference on Learning Representations_ , 2022 URL: https://openreview.net/forum?id=cw-EmNq5zfD
  * [36] Biao Zhang and Rico Sennrich “Root Mean Square Layer Normalization”, 2019 arXiv:1910.07467 [cs.LG]