# Paired operators and paired kernels

M. Cristina Câmara  Center for Mathematical Analysis, Geometry and Dynamical
Systems, Instituto Superior Técnico, Universidade de Lisboa, Av. Rovisco Pais,
1049-001 Lisboa, Portugal<EMAIL_ADDRESS>André Guimarães Instituto
Superior Técnico, Universidade de Lisboa, Av. Rovisco Pais, 1049-001 Lisboa,
Portugal<EMAIL_ADDRESS>Jonathan R. Partington School of
Mathematics, University of Leeds, Leeds LS2 9JT, U.K.
<EMAIL_ADDRESS>
###### Abstract

This paper is concerned with paired operators in the context of the Lebesgue
Hilbert space on the unit circle and its subspace, the Hardy space. By
considering when such operators commute, generalizations of the Brown–Halmos
results for Toeplitz operators are derived. Further, the kernels of such
operators are described, giving results on invariant and nearly-invariant
subspaces, together with a generalization of Coburn’s theorem on Toeplitz
kernels.

Keywords: paired operator; paired kernel; Toeplitz operator; invariant
subspace; nearly-invariant subspace; kernel; Brown–Halmos theorem; Coburn’s
theorem.

MSC (2010): 30H10, 47B35, 47B38

##  1 Introduction

Let $X$ be a Hilbert space and $P,Q$ complementary orthogonal projections on
$X$. For $A,B$ bounded linear operators on $X$ we define the paired operators
$S_{A,B}=AP+BQ$ and $\Sigma_{A,B}=PA+QB$ (see [15]). These first appeared in
the context of singular integral equations, and have been the object of
renewed interest because of the recently-established connection with problems
in diffraction theory [17], dual truncated Toepliz operators [4], and the
characterization of block Toeplitz operators whose kernels present a scalar-
type structure [7, Cor. 3.4].

Here we work in the particular context of $L^{2}(\mathbb{T})$, where
$\mathbb{T}$ is the unit circle, and $P$ is $P^{+}$, the orthogonal projection
onto the Hardy space $H^{2}$ of the unit disc and $Q$ is $P^{-}$, the
complementary projection onto $(H^{2})^{\perp}=\overline{H^{2}_{0}}$. The
operators $A$ and $B$ are multiplication operators $A=M_{a}$ and $B=M_{b}$
with $a,b\in L^{\infty}(\mathbb{T})$. This is the most common setting where
these operators are studied, and also that with the most applications. They
have been studied usually under an extra condition that either $a$ or $b$ is
invertible in $L^{\infty}(\mathbb{T})$ or, at most, has zeros of “integer
order” at a finite set of points of $\mathbb{T}$ (see [15, Sec. VI.8] for a
more precise definition).

Paired operators are also closely connected with Toeplitz operators: in fact,
they are dilations of Toeplitz operators. Moreover, it can be shown that if
$b^{\pm 1}\in L^{\infty}(\mathbb{T})$, then the paired operator
$aP^{+}+bP^{-}$ on $L^{2}(\mathbb{T})$ is equivalent after extension (as
defined in [1]) to the Toeplitz operator $T_{a/b}$ on $H^{2}$ (see [3]), and
an analogous result can be obtained for dual Toeplitz operators if $a^{\pm
1}\in L^{\infty}(\mathbb{T})$. Thus it is natural to ask whether various well
known properties of Toeplitz operators have analogues for paired operators.

In this paper we explore new connections of paired operators with Toeplitz and
Hankel operators. We study their algebraic properties, such as when two such
operators commute, providing a generalization of the Brown–Halmos results for
Toeplitz operators. Then we consider the kernels of paired operators: this
theme leads off in two directions, one of which concerns invariant and nearly-
invariant subspaces, while in the other direction we arrive at a
generalization of Coburn’s theorem on Toeplitz kernels.

##  2 Paired operators: basic properties

From now on we write $L^{p}$ to denote $L^{p}(\mathbb{T})$ for $p=2$ and
$p=\infty$.

Let $a,b\in L^{\infty}$. The paired operator $S_{a,b}:L^{2}\to L^{2}$ is
defined by

$S_{a,b}f=aP^{+}f+bP^{-}f,\qquad(f\in L^{2}).$ (2.1)

We call $\Sigma_{a,b}:L^{2}\to L^{2}$, defined by

$\Sigma_{a,b}f=P^{+}af+P^{-}bf,\qquad(f\in L^{2}),$ (2.2)

a transposed paired operator. We say that $S_{a,b}$ and $\Sigma_{a,b}$ are
dual to each other, while $S_{a,b}$ and $\Sigma_{b,a}$ are said to be
transposed to each other (see [15]).

######  Definition 2.1.

We say that a pair $\\{a,b\\}$ is nondegenerate if $a,b\in L^{\infty}$ are
nonzero a.e. on $\mathbb{T}$, and $a-b$ is too.

Nondegeneracy will be a common hypothesis in our work, although sometimes the
weaker condition that $a-b\neq 0$ will be sufficient.

Note that we identify $a$ with $M_{a}$, the operator of multiplication by $a$
whenever the context is clear. In particular, we write $P^{+}a+P^{-}b$ instead
of $P^{+}aI+P^{-}bI$.

Using the orthogonal decomposition $L^{2}=H^{2}\oplus\overline{H^{2}_{0}}$ we
may write down the matrix representation of $S_{a,b}$. For convenience, write
$P^{\pm}a=a_{\pm}$ and similarly for $b$. Now since for $h_{+}\in H^{2}$ we
have $S_{a,b}h_{+}=ah_{+}$, the matrix form of $S_{a,b}$ is a matrix of Hankel
and Toeplitz operators:

$S_{a,b}=\begin{pmatrix}a_{+}+P^{+}a_{-}&P^{+}b_{+}\\\
P^{-}a_{-}&P^{-}b_{+}+b_{-}\end{pmatrix}:H^{2}\oplus\overline{H^{2}_{0}}\to
H^{2}\oplus\overline{H^{2}_{0}}.$

Note that initially this matrix is only densely defined, but since it
corresponds to a bounded operator it can be extended to the whole of $L^{2}$.

These various operators are related as follows.

######  Proposition 2.2.

(i) $S_{a,b}^{*}=\Sigma_{\bar{a},\bar{b}}$, and
(ii)
$\overline{S_{a,b}f}=zS_{\bar{b},\bar{a}}\bar{z}\bar{f}=z\Sigma_{b,a}^{*}\bar{z}\bar{f}$
for all $f\in L^{2}$.

###### Proof.

(i) is clear; (ii) follows from the relations
$\overline{P^{\pm}f}=zP^{\mp}\bar{z}\bar{f}$. ∎

######  Proposition 2.3.

Let $M=\max(\|a\|_{\infty},\|b\|_{\infty})$. Then

$M\leq\|S_{a,b}\|\leq\min(\sqrt{2}M,\|a\|_{\infty}+\|b\|_{\infty}).$ (2.3)

###### Proof.

Restricting to $f\in H^{2}$, we have $S_{a,b}f=af$ and so
$\|S_{a,b}\|\geq\|T_{a}\|=\|a\|_{\infty}$, where $T_{a}$ is the usual Toeplitz
operator defined by $T_{a}f=P^{+}af$. Analogously,
$\|S_{a,b}\|\geq\|b\|_{\infty}$ and so $\|S_{a,b}\|\geq M$.

Next, $\|S_{a,b}\|\leq\|a\|_{\infty}+\|b\|_{\infty}$ is clear. Finally, from

$\|\phi\|_{2}^{2}=\|P^{+}\phi\|^{2}_{2}+\|P^{-}\phi\|_{2}^{2},\qquad(\phi\in
L^{2}),$

we know that for each such $\phi$ there is a $\theta\in[0,\pi/2]$ such that
$\|P^{+}\phi\|_{2}=\|\phi\|_{2}\cos\theta$ and
$\|P^{-}\phi\|_{2}=\|\phi\|_{2}\sin\theta$.

Thus

$\displaystyle\|S_{a,b}\phi\|_{2}$ $\displaystyle=$
$\displaystyle\|aP^{+}\phi+bP^{-}\phi\|_{2}$ $\displaystyle\leq$
$\displaystyle(\|a\|_{\infty}\cos\theta+\|b\|_{\infty}\sin\theta)\|\phi\|_{2}$
$\displaystyle\leq$
$\displaystyle\max\\{\|a\|_{\infty},\|b\|_{\infty}\\}(\cos\theta+\sin\theta)\|\phi\|_{2}\leq
M\sqrt{2}\|\phi\|_{2}.$

So we also have

$\|S_{a,b}\|\leq\sqrt{2}M,$ (2.4)

and the second inequality in (2.3) follows. ∎

As a consequence, we can characterise the zero paired operator:

######  Corollary 2.4.

$S_{a,b}=0$ if and only if $a=b=0$ in $L^{\infty}$.

######  Remark 2.5.

The estimates obtained in Proposition 2.3 are optimal. We have nontrivial
cases where $\|S_{a,b}\|=M$, e.g., if $a=b=1$, and cases where
$\|S_{a,b}\|=\sqrt{2}M$, e.g. if $a=1$ and $b=z$. In the latter case, taking
$\phi(z)=1+\dfrac{1}{z}$, we have $S_{a,b}\phi=2$ and $\|\phi\|=\sqrt{2}$, so
$\|S_{a,b}\|\geq\sqrt{2}$, and, from (2.4), $\|S_{a,b}\|=\sqrt{2}=\sqrt{2}M$.

######  Proposition 2.6.

$\|S_{a,b}\|=\|a\|_{\infty}+\|b\|_{\infty}$ if and only if either $a$ or $b$
is $0$ a.e.

###### Proof.

First if $b=0$ then $S_{a,b}\phi=aP^{+}\phi$ and since
$\|S_{a,0}\|\geq\|T_{a}\|=\|a\|_{\infty}$ (where $T_{a}$ is the Toeplitz
operator) it is clear that $\|S_{a,0}\|=\|a\|_{\infty}$ too. Similarly if
$a=0$.

For the converse, suppose that there is a $\phi\in L^{2}$ with
$\|\phi\|_{2}=1$ and for some $\epsilon$ with $0<\epsilon<1$ we have

$\|aP^{+}\phi+bP^{-}\phi\|\geq(1-\epsilon)(\|a\|_{\infty}+\|b\|_{\infty}).$

For what we are trying to prove we may assume that $a$ and $b$ are both
nonzero. There is a $\theta\in[0,\pi/2]$ such that
$\|P^{+}\phi\|_{2}=\cos\theta$ and $\|P^{-}\phi\|_{2}=\sin\theta$. Thus, with
$A=\|a\|_{\infty}$ and $B=\|b\|_{\infty}$ we have

$A\cos\theta+B\sin\theta\geq(1-\epsilon)(A+B),$

and hence by Cauchy–Schwarz

$A^{2}+B^{2}\geq(1-\epsilon)^{2}(A^{2}+2AB+B^{2})$

or $(2\epsilon-\epsilon^{2})(A^{2}+B^{2})\geq 2(1-\epsilon)^{2}AB$. Now
suppose that $A\geq B$ (the other way round is similar). Then we have

$2(2\epsilon-\epsilon^{2})A^{2}\geq 2(1-\epsilon)^{2}AB$

and so $B\leq(2\epsilon-\epsilon^{2})A/(1-\epsilon)^{2}$, and since we can
take $\epsilon>0$ arbitrary, this means that $B=0$. ∎

##  3 Brown–Halmos type theorems

Paired operators are closely related to Toeplitz operators in various ways.
For instance, Toeplitz operators $T_{a}$, with $a\in L^{\infty}$, can be seen
as compressions to $H^{2}$ of paired operators of the form
$S_{a,b}=aP^{+}+bP^{-}$ or their duals $\Sigma_{a,b}=P^{+}a+P^{-}b$, with
$b\in L^{\infty}$. Given that, by the Brown–Halmos theorems [2] we have for
$a,\tilde{a}\in L^{\infty}$ that

$T_{a}T_{\tilde{a}}=T_{a\tilde{a}}\iff a\in\overline{H^{\infty}}\hbox{ or
}\tilde{a}\in H^{\infty},$

one may ask, similarly, when the composition of two paired operators is a
paired operator or, more precisely, when the equations

$\displaystyle(aP^{+}+bP^{-})(\tilde{a}P^{+}+\tilde{b}P^{-})$ $\displaystyle=$
$\displaystyle a\tilde{a}P^{+}+b\tilde{b}P^{-}$ (3.1)
$\displaystyle(P^{+}a+P^{-}b)(P^{+}\tilde{a}+P^{-}\tilde{b})$ $\displaystyle=$
$\displaystyle P^{+}(a\tilde{a})+P^{-}(b\tilde{b})$ (3.2)

hold. It is clear that (3.1) and (3.2) hold if $a=b$ and $\tilde{a}=\tilde{b}$
respectively. Otherwise we have the following (recall that nondegeneracy was
defined in Definition 2.1).

######  Proposition 3.1.

Let $a,\tilde{a},b,\tilde{b}\in L^{\infty}$ and suppose that $\\{a,b\\}$ and
$\\{\tilde{a},\tilde{b}\\}$ are nondegenerate. Then

$\displaystyle S_{a,b}S_{\tilde{a},\tilde{b}}=S_{a\tilde{a},b\tilde{b}}$
$\displaystyle\iff$ $\displaystyle\tilde{a}\in H^{\infty}\hbox{ and
}\tilde{b}\in\overline{H^{\infty}}$ (3.3)
$\displaystyle\Sigma_{a,b}\Sigma_{\tilde{a},\tilde{b}}=\Sigma_{a\tilde{a},b\tilde{b}}$
$\displaystyle\iff$ $\displaystyle a\in\overline{H^{\infty}}\hbox{ and }b\in
H^{\infty}.$ (3.4)

###### Proof.

We have that

$(aP^{+}+bP^{-})(\tilde{a}P^{+}+\tilde{b}P^{-})=a\tilde{a}P^{+}+b\tilde{b}P^{-}+T,$

where

$\displaystyle T$ $\displaystyle=$ $\displaystyle-
aP^{-}\tilde{a}P^{+}+aP^{+}\tilde{b}P^{-}+bP^{-}\tilde{a}P^{+}-bP^{+}\tilde{b}P^{-}$
$\displaystyle=$
$\displaystyle(a-b)(P^{+}\tilde{b}P^{-}-P^{-}\tilde{a}P^{+}).$

If $\tilde{a}\in H^{\infty}$ and $\tilde{b}\in\overline{H^{\infty}}$ then
$T=0$. Conversely, suppose that $T=0$. Then, for all $f\in L^{2}$ we have
$P^{+}\tilde{b}P^{-}f=P^{-}\tilde{a}P^{+}f=0$ and therefore, taking $f=1$, we
have $P^{-}\tilde{a}=0$, i.e., $\tilde{a}\in H^{2}\cap L^{\infty}=H^{\infty}$.
Similarly, taking $f=\bar{z}$, we have $P^{+}\tilde{b}\bar{z}=0$, i.e.,
$\tilde{b}\bar{z}\in\overline{H^{2}_{0}}\cap\bar{z}L^{\infty}=\bar{z}\overline{H^{\infty}}$.

The equivalence in (3.4) is proved similarly, or by taking adjoints. ∎

It follows from Proposition 3.1 that
$S_{a,b}S_{\tilde{a},\tilde{b}}=S_{\tilde{a},\tilde{b}}S_{a,b}$ when
$a,\tilde{a}\in H^{\infty}$ and $b,\tilde{b}\in\overline{H^{\infty}}$, and
that
$\Sigma_{a,b}\Sigma_{\tilde{a},\tilde{b}}=\Sigma_{\tilde{a},\tilde{b}}\Sigma_{a,b}$
whenever $a,\tilde{a}\in\overline{H^{\infty}}$ and $b,\tilde{b}\in
H^{\infty}$. More generally, we have the following.

######  Proposition 3.2.

Let $a,\tilde{a},b,\tilde{b}\in L^{\infty}$ with
$\\{a,b\\},\\{\tilde{a},\tilde{b}\\}$ nondegenerate. Then

$S_{a,b}S_{\tilde{a},\tilde{b}}=S_{\tilde{a},\tilde{b}}S_{a,b}\iff(a-b)(P^{-}\tilde{a}P^{+}-P^{+}\tilde{b}P^{-})=(\tilde{a}-\tilde{b})(P^{-}aP^{+}-P^{+}bP^{-}).$

###### Proof.

$S_{a,b}S_{\tilde{a},\tilde{b}}-S_{\tilde{a},\tilde{b}}S_{a,b}$
$\displaystyle=-aP^{-}\tilde{a}P^{+}+aP^{+}\tilde{b}P^{-}$ $\displaystyle+$
$\displaystyle bP^{-}\tilde{a}P^{+}-bP^{+}\tilde{b}P^{-}$ $\displaystyle+$
$\displaystyle\tilde{a}P^{-}aP^{+}-\tilde{a}P^{+}bP^{-}\quad-\quad\tilde{b}P^{-}aP^{+}+\tilde{b}P^{+}bP^{-}$
$=-(a-b)(P^{-}\tilde{a}P^{+}-P^{+}\tilde{b}P^{-})-(\tilde{a}-\tilde{b})(P^{+}bP^{-}-P^{-}aP^{+}).$

∎

######  Corollary 3.3.

For $\eta\in L^{\infty}$ and $\\{a,b\\}$ nondegenerate,

$\eta S_{a,b}=S_{a,b}\eta\iff\eta\in\mathbb{C}.$

###### Proof.

$\eta I=S_{\eta,\eta}$, and applying the calculation in the proof of
Proposition 3.2 we have

$\eta S_{a,b}=S_{a,b}\eta\iff P^{-}\eta P^{+}=P^{+}\eta P^{-},$

which is true if and only if $\eta\in\mathbb{C}$. ∎

One may then ask for which functions $f\in L^{2}$ do multiplication by $\eta$
and $S_{a,b}$ commute.

######  Proposition 3.4.

For $\eta\in L^{\infty}$, $\\{a,b\\}$ nondegenerate, and $f\in L^{2}$,

$\displaystyle\eta S_{a,b}f=S_{a,b}(\eta f)$ $\displaystyle\iff$
$\displaystyle f\in\mathop{\rm ker}\nolimits H_{\eta}\oplus\mathop{\rm
ker}\nolimits\tilde{H}_{\eta}$ $\displaystyle\iff$ $\displaystyle\eta f_{+}\in
H^{2},\eta f_{-}\in\overline{H^{2}_{0}}$ $\displaystyle\iff$
$\displaystyle\eta f_{+}=P^{+}(\eta f)$ $\displaystyle\iff$ $\displaystyle\eta
f_{-}=P^{-}(\eta f),$

where the Hankel operators $H_{\eta}$ and $\tilde{H}_{\eta}$ are defined by
$H_{\eta}=P^{-}\eta P^{+}$ and $\tilde{H}_{\eta}=P^{+}\eta P^{-}$, and we
write $f_{\pm}=P^{\pm}f$.

###### Proof.

From Proposition 3.2 we have

$\eta S_{a,b}f=S_{a,b}(\eta f)\iff P^{-}\eta f_{+}=P^{+}\eta f_{-},$

which happens if and only if $P^{-}\eta f_{+}=P^{+}\eta f_{-}=0$, or
equivalently $\eta f_{+}\in H^{2}$ and $\eta f_{-}\in\overline{H^{2}_{0}}$.

Now $P^{+}(\eta f)=P^{+}(\eta f_{+})+P^{+}(\eta f_{-})$; if $\eta f_{+}\in
H^{2}$ and $\eta f_{-}\in\overline{H^{2}_{0}}$, then we have $P^{+}(\eta
f)=\eta f_{+}$.

Conversely, if $\eta f_{+}=P^{+}(\eta f)$ then $\eta f_{+}\in H^{2}$; on the
other hand, $\eta f_{+}=P^{+}(\eta f)=P^{+}(\eta f_{+})+P^{+}(\eta f_{-})=\eta
f_{+}+P^{+}(\eta f_{-})$, so $P^{+}(\eta f_{-})=0$ and we conclude that $\eta
f_{-}\in\overline{H^{2}_{0}}$.

The last equivalence is proved analogously. ∎

At this point we require the theory of inner and outer functions, and the
inner–outer factorization in $H^{2}$. A suitable reference is [16, Chap. 1].

As an example of a class of functions $\eta\in L^{\infty}$ and $f\in L^{2}$
such that $P^{\pm}(\eta f)=\eta P^{\pm}f$ we have the following, where
$\theta$ denotes an inner function and $K_{\theta}$ is its associated model
space $K_{\theta}=H^{2}\ominus\theta H^{2}$.

######  Proposition 3.5.

If $\eta\in\overline{H^{\infty}}$ is such that $\eta=\alpha\bar{\theta}h$ with
$h\in H^{\infty}$ and $\alpha$ an inner function, then for all $f\in
K^{\perp}_{\theta}$ we have $P^{\pm}(\eta f)=\eta P^{\pm}f$.

###### Proof.

Since $f\in K^{\perp}_{\theta}$ can be written in the form $f=f_{-}+\theta
f_{+}$ with $f_{-}\in\overline{H^{2}_{0}}$ and $f+\in H^{2}$, we have that

$\eta f=\eta(f_{-}+\theta f_{+})=\eta f_{-}+\alpha\bar{\theta}h\theta
f_{+}=\eta f_{-}+\alpha hf_{+},$

where $\eta f_{-}\in\overline{H^{2}_{0}}$ and $\alpha hf_{+}\in H^{2}$.
Therefore $P^{+}(\eta f)=\alpha hf_{+}=\eta\theta f_{+}=\eta P^{+}f$ and
$P^{-}(\eta f)=\eta f_{-}=\eta P^{-}f$. ∎

######  Remark 3.6.

We have that $w\in L^{\infty}$ is a multiplier from a model space
$K_{\theta_{1}}$ onto another model space $K_{\theta_{2}}$, i.e.,
$wK_{\theta_{1}}=K_{\theta_{2}}$, if and only if $w\in\mathop{\rm
Hol}\nolimits(\mathbb{D})$ with
$w=\gamma\theta_{2}\overline{\theta_{1}}\bar{w}$ where $|\gamma|=1$ (see [9]
and [6]). Thus conjugates of the surjective multipliers between two model
spaces provide examples of the functions $\eta$ described in Proposition 3.5.

##  4 Paired kernels

We denote the kernel of a paired operator, which we call a paired kernel, by

${\rm ker}_{a,b}=\mathop{\rm ker}\nolimits S_{a,b}.$ (4.1)

Paired kernels are closely connected with Toeplitz kernels. To obtain this
relation, we start by recalling that two operators $T$ and $S$, on the Banach
spaces $X$ and $Y$ respectively, are said to be equivalent if and only if
there exist invertible operators $E$ and $F$ such that $T=ESF$. Clearly, in
this case we have the isomorphism $\mathop{\rm ker}\nolimits
T\simeq\mathop{\rm ker}\nolimits S$ since we have $\mathop{\rm ker}\nolimits
T=F^{-1}\mathop{\rm ker}\nolimits S$. The operators $T$ and $S$ are said to be
equivalent after extension if and only if there are Banach spaces $\tilde{X}$
and $\tilde{Y}$ such that the operators

$\tilde{T}=\left[\begin{matrix}T&0\\\
0&I_{\tilde{X}}\end{matrix}\right]\qquad\hbox{and}\qquad\tilde{S}=\left[\begin{matrix}S&0\\\
0&I_{\tilde{Y}}\end{matrix}\right]$

are equivalent (see [1]).

If $b\in\mathcal{G}L^{\infty}$ (that is, if it is invertible in $L^{\infty}$),
then it is clear that $S_{a,b}$ is equivalent to $S_{G,1}$, where $G=a/b$. On
the other hand, letting again $T_{G}$ denote the Toeplitz operator with symbol
$G\in L^{\infty}$,

$T_{G}:H^{2}\to H^{2},\qquad T_{G}={P^{+}GP^{+}}_{|H^{2}},$ (4.2)

we have that $T_{G}$ is equivalent after extension to the paired operator
$S_{G,1}$ (see [3]). This means in particular that the kernels of the two
operators are isomorphic,

$\mathop{\rm ker}\nolimits T_{G}\simeq\mathop{\rm ker}\nolimits S_{G,1}={\rm
ker}_{G,1},$ (4.3)

the isomorphism in this case being induced by the mapping $P^{+}$, and we have

$\mathop{\rm ker}\nolimits T_{G}=P^{+}\mathop{\rm ker}\nolimits_{G,1}=:{\rm
ker}_{G,1}^{+}.$ (4.4)

More generally, we denote

${\rm ker}^{\pm}_{a,b}=P^{\pm}{\rm ker}_{a,b}=P^{\pm}\mathop{\rm ker}\nolimits
S_{a,b}.$ (4.5)

As happens with Toeplitz kernels, paired kernels and their projections (4.5)
possess many interesting properties which, in the case of ${\rm
ker}^{\pm}_{a,b}$, generalize those of Toeplitz kernels.

It is well known that, for $a\in L^{\infty}\setminus\\{0\\}$ we have
$\mathop{\rm ker}\nolimits T_{a}=\\{0\\}$ if $a\in H^{\infty}$ and
$\mathop{\rm ker}\nolimits T_{a}\not=\\{0\\}$ if $a\in\overline{H^{2}_{0}}$. A
corresponding result for paired kernels is the following.

######  Proposition 4.1.

Let $a,b\in L^{\infty}\setminus\\{0\\}$. Then:

  1. (i)

${\rm ker}_{a,b}=\\{0\\}$ if $a\in H^{\infty}$ and
$b\in\overline{H^{\infty}}$;

  2. (ii)

${\rm ker}_{a,b}\neq\\{0\\}$ if $a\in\overline{H^{\infty}}$ and $b\in
H^{\infty}$ has a nontrivial inner factor;

  3. (iii)

${\rm ker}_{a,b}\neq\\{0\\}$ if $a\in\overline{H^{\infty}_{0}}$ and $b\in
H^{\infty}$.

###### Proof.

(i) $aP^{+}f+bP^{-}f=0\iff aP^{+}f=-bP^{-}f$ and, since the left-hand side of
this equation is in $H^{2}$ while the right-hand side is in
$\overline{H^{2}_{0}}$, both are zero and it follows that $P^{\pm}f=0$, so
$f=0$.

(ii) Let $b=b_{i}b_{o}$ be the inner-outer factorization of $b$. Then we have
$af_{+}+bf_{-}=0$ for $f_{+}=\dfrac{b_{i}-b_{i}(0)}{z}b_{o}$ and
$f_{-}=-a\dfrac{1-b_{i}(0)\overline{b_{i}}}{z}$.

(iii) Clearly, taking $f_{+}=b$ and $f_{-}=-a$ gives a function
$f=f_{+}+f_{-}$ in ${\rm ker}_{a,b}$. Alternatively, the result follows from
part (ii), since ${\rm ker}_{a,b}={\rm ker}_{za,zb}$. ∎

We may see (iii) as a generalization of the property that $\mathop{\rm
ker}\nolimits T_{a}\neq\\{0\\}$ if $a\in\overline{H^{\infty}_{0}}$, as
follows:

If $a\in\overline{H^{\infty}_{0}}$, then saying that $\phi_{+}\in\mathop{\rm
ker}\nolimits T_{a}$ is equivalent to saying that there is a
$\phi_{-}\in\overline{H^{2}_{0}}$ such that $a\phi_{+}=\phi_{-}$. That is,
$a\phi_{+}+b\phi_{-}=0$, where $a\in\overline{H^{2}_{0}}$ and $b=-1\in
H^{\infty}$.

We now use the notation

$\phi_{\pm}=P^{\pm}\phi,\qquad(\phi\in L^{2}).$ (4.6)

######  Remark 4.2.

Note that, with the assumption that $\\{a,b\\}$ is nondegenerate, if
$\phi\in{\rm ker}_{a,b}$ and either $\phi_{+}=0$ or $\phi_{-}=0$ on a set of
positive measure then, by the Luzin–Privalov theorem, we must have $\phi=0$.

Clearly, for any measurable complex-valued function $\eta$ defined a.e. on
$\mathbb{T}$ such that $\eta\neq 0$ a.e. on $\mathbb{T}$, we have the
implication

$\tilde{a}=\eta a,\quad\tilde{b}=\eta b\quad\implies\quad{\rm ker}_{a,b}={\rm
ker}_{\tilde{a},\tilde{b}}.$ (4.7)

It is thus natural to ask when two paired kernels are equal or related by
inclusion, and whether the converse of (4.7) holds.

######  Proposition 4.3.

For a nondegenerate pair $\\{a,b\\}$ if ${\rm ker}_{a,b}\neq\\{0\\}$, then,
for any $\tilde{a},\tilde{b}\in L^{\infty}$, it holds that

${\rm ker}_{a,b}={\rm ker}_{\tilde{a},\tilde{b}}\quad\iff\quad
a\tilde{b}=\tilde{a}b.$

###### Proof.

Suppose that ${\rm ker}_{a,b}\neq\\{0\\}$ and ${\rm ker}_{a,b}={\rm
ker}_{\tilde{a},\tilde{b}}$. Then, for every $\phi\in{\rm ker}_{a,b}$ with
$\phi\neq 0$, we have

$a\phi_{+}+b\phi_{-}=\tilde{a}\phi_{+}+\tilde{b}\phi_{-}=0,$ (4.8)

which implies that $(a\tilde{b}-\tilde{a}b)\phi_{+}\phi_{-}=0$.

If $a\tilde{b}-\tilde{a}b\neq 0$ on a set of positive measure, then $\phi_{+}$
or $\phi_{-}$ must vanish on a set of positive measure, and by Remark 4.2 we
have $\phi=0$, which is a contradiction. We conclude then that
$a\tilde{b}-\tilde{a}b=0$.

Conversely, suppose that $a\tilde{b}=\tilde{a}b$. Taking
$\eta=\tilde{a}/a=\tilde{b}/b$ we have ${\rm ker}_{a,b}={\rm
ker}_{\tilde{a},\tilde{b}}$ by (4.7). ∎

This shows that the converse of (4.7) holds. We also have the following.

######  Corollary 4.4.

Each nonzero ${\rm ker}_{a,b}$ with $\\{a,b\\}$ nondegenerate is uniquely
determined by any of its nonzero elements and

${\rm ker}_{a,b}\cap{\rm
ker}_{\tilde{a},\tilde{b}}\neq\\{0\\}\quad\iff\quad{\rm ker}_{a,b}={\rm
ker}_{\tilde{a},\tilde{b}}.$

if $\tilde{a}$ and $\tilde{b}$ are also in $L^{\infty}$.

######  Corollary 4.5.

With the same assumptions as in Proposition 4.3, we have

${\rm ker}_{a,b}\subseteq{\rm
ker}_{\tilde{a},\tilde{b}}\quad\iff\quad\left\\{{\rm ker}_{a,b}={\rm
ker}_{\tilde{a},\tilde{b}}\quad\vee\quad{\rm ker}_{a,b}=\\{0\\}\right\\}.$

The following result shows that, in its turn, each nonzero $\phi\in L^{2}$
uniquely determines a paired kernel.

######  Theorem 4.6.

For each $\phi\in L^{2}\setminus\\{0\\}$, there is one and only one paired
kernel to which $\phi$ belongs.

###### Proof.

Take $\phi=\phi_{+}+\phi_{-}\in L^{2}\setminus\\{0\\}$ and let
$\phi_{+}=I_{+}O_{+}$ and $\phi_{-}=I_{-}O_{-}$, where $I_{+}$ and
$\overline{I_{-}}$ are inner functions and $O_{+}$ and
$\bar{z}\overline{O_{-}}$ are outer functions in $H^{2}$. Then we have

$\frac{I_{-}O_{-}}{I_{+}O_{+}}\phi_{+}=\phi_{-};\qquad\hbox{that
is,}\qquad\overline{I_{+}}\frac{\phi_{+}}{O_{+}}=\overline{I_{-}}\frac{\phi_{-}}{O_{-}}.$
(4.9)

Now $1/O_{+}$ belongs to the Smirnoff class ${\mathcal{N}}_{+}$, so that there
are $H_{1+}$ and $H_{2+}$ in $H^{\infty}$ such that

$\frac{1}{O_{+}}=\frac{H_{1+}}{H_{2+}}$ (4.10)

(see, for example, [16]). Analogously,

$\frac{1}{\bar{z}\overline{O_{-}}}=\frac{h_{1+}}{h_{2+}}$ (4.11)

with $h_{1+},h_{2+}\in H^{\infty}$. Thus, from (4.9)–(4.11) we have

$(\overline{I_{+}}H_{1+}\overline{h_{2+}})\phi_{+}-(\overline{I_{-}}z\overline{h_{1+}}H_{2+})\phi_{-}=0$

so $\phi\in{\rm ker}_{a,b}$ with $a=\overline{I_{+}}H_{1+}\overline{h_{2+}}$
and $b=-\overline{I_{-}}z\overline{h_{1+}}H_{2+}$, which are $L^{\infty}$
functions. By Corollary 4.4, ${\rm ker}_{a,b}$ is the only paired kernel
containing $\phi$. ∎

##  5 Invariance and near invariance

Toeplitz kernels are $S^{*}$-invariant, where $S^{*}$ is the backward shift
$S^{*}=T_{\bar{z}}$, whenever the associated symbol is in
$\overline{H^{\infty}}$, that is

$S^{*}(\mathop{\rm ker}\nolimits T_{\phi})\subseteq\mathop{\rm ker}\nolimits
T_{\phi}\quad\hbox{for}\quad\phi\in\overline{H^{\infty}}.$ (5.1)

These Toeplitz kernels are the model spaces, and they are the nontrivial
subspaces of $H^{2}$ that are invariant for $S^{*}$.

Recall that a subspace $M\subseteq H^{2}$ is said to be nearly
$S^{*}$-invariant if

$f\in M,\quad f(0)=0\implies S^{*}f\in M.$ (5.2)

While not all Toeplitz kernels are $S^{*}$-invariant subspaces of $H^{2}$,
they are all nearly $S^{*}$-invariant (see [5] for more on this subject). Note
that for $f\in H^{2}$ we have $f(0)=0$ if and only if $f\in\mathop{\rm
ker}\nolimits H_{\bar{z}}$, where $H_{\bar{z}}$ is the Hankel operator
$H_{\bar{z}}=P^{-}\bar{z}P^{+}_{|H^{2}}$. Thus (5.2) for $M=\mathop{\rm
ker}\nolimits T_{\phi}$ can equivalently be written as

$f\in\mathop{\rm ker}\nolimits T_{\phi}\cap\mathop{\rm ker}\nolimits
H_{\bar{z}}\implies T_{\bar{z}}f\in\mathop{\rm ker}\nolimits T_{\phi}.$ (5.3)

This is an important property of Toeplitz kernels which can be seen as
describing, in the case when $\mathop{\rm ker}\nolimits T_{G}$ is not
invariant for $S^{*}$, the part of $\mathop{\rm ker}\nolimits T_{\phi}$ that
is mapped onto $\mathop{\rm ker}\nolimits T_{\phi}$ by $T_{\bar{z}}=S^{*}$.

More generally, Toeplitz kernels are nearly $T_{\eta}$-invariant for every
$\eta\in\overline{H^{\infty}}$, i.e.,

$f\in\mathop{\rm ker}\nolimits T_{\phi}\cap\mathop{\rm ker}\nolimits
H_{\eta}\implies T_{\eta}f\in\mathop{\rm ker}\nolimits T_{\phi}$ (5.4)

(see [5, Sec. 3]), and if in addition $\phi\in\overline{H^{\infty}}$, then
$\mathop{\rm ker}\nolimits T_{\phi}$ is invariant for $T_{\eta}$. It is
natural to ask whether similar invariance properties hold for paired kernels.
We start by recalling that, for two operators $A$ and $B$,

$AB=BA\implies\mathop{\rm ker}\nolimits A\hbox{ is invariant for }B,$ (5.5)

since, if $f\in\mathop{\rm ker}\nolimits A$, then $A(Bf)=B(Af)=0$ and so
$Bf\in\mathop{\rm ker}\nolimits A$. From this and Proposition 3.2 we get the
following.

######  Proposition 5.1.

Let $a,\tilde{a},b,\tilde{b}\in L^{\infty}$ with
$\\{a,b\\},\\{\tilde{a},\tilde{b}\\}$ nondegenerate. Then $\mathop{\rm
ker}\nolimits S_{a,b}$ is invariant for $S_{\tilde{a},\tilde{b}}$ if
$a,\tilde{a}\in H^{\infty}$ and $b,\tilde{b}\in\overline{H^{\infty}}$; also
$\mathop{\rm ker}\nolimits\Sigma_{a,b}$ is invariant for
$\Sigma_{\tilde{a},\tilde{b}}$ if $a,\tilde{a}\in\overline{H^{\infty}}$ and
$b,\tilde{b}\in H^{\infty}$.

Regarding the invariance properties with respect to $S_{\eta,\eta}=\eta I$ we
have:

######  Proposition 5.2.

For $\eta\in L^{\infty}$ and $\\{a,b\\}$ nondegenerate we have that if
$f\in{\rm ker}_{a,b}$ then we have $\eta f\in{\rm ker}_{a,b}$ if and only if

$f\in\mathop{\rm ker}\nolimits H_{\eta}\oplus\mathop{\rm
ker}\nolimits\tilde{H}_{\eta},$ (5.6)

where $H_{\eta}=P^{-}\eta P^{+}{}_{|H^{2}}$ and $\tilde{H}_{\eta}=P^{+}\eta
P^{-}{}_{|\overline{H^{2}_{0}}}$.

###### Proof.

Let $f,\eta f\in{\rm ker}_{a,b}$. Then

$\displaystyle 0$ $\displaystyle=$ $\displaystyle aP^{+}\eta f+bP^{-}\eta f$
$\displaystyle=$ $\displaystyle aP^{+}\eta f_{+}+aP^{+}\eta f_{-}+bP^{-}\eta
f_{+}+bP^{-}\eta f_{-}$ $\displaystyle=$ $\displaystyle a\eta f_{+}-aP^{-}\eta
f_{+}+aP^{+}\eta f_{-}+bP^{-}\eta f_{+}+b\eta f_{-}-bP^{+}\eta f_{-}$
$\displaystyle=$ $\displaystyle-(a-b)(P^{-}\eta f_{+}-P^{+}\eta f_{-}),$

so $P^{-}\eta f_{+}=P^{+}\eta f_{-}=0$, which is equivalent to
$f\in\mathop{\rm ker}\nolimits H_{\eta}\oplus\mathop{\rm
ker}\nolimits\tilde{H}_{\eta}$.

The converse follows from the same calculation. ∎

We get an analogue of (5.4) as follows:

######  Proposition 5.3.

For $\eta\in L^{\infty}$ and $\\{a,b\\}$ nondegenerate and $f\in L^{2}$,

$f\in{\rm ker}_{a,b}\cap(\mathop{\rm ker}\nolimits H_{\eta}\oplus\mathop{\rm
ker}\nolimits\tilde{H}_{\eta})\implies S_{\eta,\eta}f\in{\rm ker}_{a,b}.$

##  6 Coburn’s lemma

Coburn’s lemma for Toeplitz operators [8] asserts that for $g\in L^{\infty}$
at least one of $\mathop{\rm ker}\nolimits T_{g}$ and $\mathop{\rm
ker}\nolimits T^{*}_{g}$ is $\\{0\\}$. To obtain a generalization of this
lemma we study the relations between the kernels of $S_{a,b}$,
$S_{\bar{b},\bar{a}}$, and $\Sigma_{\bar{a},\bar{b}}=S^{*}_{a,b}$.

######  Lemma 6.1.

$P^{\pm}(\bar{z}\bar{\phi})=\bar{z}\overline{\phi_{\mp}}$.

###### Proof.

Observe that
$\bar{z}\bar{\phi}=\underbrace{\bar{z}\overline{\phi_{+}}}_{\in\overline{H^{2}_{0}}}+\underbrace{\bar{z}\overline{\phi_{-}}}_{\in
H^{2}}$ and the result follows. ∎

######  Lemma 6.2.

$\overline{{\rm ker}_{a,b}}=z{\rm ker}_{\bar{b},\bar{a}}$.

###### Proof.

We have

$\displaystyle\phi\in{\rm ker}_{a,b}$ $\displaystyle\iff$ $\displaystyle
a\phi_{+}+b\phi_{-}=0$ $\displaystyle\iff$
$\displaystyle\bar{a}\bar{z}\overline{\phi_{+}}+\bar{b}\bar{z}\overline{\phi_{-}}=0$
$\displaystyle\iff$
$\displaystyle\bar{b}P^{+}(\bar{z}\bar{\phi})+\bar{a}P^{-}(\bar{z}\bar{\phi})=0$
$\displaystyle\iff$ $\displaystyle\bar{z}\bar{\phi}\in{\rm
ker}_{\bar{b},\bar{a}},$

where the third equivalence follows from Lemma 6.1. ∎

We can thus define an antilinear isomorphism between ${\rm ker}_{a,b}$ and
${\rm ker}_{\bar{b},\bar{a}}$, as follows.

######  Proposition 6.3.

The antilinear operator

$J:{\rm ker}_{a,b}\to{\rm ker}_{\bar{b},\bar{a}},\qquad
J\phi=\bar{z}\bar{\phi}$ (6.1)

is well defined and bijective.

Next, we construct an isomorphism from ${\rm ker}^{*}_{a,b}:=\mathop{\rm
ker}\nolimits S_{a,b}^{*}$ onto $\mathop{\rm ker}\nolimits_{\bar{a},\bar{b}}$.

######  Proposition 6.4.

For nondegenerate $\\{a,b\\}$ the operator $\tilde{J}:{\rm
ker}^{*}_{a,b}\to\mathop{\rm ker}\nolimits_{\bar{a},\bar{b}}$ defined by

$\tilde{J}\psi=(\bar{a}-\bar{b})\psi,\qquad(\psi\in{\rm ker}^{*}_{a,b})$ (6.2)

is injective, and if

$a\in\mathcal{G}L^{\infty},\hbox{ or }b\in\mathcal{G}L^{\infty},\hbox{ or
}(a-b)\in\mathcal{G}L^{\infty},$ (6.3)

then $\tilde{J}$ is an isomorphism from ${\rm ker}^{*}_{a,b}$ onto
$\mathop{\rm ker}\nolimits_{\bar{a},\bar{b}}$.

###### Proof.

We begin by showing that the operator is well defined, i.e.,

$(\bar{a}-\bar{b})\mathop{\rm ker}\nolimits S_{a,b}^{*}\subseteq\mathop{\rm
ker}\nolimits S_{\bar{a},\bar{b}}.$ (6.4)

Let $\psi\in{\rm ker}_{a,b}^{*}=\mathop{\rm ker}\nolimits S_{a,b}^{*}$, which
means that

$P^{+}(\bar{a}\psi)=-P^{-}(\bar{b}\psi)=0,$ (6.5)

since $H^{2}\cap\overline{H^{2}_{0}}=\\{0\\}$. Then

$\displaystyle S_{\bar{a},\bar{b}}(\bar{a}-\bar{b})\psi$ $\displaystyle=$
$\displaystyle\bar{a}P^{+}(\bar{a}\psi)-\bar{a}P^{+}(\bar{b}\psi)+\bar{b}P^{-}(\bar{a}\psi)-\bar{b}P^{-}(\bar{b}\psi)$
$\displaystyle=$
$\displaystyle-\bar{a}P^{+}(\bar{b}\psi)+\bar{b}P^{-}(\bar{a}\psi)$
$\displaystyle=$
$\displaystyle-\bar{a}(I-P^{-})(\bar{b}\psi)+\bar{b}(I-P^{+})(\bar{a}\psi)$
$\displaystyle=$ $\displaystyle-\bar{a}\bar{b}\psi+\bar{b}\bar{a}\psi=0,$

so $(\bar{a}-\bar{b})\psi\in{\rm ker}_{\bar{a},\bar{b}}$.

Now the operator $\tilde{J}$ is injective because we assume that $a-b\neq 0$
a.e. on $\mathbb{T}$, so it is left to show that it is also surjective when
(6.3) holds.

This is obvious if $a-b\in\mathcal{G}L^{\infty}$; suppose now that
$a\in\mathcal{G}L^{\infty}$ or $b\in\mathcal{G}L^{\infty}$. We have, for
$\psi\in{\rm ker}_{a,b}^{*}$, which must satisfy (6.5),

$(\bar{a}-\bar{b})\psi=\phi\iff\underbrace{\bar{a}\psi}_{\in\overline{H^{2}_{0}}}-\underbrace{\bar{b}\psi}_{\in
H^{2}}=\phi\iff\bar{a}\psi=\phi_{-}\hbox{ and }\bar{b}\psi=-\phi_{+}.$

Thus we have, since $\bar{a}\phi_{+}+\bar{b}\phi_{-}=0$,

$\psi=\frac{\phi_{-}}{\bar{a}}=\tilde{J}^{-1}\phi\qquad\hbox{if}\quad
a\in\mathcal{G}L^{\infty}$

and

$\psi=-\frac{\phi_{+}}{\bar{b}}=\tilde{J}^{-1}\phi\qquad\hbox{if}\quad
b\in\mathcal{G}L^{\infty}.$

∎

######  Corollary 6.5.

With the same assumptions as in Proposition 6.4, if (6.3) holds, then

$\tilde{J}^{-1}:{\rm ker}_{\bar{a},\bar{b}}\to{\rm ker}_{a,b}^{*}$

is defined by

$\tilde{J}^{-1}\phi=\begin{cases}\dfrac{\phi}{\bar{a}-\bar{b}}&\hbox{if}\quad
a-b\in\mathcal{G}L^{\infty},\\\ \\\ \dfrac{P^{-}\phi}{\bar{a}}&\hbox{if}\quad
a\in\mathcal{G}L^{\infty},\\\ \\\ -\dfrac{P^{+}\phi}{\bar{b}}&\hbox{if}\quad
b\in\mathcal{G}L^{\infty}.\end{cases}$

######  Corollary 6.6.

If ${\rm ker}_{\bar{a},\bar{b}}=\\{0\\}$, then ${\rm ker}_{a,b}^{*}=\\{0\\}$.

Taking Proposition 6.3 into account, we also have:

######  Corollary 6.7.

The following are equivalent:
(i) ${\rm ker}_{b,a}=\\{0\\}$;
(ii) ${\rm ker}_{\bar{a},\bar{b}}=\\{0\\}$;
and, if at least one of $a,b$ and $a-b$ lies in $\mathcal{G}L^{\infty}$,
(iii) ${\rm ker}_{a,b}^{*}=\\{0\\}$.

We can now formulate a generalization of Coburn’s lemma for Toeplitz operators
[8].

######  Theorem 6.8.

Suppose that $a,b\neq 0$ a.e. on $\mathbb{T}$. Then
(i) We have ${\rm ker}_{a,b}=\\{0\\}$ or ${\rm ker}_{b,a}=\\{0\\}$;
equivalently ${\rm ker}_{a,b}=\\{0\\}$ or ${\rm
ker}_{\bar{a},\bar{b}}=\\{0\\}$.
(ii) If at least one of $a,b$ and $a-b$ lies in $\mathcal{G}L^{\infty}$, then
${\rm ker}_{a,b}=\\{0\\}$ or ${\rm ker}_{a,b}^{*}=\\{0\\}$.

###### Proof.

Suppose that there exist $\phi,\psi\neq 0$ with $\phi\in{\rm
ker}_{a,b},\psi\in{\rm ker}_{b,a}$. Then clearly

$a\phi_{+}=-b\phi_{-}\qquad\hbox{and}\qquad b\psi_{+}=-a\psi_{-}.$

Therefore $ab\phi_{+}\psi_{+}=ab\phi_{-}\psi_{-}$, and, since $a,b\neq 0$ a.e.
on $\mathbb{T}$, we have

$\underbrace{\phi_{+}\psi_{+}}_{\in
H^{1}}=\underbrace{\phi_{-}\psi_{-}}_{\in\overline{H^{1}_{0}}},$

so both the terms above are $0$. This implies that either $\phi_{+}=0$ or
$\psi_{+}=0$ and therefore, by Remark 4.2, $\phi=0$ or $\psi=0$, which is a
contradiction. Therefore we cannot have both ${\rm ker}_{a,b}\neq\\{0\\}$ and
${\rm ker}_{b,a}\neq\\{0\\}$; by Corollary 6.7, ${\rm ker}_{b,a}\neq\\{0\\}$
is equivalent to ${\rm ker}_{\bar{a},\bar{b}}\neq\\{0\\}$.

Finally, (ii) follows from Corollary 6.7 (iii). ∎

Clearly the theorem does not apply to the cases $a=0$ a.e. or $b=0$ a.e., as
is easily verified.

We recover Coburn’s lemma when $b=1$. In that case we have $\mathop{\rm
ker}\nolimits(aP^{+}+P^{-})=\\{0\\}$ or $\mathop{\rm
ker}\nolimits(\bar{a}P^{+}+P^{-})=\\{0\\}$, where $\mathop{\rm
ker}\nolimits(aP^{+}+P_{-})\simeq\mathop{\rm ker}\nolimits T_{a}$ and
$\mathop{\rm ker}\nolimits(\bar{a}P^{+}+P^{-})\simeq\mathop{\rm ker}\nolimits
T_{a}^{*}$ (cf. (4.3)).

##  7 Final comments

In [11, 12, 13], a new class of Toeplitz-like operators $T_{\omega}$ is
presented, whose symbols are complex rational functions $\omega$. The domain
of $T_{\omega}$ is

$\\{g_{+}\in H^{p}:\omega g_{+}=f+\rho\\},$

where $f\in L^{p}$ and $\rho$ is a strictly proper (i.e., zero at $\infty$)
rational function with all poles on $\mathbb{T}$. Then the authors write
$T_{\omega}g_{+}=P^{+}f$. The operator is densely-defined as its domain
contains all polynomials. Its kernel is then defined to be those $g_{+}$ with
$\omega g_{+}=f_{-}+\rho$.

Now, in the context of paired operators, we may take a more general
$\omega=\phi/p$ where $\phi\in L^{\infty}$ and $p$ is a polynomial with all
zeros on $\mathbb{T}$. We would naturally consider the kernel associated with
the bounded paired operator $S_{\phi,-p}$, since we then have, for
$S_{\phi,-p}\,g=0$,

$\phi g_{+}=pg_{-},\qquad\hbox{that is,}\qquad\omega g_{+}=g_{-}.$

In general this gives a different outcome, as can be seen with the example
$\omega(z)=1/(z-1)$. Its $T_{\omega}$ kernel is the space of all $g_{+}$ with
$g_{+}/(z-1)=f_{-}+\rho$.

However, the kernel of $S_{1,1-z}$ is the space of all $g$ with
$g_{+}+(1-z)g_{-}=0$. Thus the kernels are different, namely $\mathbb{C}$ in
the first case and $\\{0\\}$ in the second.

Finally, another recent paper that studies pairs of projections (but in the
context of model spaces, giving insights into truncated Toeplitz operators) is
[10].

## Acknowledgements

This work was partially supported by FCT/Portugal through CAMGSD, IST-ID,
projects UIDB/04459/2020 and UIDP/04459/2020. The second author would like to
thank the Calouste Gulbenkian Foundation for partially funding this work under
the Talentos Científicos Matemática 2021 scholarship.

## References

  * [1] H. Bart and V.È. Tsekanovskiĭ, Matricial coupling and equivalence after extension. Operator theory and complex analysis (Sapporo, 1991), 143–160, Oper. Theory Adv. Appl., 59, Birkhäuser, Basel, 1992.
  * [2] A. Brown and P.R. Halmos, Algebraic properties of Toeplitz operators. J. Reine Angew. Math. 213 (1963/64), 89–102.
  * [3] M.C. Câmara, Toeplitz operators and Wiener–Hopf factorisation: an introduction. Concr. Oper. 4 (2017), no. 1, 130–145.
  * [4] M.C. Câmara, K. Kliś-Garlicka, and M. Ptak, (Asymmetric) dual truncated Toeplitz operators. Operator and norm inequalities and related topics, 429–460, Trends Math., Birkhäuser/Springer, Cham, 2022.
  * [5] M.C. Câmara and J.R. Partington, Near invariance and kernels of Toeplitz operators. J. Anal. Math. 124 (2014), 235–260.
  * [6] M.C. Câmara and J.R. Partington, Multipliers and equivalences between Toeplitz kernels. J. Math. Anal. Appl. 465 (2018), no. 1, 557–570.
  * [7] M.C. Câmara and J.R. Partington, Scalar-type kernels for block Toeplitz operators. J. Math. Anal. Appl. 489 (2020), no. 1, 124111, 25 pp.
  * [8] L. Coburn, Weyl’s theorem for nonnormal operators. Michigan Math. J. 13 (1966), 285–288.
  * [9] R.B. Crofoot, Multipliers between invariant subspaces of the backward shift. Pacific J. Math. 166 (1994), no. 2, 225–246.
  * [10] R. Debnath, D.K. Pradhan and J. Sarkar, Pairs of inner projections and two applications. J. Func. Anal. 286 (2024), 110216.
  * [11] G.J. Groenewald, S. ter Horst, J. Jaftha and A.C.M. Ran, A Toeplitz-like operator with rational symbol having poles on the unit circle I: Fredholm properties.Operator theory, analysis and the state space approach, 239–268, Oper. Theory Adv. Appl., 271, Birkhäuser/Springer, Cham, 2018.
  * [12] G.J. Groenewald, S. ter Horst, J. Jaftha and A.C.M. Ran, A Toeplitz-like operator with rational symbol having poles on the unit circle II: the spectrum. Interpolation and realization theory with applications to control theory, 133–154, Oper. Theory Adv. Appl., 272, Birkhäuser/Springer, Cham, 2019.
  * [13] G.J. Groenewald, S. ter Horst, J. Jaftha and A.C.M. Ran, A Toeplitz-like operator with rational symbol having poles on the unit circle III: The adjoint. Integral Equations Operator Theory 91 (2019), no. 5, Paper No. 43, 23 pp.
  * [14] O. Karlovych and E. Shargorodsky, The Coburn lemma and the Hartman–Wintner–Simonenko theorem for Toeplitz operators on abstract Hardy spaces, Integral Equations Operator Theory 95 (2023), no. 1, Paper No. 6, 17 pp.
  * [15] S. Mikhlin and S. Prossdorf, Singular integral operators. Springer-Verlag, Berlin, 1986. Translated from German by Albrecht Böttcher and Reinhard Lehmann, 1986.
  * [16] N.K. Nikolski, Operators, functions, and systems: an easy reading. Vol. 1. Hardy, Hankel, and Toeplitz. Mathematical Surveys and Monographs, 92. American Mathematical Society, Providence, RI, 2002.
  * [17] F.-O. Speck, Paired operators in asymmetric space setting. Large truncated Toeplitz matrices, Toeplitz operators, and related topics, 681–702, Oper. Theory Adv. Appl., 259, Birkhäuser/Springer, Cham, 2017.


# AB-Training: A Communication-Efficient Approach for Distributed Low-Rank
Learning

Daniel Coquelin<EMAIL_ADDRESS>0000-0001-8552-5153 Karlsruhe
Institute of Technology (KIT) - Scientific Computing Center (SCC)Hermann-von-
Helmholtz-Platz 1Eggenstein-LeopoldshafenBaden-WürttembergGermany76344 ,
Katherina Flügel  Karlsruhe Institute of Technology (KIT) - Scientific
Computing Center (SCC)Hermann-von-Helmholtz-Platz 1Eggenstein-
LeopoldshafenBaden-WürttembergGermany76344<EMAIL_ADDRESS>,  Marie
Weiel  Karlsruhe Institute of Technology (KIT) - Scientific Computing Center
(SCC)Hermann-von-Helmholtz-Platz 1Eggenstein-LeopoldshafenBaden-
WürttembergGermany76344<EMAIL_ADDRESS>,  Nicholas Kiefer  Karlsruhe
Institute of Technology (KIT) - Scientific Computing Center (SCC)Hermann-von-
Helmholtz-Platz 1Eggenstein-LeopoldshafenBaden-WürttembergGermany76344
<EMAIL_ADDRESS>,  Muhammed Öz  Karlsruhe Institute of Technology
(KIT) - Scientific Computing Center (SCC)Hermann-von-Helmholtz-Platz
1Eggenstein-LeopoldshafenBaden-WürttembergGermany76344<EMAIL_ADDRESS>,
Charlotte Debus  Karlsruhe Institute of Technology (KIT) - Scientific
Computing Center (SCC)Hermann-von-Helmholtz-Platz 1Eggenstein-
LeopoldshafenBaden-WürttembergGermany76344<EMAIL_ADDRESS>,  Achim
Streit  Karlsruhe Institute of Technology (KIT) - Scientific Computing Center
(SCC)Hermann-von-Helmholtz-Platz 1Eggenstein-LeopoldshafenBaden-
WürttembergGermany76344<EMAIL_ADDRESS>and  Markus Götz  Karlsruhe
Institute of Technology (KIT) - Scientific Computing Center (SCC)Hermann-von-
Helmholtz-Platz 1Eggenstein-LeopoldshafenBaden-WürttembergGermany76344
<EMAIL_ADDRESS>
(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Communication bottlenecks hinder the scalability of distributed neural network
training, particularly on distributed-memory computing clusters. To
significantly reduce this communication overhead, we introduce _AB-training_ ,
a novel data-parallel training method that decomposes weight matrices into
low-rank representations and utilizes independent group-based training. This
approach consistently reduces network traffic by 50% across multiple scaling
scenarios, increasing the training potential on communication-constrained
systems. Our method exhibits regularization effects at smaller scales, leading
to improved generalization for models like VGG16, while achieving a remarkable
44.14 : 1 compression ratio during training on CIFAR-10 and maintaining
competitive accuracy. Albeit promising, our experiments reveal that large
batch effects remain a challenge even in low-rank training regimes.

Distributed training, Low-rank representation, Neural networks, Communication
efficiency

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
53rd International Conference on Parallel Processing; August 12–15, 2024;
Gotland, Sweden††ccs: Computing methodologies Neural networks††ccs: Computing
methodologies Machine learning algorithms††ccs: Networks Network performance
analysis††ccs: Computing methodologies Distributed artificial
intelligence††ccs: Computing methodologies Batch learning

Figure 1. Average total interconnect traffic measured during the training of a
Vision Transformer (ViT) B/16 on the ImageNet-2012 dataset for various node
counts in $\mathrm{G}\mathrm{B}\mathrm{/}\mathrm{s}$ with traditional,
synchronous, full-rank data parallel training as implemented by PyTorch’s
DistributedDataParallel (DDP) class and our AB training method. The global
batch size for each training step is 4,096 images and is split evenly between
the GPUs on each node.

Average total interconnect traffic measured during the training of a Vision
Transformer B/16 on the ImageNet-2012 dataset for various node counts in
$\mathrm{G}\mathrm{B}\mathrm{/}\mathrm{s}$. The global batch size for each
training step is 4,096 images and is split evenly between the GPUs on each
node.

##  1\. Introduction

The relentless pursuit of predictive performance in machine learning has led
to neural networks of unprecedented scale and complexity. These massive state-
of-the-art models excel in tasks ranging from image classification and natural
language processing to scientific simulations. However, this performance comes
at a steep computational cost, as training large models often demands large
datasets. These datasets often exceed the capabilities of individual devices,
necessitating distributed training approaches.

Data parallelism (DP) is the most common distributed training technique for
neural networks (Ben-Nun and Hoefler, 2019). It replicates the model across
multiple devices, each of which processes a subset of the data, known as the
local batch, before aggregating the gradients and updating model parameters. A
persistent challenge in data parallel (DP) training is the communication
bottleneck created by synchronizing large model representations across compute
nodes (Niu et al., 2011). This bottleneck severely limits scalability,
particularly on distributed-memory computing systems with potential network
constraints. Large batch effects (Goyal et al., 2018), pose an additional
challenge for maintaining predictive performance, as accuracy degrades once
the global batch size crosses a critical threshold.

To address these challenges, we propose a novel low-rank training method based
on the singular value decomposition (SVD) and a hierarchical training scheme
designed to significantly reduce communication overheads in distributed
settings. Our method leverages the observation that low-rank representations
can promote regularization (Coquelin et al., 2024) and minimize the amount of
data transferred. Key contributions of our work include:

  * •

We find a 50% reduction in network traffic during distributed training on
average as compared to traditional synchronous DP training, enabling
experimentation with larger batch sizes and potentially accelerating training
within distributed computing environments without high-speed interconnects.

  * •

Our method demonstrates improved regularization effects at smaller scales,
particularly on the Vision Transformer (ViT), leading to better predictive
performance on unseen data. However, it also sheds light onto the challenge of
reducing large batch effects without hyperparameter tuning.

  * •

In an ideal scenario, AB training achieves a compression ratio of 44.14 : 1
for the VGG16 (Liu and Deng, 2015) model when trained on CIFAR-10 (Krizhevsky,
2009) while maintaining a competitive accuracy. In more standard scenarios, it
achieved compression values ranging from 1.19 : 1 to 2.54 : 1 when exceeding
the accuracy of traditional DP training.

Our work highlights promising directions for how to fully harness the
potential of low-rank training in massively scaled scenarios, with
implications for both distributed-memory computing and the broader ML
community.

##  2\. Related Work

###  2.1. Distributed Training of Neural Networks

The computational demands of training large neural networks have driven the
development of distributed training paradigms, with data parallelism as a
cornerstone approach. In a traditional data parallel (DP) training scheme, the
model is replicated across multiple devices, each processing a subset of the
training data independently. The data subset that one model instance processes
is the local batch, while the global batch is the set of all local batches.
After each forward-backward pass, gradients are aggregated across all model
instances before the model is updated by the optimizer. As the number of
compute resources and the model size increase, communication overheads
incurred during gradient synchronization become a significant constraint,
particularly in systems without high-bandwidth interconnects. Other
parallelization strategies, such as model parallelism, introduce alternative
ways to distribute training across devices, but come with their own
communication and load-balancing complexities.

Several techniques have been proposed to mitigate the communication bottleneck
in data parallel training, including gradient accumulation, topology-aware
communication patterns, asynchronous methods, and gradient compression (Vogels
et al., 2019; Abrahamyan et al., 2022). Methods like H-SGD (Lin et al., 2020b)
and DASO (Coquelin et al., 2022) utilize localized synchronization within
smaller groups between global updates, potentially reducing communication
costs by leveraging the network’s topology. Asynchronous approaches typically
make use of a parameter server for aggregating gradients (Niu et al., 2011)
and often require extensive hyperparameter tuning. Despite these advancements,
the scalability of data parallelism remains constrained by communication
bottlenecks and large batch effects, where increasing the global batch size
negatively impacts generalization performance.

###  2.2. Low-Rank Neural Network Training

A promising avenue for reducing both the memory footprint and computational
cost of neural networks lies in low-rank training. Singular Value
Decomposition (SVD) plays a crucial role in this domain. By factoring a weight
matrix $W$ into three matrices, $U$, $\Sigma$, and $V^{T}$, and retaining only
the $k$ largest singular values, a low-rank approximation with significantly
fewer parameters can be obtained given a sufficiently large $k$ value. Note
that aggressive compression can lead to a loss of approximation quality.

Several methods leverage SVD for compressing neural networks during training.
Some train directly on the $U$, $\Sigma$, and $V$ matrices (Schotthöfer et
al., 2022; Coquelin et al., 2024), while others create different
representations using the SVD and training on them (Liebenwein et al., 2021;
Yang et al., 2020a). Both approaches can offer regularization-like benefits,
potentially improving generalization (Winata et al., 2020; Cahyawijaya et al.,
2021; Phan et al., 2020).

Gradient compression using SVD has found application in federated learning
scenarios (Wang et al., 2023b), where bandwidth constraints necessitate
reduced communication. While these techniques address computational challenges
in neural network training, their integration within distributed environments
to specifically optimize communication remains an active area of research.

Low-rank factorizations can heavily reduce the communication required during
distributed training. Existing approaches simply train low-rank models in
parallel, but do not make any efforts to use the distributed system to their
advantage (Coquelin et al., 2024; Wang et al., 2023a, 2021). To the best of
our knowledge, there exist no methods that leverage both distributed
architectures and low-rank representations to train neural networks.

##  3\. Methodology

Data parallel training of neural networks is constrained by the the
communication bottleneck caused by synchronizing massive model representations
across compute nodes and large batch effects, a phenomenon where accuracy
degrades at large global batch sizes. In an attempt to tackle these
challenges, our proposed training method integrates low-rank weight
representations within a data parallel framework. It leverages the reduced
network capacity stemming from topologically informed communication patterns
and low-rank training while escaping local minima using group training.

We utilize SVD to decompose weight matrices into low-rank representations,
significantly reducing the amount of data communicated during training. We
decompose the weights of a given layer as

(1) $W_{m\times n}=A_{m\times k}B_{k\times n}=\left(U_{m\times
k}\Sigma_{k\times k}^{1/2}\right)\left(\Sigma_{k\times k}^{1/2}V_{k\times
n}^{\top}\right)$

where $U$, $\Sigma$, and $V$ are determined by the SVD of $W$ and
$k<\texttt{min}\left(m,n\right)$. Using a low-rank representation in training
has the added benefit of reducing overfitting, a common problem in training
neural networks (Yang et al., 2020b; Hu et al., 2021).

Large batch effects can be trickier to deal with. Consider the standard batch
stochastic gradient descent (SGD) optimization step

(2) $W_{t}=W_{t-1}-\frac{\eta}{N}\sum_{i=1}^{B}\nabla
Q_{i}\left(W_{t-1}\right),$

where $W_{t}$ is a parameter at time $t$ in training, $\eta$ is the learning
rate, $N$ is the batch size, and $Q_{i}\left(W\right)$ is the value of the
loss function for the $i$-th data element. It is easily shown that for a given
iteration $k>0$, the weights are simply the sum of all the previous update
steps or

(3) $W_{k}=W_{0}-\frac{\eta}{B}\sum_{s=1}^{k-1}\sum_{i=1}^{B}\nabla
Q_{i}\left(W_{s}\right)$

With a larger batch size, $B$, the magnitude of the gradients shrinks. This
removes the randomness inherent to SGD’s success and makes the network more
prone to getting stuck in local minima (Chaudhari et al., 2019).

To mitigate this effect, we draw motivation from ensemble approaches. By
dividing the DP model replicas into subgroups to be trained on independent
data subsets, we essentially form an ensemble of models. Individually, these
groups converge towards different local minima compared to full-system
training, promoting exploration of the loss landscape.

After training in subgroups, we average the independently trained models. This
merging strategy has seen some success in federated learning methods (Giuseppi
et al., 2022). Compared to federated use-cases, which typically deal with
models trained independently on biased datasets, AB training benefits from
randomly sampling from the same dataset and thus has unbiased batches during
our independent training phases.

During the independent training phases, each group trains on either $A$ or
$B$, while the other component remains fixed. As the orthogonal component of
$W$ is stable later in training (Coquelin et al., 2024), we only train on one
of $A$ or $B$ in an attempt to reduce the divergence between the independent
groups. At the end of the independent training phase, we average $A$ and $B$
across all workers to merge the independently trained models together.

Our AB training procedure consists of the following phases:

Distributed warmup:

Initial training with full-rank data parallelism allows the weights to adapt
rapidly away from their random initialization. To avoid early divergence, we
employ a traditional, full-rank data parallel warmup phase for a given number
of iterations (warmupSteps).

Independent AB decomposition:

Each model instance independently computes an AB decomposition of its weight
matrices as per Equation 1. This AB decomposition allows us to represent the
weight matrices more compactly, reducing the amount of data to be communicated
during training. The rank of this approximation is determined by a user-
provided hyperparameter (sigmaCutoff) times the largest singular value. Values
smaller than this are removed.

Group-only training:

Half of the independent groups will train the $A$ matrix, denoted as the $A$
groups, while the other groups will train $B$, the $B$ groups. For a given
number of iterations (numABSteps), processes in the $A$ group only train the
$A$ matrices, while processes in the $B$ group only train the $B$ matrices,
the matrix not being trained remains frozen in this phase.

Synchronization and update:

We average both the trained and frozen $A$ and $B$ matrices independently to
synchronize the model instances across all processes. This is crucial for
ensuring that the overall model remains consistent across all compute nodes.
Afterwards, we reconstruct the full-rank weight matrices from the updated $A$
and $B$ components.

Full-rank rebound:

We then train the full-rank network with traditional data parallelism for a
short period, which we refer to as the fullRankReboundSteps. This phase
promotes convergence and helps mitigate potential accuracy degradation
observed in low-rank training scenarios.

A formalized algorithm is shown in Algorithm 1 and the method diagram is shown
in Figure 2.

Figure 2. The AB training procedure.

The AB training procedure. See pseudocode or description in text.

Input : Model, $M$ , training data, $numTrainingSteps$, hyperparameters
($warmupSteps$, $numABSteps$, $fullRankReboundSteps$)

1

2if  _procId $\leq$ worldSize / 2_ then

3 $groupTrainTarget\leftarrow A$

4else

5 $groupTrainTarget\leftarrow B$

6

7for  _$i\leftarrow 1$ to warmupSteps_ do

8 traditionalFullRankDPTraining()

9repeat

10 foreach  _$W$ in $M$_ do

11 $A,B\leftarrow\texttt{abDecomposition}(W)$ $\triangleright$ Equation 1

12 removeSmallSingularValues()

13 startLearningRateRebound()

14 $groupTrainTarget$.setTrainable(True)

15

16 for  _$i\leftarrow 1$ to numABSteps_ do

17 independentSubgroupTraining()

18 $A\leftarrow\texttt{allReduce}(A/worldSize$)

19 $B\leftarrow\texttt{allReduce}(B/worldSize$)

20 foreach  _$W$ in $M$_ do

21 $W\leftarrow AB$

22 startLearningRateRebound()

23 for  _$i\leftarrow 1$ to fullRankReboundSteps_ do

24 traditionalFullRankDPTraining()

25

26until  _numTrainingSteps = completed steps_

Algorithm 1 The AB training method. $W$ is a parameter of the input model $M$
and $worldSize$ is the number of workers used for traditional DP training.

Popular neural network optimizers like AdamW (L. and H., 2018) rely on second-
order derivative approximations. When we change the network’s structure (e.g.,
between low-rank and full-rank representations), these optimizer states become
incompatible due to their dimensionally. To address this, we use a “learning
rate rebound” strategy. This involves reducing the learning rate to near-zero
and gradually increasing it back to its scheduled value over a user-provided
number of steps, allowing the optimizer to adjust to the new parameter
representations. We do a learning rate rebound after each time the network’s
parameters change shape.

##  4\. Experimental Evaluation

To evaluate the effectiveness of our AB training approach, we conducted
experiments on established neural network architectures. Our primary
objectives were to demonstrate significant reductions in communication
overhead, potential regularization benefits, scalability, and achievable
compression during training. We focused on the ResNet-50 (He et al., 2016) and
Vision Transformer (Dosovitskiy et al., 2021) B/16 models trained on the
ImageNet-2012 dataset (Krizhevsky et al., 2012) for image classification as
well as VGG16 (Liu and Deng, 2015) trained on CIFAR-10 (Krizhevsky, 2009). We
compare our results against traditional DP training as implemented by
PyTorch’s DDP package, referred to as ‘Traditional DDP’ or ‘Trad. DDP’, using
the same hyperparameters (HPs). We also show results for the AB training setup
without any individual trained groups which are denoted as ‘No Group’. In
these experiments, the $B$ matrix is trained using traditional DP training in
place of the full-rank weights. The ViT HPs are those used in the original
source, while the ResNet-50 parameters utilize the same learning rate
scheduler and the default values for the AdamW (L. and H., 2018) optimizer.
The VGG16 HPs are as shown in (Coquelin et al., 2024).

We analyze our training method using two scaling strategies: constant local
batch size scaling and constant global batch size scaling. In constant local
batch size scaling, the global batch size increases with the number of GPUs.
In constant global batch size scaling, we maintain a constant global batch
size while proportionally decreasing the local batch size as we increase the
number of GPUs. This emphasizes the communication overhead reductions achieved
by our method. Measurements represent the average of three runs, each of which
utilizes a different random seed. Models are initialized using orthogonal
initialization (Hu et al., 2020).

###  4.1. Computational Environment

We ran all experiments on a distributed-memory, parallel hybrid supercomputer.
Each compute node is equipped with two 38-core Intel Xeon Platinum 8368
processors at $2.4\text{\,}\mathrm{GHz}$ base and $3.4\text{\,}\mathrm{GHz}$
maximum turbo frequency, $512\text{\,}\mathrm{GB}$ local memory, a local
$960\text{\,}\mathrm{GB}$ NVMe SSD disk, two network adapters, and four NVIDIA
A100-40 GPUs with $40\text{\,}\mathrm{GB}$ memory connected via NVLink. Inter-
node communication uses a low-latency, non-blocking NVIDIA Mellanox InfiniBand
4X HDR interconnect with
$200\text{\,}\mathrm{G}\mathrm{bit}\mathrm{/}\mathrm{s}$ per port. All
experiments used Python 3.10.6 with CUDA-enabled PyTorch 2.0.0 (Paszke et al.,
2017).

###  4.2. Datasets and Models

We used the ImageNet-2012 dataset, which contains 1.2 million images, for our
scaling experiments. Basic image augmentation was applied, including
normalization, random resizing, random cropping, and a random flip. We trained
two models on this dataset: ResNet-50 (He et al., 2016) and Vision Transformer
B/16 (ViT) (Dosovitskiy et al., 2021), chosen for their widespread use and
distinct architectures.

We also trained VGG16 on the CIFAR-10 dataset, which contains 50,000 images.
Basic image augmentation was applied in the same way as for the ImageNet
experiments.

All models were trained with the AdamW (Loshchilov and Hutter, 2019)
optimizer. Other training hyperparameters can be found in Appendix 1.

###  4.3. Hyperparameter Considerations

Careful hyperparameter tuning is essential for balancing the communication
reduction achieved through our method with potential impacts on training
stability and accuracy. Key hyperparameters (HPs) include the warmup and full-
rank rebound phase durations, the number of AB training iterations, the AB
decomposition rank-reduction parameter, and the frequency of SVD and
synchronization steps. Additionally, the latter will influence the trade-offs
between regularization and communication efficiency.

Our hyperparameter search on CIFAR-100 (Krizhevsky, 2009) with a minified
Vision Transformer model (patch size of eight instead of 16, six heads instead
of twelve, depth of six instead of twelve) using Propulate (Taubert et al.,
2023) suggested the following guidelines:

  * •

warmupSteps: 20% of the total training steps

  * •

numABSteps: 3.33% of the total training steps

  * •

fullRankReboundSteps: $0.25\cdot\texttt{numABSteps}$

  * •

Learning rate rebound steps: $0.5\cdot\texttt{numABSteps}$

###  4.4. Constant Local Batch Size Scaling

In this experiment, we maintain a fixed local batch size while increasing the
available compute resources and decreasing the number of training steps per
epoch, resulting in an increasing global batch size but less training
iterations. For example, if the global batch size for a two-node (eight-GPU)
run is 2,048 and it is trained for 600 iterations in each epoch, the
corresponding four-node (16-GPU) run would have a global batch size of 4,096
and be trained for 300 iterations each epoch. This strategy investigates the
communication efficiency gains achievable as we increase the available compute
resources, even as the potential negative impacts of large batch sizes become
more pronounced. It is the most likely method by which a non-expert user will
scale their training.

This experiment tests how well AB training reduces communication requirements
and if the independent training groups can effectively train networks
independently. We expect that if the independent training groups make similar
updates, the averaged model will maintain accuracy and exhibit increased
compression. However, significant divergence in individual groups could lead
to lower accuracy and reduced compression as the individual updates conflict.
If this is the case, then AB training without independent groups will begin to
outperform AB training with groups.

Figure 3 shows the highest top 1 accuracy achieved by each model during
training. Figure 4 shows the lowest binary cross-entropy loss values achieved
across all epochs. The compression values for the trained models are shown in
Figure 5.

(a) Vision Transformer B/16

(b) ResNet-50

Figure 3. Highest top 1 accuracy measurements for each training run on
ImageNet-2012 for two network architectures with a constant local batch size
of 256. Global batch sizes range from 2,048 to 32,768 in powers of 2. Error
are plotted, though not always visible.

Highest top 1 accuracy measurements for training runs on ImageNet-2012 for two
network architectures with a constant local batch size of 256. Global batch
sizes range from 2,048 to 32,768 in powers of 2.

(a) Vision Transformer B/16

(b) ResNet-50

Figure 4. Lowest binary cross-entropy (the loss function used during training)
measurements during each training run on ImageNet-2012 for two network
architectures with a constant local batch size of 256. Global batch sizes
range from 2,048 to 32,768 in powers of 2. Error are plotted, though not
always visible.

Lowest binary cross-entropy (the loss function used during training)
measurements for training runs on ImageNet-2012 for two network architectures
with a constant local batch size of 256. Global batch sizes range from 2,048
to 32,768 in powers of 2. AB training is shown to have a regularization effect
for both ResNet-50 and the Vision Transformer. Large batch effects are also
visible as the loss values get worse (larger) as the global batch size
increases.

Figure 5. Average compression ratios for AB trained models against the
traditional data parallel baseline trained models on ImageNet-2012 for two
network architectures with a constant local batch size of 256.

Compression ratios for AB trained models against the traditional data parallel
baseline trained models on ImageNet-2012 for two network architectures with a
constant local batch size of 256.

###  4.5. Constant Global Batch Size Scaling

In this experiment, we maintain a fixed global batch size while scaling
compute resources. This contrasts with the standard approach of keeping local
batch sizes constant, emphasizing the communication overhead reductions
achieved by AB training. With a fixed global batch size of 4,096, a four-node
run would have a local batch size of 256, whereas an eight-node run would have
a local batch size of 128.

This experiment directly quantifies the communication benefits of our method
in scenarios where local batch sizes become very small due to scaling. It also
provides insight into the ability of independent training groups to learn
meaningful representations when the batch size used for independent training
is much smaller than the global batch size. Results for these runs are shown
in Figures 6 and 7.

(a) Vision Transformer B/16

(b) ResNet-50

Figure 6. Highest top 1 accuracy measurements for each training run on
ImageNet-2012 for two network architectures with a constant global batch size
of 4,096. Error are plotted, though not always visible.

Highest top 1 accuracy measurements for training runs on ImageNet-2012 for two
network architectures with a constant global batch size of 4,096.

Figure 7. Compression ratios for AB trained models against the traditional
data parallel baseline trained models on ImageNet-2012 for two network
architectures with a constant global batch size of 4,096.

Compression ratios for AB trained models against the traditional data parallel
baseline trained models on ImageNet-2012 for two network architectures with a
constant global batch size of 4,096.

##  5\. Discussion

Table 1. Average interconnect traffic, in GB/s, on each processes. $N$ represents either the number of nodes for the constant global batch size experiments or the global batch size in thousands for the constant local batch size experiments. |  |  | Constant Local
---
Batch Size
| Constant Global
---
Batch Size
| $N$ | Baseline | AB | Baseline | AB
ViT | 2 | $4.67\pm 0.00$ | $2.10\pm 0.07$ | – | –
4 | $4.77\pm 0.06$ | $2.12\pm 0.01$ | $4.81\pm 0.05$ | $1.86\pm 0.39$
8 | $4.58\pm 0.00$ | $1.76\pm 0.29$ | $7.05\pm 1.03$ | $3.45\pm 0.18$
16 | $4.44\pm 0.10$ | $1.74\pm 0.07$ | $13.66\pm 0.16$ | $5.33\pm 0.70$
32 | $4.07\pm 0.20$ | $1.58\pm 0.24$ | $14.09\pm 3.47$ | $7.88\pm 0.02$
ResNet-50 | 2 | $2.06\pm 0.01$ | $1.02\pm 0.00$ | – | –
4 | $2.07\pm 0.01$ | $0.98\pm 0.05$ | $2.06\pm 0.01$ | $1.00\pm 0.01$
8 | $2.00\pm 0.05$ | $1.05\pm 0.05$ | $2.67\pm 0.05$ | $1.16\pm 0.14$
16 | $1.96\pm 0.12$ | $1.15\pm 0.00$ | $3.28\pm 0.25$ | $1.39\pm 0.21$
32 | $1.62\pm 0.03$ | $1.05\pm 0.01$ | $3.50\pm 1.00$ | $1.76\pm 0.47$

Across both scaling scenarios, we find a consistent reduction of approximately
50% on average network traffic, demonstrating the effectiveness of our AB
training method in reducing communication overheads. Comprehensive results are
listed in Table 1, which further highlights the significant bandwidth
requirements for training even moderately sized neural networks, a challenge
that would be exacerbated for larger, state-of-the-art models.

In 13 of the 18 experiments AB training maintains competitive performance
compared to the traditional baseline using the same hyperparameters, Figures 3
and 6. Without the independently trained groups, the performance is noticeably
reduced, see Figure 3(a), while the compression values are similar, Figure 5.

The interconnect on the test system has a bandwidth of
$25\text{\,}\mathrm{G}\mathrm{B}\mathrm{/}\mathrm{s}$ per port. The baseline
experiment which required the most bandwidth, the ViT constant global batch
size experiment using 32 nodes, used only
$14.10\text{\,}\mathrm{G}\mathrm{B}\mathrm{/}\mathrm{s}$. As we did not fully
saturate the interconnect, the runtimes remained similar across all three
methods. However, in environments with more severe communication bottlenecks,
the benefits of our method would become more pronounced.

Across all experiments, we observed favorable compression ratios (Figures 7
and 5). For runs which reached the same or better loss value, the Vision
Transformer’s compression ratios range from 1.89:1 to 2.54:1. ResNet-50’s
compression was less impressive, although still respectable (ranging from
1.19:1 to 1.72:1 for outperforming runs). This suggests a potential
interaction between model architecture and the effectiveness of our method.

Table 2 demonstrates the effectiveness of AB training compared to existing
low-rank and pruning methods. We present both a standard benchmark (ResNet-50
on ImageNet-2012) and an idealized scenario (VGG16 on CIFAR-10). The latter
highlights the potential of compression techniques due to VGG16’s over-
parameterization and CIFAR-10’s relative simplicity (Chang et al., 2023). AB
training outperformed expectations in this setting, achieving a 44.14:1
compression ratio (reducing the model to 2.27% of its original size) with
negligible accuracy loss. This surpasses the compression achieved by ICP
(3.67% of original size) and ABCPrune (11.26%).

To compare the communication required for training the other model compression
methods shown in Table 2 with traditional DP, we estimate their maximum
communication savings achievable. As all of these methods iteratively reduce
the network size, we assume that the all start with the full-rank
representation and remove parameters during training. In out experiments, we
found that once the network began to compress the network, it removed much of
the network quickly, then removed small amounts during the remaining training
steps. However, we do not know the rate at which parameters are removed.
Taking this into account, we assume that models train close to full rank for
25% of training then close to their final model state for 75% of training. We
describe the estimated communication reduction (ECR) as

(4) $\text{ECR}=100\%-F-Lc$

where $F$ and $L$ are the percentages of training spent near full-rank and the
most-compressed network state respectively and $c$ is the final compression
ratio expressed as a percentage.

AB training demonstrates the greatest communication reduction in the ResNet-50
benchmark with the ECR. In practice, we measured an approximate reduction of
50% for all use-cases. This discrepancy is caused by other communications
needed during training outside of the model synchronization. With the
measurement, we outperform all models except OIALR and DLRT, which reduce the
number of trainable parameters by training the $\Sigma$ matrix from a
network’s weight’s SVD. In the VGG16 scenario, ICP (Chang et al., 2023) and AB
training without using individual groups show slightly higher extimated
communication reductions than AB training itself. This suggests that AB
training offers superior communication efficiency until models reach extremely
high compression levels.

While promising, our method also reveals challenges. As expected, we observe
generalization improvements in all experiments, these are most noticable in
Figure 4. This is likely due to the low-rank representation in combination
with the independently trained groups; both of which have regularizing effects
(see ‘No Groups’ measurements in Figure 6). However, at larger scales, both
loss and accuracy degrade significantly. Combined with the decreasing
compression ratios seen with increasing global batch size (Figure 5), this
suggests that the averaging of independently trained subgroups becomes
detrimental at extreme scales.

Similar trends in the fixed global batch size scenario (Figure 6) suggest an
incompatibility between the very large global batch size and the much smaller
batches used during the independent training phases. This likely leads to
overly large updates in the independent groups, causing them to diverge from
each other. Although this divergence increases the regularization effect of
independent group training, is reduces the accuracy of the network. Our
compression results (Figure 7) support this hypothesis. As the regularization
effect strengthens with scale, the divergences between independent groups
occur primarily in the smaller, yet important, singular values. When these
divergences occur, averaging them together will cause the divergent singular
values to shrink relative to the other values.

These challenges likely stem from suboptimal model merging or overly
aggressive learning rate schedules. Since it is common practice to adjust the
learning rate based on global batch size in traditional DP training, a similar
strategy may be required for our method.

Our findings highlight the need for further exploration across diverse models,
datasets, and hardware configurations. Understanding the interplay between
network bandwidth limitations, and our method’s potential benefits in real-
world environments is crucial. Finally, the observed accuracy degradation at
scale motivates research into improved update mechanisms, potentially
exploring non-average update rules, mixing matrices, or loss-based weighted
averaging schemes.

Table 2.  Comparison of low-rank and pruning methods for ResNet-50 (on ImageNet-2012) and VGG16 (on CIFAR10). ’Difference to Baseline’ indicates validation top 1 performance relative to the original full-rank model in each study, with positive values denoting better performance. AB training used a global batch size of 4,096 for ImageNet and 1,024 for CIFAR10, achieving maximum top 1 accuracies of 75.67% and 91.87% respectively. The estimated communication reduction (ECR) is defined by Equation 4. AB training’s ECR assumes independent groups do not utilize the compute system’s interconnect. OIALR’s and DLRT’s ECR uses the compression of the trainable parameters as they report. | Method |  | Difference
---
to Baseline
| Compression
---
Ratio
ECR
ResNet-50 ImageNet-2012 | AB | +1.55 % | $1.39:1$ |  73.29 %*
AB (no groups) | -0.02 % | $1.27:1$ | 15.94 %
OIALR (Coquelin et al., 2024) | -1.72 % | $1.21:1$ | 63.64 %
DLRT (Schotthöfer et al., 2022) | -0.56 % | $1.85:1$ | 64.35 %
PP-1 (Singh et al., 2019) | -0.20 % | $2.26:1$ | 41.81 %
CP (He et al., 2017) | -1.40 % | $2.00:1$ | 37.5 %
SFP (He et al., 2018) | -0.20 % | $2.39:1$ | 43.62 %
ThiNet (Luo et al., 2017) | -1.50 % | 2.71 : 1 | 47.32 %
ABCPrune (Lin et al., 2020a) | -2.15 % | 1.84 : 1 | 34.24 %
VGG16 CIFAR-10 | AB | -0.23 % | 44.14 : 1 | 65.60 %*
AB (no groups) | -0.89 % | 36.63 : 1 | 72.95 %
OIALR (Coquelin et al., 2024) | +0.10 % | 3.70 : 1 | 64.59 %
DLRT (Schotthöfer et al., 2022) | -1.89 % | 1.79 : 1 | 16.88 %
ABCPrune (Lin et al., 2020a) | +0.06 % | 8.83 : 1 | 66.51 %
ICP (Chang et al., 2023) | -0.31 % | 27.25 : 1 | 72.25 %

##  6\. Conclusion

Our experimental results demonstrate the significant potential of our AB
training method to reduce communication overheads by utilizing low-rank
representations and independently trained worker subgroups. The consistent 50%
reduction in network traffic has the potential to unlock new possibilities
within distributed-memory computing environments. This could be particularly
impactful for scientific research on machines without high-speed interconnects
and for exploring even larger, more complex models. Additionally, the
pronounced regularization effects observed at smaller scales offer a promising
direction for improving generalization and performance.

While our method successfully addresses communication challenges, its
performance at extreme scales highlights the complex interplay between low-
rank representations, large batch effects, and hyperparameter optimization.
Our findings emphasize the need for further research into tailored
hyperparameter strategies and novel update mechanisms to fully harness the
potential of this approach in massively scaled scenarios.

This work represents a significant step towards communication-efficient
distributed training. Our results offer valuable insights and highlight
promising areas for future investigation. By addressing the limitations
uncovered in our study, we believe the research community can pave the way for
even more efficient and scalable training of large-scale neural networks,
ultimately furthering scientific discovery across a variety of domains.

###### Acknowledgements.

This work was performed on the HoreKa supercomputer funded by the Ministry of
Science, Research and the Arts Baden-Württemberg and by the Federal Ministry
of Education and Research. This work is supported by the Helmholtz Association
Initiative and Networking Fund under the Helmholtz AI platform grant and the
HAICORE@KIT partition.

## References

  * (1)
  * Abrahamyan et al. (2022) Lusine Abrahamyan, Yiming Chen, Giannis Bekoulis, and Nikos Deligiannis. 2022.  Learned Gradient Compression for Distributed Deep Learning.  _IEEE Transactions on Neural Networks and Learning Systems_ 33, 12 (2022), 7330–7344.  https://doi.org/10.1109/TNNLS.2021.3084806
  * Ben-Nun and Hoefler (2019) Tal Ben-Nun and Torsten Hoefler. 2019.  Demystifying Parallel and Distributed Deep Learning: An In-depth Concurrency Analysis.  _ACM Comput. Surv._ 52, 4, Article 65 (aug 2019), 43 pages.  https://doi.org/10.1145/3320060
  * Cahyawijaya et al. (2021) S. Cahyawijaya, G. I. Winata, H. Lovenia, et al. 2021\.  Greenformer: Factorization Toolkit for Efficient Deep Neural Networks.  https://doi.org/10.48550/arXiv.2109.06762 arXiv:2109.06762 [cs].
  * Chang et al. (2023) Jingfei Chang, Yang Lu, Ping Xue, Yiqun Xu, and Zhen Wei. 2023.  Iterative clustering pruning for convolutional neural networks.  _Knowledge-Based Systems_ 265 (2023), 110386.  https://doi.org/10.1016/j.knosys.2023.110386
  * Chaudhari et al. (2019) Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. 2019.  Entropy-SGD: biasing gradient descent into wide valleys*.  _Journal of Statistical Mechanics: Theory and Experiment_ 2019, 12 (dec 2019), 124018.  https://doi.org/10.1088/1742-5468/ab39d9
  * Coquelin et al. (2022) Daniel Coquelin, Charlotte Debus, Markus Götz, Fabrice von der Lehr, James Kahn, Martin Siggel, and Achim Streit. 2022.  Accelerating neural network training with distributed asynchronous and selective optimization (DASO).  _Journal of Big Data_ 9, 1 (Feb. 2022), 14.  https://doi.org/10.1186/s40537-021-00556-1
  * Coquelin et al. (2024) Daniel Coquelin, Katharina Flügel, Marie Weiel, Nicholas Kiefer, Charlotte Debus, Achim Streit, and Markus Götz. 2024.  Harnessing Orthogonality to Train Low-Rank Neural Networks.  https://doi.org/10.48550/arXiv.2401.08505 arXiv:2401.08505 [cs].
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  https://doi.org/10.48550/arXiv.2010.11929 arXiv:2010.11929 [cs].
  * Giuseppi et al. (2022) Alessandro Giuseppi, Sabato Manfredi, and Antonio Pietrabissa. 2022.  A Weighted Average Consensus Approach for Decentralized Federated Learning.  _Machine Intelligence Research_ 19, 4 (Aug. 2022), 319–330.  https://doi.org/10.1007/s11633-022-1338-z
  * Goyal et al. (2018) Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2018.  Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.  https://doi.org/10.48550/arXiv.1706.02677 arXiv:1706.02677 [cs].
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep Residual Learning for Image Recognition. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 770–778.  https://doi.org/10.1109/CVPR.2016.90 ISSN: 1063-6919.
  * He et al. (2018) Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. 2018.  Soft filter pruning for accelerating deep convolutional neural networks. In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ _(IJCAI’18)_. AAAI Press, Stockholm, Sweden, 2234–2240.
  * He et al. (2017) Yihui He, Xiangyu Zhang, and Jian Sun. 2017.  Channel Pruning for Accelerating Very Deep Neural Networks. In _2017 IEEE International Conference on Computer Vision (ICCV)_. 1398–1406.  https://doi.org/10.1109/ICCV.2017.155 ISSN: 2380-7504.
  * Hu et al. (2020) Wei Hu, Lechao Xiao, and Jeffrey Pennington. 2020.  Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=rkgqN1SYvr
  * Hu et al. (2021) Zhanxuan Hu, Feiping Nie, Rong Wang, and Xuelong Li. 2021.  Low Rank Regularization: A review.  _Neural Networks: The Official Journal of the International Neural Network Society_ 136 (April 2021), 218–232.
  * Krizhevsky (2009) A. Krizhevsky. 2009.  Learning Multiple Layers of Features from Tiny Images.  https://www.semanticscholar.org/paper/Learning-Multiple-Layers-of-Features-from-Tiny-Krizhevsky/5d90f06bb70a0a3dced62413346235c02b1aa086
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012.  ImageNet Classification with Deep Convolutional Neural Networks. In _Advances in Neural Information Processing Systems_ , Vol. 25. Curran Associates, Inc.  https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
  * L. and H. (2018) Ilya L. and Frank H. 2018.  Fixing Weight Decay Regularization in Adam.  https://openreview.net/forum?id=rk6qdGgCZ
  * Liebenwein et al. (2021) Lucas Liebenwein, Alaa Maalouf, Dan Feldman, and Daniela Rus. 2021.  Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition. In _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 5328–5344.  https://proceedings.neurips.cc/paper_files/paper/2021/file/2adcfc3929e7c03fac3100d3ad51da26-Paper.pdf
  * Lin et al. (2020a) Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, and Yonghong Tian. 2020a.  Channel Pruning via Automatic Structure Search. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , Christian Bessiere (Ed.). International Joint Conferences on Artificial Intelligence Organization, 673–679.  https://doi.org/10.24963/ijcai.2020/94 Main track.
  * Lin et al. (2020b) Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. 2020b.  Don’t Use Large Mini-Batches, Use Local SGD.  https://doi.org/10.48550/arXiv.1808.07217 arXiv:1808.07217 [cs, stat].
  * Liu and Deng (2015) Shuying Liu and Weihong Deng. 2015.  Very deep convolutional neural network based image classification using small training sample size. In _2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)_. 730–734.  https://doi.org/10.1109/ACPR.2015.7486599
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled Weight Decay Regularization.  https://doi.org/10.48550/arXiv.1711.05101 arXiv:1711.05101 [cs, math].
  * Luo et al. (2017) Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. 2017.  ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression. In _2017 IEEE International Conference on Computer Vision (ICCV)_. 5068–5076.  https://doi.org/10.1109/ICCV.2017.541 ISSN: 2380-7504.
  * Niu et al. (2011) Feng Niu, Benjamin Recht, Christopher Re, and Stephen J. Wright. 2011.  HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent.  https://doi.org/10.48550/arXiv.1106.5730 arXiv:1106.5730 [cs, math].
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017.  Automatic differentiation in PyTorch.  (Oct. 2017).  https://openreview.net/forum?id=BJJsrmfCZ
  * Phan et al. (2020) A.-H. Phan, K. Sobolev, K. Sozykin, et al. 2020\.  Stable Low-Rank Tensor Decomposition for Compression of Convolutional Neural Network. In _Computer Vision – ECCV 2020_ _(Lecture Notes in Computer Science)_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm (Eds.). Springer International Publishing, Cham, 522–539.  https://doi.org/10.1007/978-3-030-58526-6_31
  * Schotthöfer et al. (2022) Steffen Schotthöfer, Emanuele Zangrando, Jonas Kusch, Gianluca Ceruti, and Francesco Tudisco. 2022.  Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations.  _Advances in Neural Information Processing Systems_ 35 (Dec. 2022), 20051–20063.  https://papers.nips.cc/paper_files/paper/2022/hash/7e98b00eeafcdaeb0c5661fb9355be3a-Abstract-Conference.html
  * Singh et al. (2019) Pravendra Singh, Vinay Kumar Verma, Piyush Rai, and Vinay P. Namboodiri. 2019.  Play and Prune: Adaptive Filter Pruning for Deep Model Compression.  (2019), 3460–3466.  https://www.ijcai.org/Proceedings/2019/480
  * Taubert et al. (2023) Oskar Taubert, Marie Weiel, Daniel Coquelin, Anis Farshian, Charlotte Debus, Alexander Schug, Achim Streit, and Markus Götz. 2023.  Massively Parallel Genetic Optimization Through Asynchronous Propagation of Populations. In _High Performance Computing_ , Abhinav Bhatele, Jeff Hammond, Marc Baboulin, and Carola Kruse (Eds.). Springer Nature Switzerland, Cham, 106–124.  https://doi.org/10.1007/978-3-031-32041-5_6
  * Vogels et al. (2019) Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. 2019.  PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization. In _Advances in Neural Information Processing Systems_ , Vol. 32. Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2019/hash/d9fbed9da256e344c1fa46bb46c34c5f-Abstract.html
  * Wang et al. (2021) Hongyi Wang, Saurabh Agarwal, and Dimitris Papailiopoulos. 2021.  Pufferfish: Communication-efficient Models At No Extra Cost.  _ArXiv_ abs/2103.03936 (2021).  https://api.semanticscholar.org/CorpusID:232148049
  * Wang et al. (2023a) Hongyi Wang, Saurabh Agarwal, Pongsakorn U-chupala, Yoshiki Tanaka, Eric P. Xing, and Dimitris Papailiopoulos. 2023a.  Cuttlefish: Low-Rank Model Training without All the Tuning.  _ArXiv_ abs/2305.02538 (2023).  https://api.semanticscholar.org/CorpusID:258480187
  * Wang et al. (2023b) Haolin Wang, Xuefeng Liu, Jianwei Niu, and Shaojie Tang. 2023b.  SVDFed: Enabling Communication-Efficient Federated Learning via Singular-Value-Decomposition. In _IEEE INFOCOM 2023 - IEEE Conference on Computer Communications_. 1–10.  https://doi.org/10.1109/INFOCOM53939.2023.10229042
  * Winata et al. (2020) G. I. Winata, S. Cahyawijaya, Z. Lin, Z. Liu, and P. Fung. 2020.  Lightweight and Efficient End-To-End Speech Recognition Using Low-Rank Transformer. In _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 6144–6148.  https://doi.org/10.1109/ICASSP40776.2020.9053878 ISSN: 2379-190X.
  * Yang et al. (2020a) Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. 2020a.  Learning Low-Rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_.
  * Yang et al. (2020b) Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen. 2020b.  Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. 2899–2908.  https://doi.org/10.1109/CVPRW50498.2020.00347