# LoRA+: Efficient Low Rank Adaptation
of Large Models

Soufiane Hayou
Simons Institute
UC Berkeley
<EMAIL_ADDRESS>
&Nikhil Ghosh∗
Dept of Statistics
UC Berkeley
<EMAIL_ADDRESS>
&Bin Yu
Dept of Statistics
UC Berkeley
<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

In this paper, we show that Low Rank Adaptation (LoRA) as originally
introduced in [17] leads to suboptimal finetuning of models with large width
(embedding dimension). This is due to the fact that adapter matrices $A$ and
$B$ in LoRA are updated with the same learning rate. Using scaling arguments
for large width networks, we demonstrate that using the same learning rate for
$A$ and $B$ does not allow efficient feature learning. We then show that this
suboptimality of LoRA can be corrected simply by setting different learning
rates for the LoRA adapter matrices $A$ and $B$ with a well-chosen fixed
ratio. We call this proposed algorithm LoRA$+$. In our extensive experiments,
LoRA$+$ improves performance ($1\%-2\%$ improvements) and finetuning speed (up
to $\sim 2$X SpeedUp), at the same computational cost as LoRA.

##  1 Introduction

State-of-the-art (SOTA) deep learning models all share a common
characteristic: they all have an extremely large number of parameters (10’s if
not 100’s of billions parameters). Currently, only a few industry labs can
pretrain large language models due to their high training cost. However, many
pretrained models are accessible either through an API (GPT4, [32]) or through
open-source platforms (Llama, [33]). Most practitioners are interested in
using such models for specific tasks and want to _adapt_ these models to a
new, generally smaller task. This procedure is known as _finetuning_ , where
one adjusts the weights of the pretrained model to improve performance on the
new task. However, due to the size of SOTA models, adapting to down-stream
tasks with full finetuning (finetuning all model parameters) is
computationally infeasible as it requires modifying the weights of the
pretrained models using gradient methods which is a costly process.

Besides, a model that has already learned generally useful representations
during pretraining would not require in-principle significant adaptation of
all parameters. With this intuition, researchers have proposed a variety of
resource-efficient finetuning methods which typically freeze the pretrained
weights and tune only a small set of newly inserted parameters. Such methods
include prompt tuning ([18]) where a “soft prompt" is learned and appended to
the input, the adapters method ([10]) where lightweight “adapter" layers are
inserted and trained, and $(IA)^{3}$ ([21]) where activation vectors are
modified with learned scalings. Another resource-efficient method is known as
_Low Rank Adaptation_ ([17]), or simply LoRA. In LoRA finetuning, only a low
rank matrix, called an adapter, that is added to the pretrained weights is
trainable. The training can be done with any optimizer and in practice a
common choice is Adam [3]. Since the trained adapter is low-rank, this
effectively reduces the number of trainable parameters in the fine-tuning
process, significantly decreasing the training cost. On many tasks such as
instruction finetuning, LoRA has been shown to achieve comparable or better
performance compared with full-finetuning [34, 29], although on complicated,
long form generation tasks, it is not always as performant. The generally high
performance level and the computational savings of LoRA have contributed to it
becoming an industry standard finetuning method.

Efficient use of LoRA requires a careful choice of multiple hyperparameters
including the rank and the learning rate. While some theoretical guidelines on
the choice of the rank in LoRA exist in the literature (see [37] where the
authors give a lower bound on the rank of LoRA layers in order to adapt a
given network to approximate a network of smaller size), there are no
principled guidelines on how to set the learning rate, apart from common
choices of learning rates of order $1e$-$4$.

Figure 1: The key difference between standard LoRA and LoRA$+$ is in how
learning rates are set (the matrices $G_{A}$ and $G_{B}$ are ‘effective’
gradients after Adam processing) With standard LoRA, the learning rate is the
same for $A$ and $B$, which provably leads to inefficient finetuning when
embedding dimension is large. In LoRA$+$, we set the learning rate of $B$ to
be $\lambda\times$ that of $A$, where $\lambda\gg 1$ is a fixed ratio. We
later provide guidelines on how to choose $\lambda$.

##### Related Work.

Building upon the LoRA recipe, [24] introduced a quantized version of LoRA (or
QLoRA), which further reduces computation costs substantially by quantizing
pretrained weights down to as few as four bits. Using QLoRA enables fine-
tuning of models as large as Llama-65b [33], on a single consumer GPU while
achieving competitive performance with full-finetuning. To further improve
LoRA training with quantization, [28] introduced a new method called LoftQ for
computing a better initialization for quantized training. Additional
variations of LoRA have been proposed such as VeRA ([27]) which freezes random
weight tied adapters and learns vector scalings of the internal adapter
activations. This achieves a further reduction in the number of trainable
parameters while achieving comparable performance to LoRA on several NLP
finetuning tasks. However, to the best of our knowledge, there is no
principled guidance for setting LoRA learning rate which is the focus of our
work.

##### Contributions.

In this paper, we propose to provide guidelines for setting the learning rate
through a theory of scaling for neural networks. There is a significant number
of works on the scaling of neural networks from the infinite width/depth
perspective. The approach is simple: take the width/depth of a neural network
to infinity,111Depending on the model, one might want to scale width with
fixed depth and vice-versa, or both at the same time. See Section A.1 for more
details. understand how the behavior of the limit depends on the choice of the
hyperparameters in the training process such as the learning rate and
initialization variance, then derive principled choices for these
hyperparameters to achieve some desired goal (e.g. improve feature learning).
Examples of the infinite-width limit include works on initialization schemes
such as [4, 13], or more holistically network parametrizations such as [19]
where the authors introduced $\mu$P, a neural network parameterization
ensuring feature learning in the infinite-width limit, offering precise
scaling rules for architecture and learning rates to maximize feature
learning. Examples for the depth limit include initialization strategies [6,
26, 9], block scaling (see e.g. [16, 25, 31]), depth parametrizations [36, 23]
etc. Here we propose to use the same strategy to derive scaling rules for the
learning rate in LoRA for finetuning. More precisely, we study the infinite-
width limit of LoRA finetuning dynamics and show that standard LoRA setup is
suboptimal. We correct this by introducing a new method called LoRA$+$ that
improves feature learning in low rank adaptation in the this limit. The key
innovation in LoRA$+$ is setting different learning rates for $A$ and $B$
modules (LoRA modules) as explained in Figure 1. Our theory is validated with
extensive empirical results with different language of models and tasks.

##  2 Setup and Definitions

Our methodology in this paper is model agnostic and applies to general neural
network models. Let us consider a neural network of the form

$\begin{cases}Y_{in}(x)=W_{in}x,\\\
Y_{l}(x)=\mathcal{F}_{l}(W_{l},Y_{l-1}(x)),\;l\in[L],\\\
Y_{out}(x)=W_{out}Y_{L}(x),\end{cases}$ (1)

where $x\in\mathbb{R}^{d}$ is the input, $L\geq 1$ is the network depth,
$(\mathcal{F}_{l})_{l\in[L]}$ are mappings that define the layers,
$W_{l}\in\mathbb{R}^{n\times n}$ are the hidden weights, where $n$ is the
network width, and $W_{in},W_{out}$ are input and output embedding weights.

Model (1) is _pretrained_ on some dataset $\mathcal{D}$ to perform some
specified task (e.g. next token prediction). Let
$\mathbf{W}=(W_{in},(W_{l})_{l\in[L]},W_{out})$. Pretraining is done by
minimizing the empirical loss function

$\mathcal{L}(\mathbf{W})=\hat{\mathbb{E}}_{(x,y)\sim\mathcal{D}}[\ell(Y_{out}(x),y),$

where $\ell$ is a loss function (e.g. cross-entropy), $Y_{out}$ is the output
of the network, and $\hat{\mathbb{E}}$ refers to the empirical expectation
(average over training data).

Once the model is pretrained, one can finetune it to improve performance on
some downstream task. To achieve this with relatively small devices (limited
GPUs), resource-efficient finetuning methods like LoRA significantly reduce
the computational cost by considering low rank weight matrices instead of full
rank finetuning (or simply full finetuning).

######  Definition 1 (Low Rank Adapters (LoRA) from [17]).

For any weight matrix $W\in\mathbb{R}^{n_{1}\times n_{2}}$ in the pretrained
model, we constrain its update in the fine-tuning process by representing the
latter with a low-rank decomposition $W=W^{*}+\frac{\alpha}{r}BA$. Here, only
the weight matrices $B\in\mathbb{R}^{n_{1}\times r}$, $A\in\mathbb{R}^{r\times
n_{2}}$ are trainable. The rank $r\ll\min(n_{1},n_{2})$ and
$\alpha\in\mathbb{R}$ are tunable constants.

##### Scaling of Neural Networks.

It is well known that as the width $n$ grows, the network initialization
scheme and the learning should be adapted to avoid numerical instabilities and
ensure efficient learning. For instance, the variance of the initialization
weights (in hidden layers) should scale $1/n$ to prevent arbitrarily large
pre-activations as we increase model width $n$ (e.g. He init [4]). To derive
such scaling rules, a principled approach consist of analyzing statistical
properties of key quantities in the model (e.g. pre-activations) as $n$ grows
and then adjust the initialization, the learning rate, and the architecture
itself to achieve desirable properties in the limit $n\to\infty$ [9, 5, 13,
35]. This approach is used in this paper to study feature learning dynamics
with LoRA in the infinite-width limit. This will allow us to derive scaling
rules for the learning rates of LoRA modules. For more details about the
theory of scaling of neural networks, see Section A.1.

Throughout the paper, we will be using asymptotic notation to describe the
behaviour of several quantities as the width $n$ grows.

##### Notation.

Hereafter, we use the following notation to describe the asymptotic behaviour
as the width $n$ grows. Given sequences $c_{n}\in\mathbb{R}$ and
$d_{n}\in\mathbb{R}^{+}$, we write $c_{n}=\mathcal{O}(d_{n})$, resp.
$c_{n}=\Omega(d_{n})$, to refer to $c_{n}<\kappa d_{n}$, resp. $c_{n}>\kappa
d_{n}$, for some constant $\kappa>0$. We write $c_{n}=\Theta(d_{n})$ if both
$c_{n}=\mathcal{O}(d_{n})$ and $c_{n}=\Omega(d_{n})$ are satisfied. For vector
sequences $c_{n}=(c_{n}^{i})_{1\leq i\leq k}\in\mathbb{R}^{k}$ (for some
$k>0$), we write $c_{n}=\mathcal{O}(d_{n})$ when
$c_{n}^{i}=\mathcal{O}(d_{n}^{i})$ for all $i\in[k]$, and same holds for other
asymptotic notations. Finally, when the sequence $c_{n}$ is a vector of random
variables, convergence is understood to be convergence in second moment
($L_{2}$ norm).

##  3 An Intuitive Analysis of LoRA

In this section, we provide an intuitive analysis that naturally leads to our
proposed method. Our intuition is simple: the matrices $A$ and $B$ have
“transposed” shapes and one would naturally ask whether the learning rate
should be set differently for the two matrices. In practice, most SOTA models
have large width (embedding dimension). Thus, it makes sense to study the
training dynamics when the width goes to infinity. More precisely, we are
interested in the question:

_How should we set the learning rates of LoRA modules $A$ and $B$ as model
width $n\to\infty$ ?_

Using a simple linear model, we characterize the infinite-width finetuning
dynamics and infer how ‘optimal’ learning rates should behave as a function of
width.

###  3.1 LoRA with a Toy Model

Consider the following linear model

$f(x)=(W^{*}+ba^{\top})x,$ (2)

where $W^{*}\in\mathbb{R}^{1\times n}$ are the pretrained weights,
$b\in\mathbb{R},a\in\mathbb{R}^{n}$ are LoRA weights,222Here, we consider
$n_{2}=1$ to simplify the analysis. All the conclusions remain essentially
valid when $n_{2}=n_{1}=n$. $x\in\mathbb{R}^{n}$ is the model input. This
setup corresponds to $n_{1}=1,n_{2}=n,r=1$ in 1. We assume that the weights
$W^{*}$ are fixed (from pretraining). The goal is to minimize the loss
$\mathcal{L}(\theta)=\frac{1}{2}(f(x)-y)^{2}$ where $\theta=(a,b)$ and $(x,y)$
is an input-output pair.333For simplicity, we assume that the finetuning
dataset consists of a single sample. Our analysis is readily generalizable to
multiple samples. We assume that $x=\Theta_{n}(1)$ which means that input
coordinates remain roughly of the same order as we increase width. In the
following, we analyze the behaviour of the finetuning dynamics as model width
$n$ grows. We begin by the behaviour at initialization.

##### Initialization.

We consider a Gaussian initialization of the weights as follows:
$a_{i}\sim\mathcal{N}(0,\sigma_{a}^{2})$,
$b\sim\mathcal{N}(0,\sigma_{b}^{2})$.444The Gaussian distribution can be
replaced by any other distribution with finite variance. With LoRA, we
generally want to initialize the product $ba^{\top}$ to be $0$ so that
finetuning starts from the pretrained model. This implies at least one of the
weights $a$ and $b$ is initialized to $0$. If both are initialized to $0$, it
is trivial that no learning occurs in this case since this is a saddle point.
Thus, we should initialize one of the parameters $a$ and $b$ to be non-zero
and the other to be zero. If we choose a non-zero initialization for $a$, then
following standard initialization schemes (e.g., He Init [4], LeCun Init [1]),
one should set $\sigma_{a}^{2}=\Theta(n^{-1})$ to ensure $a^{\top}x$ does not
explode with width. This is justified by the Central Limit Theorem
(CLT).555Technically, the CLT only ensures the almost sure convergence, the
$L_{2}$ convergence follows from the Dominated Convergence Theorem. We omit
these technical details in this paper. On the other hand, if we choose a non-
zero initialization for $b$, one should make sure that
$\sigma_{b}^{2}=\Theta(1)$. This leaves us with two possible initialization
schemes:

  * •

Init[1]: $\sigma_{b}^{2}=0,\sigma_{a}^{2}=\Theta(n^{-1})$.

  * •

Init[2]: $\sigma_{b}^{2}=\Theta(1),\sigma_{a}^{2}=0$.

Hereafter, our analysis will only consider these two initialization schemes
for LoRA modules, although the results should in-principle hold for other
schemes, providing that stability (as discussed above) is satisfied.

##### Learning rate.

Without loss of generality, we can simplify the analysis by assuming that
$W^{*}=0$. This can be achieved by setting $\tilde{y}=y-W^{*}x$. The gradients
are given by

$\displaystyle\frac{\partial\mathcal{L}}{\partial b}$
$\displaystyle=a^{\top}x(f(x)-y),$
$\displaystyle\frac{\partial\mathcal{L}}{\partial a}$
$\displaystyle=b(f(x)-y)x,$

Now let us look at what happens to the model output $f$ after the first
finetuning step with learning rate $\eta>0$. We use subscript $t$ to denote
the finetuning step. We have

$\displaystyle\Delta f_{t}$ $\displaystyle\overset{def}{=}f_{t}(x)-f_{t-1}(x)$
$\displaystyle=-\underset{\delta_{t}^{1}}{\underbrace{\eta
b_{t-1}^{2}(f_{t-1}(x)-y)\|x\|^{2}}}-\underset{\delta_{t}^{2}}{\underbrace{\eta(a_{t-1}^{\top}x)^{2}(f_{t-1}(x)-y)}}$
$\displaystyle\hskip
56.9055pt+\underset{\delta_{t}^{3}}{\underbrace{\eta^{2}(f_{t-1}(x)-y)^{2}b_{t-1}(a_{t-1}^{\top}x)\|x\|^{2}}}.$

The update in model output is driven by the three terms
$(\delta_{t}^{i})_{i\in\\{1,2,3\\}}$. The first two terms represent “linear”
contributions to the update, i.e. change in model output driven by fixing $b$
and updating $a$ and vice-versa. These terms are order one in $\eta$. The
third term $\delta_{t}^{3}$ represents a multiplicative update, compounding
the updates in $a$ and $b$, and is an order two term in $\eta$. As $n$ grows,
a desirable property is that $\Delta f_{t}=\Theta(1)$. Intuitively, this means
that as we scale the model, feature updates do not ‘suffer’ from this scaling
(see Section A.1 for more details). An example of such scenario where feature
learning is affected by scaling is the lazy training regime [7], where feature
updates are of order $\Theta(n^{-1/2})$ which implies that no feature learning
occurs in the limit $n\to\infty$. The condition $\Delta f_{t}=\Theta(1)$ also
implies that the update does not explode with width, which is also a desirable
property.

Having $\Delta f_{t}=\Theta(1)$ satisfied implies that at least one of the
three terms $(\delta_{t}^{i})_{i\in\\{1,2,3\\}}$ is $\Theta(1)$. Ideally, we
want both $\delta^{1}_{t}$ and $\delta^{2}_{t}$ to be $\Theta(1)$ because
otherwise it means that either $a$ or $b$ is not efficiently updated. For
instance, if $\delta^{1}_{t}=o(1)$, it means that as $n\to\infty$, the model
acts as if $a$ is fixed and only $b$ is trained. Similar conclusions hold when
$\delta^{2}_{t}=o(1)$. Having both $\delta^{1}_{t}$ and $\delta^{2}_{t}$ being
$\Theta(1)$ in width means that both $a$ and $b$ parameter updates
significantly contribute to the change in $f_{t}(x)$, and we say that feature
learning with LoRA is _efficient_ when this is the case, i.e.
$\delta_{i}^{t}=\Theta(1)$ for $i\in\\{1,2\\}$ and all $t>1$. We will
formalize this definition of efficiency in the next section. The reader might
wonder why we do not require that $\delta^{3}_{t}$ be $\Theta(1)$. We will see
that when both $\delta^{1}_{t}$ and $\delta^{2}_{t}$ are $\Theta(1)$, the term
$\delta^{3}_{t}$ is also $\Theta(1)$.

##### Efficiency Analysis.

To understand the contribution of each term in $\Delta f_{t}$ as the width $n$
grows, let us assume that we train the model with gradient descent with
learning rate $\eta=\Theta(n^{c})$ for some $c\in\mathbb{R}$, and suppose that
we initialize the model with Init[1]. Since the training dynamics are mainly
simple linear algebra operation (matrix vector products, sum of
vectors/scalars etc, see the Tensor Programs framework for more details
[22]),666A crucial assumption for this to hold is also to have that for any
matrix/vector product in the training dynamics, the product dimension (the
dimension along which the matrix/vector product is calculated) is
$\Theta(n^{\alpha})$ for some $\alpha>0$. For instance, in the case of
Transformers, this is satisfied since the MLP embedding dimension is generally
$k\times n$. However, this condition would be violated if for instance one
considers MLP embedding dimension $kn\log(n)$. Such non-standard scaling
choices require a particular treatment, but the conclusions remain the same.
it is easy to see that any quantity in the training dynamics should roughly
have a magnitude of order $n^{\gamma}$ in width for some
$\gamma\in\mathbb{R}$. For any quantity $v$ in the training dynamics, we
therefore write $v=\Theta(n^{\gamma[v]})$. When $v$ is a vector, we use the
same notation when all entries of $v$ are $\Theta(n^{\gamma[v]})$. The
$\gamma$ notation is formally defined in Appendix A.

Starting from initialization, we have $f_{0}(x)=0$. LoRA finetuning is
efficient when $\delta_{t}^{1}=\Theta(1)$ and $\delta^{2}_{t}=\Theta(1)$ for
all $t>1$,777Here we use the $t>1$ instead of $t>0$ because starting from
Init[1] or Init[2], at least one the terms $\delta_{1}^{1}$ or
$\delta_{1}^{2}$ will be zero. and $f_{t}(x)=\Theta(1)$ for $t>1$. This
translate to

$\begin{cases}c+2\gamma[b_{t-1}]+1=0\quad(\delta^{1}_{t}=\Theta(1))\\\
c+2\gamma[a_{t-1}^{\top}x]=0\quad(\delta^{2}_{t}=\Theta(1))\\\
\gamma[b_{t-1}]+\gamma[a_{t-1}^{\top}x]=0\quad(f_{t-1}(x)=\Theta(1))\\\
\end{cases}$

Solving this equation yields $c=-1/2$, i.e. the learning rate should scale as
$\eta=\Theta(n^{-1/2})$ in order to achieve efficient feature learning. At
initialization, $b_{0}=0$ and $a_{0}^{\top}x=\Theta(1)$ (by Central Limit
Theorem). Through an inductive argument, for $t>0$, $b_{t}$ will be of order
$\Theta(n^{-1/2})$ and $a_{t}^{\top}x$ will be of order $\Theta(1)$, yielding
$f_{t}(x)=\Theta(n^{-1/2})$. Indeed, at each iteration the update to $b_{t}$
will be of order $\Theta(\eta ya_{t-1}^{\top}x)=\Theta(n^{-1/2})$ and the
updates to $a_{t}$ are of order $\Theta(\eta b_{t-1}yx)=\Theta(n^{-1})$. As
$f_{t}=\Theta(n^{-1/2})$, this yields a contradiction towards learning
$\Theta(1)$ features.

This analysis shows that we cannot have both $\delta_{t}^{1}$ and
$\delta_{t}^{2}$ to be $\Theta(1)$ with this parametrization. This remains
true with Init[2]. We formalize this result in the next proposition and refer
the reader to Appendix A for further technical details.

######  Proposition 1 (Inefficiency of LoRA fine-tuning).

Assume that LoRA weights are initialized with Init[1] or Init[2] and trained
with gradient descent with learning rate $\eta=\Theta(n^{c})$ for some
$c\in\mathbb{R}$. Then, it is impossible to have $\delta_{t}^{i}=\Theta(1)$
for $i\in\\{1,2\\}$ for any $t>0$, and therefore, fine-tuning with LoRA in
this setup is inefficient.

In conclusion, efficiency cannot be achieved with this parametrization of the
learning rate. This suggests that standard LoRA finetuning as currently used
by practitioners is suboptimal, especially when model width is large, which is
a property that is largely satsified in practice ($n\approx 700$ for GPT2 and
$n\approx 4000$ for LLama). Therefore, it is natural to ask:

——————————————-

_How can we ensure efficient LoRA finetuning?_

——————————————-

The previous analysis suggests that _we are missing crucial hyperparameters_
in the standard LoRA setup. Indeed, we show that by decoupling the learning
rate for $a$ and $b$, we can have $\delta_{t}^{i}=\Theta(1)$ for
$i\in\\{1,2,3\\}$. We write $\eta_{a},\eta_{b}$ to denote the learning rates.
The analysis conducted above remains morally the same with the only difference
being in the learning rates. Let $\eta_{a}=\Theta(n^{c_{a}})$ and
$\eta_{b}=\Theta(n^{c_{b}})$, and assume that weights are initialized with
Init[1]. A similar analysis to the one conducted above show that having
$f_{t}(x)=\Theta(1)$ and $\delta_{t}^{i}=\Theta(1)$ for $i\in\\{1,2\\}$ and
$t>0$ implies that for all $t>1$

$\begin{cases}c_{a}+2\gamma[b_{t-1}]+1=0\quad(\delta^{1}_{t}=\Theta(1))\\\
c_{b}+2\gamma[a_{t-1}^{\top}x]=0\quad(\delta^{2}_{t}=\Theta(1))\\\
\gamma[b_{t-1}]+\gamma[a_{t-1}^{\top}x]=0\quad(f_{t-1}(x)=\Theta(1))\end{cases}$

which, after simple calculations, implies that $c_{a}+c_{b}=-1$. This is only
a necessary condition (does not imply efficiency as defined above). In the
next result, taking also some elements of stability into consideration, we
fully characterize the choice of $\eta_{a}$ and $\eta_{b}$ to ensure efficient
LoRA fine-tuning.

######  Proposition 2 (Efficient Fine-Tuning with LoRA).

In the case of model (2), with $\eta_{a}=\Theta(n^{-1})$ and
$\eta_{b}=\Theta(1)$, we have for all $t>1$, $i\in\\{1,2,3\\}$,
$\delta_{t}^{i}=\Theta(1)$.

We refer the reader to Appendix A for more details on the proof of 2. In
conclusion, scaling the learning rates as $\eta_{a}=\Theta(n^{-1})$ and
$\eta_{b}=\Theta(1)$ ensures stability ($\Delta f_{t}=\Theta(1)$) and
efficiency of LoRA finetuning ($\delta^{i}_{t}=\Theta(1)$ for $i\in\\{1,2\\}$
and $t>1$) in the infinite-width limit. In practice, this means that the
learning rate for $b$ should be generally much larger than that of $a$. This
remains true even if $b\in\mathbb{R}^{r}$ for general $r$. We will later see
that this scaling is valid for general neural network models.

###  3.2 Verifying the Results on a Toy Model

Before generalizing the results to neural networks and conducting experiements
with language models, we would like to first validate these conclusions on a
toy model. The previous analysis considers a simple linear model. To assess
the validity of the scaling rules in a non-linear setting, we consider a
simple neural network model given by

$f(x)=W_{out}\phi(BA\phi(W_{in}x)),$ (3)

where $W_{in}\in\mathbb{R}^{n\times d},W_{out}\in\mathbb{R}^{1\times
n},A\in\mathbb{R}^{r\times n},B\in\mathbb{R}^{n\times r}$ are the weights, and
$\phi$ is the ReLU function.

Figure 2: (Top) Train/Test accuracy of toy model Equation 3 averaged over 3
random seeds. Orange dashed line represents the line $\eta_{A}=\eta_{B}$, and
red dots represents all values of $(\eta_{A},\eta_{B})$ for which
$d_{\min}(\eta_{A},\eta_{B}):=\mathcal{L}_{(\eta_{A},\eta_{B})}/\mathcal{L}^{*}-1\leq
1\%$, where $\mathcal{L}^{*}$ is the best loss. (Bottom) Train/Test curves for
two sets of learning rates: the optimal choice
$(\eta_{A}^{*},\eta_{B}^{*})=(2.78,1.29\mathrm{e}{-4})$ overall at $t=200$ in
terms of test loss (Blue) and the optimal choice when $\eta_{A}=\eta_{B}$
which is given by
$(\eta_{A},\eta_{B})=(2.15\mathrm{e}{-2},2.15\mathrm{e}{-2})$ (Orange). All
values are averaged oevr three runs and confidence interval are shown
(shaded).

We train the model on a synthetic dataset generated with

$\begin{cases}X\sim\mathcal{N}(0,I_{d})\\\
Y=\sin(d^{-1}\sum_{i=1}^{d}X_{i}).\end{cases}$

The weights are initialized as follows:

  * •

$W_{in}\sim\mathcal{N}(0,1),W_{out}\sim\mathcal{N}(0,1/n)$.

  * •

$A\sim\mathcal{N}(0,1/n),B\sim\mathcal{N}(0,1)$

Here, only the weight matrices $A,B$ are trained and $W_{in},W_{out}$ are
fixed to their initial value. We use $d=5,n=100,r=4$, train dataset size
$N_{train}=1000$ and a test dataset of size $N_{test}=100$.888See Appendix C
for more details about the experimental setup. The train/test loss for varying
$\eta_{A}$ and $\eta_{B}$ is reported in Figure 2 at the early stages of the
training ($t=10$) and after convergence (we observed convergence around
$t\approx 200$ for reasonable choices of learning rates). The red ’$+$’ signs
represents learning rates $(\eta_{A},\eta_{B})$ for which the loss is within
$1\%$ range from the best loss and dashed line represents the case where the
learning rates are set equal. We observe that both the best train and test
losses are consistently achieved by a combination of learning rates where
$\eta_{b}\gg\eta_{a}$, which validates our analysis in the previous section.
Notice also that optimal learning rates $(\eta_{A},\eta_{B})$ are generally
close to the edge of stability, a well-known behaviour in training dynamics of
deep networks [15].

##  4 Stability and Feature Learning with LoRA in the $\infty$-width Limit

In this section, we extend the analysis above to general neural architectures
with LoRA layers. We show that the conclusions from the analysis on the linear
model hold for general neural architectures: 1) using the same learning rate
for both $A$ and $B$ leads to suboptimal feature learning when model width is
large, and 2) this problem can be fixed by setting different learning rates
for $A$ and $B$.

Since our aim in this paper is primarily methodological, the theoretical
results in this section are of a physics level of rigor, omitting technical
assumptions that would otherwise make the analysis rigorous but unnecessarily
complicated. In all the results, LoRA rank $r$ is considered fixed and
finetuning dynamics are analyzed in the limit of infinite-width. This setup
fairly represents practical scenarios where $r\ll n$ and $r$ is generally
small.

##### Notation.

The LoRA weights are initialized with
$A_{ij}\sim\mathcal{N}(0,\sigma_{A}^{2}),B_{ij}\sim\mathcal{N}(0,\sigma_{B}^{2})$
for some $\sigma_{A},\sigma_{B}\geq 0$.999In [17], $B$ is initialized to $0$,
which corresponds to setting $\sigma_{B}=0$. Here also, we assume that either
$\sigma_{B}^{2}=0$ and $\sigma_{A}^{2}=\Theta(n^{-1})$ (Init[1]), or
$\sigma_{B}^{2}=\Theta(1)$ and $\sigma_{A}^{2}=0$ (Init[2]). Given a LoRA
layer in the model, $\underline{Z}$ denotes the input to that layer and
$\bar{Z}$ the output after adding the pretrained weights. More precisely, we
write $\bar{Z}=W^{*}\underline{Z}+\frac{\alpha}{r}BA\,\underline{Z}$.

Our main analysis relies on a careful estimation of the magnitude of several
quantities including _LoRA features_. Let us first give a formal definition.

######  Definition 2 (LoRA Features).

Given a general neural architecture and a LoRA layer (1), we define LoRA
features $(Z_{A},Z_{B})$ as

$\begin{cases}Z_{A}=A\underline{Z}\\\
Z_{B}=BZ_{A}=BA\underline{Z},\end{cases}$

At fine-tuning step $t$, we use the superscript $t$ to denote the value of
LoRA features $Z_{A}^{t},Z_{B}^{t}$, and the subscript $t$ to denote the
weights $A_{t},B_{t}$.

LoRA layers are 2-layers linear networks with a “bottleneck” in the middle
(since generally $r\ll n$). This bottleneck shape might induce some numerical
challenges in training stability and efficiency (which we later define in 3
and 5) without proper care. Let us first describe a simplified setting that
will be subsequently used in our analysis.

##### Finetuning Dataset.

To simplify the analysis, we assume that the finetuning dataset comprises a
single sample $(x,y)$,101010This assumption on the finetuning dataset is for
simplification purposes only. All our analysis can be re-written with
‘batched’ gradients and the conclusions remain the same. However, some
additonal assumptions are required to make the analysis rigorous. and the goal
is to minimize the loss $\mathcal{L}(\bm{\theta},(x,y))$ computed with the
underlying model where the adjusted weights are given by $W^{*}+BA$ for all
LoRA layers (here $\bm{\theta}=\\{A,B,\textrm{ for all LoRA layers in the
model}\\}$). At training step $t$, and for any LoRA layer in the model,
$\underline{Z}^{t}$ is the input to the LoRA layer, computed with data input
$x$. Similarly, we write $d\bar{Z}^{t}$ to denote the gradient of the loss
function with respect to the layer output features $\bar{Z}$ evaluated at data
point $(x,y)$.

The notion of stability of LoRA as discussed in Section 3 can be generalized
to any neural network model as follows.

######  Definition 3 (Stability).

We say that LoRA finetuning is stable if for all LoRA layers in the model, and
all training steps $t$, we have $\underline{Z},Z_{A},Z_{B}=\mathcal{O}(1)$ as
the width goes to infinity.

Stability implies that no quantity in the network explodes as width grows, a
desirable property as we scale the model.111111It is possible to define
stability as $\underline{Z},Z_{B}=\mathcal{O}(1)$ and exclude $Z_{A}$ from the
condition. This would allow scenarios where for instance the entries of $A$
explode with width but their magnitude is compensated with a smaller magnitude
of $B$. This system has one degree of freedom because of the homogeneity of
the product $BA$, and by imposing that $Z_{A}=\mathcal{O}(1)$, we avoid having
such scenarios. Naturally, in order to ensure stability, one has to scale
hyperparameters (initialization, learning rate) as $n$ grows. Scaling rules
for initialization are fairly easy to infer and were already discussed in
Section 3 where we obtained two plausible initialization schemes (Init[1] and
Init[2]). More importantly, if we arbitrarily scale the learning rate with
width, we might end up with suboptimal learning as width grows even if the
finetuning is stable. This is the case for instance when we aggressively
downscale the learning rate with width, or inadequately parameterize the
network (e.g. Neural Tangent Kernel parametrization which leads to the kernel
regime in the infinite width limit, [7]). To take this into account, we define
a notion of feature learning with LoRA.

######  Definition 4 (Stable Feature Learning with LoRA).

We say that LoRA finetuning induces stable feature learning if it is stable
(3), and for all LoRA layers and finetuning step $t$, we have $\Delta
Z_{B}^{t}\overset{def}{=}Z_{B}^{t+1}-Z_{B}^{t}=\Theta(1)$.

A similar definition of feature learning was introduced in [35] for
pretraining. This definition of feature learning ensures that the network is
not ‘stuck’ in a kernel regime where feature updates are of order
$\mathcal{O}(n^{-\epsilon})$ in the infinite-width limit for some
$\epsilon>0$, which implies that no feature learning occurs in the limit. The
authors introduced the $\mu$-parameterization (or maximal update
parametrization), a specific network parameterization (initialization +
learning rate scaling), that ensures that feature updates are $\Theta(1)$.
Note that here we added stability in the definition, but in principle, one
could define feature learning with $\Omega$ instead of $\Theta$. The latter
covers unstable scenarios (e.g. when $\Delta Z_{B}^{t}=\Theta(n)$ due to
improper scaling of initialization and learning rate), so we omit it here and
focus on stable feature learning. Also, notice that we only consider
finetuning dynamics and not the pretraining dynamics. However, since our
analysis depends on weights $W^{*}$ from pretraining, we assume that
pretraining parameterization ensures stability and feature learning as width
grows (see Appendix A for more details).121212When taking the infinite width
limit, we assume that pretraining parameterization is $\mu$P. This is just a
technicality for the infinite-width limit and does not have any implications
on practical scenarios where the width is finite. The most important
implications of this assumption is that in the pretrained network (before
introducing LoRA layers), we have $\underline{Z}=\Theta(1),\bar{Z}=\Theta(1)$,
which holds for a general input-output pair $(x,y)$.

We are now ready to investigate the impact of the learning rate on finetuning.
At finetuning step $t$, the gradients are given by

$\displaystyle\frac{\partial\mathcal{L}_{t}}{\partial B}$
$\displaystyle=\frac{\alpha}{r}d\bar{Z}^{t-1}\otimes
A_{t-1}\underline{Z}^{t-1}$
$\displaystyle\frac{\partial\mathcal{L}_{t}}{\partial A}$
$\displaystyle=dZ_{A}^{t-1}\otimes\underline{Z}^{t-1}=\frac{\alpha}{r}B^{\top}_{t-1}d\bar{Z}^{t-1}\otimes\underline{Z}^{t-1},$

where $u\otimes v$ denotes the outer product $uv^{\top}$ of vectors $u$, $v$,
and the weights are updated as follows

$A_{t}=A_{t-1}-\eta_{A}g_{A}^{t-1},\quad B_{t}=B_{t-1}-\eta_{B}g_{B}^{t-1},$

where $g_{A},g_{B}$ are processed gradients (e.g. normalized gradients with
momentum as in AdamW etc). Hereafter, we assume that the gradients are
processed in a way that makes their entries $\Theta(1)$. This is generally
satisfied in practice (with Adam for instance) and has been considered in [35]
to derive the $\mu$-parametrization for general gradient processing functions.

Unlike the linear model in Section 3, LoRA feature updates are not only driven
by the change in the $A,B$ weights, but also $\underline{Z},d\bar{Z}$ which
are updated as we finetune the model (assuming there are multiple LoRA
layers). To isolate the contribution of individual LoRA layers to feature
learning, we assume that only a _single LoRA layer is trainable_ and all other
LoRA layers are frozen.131313This is equivalent to having only a single LoRA
layer in the model since LoRA layers are initialized to zero. In this way, we
can quantify feature learning induced by the LoRA layer as we finetune the
model.. In this setting, considering the only trainable LoRA layer in the
model, the layer input $\underline{Z}$ is fixed and does not change with $t$,
while $d\bar{Z}$ changes with step $t$ (because
$\bar{Z}^{t}=(W^{*}+\frac{\alpha}{r}B_{t}A_{t})\underline{Z}$). After step
$t$, $Z_{B}$ is updated as follows

$\Delta Z_{B}^{t}=\underset{\delta_{t}^{1}}{\underbrace{B_{t-1}\Delta
Z_{A}^{t}}}+\underset{\delta_{t}^{2}}{\underbrace{\Delta
B_{t}Z_{A}^{t-1}}}+\underset{\delta^{3}_{t}}{\underbrace{\Delta B_{t}\Delta
Z_{A}^{t}}}$

As discussed in Section 3, the terms $\delta^{1}_{t},\delta^{2}_{t}$ represent
the ‘linear’ feature updates that we obtain if we fix one weight matrix and
only train the other. The third term $\delta^{3}_{t}$ represents the
‘multiplicative’ feature update which captures the compounded update due to
updating both $A$ and $B$.

##### Analysis of the Role of $A$ and $B$.

As discussed above, we want to ensure that $\delta_{t}^{1}=\Theta(1)$ and
$\delta_{t}^{2}=\Theta(1)$ which means that both weight matrices contribute to
the update in $Z_{B}$. To further explain why this is a desirable property,
let us analyze how changes in matrices $A$ and $B$ affect LoRA feature
$Z_{B}=BA\,\underline{Z}$.

Let $(B_{:,i})_{1\leq i\leq r}$ denote the columns of $B$. We have the
following decomposition of $Z_{B}$:

$Z_{B}=\sum_{i=1}^{r}(A\,\underline{Z})_{i}B_{:,i},$

where $(A\underline{Z})_{i}$ is the $i^{th}$ coordinate of $A\underline{Z}$.
This decomposition suggests that the _direction_ of $Z_{B}$ is a weighted sum
of the columns of $B$, and $A$ modulates the _weights_. With this, we can also
write

$\begin{cases}\delta^{1}_{t}=\sum_{i=1}^{r}(\Delta
A_{t}\underline{Z})_{i}(B_{:,i})_{t-1}\\\
\delta^{2}_{t}=\sum_{i=1}^{r}(A_{t-1}\underline{Z})_{i}(\Delta
B_{:,i})_{t-1},\end{cases}$

where $(B_{:,i})_{t}$ refers to the columns of $B$ at time step $t$. Having
both $\delta_{t}^{1}$ and $\delta_{t}^{2}$ of order $\Theta(1)$ means that
both $A$ and $B$ are ‘sufficiently’ updated to induce a change in weights
$(A\underline{Z})_{i}$ and directions $B_{:,i}$. If one of the matrices $A,B$
is not efficiently updated, we might end up with suboptimal finetuning,
leading to either non updated directions $B$ or direction weights
$(A_{t-1}Z)$. For instance, assuming that the model is initialized with
Init[2], and that $B$ is not efficiently updated, the direction of $Z_{B}$
will be mostly determined by the vector (sub)space of dimension $r$ generated
by the columns of $B$ at initialization. This analysis leads to the following
definition of efficient learning with LoRA.

######  Definition 5 (Efficient Learning with LoRA).

We say that LoRA fine-tuning is efficient if it is stable (3), and for all
LoRA layers in the model, and all fine-tuning steps $t>1$, we have

$\delta_{t}^{i}=\Theta(1),\quad i\in\\{1,2\\}.$

Note that it is possible to achieve stable feature learning (4) without
necessarily having efficient learning. This is the case when for instance $B$
is not updated (fixed to a non-zero init with Init[2]) and only $A$ is
updated, which corresponds to simply setting $\eta_{B}=0$. This is a trivial
case, but other non-trivial cases of inefficiency are common in practice, such
as the use of the same learning rate for $A$ and $B$ which is a standard
practice. In the next theorem, we characterize the optimal scaling of learning
rates $\eta_{A}$ and $\eta_{B}$, a conclusion similar to that of Section 3.

######  Theorem 1 (Efficient LoRA (Informal)).

Assume that weight matrices $A$ and $B$ are trained with Adam with respective
learning rates $\eta_{A}$ and $\eta_{B}$. Then, it is impossible to achieve
efficiency with $\eta_{A}=\eta_{B}$. However, LoRA Finetuning is efficient
with $\eta_{A}=\Theta(n^{-1})$ and $\eta_{B}=\Theta(1)$.

The result of 1 suggests that efficiency can only be achieved with a pair of
learning rates satisfying $\eta_{B}/\eta_{A}=\Theta(n)$. In practice, this
translates to setting $\eta_{B}\gg\eta_{A}$, but does not provide a precise
ratio $\eta_{B}/\eta_{A}$ to be fixed while tuning the learning rate (the
constant in ‘$\Theta$’ is generally intractable), unless we tune both
$\eta_{B}$ and $\eta_{A}$ which is not efficient from a computational
perspective as it becomes a 2D tuning problem. It is therefore natural to set
a fixed ratio $\eta_{B}/\eta_{A}$ and tune only $\eta_{A}$ (or $\eta_{B}$),
which would effectively reduce the tuning process to a 1D grid search,
achieving the same computational cost of standard LoRA where the learning rate
is the same for $A$ and $B$. We call this method LoRA$+$.

LoRA$+$ : set the learning rates for LoRA modules $A,B$ such that
$\eta_{B}=\lambda\eta_{A}$ with $\lambda>1$ fixed and tune $\eta_{A}$.

The choice of the ratio $\lambda$ is crucial in LoRA$+$. A ratio closer to $1$
will mimic standard LoRA, while a choice of $\lambda\gg 1$ should generally
improve the results (as we will see in the next section) when $n\gg r$. In the
next section, through extensive empirical evaluations, we first validate our
theoretical result and show that optimal pairs $(\eta_{A},\eta_{B})$ (in terms
of test accuracy) generally satisfy $\eta_{B}\gg\eta_{A}$. We then investigate
the optimal ratio $\lambda$ for LoRA$+$ and suggest a default ratio that was
empirically found to generally improve performance compared to standard LoRA.

##### Comparison of 1 and 2.

Although the conclusions of 1 and 2 are similar, the proof techniques are
different. In 2, the linear model is trained with gradient descent, while in
1, the training algorithm is Adam-type in the sense that it normalizes the
gradients before updating the weights. The formal statement of 1 requires an
additional assumption on the alignment of the processed gradients $g_{A}$ with
LoRA input $\underline{Z}$. This technical detail is introduced and discussed
in Appendix A.

##  5 Experiments with Language Models

In this section, we report our empirical results using LoRA to finetune a set
of language models on different benchmarks. Details about the experimental
setup and more empirical results are provided in Appendix C. We also identify
a default value for the ratio $\lambda=\eta_{B}/\eta_{A}$ that generally
improves performance as compared to standard LoRA. The code for our
experiments is available at https://github.com/nikhil-ghosh-berkeley/loraplus.

###  5.1 GLUE tasks with GPT-2 and RoBERTa

The GLUE benchmark (General Language Understanding Evaluation) consists of
several language tasks that evaluate the understanding capabilities of
langugage models [8]. Using LoRA, we finetune Roberta-base from the RoBERTa
family [11] and GPT-2 [12] on MNLI, QQP, SST2, and QNLI tasks (Other tasks are
smaller and generally require an already finetuned model e.g. on MNLI as
starting checkpoint) with varying learning rates $(\eta_{A},\eta_{B})$ to
identify the optimal combination. Empirical details are provided in Appendix
C.

Figure 3: Test accuracy of Roberta-base finetuning for $3$ epochs on MNLI,
QQP, QNLI, and $10$ epochs on SST2, with sequence length $T=128$ and half
precision (FP16). LoRA hyperparameters are set to $\alpha=r=8$. All values are
averaged over 3 random seeds (we do not show confidence intervals for better
visualizations, but fluctuations are of order $0.1\%$, see Figure 7 for
instance). For better visualization, when accuracy is lower than a fixed
threshold, we set it to threshold. Values shown in red are: 1) the best
accuracy (overall) and 2) the accuracy for a set of learning rates where
$\eta_{B}$ and $\eta_{A}$ are close in order of magnitude
($\eta_{B}/\eta_{A}\in[1,1.25]$).

##### Roberta-base.

Figure 3 shows the results of Roberta-base finetuning with $\alpha=r=8$,
trained with half precision (FP16). We observe that test accuracy is
consistently maximal for some set of learning rates satisfying
$\eta_{B}\gg\eta_{A}$, outperforming the standard practice where $\eta_{A}$
and $\eta_{B}$ are usually set equal. Interestingly, the gap between the
optimal choice of learning rates overall and the optimal choice when
$\eta_{A}\approx\eta_{B}$ is more pronounced for ‘harder’ tasks like MNLI and
QQP, as compared to SST2 and QNLI. This is probably due to the fact that
harder tasks require more efficient feature learning. It is also worth
mentioning that in our experiments, given limited computational resources, we
use sequence length $T=128$ and finetune for only $3$ epochs for MNLI and QQP,
so it is expected that we obtain test accuracies lower that those reported in
[17] where the authores finetune Roberta-base with $T=512$ sequence length
(for MNLI) and more epochs ($30$ for MNLI). In Appendix C, we provide
additional results with Test/Train accuracy/loss.

Figure 4: Test accuracy of GPT-2 model after finetuning for $3$ epochs on
MNLI, QQP, with FP16 precision. LoRA hyperparameters are set to $\alpha=r=8$.
Both train/test accuracy are consistently maximal for some choice of learning
rates where $\eta_{B}\gg\eta_{A}$. See Appendix C for more numerical results
with GPT2.

##### GPT-2.

Figure 4 shows the results of finetuning GPT-2 model with LoRA on MNLI and QQP
with half precision FP16 (other tasks and full precision training are provided
in Appendix C). Similar to the conclusions from Roberta-base, we observe that
maximal test accuracies are achieved with some $(\eta_{A},\eta_{B})$
satisfying $\eta_{B}\gg\eta_{A}$. Further GPT-2 results with different tasks
are provided in Appendix C. Here also, we observed that the harder the task,
the larger the gap between model performance when $\eta_{B}\gg\eta_{A}$ and
when $\eta_{A}\approx\eta_{B}$.

###  5.2 Llama

To further validate our theoretical findings, we report the results of
finetuning the Llama-7b model [33] on the MNLI dataset and flan-v2 dataset
[30] using LoRA. Each trial is averaged over two seeds.

##### Flan-v2.

We examine LoRA training of Llama on the instruction finetuning dataset
flan-v2 [30]. To make the experiments computationally feasible, we train for
one epoch on a size $100,000$ subset of the flan-v2 dataset. We record the
test accuracy of the best checkpoint every 500 steps. The LoRA hyperparameters
are set to $\alpha=16$ and $r=64$. The adapters are added to every linear
layer (excluding embedding layers) and we use a constant learning rate
schedule. The full training details are in Appendix C.

Figure 5: Left: MMLU accuracy of Llama-7b trained for one epoch on a 100k
subset of flan-v2. Right: Test accuracy of the best checkpoint of Llama-7b
trained on MNLI for one epoch. Values are averaged over two seeds.

We evaluate the final model on the MMLU benchmark [14]. The results in Figure
5 show that for this benchmark taking $\eta_{B}\gg\eta_{A}$ is advantageous
and results in a roughly 1.3% gain compared with the optimal
$\eta_{B}=\eta_{A}$.

##### MNLI.

The right panel of Figure 5 shows the results of finetuning Llama-7b with LoRA
on MNLI, with $\alpha=16$, $r=8$. We train using half precision (FP16) and
constant learning rate schedule, with a sequence length $T=128$. Since MNLI is
a relatively easy task for Llama, we finetune for only one epoch, which is
sufficient for the model to reach its peak test accuracy. In Figure 5 we see
that $\eta_{B}=\eta_{A}$ is nearly optimal for all $\eta_{B}\geq\eta_{A}$.
This is consistent with the intuition that efficient feature learning is not
required for easy tasks and that having $\eta_{B}/\eta_{A}\gg 1$ does not
significantly enhance performance. Additionally, the magnitude of stable
learning rates for Llama is much smaller than for GPT-2 and RoBERTa on MNLI
further supporting that Llama requires less adaptation. Analogous plots for
the train and test loss are shown in Figure 17 in Appendix C.

###  5.3 How to set the ratio $\lambda=\eta_{B}/\eta_{A}$ in LoRA$+$?

Naturally, the optimal ratio $\lambda$ depends on the architecture and the
finetuning task via the constants in ‘$\Theta$’ (1). This is a limitation of
these asymptotic results since they do not offer any insights on how the
constants are affected by the task and the neural architecture. To overcome
this limitations, we proceed with an empirical approach to estimate a
reasonable value for $\lambda$.

Figure 6: Distribution of the ratio $\eta_{B}/\eta_{A}$ for the top 4 learning
rate for each pair (model, task). The 4 learning rates are selected using the
test loss at the end of finetuning (i.e. top 4 learning rates
$(\eta_{B},\eta_{A})$ in terms of test loss). The distribution shows the
interquartile range ( $25\%-75\%$ quantiles) and the median.

In Figure 6, we show the distribution of the ratio $\eta_{B}/\eta_{A}$ for the
top $4$ runs in terms of test accuracy for different pairs of (model, task).
This is the same experimental setup of Figure 3 and Figure 4. The optimal
ratio is model and task sensitive, but the median log-ratio hovers around $4$.
We estimate that generally setting a ratio of
$\lambda=\eta_{B}/\eta_{A}\approx 2^{4}$ improves performance. This can be
achieved by setting $\eta_{A}=\eta,\eta_{B}=2^{4}\eta$, and tuning the
hyperparameter $\eta$.

To examine the effectiveness of this method, we finetuned Roberta-base on MNLI
task for 3 epochs with two different setups: (LoRA+) $\eta_{B}=2^{4}\eta_{A}$
and (Standard) $\eta_{B}=\eta_{A}$. In both setups, we tune $\eta_{A}$ via a
grid search. The results are reported in Figure 7.

Figure 7: Test accuracy of Roberta-base finetuned on the MNLI task in two
setups: (LoRA+) $\eta_{B}=2^{4}\eta_{A}$ and (Standard) $\eta_{B}=\eta_{A}$.
$\eta_{A}$ is tuned using a grid search.

Our method LoRA$+$ with $\lambda=2^{4}$ shows a significant improvement both
in final test accuracy and training speed. Using LoRA$+$, in only $1.6$
epochs, we reach the final accuracy obtained with standard setup after $3$
epochs.

###  5.4 Guide for practitioners: Using LoRA$+$ for finetuning

Integrating LoRA$+$ in any finetuning code is simple and can be achieved with
a single line of code by importing the custom trainer LoraPlusTrainer from the
code provided in this link. In the trainer config, the value of $\lambda$ is
set by default to $2^{4}$ which we found to generally improve performance
compared to LoRA. However, note that the optimal value of $\lambda$ is task
and model dependent: if the task is relatively hard for the pretrained model,
the choice of $\lambda$ will be crucial since efficient training is required
to align the model with finetuning task. On the contrary, the impact of
$\lambda$ is less noticeable when the finetuning task is relatively easy for
the pretrained model. As a rule of thumb, setting $\lambda=2^{4}$ is a good
starting point.

##  6 Conclusion and Limitations

Employing a scaling argument, we showed that LoRA finetuning as it is
currently used in practice is not efficient. We proposed a method, LoRA$+$,
that resolves this issue by setting different learning rates for LoRA adapter
matrices. Our analysis is supported by extensive empirical results confirming
the benefits of LoRA$+$ for both training speed and performance. These
benefits are more significant for ‘hard’ tasks such as MNLI for Roberta/GPT2
(compared to SST2 for instance) and MMLU for LLama-7b (compared to MNLI for
instance). However, as we depicted in Figure 7, a more refined estimation of
the optimal ratio $\eta_{B}/\eta_{A}$ should take into account task and model
dependent, and our analysis in this paper lacks this dimension. We leave this
for future work.

##  7 Acknowledgement

We thank Amazon Web Services (AWS) for cloud credits under an Amazon Research
Award. We also gratefully acknowledge partial support from NSF grants
DMS-2209975, 2015341, NSF grant 2023505 on Collaborative Research: Foundations
of Data Science Institute (FODSI), the NSF and the Simons Foundation for the
Collaboration on the Theoretical Foundations of Deep Learning through awards
DMS-2031883 and 814639, and NSF grant MC2378 to the Institute for Artificial
CyberThreat Intelligence and OperatioN (ACTION).

## References

  * [1] Yann LeCun, Léon Bottou, Genevieve B Orr and Klaus-Robert Müller  “Efficient backprop”  In _Neural networks: Tricks of the trade_ Springer, 2002, pp. 9–50
  * [2] Liu Yang, Steve Hanneke and Jaime Carbonell  “A theory of transfer learning with applications to active learning”  In _Machine learning_ 90 Springer, 2013, pp. 161–189
  * [3] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _arXiv preprint arXiv:1412.6980_ , 2014
  * [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [5] S.S. Schoenholz, J. Gilmer, S. Ganguli and J. Sohl-Dickstein  “Deep Information Propagation”  In _International Conference on Learning Representations_ , 2017
  * [6] Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli and Jascha Sohl-Dickstein  “Deep Information Propagation”, 2017  arXiv:1611.01232 [stat.ML]
  * [7] Arthur Jacot, Franck Gabriel and Clément Hongler  “Neural tangent kernel: Convergence and generalization in neural networks”  In _Advances in neural information processing systems_ 31, 2018
  * [8] Alex Wang et al.  “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding”, 2018  arXiv:1804.07461 [cs.CL]
  * [9] Soufiane Hayou, Arnaud Doucet and Judith Rousseau  “On the Impact of the Activation function on Deep Neural Networks Training”  In _Proceedings of the 36th International Conference on Machine Learning_ 97, Proceedings of Machine Learning Research  PMLR, 2019, pp. 2672–2680  URL: https://proceedings.mlr.press/v97/hayou19a.html
  * [10] Neil Houlsby et al.  “Parameter-efficient transfer learning for NLP”  In _International Conference on Machine Learning_ , 2019, pp. 2790–2799  PMLR
  * [11] Yinhan Liu et al.  “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, 2019  arXiv:1907.11692 [cs.CL]
  * [12] Alec Radford et al.  “Language Models are Unsupervised Multitask Learners”, 2019
  * [13] G. Yang  “Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation”  In _arXiv preprint arXiv:1902.04760_ , 2019
  * [14] Dan Hendrycks et al.  “Measuring massive multitask language understanding”  In _arXiv preprint arXiv:2009.03300_ , 2020
  * [15] Jeremy Cohen et al.  “Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability”  In _International Conference on Learning Representations_ , 2021  URL: https://openreview.net/forum?id=jh-rTtvkGeM
  * [16] Soufiane Hayou et al.  “Stable ResNet”  In _Proceedings of The 24th International Conference on Artificial Intelligence and Statistics_ 130, Proceedings of Machine Learning Research  PMLR, 2021, pp. 1324–1332  URL: https://proceedings.mlr.press/v130/hayou21a.html
  * [17] Edward J. Hu et al.  “LoRA: Low-Rank Adaptation of Large Language Models”  In _arXiv preprint arXiv:2106.09685_ , 2021
  * [18] Brian Lester, Rami Al-Rfou and Noah Constant  “The power of scale for parameter-efficient prompt tuning”  In _arXiv preprint arXiv:2104.08691_ , 2021
  * [19] Greg Yang and Edward J Hu  “Tensor programs iv: Feature learning in infinite-width neural networks”  In _International Conference on Machine Learning_ , 2021, pp. 11727–11737  PMLR
  * [20] Jordan Hoffmann et al.  “Training Compute-Optimal Large Language Models”, 2022  arXiv:2203.15556 [cs.CL]
  * [21] Haokun Liu et al.  “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 1950–1965
  * [22] Greg Yang et al.  “Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer”  In _arXiv preprint arXiv:2203.03466_ , 2022
  * [23] Blake Bordelon et al.  “Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit”, 2023  arXiv:2309.16620 [stat.ML]
  * [24] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer  “QLoRA: Efficient Finetuning of Quantized LLMs”  In _arXiv preprint arXiv:2305.14314_ , 2023
  * [25] Soufiane Hayou  “On the infinite-depth limit of finite-width neural networks”  In _Transactions on Machine Learning Research_ , 2023  URL: https://openreview.net/forum?id=RbLsYz1Az9
  * [26] Bobby He et al.  “Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagation”, 2023  arXiv:2302.10322 [cs.LG]
  * [27] Dawid Jan Kopiczko, Tijmen Blankevoort and Yuki Markus Asano  “VeRA: Vector-based Random Matrix Adaptation”  In _arXiv preprint arXiv:2310.11454_ , 2023
  * [28] Yixiao Li et al.  “Loftq: Lora-fine-tuning-aware quantization for large language models”  In _arXiv preprint arXiv:2310.08659_ , 2023
  * [29] Haotian Liu, Chunyuan Li, Yuheng Li and Yong Jae Lee  “Improved baselines with visual instruction tuning”  In _arXiv preprint arXiv:2310.03744_ , 2023
  * [30] Shayne Longpre et al.  “The flan collection: Designing data and methods for effective instruction tuning”  In _arXiv preprint arXiv:2301.13688_ , 2023
  * [31] Lorenzo Noci et al.  “The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit”, 2023  arXiv:2306.17759 [stat.ML]
  * [32] OpenAI  “GPT-4 Technical Report”  In _arXiv preprint arXiv:2303.08774_ , 2023
  * [33] Hugo Touvron et al.  “Llama 2: Open Foundation and Fine-Tuned Chat Models”  In _arXiv preprint arXiv:2307.09288_ , 2023
  * [34] Yizhong Wang et al.  “How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources”  In _arXiv preprint arXiv:2306.04751_ , 2023
  * [35] Greg Yang and Etai Littwin  “Tensor programs ivb: Adaptive optimization in the infinite-width limit”  In _arXiv preprint arXiv:2308.01814_ , 2023
  * [36] Greg Yang, Dingli Yu, Chen Zhu and Soufiane Hayou  “Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks”  In _arXiv preprint arXiv:2310.02244_ , 2023
  * [37] Yuchen Zeng and Kangwook Lee  “The Expressive Power of Low-Rank Adaptation”  In _arXiv preprint arXiv:2310.17513_ , 2023

##  Appendix A Proofs

In this section, we provide proofs for 1, 2, 1, and some technical details
used in the proofs.

###  A.1 Scaling of Neural Networks

Scaling refers to the process of increasing the size of one of the ingredients
in the model to improve performance (see e.g. [20]). This includes model
capacity which can be increased via width (embedding dimension) or depth
(number of layers) or both, compute (training data), number of training steps
etc. In this paper, we are interested in scaling model capacity via the width
$n$. This is motivated by the fact that most state-of-the-art language and
vision models have large width.

It is well known that as the width $n$ grows, the network initialization
scheme and the learning should be adapted to avoid numerical instabilities and
ensure efficient learning. For instance, the initialization variance should
scale $1/n$ to prevent arbitrarily large pre-activations as we increase model
width $n$ (e.g. He init [4]). To derive such scaling rules, a principled
approach consist of analyzing statistical properties of key quantities in the
model (e.g. pre-activations) as $n$ grows and then adjust the initialization,
the learning rate, and the architecture itself to achieve desirable properties
in the limit $n\to\infty$ [9, 5, 13].

In this context, [22] introduces the Maximal Update Parameterization (or
$\mu$P), a set of scaling rules for the initialization scheme, the learning
rate, and the network architecture that ensure stability and maximal feature
learning in the infinite width limit. Stability is defined by
$Y_{l}^{i}=\Theta(1)$ for all $l$ and $i$ where the asymptotic notation
‘$\Theta(.)$’ is with respect to width $n$ (see next paragraph for a formal
definition), and feature learning is defined by $\Delta Y_{l}=\Theta(1)$,
where $\Delta$ refers to the feature update after taking a gradient step.
$\mu$P guarantees that these two conditions are satisfied at any training step
$t$. Roughly speaking, $\mu$P specifies that hidden weights should be
initialized with $\Theta(n^{-1/2})$ random weights, and weight updates should
be of order $\Theta(n^{-1})$. Input weights should be initialized $\Theta(1)$
and the weights update should be $\Theta(1)$ as well. While the output weights
should be initialized $\Theta(n^{-1})$ and updated with $\Theta(n^{-1})$.
These rules ensure both stability and feature learning in the infinite-width
limit, in contrast to standard parameterization (exploding features if the
learning rate is well tuned), and kernel parameterizations (e.g. Neural
Tangent Kernel parameterization where $\Delta Y_{l}=\Theta(n^{-1/2})$, i.e. no
feature learning in the limit).

###  A.2 The Gamma Function ($\gamma[.]$)

In the theory of scaling of neural networks, one usually tracks the asymptotic
behaviour of key quantities as we scale some model ingredient. For instance,
if we scale the width, we are interested in quantifying how certain quantities
in the network behave as width $n$ grows large and the asymptotic notation
becomes natural in this case. This is a standard approach for (principled)
model scaling and it has so far been used to derive scaling rules for
initialization [5], activation function [9], network parametrization [36],
amongst other things.

With Init[1] and Init[2], the weights are initialized with
$\Theta(n^{-\beta})$ for some $\beta\geq 0$. Assuming that the learning rates
also scale polynomially with $n$, it is straightforward that preactivations,
gradients, and weight updates are all asymptotically polynomial in $n$. It is
therefore natural to introduce the Gamma function, and we write
$v=\Theta(\gamma[v])$ to capture this polynomial behaviour. Now, let us
introduce some elementary operations with the Gamma function.

##### Multiplication.

Given two real-valued variables $v,v^{\prime}$, we have $\gamma[v\times
v^{\prime}]=\gamma[v]+\gamma[v^{\prime}]$.

##### Addition.

Given two real-valued variables $v,v^{\prime}$, we generally have
$\gamma[v+v^{\prime}]=\max(\gamma[v],\gamma[v^{\prime}])$. The only case where
this is violated is when $v^{\prime}=-v$. This is generally a zero probability
event if $v$ and $v^{\prime}$ are random variables that are not perfectly
correlated, which is the case in most situations where we make use of this
formula (see the proofs below).

###  A.3 Proof of 1

Proposition 1. [Inefficiency of LoRA fine-tuning] _Assume that LoRA weights
are initialized with Init[1] or Init[2] and trained with gradient descent with
learning rate $\eta=\Theta(n^{c})$ for some $c\in\mathbb{R}$. Then, it is
impossible to have $\delta_{t}^{i}=\Theta(1)$ for all $i$ for any $t>0$, and
therefore, fine-tuning with LoRA in this setup is inefficient._

###### Proof.

Assume that the model is initialized with Init[1]. Since the training dynamics
are mainly simple linear algebra operation (matrix vector products, sum of
vectors/scalars etc), it is easy to see that any vector/scaler in the training
dynamics has a magnitude of order $n^{\gamma}$ for some $\gamma\in\mathbb{R}$
(for more details, see the Tensor Programs framework, e.g. [13]). For any
quantity $v$ in the training dynamics, we write $v=\Theta(n^{\gamma[v]})$.
When $v$ is a vector, we use the same notation when all entries of $v$ are
$\Theta(n^{\gamma[v]})$. Efficiency is defined by having
$\delta^{t}_{i}=\Theta(1)$ for $i\in\\{1,2\\}$ and $t>1$. Note that this
implies $f_{t}(x)=\Theta(1)$ for all $t>1$. Let $t>1$ and assume that learning
with LoRA is efficient. We will show that this leads to a contradiction.
Efficiency requires that $\delta_{t}^{i}=\Theta(1)$ for all $t,i\in\\{1,2\\}$.
Using the elementary formulas from Section A.2, this implies that for all $t$

$\begin{cases}\gamma[\eta]+2\gamma[b_{t-1}]+1=0\\\
\gamma[\eta]+2\gamma[a_{t-1}^{\top}x]=0\\\
\gamma[b_{t-1}]+\gamma[a_{t-1}^{\top}x]=0.\end{cases}$

Solving this equation yields $\gamma[\eta]=-1/2$, i.e. LoRA finetuning can be
efficient only if the learning rate scales as $\eta=\Theta(n^{-1/2})$. Let us
now show that this yields a contradiction. From the gradient updates and the
elementary operations from Section A.2, we have the following recursive
formulas

$\begin{cases}\gamma[b_{t}]=\max(\gamma[b_{t-1}],-1/2+\gamma[a_{t-1}^{\top}x])\\\
\gamma[a_{t}^{\top}x]=\max(\gamma[a_{t-1}^{\top}x],1/2+\gamma[b_{t-1}])\end{cases}$

Starting from $t=1$, with Init[1] we have
$\gamma[b_{1}]=\gamma[\eta(a_{0}^{\top}x)y]=-1/2$ and
$\gamma[a_{1}^{\top}x]=\gamma[a_{0}^{\top}x]=0$, we have $\gamma[b_{2}]=-1/2$
and $\gamma[a_{2}^{\top}x]=0$. Trivially, this holds for any $t$. However,
this implies that $\gamma[f_{t}]=\gamma[b_{t}]+\gamma[a_{t}^{\top}x]=-1/2$
which means that $\Delta f_{t}$ cannot be $\Theta(1)$. With Init[2], we have
$\gamma[b_{1}]=\gamma[b_{0}]=0$ and $\gamma[a_{1}^{\top}]=\gamma[\eta
b_{0}y\|x\|^{2}]=-1/2+1=1/2$. From the recursive formula we get
$\gamma[b_{2}]=0$ and $\gamma[a_{2}^{\top}x]=1/2$ which remains true for all
$t$. In this case we have $\gamma[f_{t}]=1/2$ which contradicts $\Delta
f_{t}=\Theta(1)$.

In both cases, this contradicts our assumption, and therefore efficiency
cannot be achieved in this setup.

∎

###  A.4 Proof of 2

Proposition 2. [Efficient Fine-Tuning with LoRA] _In the case of Toy model
Equation 2, with $\eta_{a}=\Theta(n^{-1})$ and $\eta_{b}=\Theta(1)$, we have
for all $t>1$, $\in\\{1,2,3\\}$, $\delta_{t}^{i}=\Theta(1)$._

###### Proof.

The proof is similar in flavor to that of 1. In this case, the set of
equations that should be satisfied so that $\delta_{t}^{i}=\Theta(1)$ are
given by

$\begin{cases}\gamma[\eta_{a}]+2\gamma[b_{t-1}]+1=0\\\
\gamma[\eta_{b}]+2\gamma[a_{t-1}^{\top}x]=0\\\
\gamma[\eta_{a}]+\gamma[\eta_{b}]+\gamma[b_{t-1}]+\gamma[a_{t-1}^{\top}x]+1=0,\end{cases}$

where we have used the elementary formulas from Section A.2. Simple
calculations yield $\gamma[\eta_{a}]+\gamma[\eta_{b}]=-1$. Using the gradient
update expression with the elementary addition from Section A.2, the recursive
formulas controlling $\gamma[b_{t}]$ and $\gamma[a_{t}^{\top}x]$ are given by

$\begin{cases}\gamma[b_{t}]=\max(\gamma[b_{t-1}],\gamma[\eta_{b}]+\gamma[a_{t-1}^{\top}x])\\\
\gamma[a_{t}^{\top}x]=\max(\gamma[a_{t-1}^{\top}x],\gamma[\eta_{a}]+\gamma[b_{t-1}]+1).\end{cases}$

Starting from $t=1$, with Init[1], we have
$\gamma[b_{1}]=\gamma[\eta_{b}(a_{0}^{\top}x)y]=\gamma[\eta_{b}]$ and
$\gamma[a_{1}^{\top}x]=\gamma[a_{0}^{\top}x]=0$. Therefore
$\gamma[b_{2}]=\max(\gamma[\eta_{b}],\gamma[\eta_{b}]+0)=\gamma[\eta_{b}]$,
and
$\gamma[a_{2}^{\top}x]=\max(0,\gamma[\eta_{a}]+\gamma[\eta_{b}]+1)=\max(0,0)=0$.
By induction, this holds for all $t\geq 1$. With Init[2], we have
$\gamma[b_{1}]=\gamma[b_{0}]=0$, and
$\gamma[a_{1}^{\top}x]=\gamma[-\eta_{a}b_{0}^{2}y\|x\|^{2}]=\gamma[\eta_{a}]+1$.
At step $t=2$, we have
$\gamma[b_{2}]=\max(0,\gamma[\eta_{b}]+\gamma[\eta_{a}]+1)=0$ and
$\gamma[a_{2}^{\top}x]=\max(\gamma[\eta_{a}]+1,\gamma[\eta_{a}]+0+1)=\gamma[\eta_{a}]+1$,
and this holds for all $t$ by induction. In both cases, to ensure that
$\gamma[f_{t}]=\gamma[b_{t}]+\gamma[a_{t}^{\top}x]=0$, we have to set
$\gamma[\eta_{b}]=0$ and $\gamma[\eta_{a}]=-1$ (straightforward from the
equation $\gamma[\eta_{b}]+\gamma[\eta_{a}]=-1$). In conclusion, setting
$\eta_{a}=\Theta(n^{-1})$ and $\eta_{b}=\Theta(1)$ ensures efficient fine-
tuning with LoRA.

∎

###  A.5 Proof of 1

In this section, we give a non-rigorous but intuitive proof of 1. The proof
relies on the following assumption on the processed gradient $g_{A}$.

######  Assumption 1.

With the same setup of Section 4, at training step $t$, we have
$g_{A}^{t}\underline{Z}=\Theta(n)$.

To see why 1 is sound in practice, let us study the product
$g_{A}^{t}\underline{Z}$ in the simple case of Adam with no momentum, a.k.a
SignSGD which is given by

$g_{A}=\textrm{sign}\left(\frac{\partial\mathcal{L}}{\partial A}\right),$

where the sign function is applied element-wise. At training step $t$, we have

$\frac{\partial\mathcal{L}_{t}}{\partial
A}=\frac{\alpha}{r}B^{\top}_{t-1}d\bar{Z}^{t-1}\otimes\underline{Z},$

Let $S^{t}=\frac{\alpha}{r}B^{\top}_{t-1}d\bar{Z}^{t-1}$. Therefore we have

$g_{A}=\textrm{sign}(S^{t}\otimes\underline{Z})=(\textrm{sign}(S^{t}_{i}\underline{Z}_{j}))_{1\leq
i,j\leq n}.$

However, note that we also have

$\textrm{sign}(S^{t}_{i}\underline{Z}_{j})=\textrm{sign}(S^{t}_{i})\textrm{sign}(\underline{Z}_{j}),$

and as a result

$g_{A}^{t}=\textrm{sign}(S^{t})\otimes\textrm{sign}(\underline{Z}).$

Hence, we obtain

$g_{A}^{t}\underline{Z}=(\textrm{sign}(\underline{Z})^{\top}\underline{Z})\textrm{sign}(S^{t})=\Theta(n),$

where we used the fact that
$\textrm{sign}(\underline{Z})^{\top}\underline{Z}=\Theta(n)$.

This intuition should in-principle hold for the general variant of Adam with
momentum as long as the gradient processing function (a notion introduced in
[2]) roughly preserves the $\textrm{sign}(\underline{Z})$ direction. This
reasoning can be made rigorous for general gradient processing function using
the Tensor Program framework and taking the infinite-width limit where the
components of $g_{A},\underline{Z},d\bar{Z}$ all become iid. However this
necessitates an intricate treatment of several quantities in the process,
which we believe is an unnecessary complication and does not serve the main
purpose of this paper.

Let us now give a proof for the main claim.

Theorem 1. _Assume that weight matrices $A$ and $B$ are trained with Adam with
respective learning rates $\eta_{A}$ and $\eta_{B}$ and that 1 is satisifed
with the Adam gradient processing function. Then, it is impossible to achieve
efficiency with $\eta_{A}=\eta_{B}$. However, LoRA Finetuning is efficient
with $\eta_{A}=\Theta(n^{-1})$ and $\eta_{B}=\Theta(1)$._

###### Proof.

With the same setup of Section 4, at step $t$, we have

$\begin{cases}\delta_{t}^{1}=B_{t-1}\Delta
Z_{A}^{t}=-\eta_{A}B_{t-1}g^{t-1}_{A}\underline{Z}\\\ \delta_{t}^{2}=\Delta
B_{t}Z_{A}^{t-1}=-\eta_{B}g^{t-1}_{B}A_{t-1}\underline{Z}\\\
\delta_{t}^{3}=\Delta B_{t}\Delta
Z_{A}^{t}=\eta_{A}\eta_{B}g^{t-1}_{B}g^{t-1}_{A}\underline{Z}\end{cases}$

The key observation here is that $g^{t-1}_{A}\underline{Z}$ has entries of
order $\Theta(n)$ as predicted and justified in 1. Having
$\delta^{i}_{t}=\Theta(1)$ for $i\in\\{1,2\\}$ and $Z_{B}^{t}=\Theta(1)$ for
$t>1$ translate to

$\begin{cases}\gamma[\eta_{A}]+\gamma[B_{t-1}]+1=0\\\
\gamma[\eta_{B}]+\gamma[A_{t-1}\underline{Z}]=0\\\
\gamma[B_{t-1}]+\gamma[A_{t-1}\underline{Z}]=0,\end{cases}$

which implies that $\gamma[\eta_{A}]+\gamma[\eta_{B}]=-1$.

With the gradient updates, we have

$\displaystyle B_{t}$ $\displaystyle=B_{t-1}-\eta_{B}g^{t-1}_{B}$
$\displaystyle A_{t}\underline{Z}$
$\displaystyle=A_{t-1}\underline{Z}-\eta_{A}g^{t-1}_{A}\underline{Z}$

which implies that

$\displaystyle\gamma[B_{t}]$
$\displaystyle=\max(\gamma[B_{t-1}],\gamma[\eta_{B}])$
$\displaystyle\gamma[A_{t}\underline{Z}]$
$\displaystyle=\max(\gamma[A_{t-1}\underline{Z}],\gamma[\eta_{A}]+1),$

Now assume that the model is initialized with Init[1]. We have
$\gamma[B_{1}]=\gamma[\eta_{B}]$ and therefore for all $t$, we have
$\gamma[B_{t}]=\gamma[\eta_{B}]$. We also have
$\gamma[A_{1}\underline{Z}]=\gamma[A_{0}\underline{Z}]=0$ (because
$A_{1}=A_{0}$, and we use the Central Limit Theorem to conclude). Hence, if we
choose the same learning rate for $A$ and $B$, given by $\eta$, we obtain
$\gamma[\eta]=-1/2$, and therefore
$\gamma[Z_{A}^{t-1}]=\gamma[A_{t-1}\underline{Z}]=1/2$ which violates the
stability condition. A similar behaviour occurs with Init[2]. Hence,
efficiency is not possible in this case. However, if we set
$\gamma[\eta_{B}]=0$ and $\gamma[\eta_{A}]=-1$, we get that
$\gamma[B_{t}]=0,\gamma[A_{t}\underline{Z}]=0$, and $\delta_{t}^{i}=\Theta(1)$
for all $i\in\\{1,2,3\\}$ and $t\geq 1$. The same result holds with Init[2].

∎

##  Appendix B Efficiency from a Loss perspective.

Consider the same setup of Section 4. At step $t$, the loss changes as follows

$\displaystyle\Delta\mathcal{L}$
$\displaystyle=\mathcal{L}((BA)_{t})-\mathcal{L}((BA)_{t-1})$
$\displaystyle\approx\langle
d\bar{Z}^{t-1}\otimes\underline{Z},(BA)_{t}-(BA)_{t-1}\rangle_{F}$
$\displaystyle=\langle d\bar{Z}^{t-1},\Delta Z_{B}^{t}\rangle,$

where $\langle.,.\rangle_{F}$ is the Frobenius inner product in
$\mathbb{R}^{n\times n}$, and $\langle.,.\rangle$ is the euclidean product in
$\mathbb{R}^{n}$. Since the direction of the feature updates are significantly
correlated with $d\bar{Z}^{t-1}$, it should be expected that having
$\delta_{t}^{i}=\Theta(1)$ for all $i$ results in more efficient loss
reduction.

##  Appendix C Additional Experiments

This section complements the empirical results reported in the main text. We
provide the details of our experimental setup, and show the acc/loss heatmaps
for several configurations.

###  C.1 Empirical Details

####  C.1.1 Toy Example

In Figure 2, we trained a simple MLP with LoRA layers to verify the results of
the analysis in Section 3. Here we provide the empirical details for these
experiments.

##### Model.

We consider a simple MLP given by

$f(x)=W_{out}\phi(BA\phi(W_{in}x)),$

where $W_{in}\in\mathbb{R}^{n\times d},W_{out}\in\mathbb{R}^{1\times
n},A\in\mathbb{R}^{r\times n},B\in\mathbb{R}^{n\times r}$ are the weights, and
$\phi$ is the ReLU activation function. Here, we used $d=5$, $n=100$, and
$r=4$.

##### Dataset.

Synthetic dataset generated by
$X\sim\mathcal{N}(0,I_{d}),Y=\sin(d^{-1}\sum_{i=1}^{d}X_{i})$ with $d=5$. The
number of training examples is $N_{train}=1000$, and the number of test
examples is $N_{test}=100$.

##### Training.

We train the model with gradient descent for a range for values of
$(\eta_{A},\eta_{B})$. The weights are initialized as follows:
$W_{in}\sim\mathcal{N}(0,1.),W_{out}\sim\mathcal{N}(0,1/n),A\sim\mathcal{N}(0,1/n),B\sim\mathcal{N}(0,1.)$.
Only the weight matrices $A,B$ are trained and $W_{in},W_{out}$ are fixed to
their initial value.

####  C.1.2 GLUE tasks with GPT2/Roberta

For our experiments with GPT2/Roberta-base models, finetuned on GLUE tasks, we
use the following setup:

##### Tasks.

MNLI, QQP, SST2, QNLI

##### Models.

GPT2, Roberta-base (sequence length 128)

##### LoRA initialization.

Init[2] (we did not observe a significant difference between Init[1] and
Init[2])

##### Training Alg.

AdamW with $\beta_{1}=0.9,\beta_{2}=0.99,\epsilon=$ 1e-8, linear schedule, no
warmup.

##### Learning rate grid.

$\eta_{A}\in\\{$4e-3, 2e-3, 1e-3, 5e-4, 2e-4, 1e-4$\\}$, $\eta_{B}\in\\{$
8e-4, 4e-4, 2e-4, 1e-4, 5e-5, 2e-5, 1e-5 $\\}$.

##### Targert Modules for LoRA.

For Roberta-base, we add LoRA layers to ‘query’ and ‘value’ weights. For GPT2,
we add LoRA layers to ‘c_attn, c_proj, c_fc’.

##### Other Hyperparameters.

Sequence length $T=128$, train batch size $bs=32$, number of train epochs
$E=3$ ($E=10$ for SST2), number of random seeds $s=3$.

##### GPUs.

Nvidia V100, Nvidia A10.

####  C.1.3 Llama MNLI

For our experiments using the Llama-7b model, finetuned on MNLI, we use
following setup

##### Training Alg.

AdamW with $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=$ 1e-6, constant
schedule.

##### Learning rate grid.

$\eta_{A}\in\\{$1e-6, 5e-6, 1e-5, 5e-5, 1e-4$\\}$, $\eta_{B}\in\\{$1e-6, 5e-6,
1e-5, 5e-5, 1e-4, 2e-4, 4e-4$\\}$

##### LoRA Hyperparameters.

LoRA rank $r=8$, $\alpha=16$, and dropout $0.1$. LoRA target modules ‘q_proj,
k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj’.

##### Other Hyperparameters.

Sequence length $T=128$, train batch size $bs=32$, number of train epochs
$E=1$, number of random seeds $s=2$ for $\eta_{A}=\eta_{B}$ and
$\eta_{A},\eta_{B}$ near test optimal, $s=1$ otherwise. Precision FP16.

##### GPUs.

Nvidia V100.

####  C.1.4 Llama flan-v2

For our experiments using the Llama-7b model, finetuned on a size 100k random
subset flan-v2, we use following setup

##### Training Alg.

AdamW with $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=$ 1e-6, constant
schedule.

##### Learning rate grid.

$\eta_{A}\in\\{$1e-5, 5e-5, 1e-4, 2e-4, 4e-4$\\}$, $\eta_{B}\in\\{$1e-5, 5e-5,
1e-4, 2e-4, 4e-4, 5e-4, 1e-3, 2e-3$\\}$

##### LoRA Hyperparameters.

LoRA rank $r=64$, $\alpha=16$, and dropout $0.1$. LoRA target modules ‘q_proj,
k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj’.

##### Other Hyperparameters.

Sequence length $T_{\text{source}}=1536$, $T_{\text{target}}=512$, train batch
size $bs=16$, number of epochs $E=1$, number of random seeds $s=2$ for
$\eta_{A}=\eta_{B}$ and $\eta_{A},\eta_{B}$ near test optimal, $s=1$
otherwise. Precision BF16.

##### MMLU Evaluation.

We evaluate average accuracy on MMLU using 5-shot prompting.

##### GPUs.

Nvidia A10.

###  C.2 Results of Roberta-base finetuning on all tasks

Figure 3 showed finetuning test accuracy for Roberta-base. To complement these
results, we show here the test/train accuracy for all tasks.

Figure 8: GLUE/Roberta-base: same as Figure 3 with test/train accuracy.

Interestingly, the optimal choice of learning rates for test accuracy differs
from that of the train accuracy, although the difference is small. This can be
due to mild overfitting occuring during finetuning (the optimal choice of
learning rates $(\eta_{A},\eta_{B})$ for train accuracy probably lead to a
some overfitting).

###  C.3 Results of GPT2 finetuning on all tasks

Figure 4 showed finetuning results for GPT2 on MNLI and QQP. To complement
these results, we show here the test/train accuracy for all tasks.

Figure 9: GLUE/GPT2: same setup as Figure 4 with additional tasks

###  C.4 GLUE tasks with full precision finetuning

Figure 10: GLUE/Roberta-base: same as Figure 3 with full precision training
instead of FP16. Figure 11: GLUE/GPT2: same setup as Figure 9 with full
precision training

###  C.5 GLUE tasks test/train loss

Figure 12: GLUE/Roberta-base: same setup as Figure 3 with
$100\times$Test/Train loss instead of accuracy Figure 13: GLUE/GPT2: same
setup as Figure 9 with $100\times$Test/Train loss instead of accuracy

###  C.6 GLUE tasks with different LoRA ranks

Figure 14: GLUE/Roberta-base: same setup as Figure 3 with $r=4$ Figure 15:
GLUE/Roberta-base: same setup as Figure 3 with $r=16$ Figure 16: GLUE/GPT2:
same setup as Figure 11 with $r=4$

###  C.7 Llama MNLI test/train loss

Figure 17: Train and test loss of Llama-7b finetuned on MNLI in the same
setting as Figure 5 right panel.

###  C.8 Llama Flan-v2 MMLU acc/train loss

Figure 18: MMLU evaluation accuracy and train loss of Llama-7b trained on
flan-v2 100k in the same setting as Figure 5 left panel. Interestingly, even
in one epoch the model can overfit. We were unable to find $\eta_{B}>\eta_{A}$
that was optimal for train loss, however it could be the case that the grid
was not fine enough or that overfitting does not require much “feature
learning" and $\eta_{B}/\eta_{A}\approx 1$ is optimal for minimizing train
loss (see the main text for more discussion).


# Factorial moments of the Poisson distribution of order $k$

S. R. Mane<EMAIL_ADDRESS>Convergent Computing Inc., P. O. Box 561,
Shoreham, NY 11786, USA

###### Abstract

The factorial moments of the standard Poisson distribution are well known. The
present note presents an explicit combinatorial sum for the factorial moments
of the Poisson distribution of order $k$. Unlike the standard Poisson
distribution (the case $k=1$), for $k>1$ the structure of the factorial
moments is much more complicated. Some properties of the factorial moments of
the Poisson distribution of order $k$ are elucidated in this note.

###### keywords:

Poisson distribution of order $k$ , moments , factorial moments , Compound
Poisson distribution , discrete distribution

###### MSC:

[2020] 60E05 , 39B05 , 11B37 , 05-08

††journal: (internal report CC23-11)

It is well known that for a Poisson distribution with rate parameter
$\lambda$, the mean and variance both equal $\lambda$ and the $n^{th}$
factorial moment is $\lambda^{n}$ for all $n\geq 1$. The Poisson distribution
of order $k$ [1] is a variant or extension of the standard Poisson
distribution. For $k=1$ it is the standard Poisson distribution. It is an
example of a compound Poisson distribution [2]. Its probability mass function
is [1]

$P_{n}(k,\lambda)=e^{-k\lambda}\sum_{n_{1}+2n_{2}+\dots+kn_{k}=n}\prod_{j=1}^{k}\frac{\lambda^{n_{j}}}{n_{j}!}\,.$
(1)

For $k=1$ this simplifies to standard result
$P_{n}(\lambda)=e^{-\lambda}\lambda^{n}/n!$. Philippou [3] derived the mean
and variance

$\displaystyle\mu_{k}$ $\displaystyle=(1+2+\dots+k)\lambda$ $\displaystyle=$
$\displaystyle\;\frac{k(k+1)}{2}\,\lambda\,,$ (2a)
$\displaystyle\sigma^{2}_{k}$ $\displaystyle=(1^{2}+2^{2}+\dots+k^{2})\lambda$
$\displaystyle=$ $\displaystyle\;\frac{k(k+1)(2k+1)}{6}\,\lambda\,.$ (2b)

Less is known about its factorial moments. Let $M_{(n)}(k,\lambda)$ denote the
$n^{th}$ factorial moment of the Poisson distribution of order $k$. The author
obtained the following combinatorial sum

$M_{(n)}(k,\lambda)=n!\sum_{n_{1}+2n_{2}+\dots+kn_{k}=n}\prod_{j=1}^{k}\frac{\lambda^{n_{j}}}{n_{j}!}\biggl{[}\binom{k+1}{j+1}\biggr{]}^{n_{j}}\,.$
(3)

On searching the literature, it was discovered that Charalambides published
the following expression for the factorial moment (eq. (4.9) in [4])

$M_{(n)}=n!\sum_{r=0}^{n}\lambda^{r}\,Q(n+r,r,k+1)/r!\,.$ (4)

The quantity $Q(n+r,r,k+1)$ was given as follows (eq. (3.19) in [4])

$T_{n,r;k}(g_{1},g_{2},\dots,g_{m})=n!k^{-r}Q(n+r,r,k+1)/r!\,.$ (5)

Here (unnumbered before eq. (3.19) in [4])

$\displaystyle g_{j}$ $\displaystyle=\frac{j!}{k}\binom{k+1}{j+1}\,,$ (6a)
$\displaystyle\sum_{n=0}^{\infty}T_{n,r;k}(g_{1},g_{2},\dots,g_{m})\frac{t^{n}}{n!}$
$\displaystyle=k^{-r}t^{-r}\biggl{(}\sum_{j=1}^{k}\binom{k+1}{j+1}t^{j+1}\biggr{)}^{r}\biggr{/}r!\,.$
(6b)

(Note: this corrects a misprint in [4], where $t^{j+1}$ was written as
$t^{i+1}$.) The subscript “$m$” on $g_{m}$ is $m=\min\\{n,k\\}$. Upon making
the relevant substitutions into eq. (4), the result coincides with eq. (3).
Hence eq. (3) can be considered as a confirmation of the formulas in [4].
However [4] did not display an explicit combinatorial sum for the factorial
moments of the Poisson distribution of order $k$.

For the standard Poisson distribution ($k=1$), the sum in eq. (3) contains
only one summand and the result simplifies to (setting $n_{1}=n$)

$M_{(n)}(1,\lambda)=n!\frac{\lambda^{n}}{n!}\biggl{[}\binom{2}{2}\biggr{]}^{n}=\lambda^{n}\,.$
(7)

This is the well known result for the standard Poisson distribution. However,
for $k>1$, the sum in eq. (3) contains many more terms. At a minimum, it is
evident that $M_{(n)}(k,\lambda)$ is a polynomial in $\lambda$ of degree $n$.
It is also evident that $M_{(n)}(k,\lambda)=1$ for $n=0$ and has no constant
term for $n>0$. Also note that if $n<k$, then terms where $j>n$ do not appear
in the sum in eq. (3) because we must have $n_{j}=0$ if $j>n$. This tells us
the following: for fixed $n\geq 1$, the expression for $M_{(n)}(k,\lambda)$ is
the same for all $k\geq n$. For example:

  1. 1.

For $n=1$, only $n_{1}$ is nonzero and the factorial moment is
$M_{(1)}(k,\lambda)=\lambda$ for all $k\geq 1$.

  2. 2.

For $n=2$, only $n_{1}$ and $n_{2}$ are nonzero. We shall see below that

$M_{(2)}(k,\lambda)=\biggl{(}\frac{k(k+1)}{2}\biggr{)}^{2}\lambda^{2}+\frac{(k-1)k(k+1)}{3}\,\lambda\,.$
(8)

This is the expression for $M_{(2)}(k,\lambda)$ for all $k\geq 2$. Observe
that for $k=1$, the second term (in $\lambda$) vanishes and also $k(k+1)/2=1$
so the result is $M_{(2)}(1,\lambda)=\lambda^{2}$.

For brevity of the notation in the formulas below, define the following (where
$\kappa_{j}=0$ if $j>k$)

$\kappa_{j}=\binom{k+1}{j+1}\,.$ (9)

Hence $\kappa_{1}=k(k+1)/2$ and $\kappa_{2}=(k-1)k(k+1)/6$, etc. Observe that
$\kappa_{j}$ is proportional to $g_{j}$ in [4] (see eq. (6a)). Let us study
the simplest case, viz. $k=2$. Set $n_{2}=s$ and $n_{1}=n-2s$ and note that
$0\leq s\leq\lfloor(n/2)\rfloor$ to obtain

$M_{(n)}(2,\lambda)=\sum_{s=0}^{\lfloor(n/2)\rfloor}\frac{n!}{(n-2s)!s!}\kappa_{1}^{n-2s}\kappa_{2}^{s}\,\lambda^{n-s}\,.$
(10)

Then for $n=1,2,\dots$ (omitting explicit mention of the arguments
$(k,\lambda)$)

$\displaystyle M_{(1)}$ $\displaystyle=\kappa_{1}\lambda\,,$ (11a)
$\displaystyle M_{(2)}$
$\displaystyle=\kappa_{1}^{2}\lambda^{2}+2\kappa_{2}\lambda\,,$ (11b)
$\displaystyle M_{(3)}$
$\displaystyle=\kappa_{1}^{3}\lambda^{3}+6\kappa_{1}\kappa_{2}\lambda^{2}\,,$
(11c) $\displaystyle M_{(4)}$
$\displaystyle=\kappa_{1}^{4}\lambda^{4}+12\kappa_{1}^{2}\kappa_{2}\lambda^{3}+12\kappa_{2}^{2}\lambda^{2}\,,$
(11d) $\displaystyle M_{(5)}$
$\displaystyle=\kappa_{1}^{5}\lambda^{5}+20\kappa_{1}^{3}\kappa_{2}\lambda^{4}+60\kappa_{1}\kappa_{2}^{2}\lambda^{3}\,,$
(11e) $\displaystyle M_{(6)}$
$\displaystyle=\kappa_{1}^{6}\lambda^{6}+30\kappa_{1}^{4}\kappa_{2}\lambda^{5}+180\kappa_{1}^{2}\kappa_{2}^{2}\lambda^{4}+120\kappa_{2}^{3}\lambda^{3}\,.$
(11f)

Note the following:

  1. 1.

Clearly, for any $k>1$, even for $k=2$, the number of terms in $M_{(n)}$ is
not bounded as a function of $n$. For $k=2$, the polynomial
$M_{(n)}(2,\lambda)$ contains all powers of $\lambda$ from $n$ down to
$\lfloor(n+1)/2\rfloor$.

  2. 2.

More generally, the polynomial $M_{(n)}(k,\lambda)$ contains all powers of
$\lambda$ from $n$ down to $\lfloor(n+k-1)/k\rfloor$. It is only for $k=1$
that the number of terms in $M_{(n)}(k,\lambda)$ is a bounded function of $n$.

  3. 3.

Stated another way, the lowest power of $\lambda$ in $M_{(n)}(k,\lambda)$ is
$\lambda$ for $n\in[1,k]$ and $\lambda^{2}$ for $n\in[k+1,2k]$ and in general
it is $\lambda^{j}$ for $n\in[jk-k+1,jk]$.

Let us write the expressions for $M_{(n)}(k,\lambda)$ for low values of $n$
and arbitrary $k$. We again omit explicit mention of the arguments
$(k,\lambda)$.

$\displaystyle M_{(1)}$ $\displaystyle=\kappa_{1}\lambda\,,$ (12a)
$\displaystyle M_{(2)}$
$\displaystyle=\kappa_{1}^{2}\lambda^{2}+2\kappa_{2}\lambda\,,$ (12b)
$\displaystyle M_{(3)}$
$\displaystyle=\kappa_{1}^{3}\lambda^{3}+6\kappa_{1}\kappa_{2}\lambda^{2}+6\kappa_{3}\lambda\,,$
(12c) $\displaystyle M_{(4)}$
$\displaystyle=\kappa_{1}^{4}\lambda^{4}+12\kappa_{1}^{2}\kappa_{2}\lambda^{3}+(12\kappa_{2}^{2}+24\kappa_{1}\kappa_{3})\lambda^{2}+24\kappa_{4}\lambda\,,$
(12d) $\displaystyle M_{(5)}$
$\displaystyle=\kappa_{1}^{5}\lambda^{5}+20\kappa_{1}^{3}\kappa_{2}\lambda^{4}+(60\kappa_{1}\kappa_{2}^{2}+60\kappa_{1}^{2}\kappa_{3})\lambda^{3}+(120\kappa_{1}\kappa_{4}+120\kappa_{2}\kappa_{3})\lambda^{2}+120\kappa_{5}\lambda\,,$
(12e) $\displaystyle M_{(6)}$
$\displaystyle=\kappa_{1}^{6}\lambda^{6}+30\kappa_{1}^{4}\kappa_{2}\lambda^{5}+(180\kappa_{1}^{2}\kappa_{2}^{2}+120\kappa_{1}^{3}\kappa_{3})\lambda^{4}+(360\kappa_{1}^{2}\kappa_{4}+120\kappa_{2}^{3}+720\kappa_{1}\kappa_{2}\kappa_{3})\lambda^{3}$
$\displaystyle\quad\qquad\;\,+(720\kappa_{1}\kappa_{5}+720\kappa_{2}\kappa_{4}+360\kappa_{3}^{2})\lambda^{2}+720\kappa_{6}\lambda\,.$
(12f)

We adopt the notation in the textbook by Graham, Knuth and Patashnik [5] and
write $[\lambda^{j}]M_{(n)}$ for the coefficient of $\lambda^{j}$ in
$M_{(n)}$. Then note the following:

  1. 1.

For all $k\geq 1$ and $n\geq 1$, the coefficient of the highest power
$\lambda^{n}$ is

$[\lambda^{n}]\,M_{(n)}=\frac{n!}{n!}\kappa_{1}^{n}=\biggl{(}\frac{k(k+1)}{2}\biggr{)}^{n}\,.$
(13)

  2. 2.

For all $k\geq 2$ and $n\geq 2$, the coefficient of the second highest power
$\lambda^{n-1}$ is

$\begin{split}[\lambda^{n-1}]\,M_{(n)}&=\frac{n!}{(n-2)!1!}\kappa_{1}^{n-2}\kappa_{2}\\\
&=\frac{n(n-1)}{3}\biggl{(}\frac{k(k+1)}{2}\biggr{)}^{n-1}(k-1)\,.\end{split}$
(14)

  3. 3.

It is more complicated for the coefficient of the third highest power
$\lambda^{n-2}$ and lower powers.

  4. 4.

For all $k\geq 3$ and $n\geq 3$, the coefficient of the third highest power
$\lambda^{n-2}$ is

$\begin{split}[\lambda^{n-2}]\,M_{(n)}&=\frac{n!}{(n-4)!2!}\kappa_{1}^{n-4}\kappa_{2}^{2}+\frac{n!}{(n-3)!1!}\kappa_{1}^{n-3}\kappa_{3}\\\
&=\frac{1}{6}\binom{n}{3}\biggl{(}\frac{k(k+1)}{2}\biggr{)}^{n-2}(k-1)((2n-3)k-2n)\,.\end{split}$
(15)

  5. 5.

For all $k\geq 4$ and $n\geq 4$, the coefficient of the fourth highest power
$\lambda^{n-3}$ is

$\begin{split}[\lambda^{n-3}]\,M_{(n)}&=\frac{n!}{(n-6)!3!}\kappa_{1}^{n-6}\kappa_{2}^{3}+\frac{n!}{(n-5)!1!1!}\kappa_{1}^{n-5}\kappa_{2}\kappa_{3}+\frac{n!}{(n-4)!1!}\kappa_{1}^{n-4}\kappa_{4}\\\
&=\frac{2}{135}\binom{n}{4}\biggl{(}\frac{k(k+1)}{2}\biggr{)}^{n-3}(k-1)\biggl{[}(10n^{2}-45n+47)k^{2}-5(4n^{2}-9n-1)k+2(5n^{2}+1)\biggr{]}\,.\end{split}$
(16)

  6. 6.

However, the coefficient of the lowest power $\lambda$ is simple, for all
$k\geq 1$ and $n\geq 1$:

$[\lambda]\,M_{(n)}=\frac{n!}{1!}\kappa_{n}=n!\binom{k+1}{n+1}=\frac{(k+1)k\dots(k-n+1)}{n+1}\,.$
(17)

The value of the coefficient is zero if $k<n$. For $k\geq 2$ and $n=2$, the
coefficient of $\lambda$ is $(k-1)k(k+1)/3$, as stated in eq. (8).

The factorial moments can also be derived from the factorial moment generating
function (FMGF). For the Poisson distribution of order $k$, the FMGF is,
omitting the arguments $(k,\lambda)$ ([2], see also [4])

$M(t)=e^{-k\lambda}e^{\lambda(t+t^{2}+\dots+t^{k})}\,.$ (18)

The $n^{th}$ factorial moment is given by

$M_{(n)}=\frac{d^{n}M(t)}{dt^{n}}\biggr{|}_{t=1}\,.$ (19)

The first few factorial moments are as follows.

  1. 1.

The first derivative is

$\frac{dM(t)}{dt}=\lambda
e^{-k\lambda}e^{\lambda(t+t^{2}+\dots+t^{k})}(1+2t+\dots+kt^{k-1})\,.$ (20)

  2. 2.

Evaluate at $t=1$ to obtain the first factorial moment

$M_{(1)}=\lambda(1+2+\dots+k)=\binom{k+1}{2}\lambda=\kappa_{1}\lambda\,.$ (21)

  3. 3.

The second derivative is

$\begin{split}\frac{d^{2}M(t)}{dt^{2}}&=\lambda e^{-k\lambda}\,\frac{d\
}{dt}\Bigl{[}\,e^{\lambda(t+t^{2}+\dots+t^{k})}(1+2t+\dots+kt^{k-1})\,\Bigr{]}\\\
&=\lambda
e^{-k\lambda}\,e^{\lambda(t+t^{2}+\dots+t^{k})}\biggl{[}\,\lambda(1+2t+\dots+kt^{k-1})^{2}+2!\sum_{j=2}^{k}\binom{j}{2}t^{j-2}\,\biggr{]}\,.\end{split}$
(22)

  4. 4.

Evaluate at $t=1$ to obtain the second factorial moment

$\begin{split}M_{(2)}&=\lambda^{2}\binom{k+1}{2}^{2}+2!\lambda\binom{k+1}{3}\\\
&=\kappa_{1}^{2}\lambda^{2}+2\kappa_{2}\lambda\\\
&=\biggl{(}\frac{k(k+1)}{2}\biggr{)}^{2}\lambda^{2}+\frac{(k-1)k(k+1)}{3}\lambda\,.\end{split}$
(23)

This agrees with eqs. (8) and (12b).

  5. 5.

The third derivative is

$\begin{split}\frac{d^{3}M(t)}{dt^{2}}&=\lambda e^{-k\lambda}\,\frac{d\
}{dt}\biggl{\\{}\,e^{\lambda(t+t^{2}+\dots+t^{k})}\biggl{[}\,\lambda(1+2t+\dots+kt^{k-1})^{2}+2!\sum_{j=2}^{k}\binom{j}{2}t^{j-2}\,\biggr{]}\biggr{\\}}\\\
&=\lambda
e^{-k\lambda}\,e^{\lambda(t+t^{2}+\dots+t^{k})}\biggl{\\{}\,\lambda^{2}(1+2t+\dots+kt^{k-1})^{3}\\\
&\qquad\qquad\qquad\qquad\qquad\quad+3\times
2!\lambda\biggl{(}\sum_{j=2}^{k}\binom{j}{2}t^{j-2}\biggr{)}(1+2t+\dots+kt^{k-1})\\\
&\qquad\qquad\qquad\qquad\qquad\quad+3!\sum_{j=2}^{k}\binom{j}{3}t^{j-3}\,\biggr{\\}}\,.\end{split}$
(24)

  6. 6.

Evaluate at $t=1$ to obtain the third factorial moment

$\begin{split}M_{(3)}&=\lambda^{3}\binom{k+1}{2}^{3}+6\lambda^{2}\binom{k+1}{2}\binom{k+1}{3}+3!\lambda\binom{k+1}{4}\\\
&=\kappa_{1}^{3}\lambda^{3}+6\kappa_{1}\kappa_{2}\lambda^{2}+6\kappa_{3}\lambda\\\
&=\biggl{(}\frac{k(k+1)}{2}\biggr{)}^{3}\lambda^{3}+2\biggl{(}\frac{k(k+1)}{2}\biggr{)}^{2}(k-1)\lambda^{2}+\frac{(k-2)(k-1)k(k+1)}{4}\lambda\,.\end{split}$
(25)

This agrees with eq. (12c) and is the same expression for all $k\geq 3$. For
$k=1$ the last two terms vanish and for $k=2$ the last term vanishes.

In conclusion, a formula for the factorial moments of the Poisson distribution
of order $k$ was published in 1986 in [4], although an explicit combinatorial
sum was not displayed. The present note displays an explicit combinatorial sum
for the factorial moments of the Poisson distribution of order $k$, and is in
agreement with the formula in [4]. Unlike the standard Poisson distribution
(the case $k=1$), for $k>1$ the structure of the factorial moments is much
more complicated. Some properties of the factorial moments of the Poisson
distribution of order $k$ were elucidated in this note.

## Acknowledgements

The author thanks Professor Charalambides for clarifying the details of some
of the formulas in [4].

## References

  * [1] A.N. Philippou, C. Georghiou and C. Philippou, “A generalized geometric distribution and some of its properties” Stat. Prob. Lett. 1, 171–175 (1983).
  * [2] R.M. Adelson, “Compound Poisson Distributions” Operational Research Quarterly 17, 73–75 (1966).
  * [3] A.N. Philippou, “Poisson and compound Poisson distributions of order $k$ and some of their properties” Journal of Soviet Mathematics 27, 3294–3297 (1984).
  * [4] Ch.A. Charalambides, “on discrete distributions of order $k$” Ann. Inst. Statist. Math. 38, 557–568 (1986).
  * [5] R.L. Graham, D.E. Knuth and O. Patashnik, Concrete Mathematics, Addison-Wesley, New York, USA, 2nd Edition, (1994).