Open AI paper about zero-shot hyperparameter transfer
https://openreview.net/pdf?id=Bx6qKuBM2AD
None
None
None
Tensor Programs V:
Tuning Large Neural Networks via
Zero-Shot Hyperparameter Transfer
Greg Yang∗× Edward J. Hu∗×† Igor Babuschkin◦ Szymon Sidor◦ Xiaodong Liu×
David Farhi◦ Nick Ryder◦ Jakub Pachocki◦ Weizhu Chen× Jianfeng Gao×
×Microsoft Corporation ◦OpenAI
Abstract
Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively
so for neural networks (NNs) with billions of parameters. We show that, in the
recently discovered Maximal Update Parametrization (µP), many optimal HPs
remain stable even as model size changes. This leads to a new HP tuning paradigm
we call µTransfer: parametrize the target model in µP, tune the HP indirectly on a
smaller model, and zero-shot transfer them to the full-sized model, i.e., without
directly tuning the latter at all. We verify µTransfer on Transformer and ResNet.
For example, 1) by transferring pretraining HPs from a model of 13M parameters,
we outperform published numbers of BERT-large (350M parameters), with a total
tuning cost equivalent to pretraining BERT-large once; 2) by transferring from
40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with
tuning cost only 7% of total pretraining cost. A Pytorch implementation of our
technique can be found at github.com/microsoft/mup.
2
1 Introduction
20 18 16 14 12 10
log2LearningRate
3.5
4.0
4.5
5.0
5.5
6.0
6.5
7.0
Training Loss
optimum shifts
Standard Practice
Width
128
256
512
1024
2048
4096
8192
20 18 16 14 12 10
log2LearningRate
optimum stable
Our Work Figure 1: Training loss against learning rate on
Transformers of varying dmodel trained with Adam.
Conventionally and in contrast with our technique,
different widths do not share the same optimal hyperparameter; wider networks do not always perform better than narrower ones; in fact they underperform the same-width networks in our technique
even after tuning learning rate. See Sections 3
and 4 for experimental setup.
Hyperparameter (HP) tuning is critical to deep
learning. Poorly chosen HPs result in subpar
performance and training instability. Many published baselines are hard to compare to one
another due to varying degrees of HP tuning.
These issues are exacerbated when training extremely large deep learning models, since stateof-the-art networks with billions of parameters
become prohibitively expensive to tune.
Recently, [45] showed that different neural network parametrizations induce different infinitewidth limits and proposed the Maximal Update
Parametrization (abbreviated µP) (summarized
in Table 3) that enables “maximal” feature learning in the limit. Intuitively, it ensures that each
layer is updated on the same order during training regardless of width.
3
In contrast, while the
standard parametrization (SP) ensures activations are of unit order at initialization, it actually causes
†Work done partly during Microsoft AI Residency Program.
∗Equal contribution. Order is random. Correspondence to {gregyang, edwardhu}@microsoft.com
2
See arxiv.org for the full, up-to-date version of this work.
3
i.e., the updates’ effect on activations becomes roughly independent of width in the large width limit.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Algorithm 1 Tuning a Large Target Model via µTransfer
1: Parametrize target model in Maximal Update Parametrization (µP)
2: Tune a smaller version (in width and/or depth) of target model
3: Copy tuned hyperparameters to target model
Table 1: Hyperparameters That Can Be µTransferred, Not µTransferred, or µTransferred
Across, with a few caveats discussed in Section 5.1. * means empirically validated only on Transformers, while all others additionally have theoretical justification.
µTransferable Not µTransferable µTransferred Across
optimization related, init, regularization width, depth*, batch size*,
parameter multipliers, etc (dropout, weight decay, etc) training time*, seq length*
them to blow up in wide models during training [45] essentially due to an imbalance of per-layer
learning rate (also see Fig. 8). We leverage µP to zero-shot transfer HPs from small models to large
models in this work – that is, we obtain near optimal HPs on a large model without directly tuning
it at all! While practitioners have always guessed HPs of large models from those of small models,
the results are hit-or-miss at best because of incorrect parametrization. For example, as shown in
Fig. 1, in a Transformer, the optimal learning rate is stable with width in µP (right) but far from
so in standard parametrization (left). In addition to width, we empirically verify that, with a few
caveats, HPs can also be transferred across depth (in Section 5.1) as well as batch size, language
model sequence length, and training time (in Appendix I.2.1). This reduces the tuning problem of
an (arbitrarily) large model to that of a (fixed-sized) small model. Our overall procedure, which we
call µTransfer, is summarized in Algorithm 1 and Fig. 2, and the HPs we cover are summarized in
Tables 1 and 2.
$$$$$$
$$
Standard
Practice Our Method
Directly tune large model
Shrink Tune Transfer
𝜇 𝜇
Figure 2: Illustration of µTransfer
There are several benefits to our approach: 1. Better Performance: µTransfer is not just about predicting how the
optimal learning rate scales in SP. In general, we expect the
µTransferred model to outperform its SP counterpart with
learning rate optimally tuned. For example, this is the case
in Fig. 1 with the width-8192 Transformer. We discuss the
reason for this in Appendices B and C. 2. Speedup: It provides massive speedup to the tuning of large models. For
example, we are able to outperform published numbers of
(350M) BERT-large [9] purely by zero-shot HP transfer,
with tuning cost approximately equal to 1 BERT-large pretraining. Likewise, we outperform the published numbers
of the 6.7B GPT-3 model [6] with tuning cost being only
7% of total pretraining cost. For models on this scale, HP
tuning is not feasible at all without our approach. 3. Tune
Once for Whole Family: For any fixed family of models with varying width and depth (such as
the BERT family or the GPT-3 family), we only need to tune a single small model and can reuse
its HPs for all models in the family.4 For example, we will use this technique to tune BERT-base
(110M parameters) and BERT-large (350M parameters) simultaneously by transferring from a 13M
model. 4. Better Compute Utilization: While large model training needs to be distributed across
many GPUs, the small model tuning can happen on individual GPUs, greatly increasing the level
of parallelism for tuning (and in the context of organizational compute clusters, better scheduling
and utilization ratio). 5. Painless Transition from Exploration to Scaling Up: Often, researchers
explore new ideas on small models but, when scaling up, find their HPs optimized during exploration
work poorly on large models. µTransfer would solve this problem.
Nevertheless, µTransfer still has several limitations. For example, while it is very effective for
pretraining, it cannot transfer regularization HPs,5so it’s generally not applicable to the finetuning of
pretrained models. We discuss other limitations carefully in Section 5.1.
4
but possibly not for different data and/or tasks.
5
It can transfer regularization HPs to the extent they help training but it may not transfer their effect on
testing.
2
Table 2: Examples of µTransferable Hyperparameters. All of the below can also be specialized
to per-layer hyperparameters.
Optimizer Related Initialization Parameter Multipliers
learning rate (LR), momentum, per-layer multiplicative constants after
Adam beta, LR schedule, etc init. variance weight/biases, etc
Our Contributions
• We demonstrate it is possible to zero-shot transfer near optimal HPs to a large model from a
small version via the Maximal Update Parametrization (µP) from [45].
• While [45] only covered SGD, here we derive µP for Adam as well (Table 3).
• We propose a new HP tuning technique, µTransfer, for large neural networks based on this
observation that provides massive speedup over conventional methods and covers both SGD
and Adam training;
• We thoroughly verify our method on machine translation and large language model pretraining (in Section 6.3) as well as image classification (in Appendix I.1);
• We release a PyTorch [27] package for implementing µTransfer painlessly. A sketch of this
package is given in Appendix J.
Terminologies Sometimes, to be less ambiguous, we often refer to the “large model” as the target
model, as it is the model we wish to ultimately tune, while we refer to the “small model” as the
proxy model, as it proxies the HP tuning process. We follow standard notation dmodel, dhead =
dk, dv, nhead, df fn regarding dimensions in a Transformer; one can see Fig. 11 for a refresher.
Tensor Programs Series This paper is the 5th installment of the Tensor Programs series. While
the target audience here are practitioners and empirical researchers, this paper presents the first major
practical payoff of the theoretical foundation built in previous works [41–46].
2 Parametrization Matters: A Primer
In this section, we give a very basic primer on why the correct parametrization can allow HP transfer
across width, but see Appendices L.1 to L.3 for more (mathematical) details.
The Central Limit Theorem (CLT) says that, if x1, . . . , xn are iid samples from a zero-mean, unitvariance distribution, then √
1
n
(x1 + · · · + xn) converges to a standard Gaussian N (0, 1) as n → ∞.
Therefore, we can say that √
1
n
is the right order of scaling factor cn such that cn(x1 + · · · + xn)
converges to something nontrivial. In contrast, if we set cn = 1/n, then cn(x1 + · · · + xn) → 0; or
if cn = 1, then cn(x1 + · · · + xn) blows up in variance as n → ∞.
Now suppose we would like to minimize the function
Fn(c)
def = E
x1,...,xn
f(c(x1 + · · · + xn)) (1)
over c ∈ R, for some bounded continuous function f : R → R. If we reparametrize c = α/√n for
α ∈ R, then by CLT, Gn(α)
def = Fn(c) → E f(N (0, α2
)) stabilizes into a function of α as n → ∞.
Then for sufficiently large n, the optimal α
∗
n
def = arg minα Gn(α) should be close to α
∗
N for any
N > n, and indeed, for N = ∞ — this precisely means we can transfer the optimal c
∗
n or α
∗
n
for a
smaller problem (say Fn) to a larger problem (say FN ): GN is approximately minimized by α
∗
n
and
FN is approximately minimized by c
∗
n
p
n/N. Because the transfer algorithm is simply copying α,
we say the parametrization c = α/√n is the correct parametrization for this problem.
In the scenario studied in this paper, x1, . . . , xn are akin to randomly initialized parameters of a
width-n neural network, c is akin to a HP such as learning rate, and f is the test-set performance
of the network after training, so that Fn gives its expectation over random initializations. Just as
in this example, if we parametrize the learning rate and other HPs correctly, then we can directly
copy the optimal HPs for a narrower network into a wide network and expect approximately optimal
3
performance — this is the hyperparameter transfer we propose here. It turns out the Maximal Update
Parametrization (µP) introduced in [45] is correct (akin to the parametrization in α above), while the
standard parametrization (SP) is incorrect (akin to the parametrization in c). We will review both
parametrizations shortly. Theoretically, a µP network has a well-defined infinite-width limit — akin
to (x1 + · · · + xn)/
√
n having a N (0, 1) limit by CLT — while a SP network does not (the limit will
blow up) [45].6In fact, based on the theoretical foundation laid in [45], we argue in Appendix L.3
that µP should also be the unique parametrization that allows HP transfer across width.
We emphasize that, to ensure transferability of any hyperparameter (such as learning rate), it’s not
sufficient to reparametrize only that hyperparameter, but rather, we need to identify and correctly
reparametrize all hyperparameters in Table 2. For example, in Fig. 1, the wide models in SP still
underperform their counterparts in µP, even with learning rate tuned optimally. This is precisely
because SP does not scale parameter multipliers and input/output layer learning rates correctly in
contrast to µP (see Table 3). See Appendix C for more intuition via a continuation of our example
here. We shall also explain this more concretely in the context of neural networks in Appendix B.
3 Hyperparameters Don’t Transfer Conventionally
In the community there seem to be conflicting assumptions about HP stability. A priori, models
of different sizes don’t have any reason to share the optimal HPs. Indeed, papers aiming for stateof-the-art results often tune them separately. On the other hand, a nontrivial fraction of papers in
deep learning fixes all HPs when comparing against baselines, which reflects an assumption that
the optimal HPs should be stable — not only among the same model of different sizes but also
among models of different designs — therefore, such comparisons are fair. Here, we demonstrate HP
instability across width explicitly in MLP and Transformers in the standard parametrization. We will
only look at training loss to exclude the effect of regularization.
MLP with Standard Parametrization We start with a 2-hidden-layer MLP with activation function φ, using the standard parametrization7 with LeCun initialization8
akin to the default in PyTorch:
f(ξ) = W3>φ(W2>φ(W1>ξ + b
1
) + b
2
)
with init. W1 ∼ N (0,
1/din), W{2,3} ∼ N (0,1/n), b{1,2} = 0,
(2)
14 12 10 8 6 4 2
log2LearningRate
0.0
0.5
1.0
1.5
2.0
Training Loss
SP / xent
width
256
512
1024
2048
4096
8192
14 12 10 8 6 4 2
log2LearningRate
0.0
0.5
1.0
1.5
2.0
P / xent
Figure 3: MLP width different hidden sizes trained
for 20 epoch on CIFAR-10 using SGD. Left uses standard parametrization (SP); right uses maximal update
parametrization (µP). µP networks exhibit better learning
rate stability than their SP counterparts.
where W1 ∈ R
din×n, b1 ∈ Rn,
W2 ∈ R
n×n, b2 ∈ Rn, W3 ∈
R
n×dout and din, n, and dout are
the input, hidden, and output dimensions. The particular MLP we use has
φ = ReLU and a cross-entropy (xent)
loss function. We define the width of
MLP as the hidden size n, which is
varied from 256 to 8192. The models are trained on CIFAR-10 for 20
epochs, which is more than enough to
ensure convergence.
As shown on the left in Fig. 3, the
optimal learning rate shifts by roughly
an order of magnitude as the width
increases from 256 to 8192; using the
optimal learning of the smallest model
on the largest model gives very bad performance, if not divergence.
Transformer with Standard Parametrization This perhaps unsurprising observation holds for
more complex architectures such as Transformer as well, as shown in Fig. 1 (left). We define width
6The more theoretically astute reader may observe that SP with a Θ(1/width) learning rate induces a
well-defined infinite-width limit exists as well. Nevertheless, this does not allow HP transfer because this limit is
in kernel regime as shown in [45]. See Appendix L.3 for more discussions.
7
i.e. the default parametrization offered by common deep learning frameworks. See Table 3 for a review.
8The key here is that the init. variance ∝ 1/fan_in, so the same insights here apply with e.g. He initialization.
4
https://openreview.net/pdf?id=Bx6qKuBM2AD
None
None
None
Tensor Programs V:
Tuning Large Neural Networks via
Zero-Shot Hyperparameter Transfer
Greg Yang∗× Edward J. Hu∗×† Igor Babuschkin◦ Szymon Sidor◦ Xiaodong Liu×
David Farhi◦ Nick Ryder◦ Jakub Pachocki◦ Weizhu Chen× Jianfeng Gao×
×Microsoft Corporation ◦OpenAI
Abstract
Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively
so for neural networks (NNs) with billions of parameters. We show that, in the
recently discovered Maximal Update Parametrization (µP), many optimal HPs
remain stable even as model size changes. This leads to a new HP tuning paradigm
we call µTransfer: parametrize the target model in µP, tune the HP indirectly on a
smaller model, and zero-shot transfer them to the full-sized model, i.e., without
directly tuning the latter at all. We verify µTransfer on Transformer and ResNet.
For example, 1) by transferring pretraining HPs from a model of 13M parameters,
we outperform published numbers of BERT-large (350M parameters), with a total
tuning cost equivalent to pretraining BERT-large once; 2) by transferring from
40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with
tuning cost only 7% of total pretraining cost. A Pytorch implementation of our
technique can be found at github.com/microsoft/mup.
2
1 Introduction
20 18 16 14 12 10
log2LearningRate
3.5
4.0
4.5
5.0
5.5
6.0
6.5
7.0
Training Loss
optimum shifts
Standard Practice
Width
128
256
512
1024
2048
4096
8192
20 18 16 14 12 10
log2LearningRate
optimum stable
Our Work Figure 1: Training loss against learning rate on
Transformers of varying dmodel trained with Adam.
Conventionally and in contrast with our technique,
different widths do not share the same optimal hyperparameter; wider networks do not always perform better than narrower ones; in fact they underperform the same-width networks in our technique
even after tuning learning rate. See Sections 3
and 4 for experimental setup.
Hyperparameter (HP) tuning is critical to deep
learning. Poorly chosen HPs result in subpar
performance and training instability. Many published baselines are hard to compare to one
another due to varying degrees of HP tuning.
These issues are exacerbated when training extremely large deep learning models, since stateof-the-art networks with billions of parameters
become prohibitively expensive to tune.
Recently, [45] showed that different neural network parametrizations induce different infinitewidth limits and proposed the Maximal Update
Parametrization (abbreviated µP) (summarized
in Table 3) that enables “maximal” feature learning in the limit. Intuitively, it ensures that each
layer is updated on the same order during training regardless of width.
3
In contrast, while the
standard parametrization (SP) ensures activations are of unit order at initialization, it actually causes
†Work done partly during Microsoft AI Residency Program.
∗Equal contribution. Order is random. Correspondence to {gregyang, edwardhu}@microsoft.com
2
See arxiv.org for the full, up-to-date version of this work.
3
i.e., the updates’ effect on activations becomes roughly independent of width in the large width limit.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Algorithm 1 Tuning a Large Target Model via µTransfer
1: Parametrize target model in Maximal Update Parametrization (µP)
2: Tune a smaller version (in width and/or depth) of target model
3: Copy tuned hyperparameters to target model
Table 1: Hyperparameters That Can Be µTransferred, Not µTransferred, or µTransferred
Across, with a few caveats discussed in Section 5.1. * means empirically validated only on Transformers, while all others additionally have theoretical justification.
µTransferable Not µTransferable µTransferred Across
optimization related, init, regularization width, depth*, batch size*,
parameter multipliers, etc (dropout, weight decay, etc) training time*, seq length*
them to blow up in wide models during training [45] essentially due to an imbalance of per-layer
learning rate (also see Fig. 8). We leverage µP to zero-shot transfer HPs from small models to large
models in this work – that is, we obtain near optimal HPs on a large model without directly tuning
it at all! While practitioners have always guessed HPs of large models from those of small models,
the results are hit-or-miss at best because of incorrect parametrization. For example, as shown in
Fig. 1, in a Transformer, the optimal learning rate is stable with width in µP (right) but far from
so in standard parametrization (left). In addition to width, we empirically verify that, with a few
caveats, HPs can also be transferred across depth (in Section 5.1) as well as batch size, language
model sequence length, and training time (in Appendix I.2.1). This reduces the tuning problem of
an (arbitrarily) large model to that of a (fixed-sized) small model. Our overall procedure, which we
call µTransfer, is summarized in Algorithm 1 and Fig. 2, and the HPs we cover are summarized in
Tables 1 and 2.
$$$$$$
$$
Standard
Practice Our Method
Directly tune large model
Shrink Tune Transfer
𝜇 𝜇
Figure 2: Illustration of µTransfer
There are several benefits to our approach: 1. Better Performance: µTransfer is not just about predicting how the
optimal learning rate scales in SP. In general, we expect the
µTransferred model to outperform its SP counterpart with
learning rate optimally tuned. For example, this is the case
in Fig. 1 with the width-8192 Transformer. We discuss the
reason for this in Appendices B and C. 2. Speedup: It provides massive speedup to the tuning of large models. For
example, we are able to outperform published numbers of
(350M) BERT-large [9] purely by zero-shot HP transfer,
with tuning cost approximately equal to 1 BERT-large pretraining. Likewise, we outperform the published numbers
of the 6.7B GPT-3 model [6] with tuning cost being only
7% of total pretraining cost. For models on this scale, HP
tuning is not feasible at all without our approach. 3. Tune
Once for Whole Family: For any fixed family of models with varying width and depth (such as
the BERT family or the GPT-3 family), we only need to tune a single small model and can reuse
its HPs for all models in the family.4 For example, we will use this technique to tune BERT-base
(110M parameters) and BERT-large (350M parameters) simultaneously by transferring from a 13M
model. 4. Better Compute Utilization: While large model training needs to be distributed across
many GPUs, the small model tuning can happen on individual GPUs, greatly increasing the level
of parallelism for tuning (and in the context of organizational compute clusters, better scheduling
and utilization ratio). 5. Painless Transition from Exploration to Scaling Up: Often, researchers
explore new ideas on small models but, when scaling up, find their HPs optimized during exploration
work poorly on large models. µTransfer would solve this problem.
Nevertheless, µTransfer still has several limitations. For example, while it is very effective for
pretraining, it cannot transfer regularization HPs,5so it’s generally not applicable to the finetuning of
pretrained models. We discuss other limitations carefully in Section 5.1.
4
but possibly not for different data and/or tasks.
5
It can transfer regularization HPs to the extent they help training but it may not transfer their effect on
testing.
2
Table 2: Examples of µTransferable Hyperparameters. All of the below can also be specialized
to per-layer hyperparameters.
Optimizer Related Initialization Parameter Multipliers
learning rate (LR), momentum, per-layer multiplicative constants after
Adam beta, LR schedule, etc init. variance weight/biases, etc
Our Contributions
• We demonstrate it is possible to zero-shot transfer near optimal HPs to a large model from a
small version via the Maximal Update Parametrization (µP) from [45].
• While [45] only covered SGD, here we derive µP for Adam as well (Table 3).
• We propose a new HP tuning technique, µTransfer, for large neural networks based on this
observation that provides massive speedup over conventional methods and covers both SGD
and Adam training;
• We thoroughly verify our method on machine translation and large language model pretraining (in Section 6.3) as well as image classification (in Appendix I.1);
• We release a PyTorch [27] package for implementing µTransfer painlessly. A sketch of this
package is given in Appendix J.
Terminologies Sometimes, to be less ambiguous, we often refer to the “large model” as the target
model, as it is the model we wish to ultimately tune, while we refer to the “small model” as the
proxy model, as it proxies the HP tuning process. We follow standard notation dmodel, dhead =
dk, dv, nhead, df fn regarding dimensions in a Transformer; one can see Fig. 11 for a refresher.
Tensor Programs Series This paper is the 5th installment of the Tensor Programs series. While
the target audience here are practitioners and empirical researchers, this paper presents the first major
practical payoff of the theoretical foundation built in previous works [41–46].
2 Parametrization Matters: A Primer
In this section, we give a very basic primer on why the correct parametrization can allow HP transfer
across width, but see Appendices L.1 to L.3 for more (mathematical) details.
The Central Limit Theorem (CLT) says that, if x1, . . . , xn are iid samples from a zero-mean, unitvariance distribution, then √
1
n
(x1 + · · · + xn) converges to a standard Gaussian N (0, 1) as n → ∞.
Therefore, we can say that √
1
n
is the right order of scaling factor cn such that cn(x1 + · · · + xn)
converges to something nontrivial. In contrast, if we set cn = 1/n, then cn(x1 + · · · + xn) → 0; or
if cn = 1, then cn(x1 + · · · + xn) blows up in variance as n → ∞.
Now suppose we would like to minimize the function
Fn(c)
def = E
x1,...,xn
f(c(x1 + · · · + xn)) (1)
over c ∈ R, for some bounded continuous function f : R → R. If we reparametrize c = α/√n for
α ∈ R, then by CLT, Gn(α)
def = Fn(c) → E f(N (0, α2
)) stabilizes into a function of α as n → ∞.
Then for sufficiently large n, the optimal α
∗
n
def = arg minα Gn(α) should be close to α
∗
N for any
N > n, and indeed, for N = ∞ — this precisely means we can transfer the optimal c
∗
n or α
∗
n
for a
smaller problem (say Fn) to a larger problem (say FN ): GN is approximately minimized by α
∗
n
and
FN is approximately minimized by c
∗
n
p
n/N. Because the transfer algorithm is simply copying α,
we say the parametrization c = α/√n is the correct parametrization for this problem.
In the scenario studied in this paper, x1, . . . , xn are akin to randomly initialized parameters of a
width-n neural network, c is akin to a HP such as learning rate, and f is the test-set performance
of the network after training, so that Fn gives its expectation over random initializations. Just as
in this example, if we parametrize the learning rate and other HPs correctly, then we can directly
copy the optimal HPs for a narrower network into a wide network and expect approximately optimal
3
performance — this is the hyperparameter transfer we propose here. It turns out the Maximal Update
Parametrization (µP) introduced in [45] is correct (akin to the parametrization in α above), while the
standard parametrization (SP) is incorrect (akin to the parametrization in c). We will review both
parametrizations shortly. Theoretically, a µP network has a well-defined infinite-width limit — akin
to (x1 + · · · + xn)/
√
n having a N (0, 1) limit by CLT — while a SP network does not (the limit will
blow up) [45].6In fact, based on the theoretical foundation laid in [45], we argue in Appendix L.3
that µP should also be the unique parametrization that allows HP transfer across width.
We emphasize that, to ensure transferability of any hyperparameter (such as learning rate), it’s not
sufficient to reparametrize only that hyperparameter, but rather, we need to identify and correctly
reparametrize all hyperparameters in Table 2. For example, in Fig. 1, the wide models in SP still
underperform their counterparts in µP, even with learning rate tuned optimally. This is precisely
because SP does not scale parameter multipliers and input/output layer learning rates correctly in
contrast to µP (see Table 3). See Appendix C for more intuition via a continuation of our example
here. We shall also explain this more concretely in the context of neural networks in Appendix B.
3 Hyperparameters Don’t Transfer Conventionally
In the community there seem to be conflicting assumptions about HP stability. A priori, models
of different sizes don’t have any reason to share the optimal HPs. Indeed, papers aiming for stateof-the-art results often tune them separately. On the other hand, a nontrivial fraction of papers in
deep learning fixes all HPs when comparing against baselines, which reflects an assumption that
the optimal HPs should be stable — not only among the same model of different sizes but also
among models of different designs — therefore, such comparisons are fair. Here, we demonstrate HP
instability across width explicitly in MLP and Transformers in the standard parametrization. We will
only look at training loss to exclude the effect of regularization.
MLP with Standard Parametrization We start with a 2-hidden-layer MLP with activation function φ, using the standard parametrization7 with LeCun initialization8
akin to the default in PyTorch:
f(ξ) = W3>φ(W2>φ(W1>ξ + b
1
) + b
2
)
with init. W1 ∼ N (0,
1/din), W{2,3} ∼ N (0,1/n), b{1,2} = 0,
(2)
14 12 10 8 6 4 2
log2LearningRate
0.0
0.5
1.0
1.5
2.0
Training Loss
SP / xent
width
256
512
1024
2048
4096
8192
14 12 10 8 6 4 2
log2LearningRate
0.0
0.5
1.0
1.5
2.0
P / xent
Figure 3: MLP width different hidden sizes trained
for 20 epoch on CIFAR-10 using SGD. Left uses standard parametrization (SP); right uses maximal update
parametrization (µP). µP networks exhibit better learning
rate stability than their SP counterparts.
where W1 ∈ R
din×n, b1 ∈ Rn,
W2 ∈ R
n×n, b2 ∈ Rn, W3 ∈
R
n×dout and din, n, and dout are
the input, hidden, and output dimensions. The particular MLP we use has
φ = ReLU and a cross-entropy (xent)
loss function. We define the width of
MLP as the hidden size n, which is
varied from 256 to 8192. The models are trained on CIFAR-10 for 20
epochs, which is more than enough to
ensure convergence.
As shown on the left in Fig. 3, the
optimal learning rate shifts by roughly
an order of magnitude as the width
increases from 256 to 8192; using the
optimal learning of the smallest model
on the largest model gives very bad performance, if not divergence.
Transformer with Standard Parametrization This perhaps unsurprising observation holds for
more complex architectures such as Transformer as well, as shown in Fig. 1 (left). We define width
6The more theoretically astute reader may observe that SP with a Θ(1/width) learning rate induces a
well-defined infinite-width limit exists as well. Nevertheless, this does not allow HP transfer because this limit is
in kernel regime as shown in [45]. See Appendix L.3 for more discussions.
7
i.e. the default parametrization offered by common deep learning frameworks. See Table 3 for a review.
8The key here is that the init. variance ∝ 1/fan_in, so the same insights here apply with e.g. He initialization.
4
https://openreview.net/pdf?id=Bx6qKuBM2AD
https://arxiv.org/abs/2203.03466
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
2022-03-07
Yang; Greg; Hu; Edward J; Babuschkin; Igor; Sidor; Szymon; Liu; Xiaodong; Farhi; David; Ryder; Nick; Pachocki; Jakub; Chen; Weizhu; Gao; Jianfeng
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/2203.03466">Download PDF</a></p><blockquote>
 Abstract: Hyperparameter (HP) tuning in deep learning is an expensive process,
prohibitively so for neural networks (NNs) with billions of parameters. We show
that, in the recently discovered Maximal Update Parametrization (muP), many
optimal HPs remain stable even as model size changes. This leads to a new HP
tuning paradigm we call muTransfer: parametrize the target model in muP, tune
the HP indirectly on a smaller model, and zero-shot transfer them to the
full-sized model, i.e., without directly tuning the latter at all. We verify
muTransfer on Transformer and ResNet. For example, 1) by transferring
pretraining HPs from a model of 13M parameters, we outperform published numbers
of BERT-large (350M parameters), with a total tuning cost equivalent to
pretraining BERT-large once; 2) by transferring from 40M parameters, we
outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7%
of total pretraining cost. A Pytorch implementation of our technique can be
found at <a href="http://github.com/microsoft/mup">this http URL</a> and installable via `pip install mup`.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Greg Yang [<a href="/show-email/8bfbd04e/2203.03466">view email</a>]
 <br />
 <strong><a href="/abs/2203.03466v1">[v1]</a></strong>
 Mon, 7 Mar 2022 15:37:35 UTC (2,817 KB)<br /><strong>[v2]</strong>
Mon, 28 Mar 2022 08:12:14 UTC (2,818 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2203.03466

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Machine Learning

 arXiv:2203.03466 (cs)
 [Submitted on 7 Mar 2022 (v1), last revised 28 Mar 2022 (this version, v2)]

 Title: Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

 Authors: Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, Jianfeng Gao
 Download PDF
 Abstract: Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify muTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at this http URL and installable via `pip install mup`.
 Comments: NeurIPS 2021  
 Subjects: Machine Learning (cs.LG) ; Disordered Systems and Neural Networks (cond-mat.dis-nn); Neural and Evolutionary Computing (cs.NE)
 Cite as: arXiv:2203.03466 [cs.LG]  
 (or arXiv:2203.03466v2 [cs.LG] for this version)  
 https://doi.org/10.48550/arXiv.2203.03466  
 Focus to learn more  
 arXiv-issued DOI via DataCite  
 

 Submission history

 From: Greg Yang [view email]
 [v1] Mon, 7 Mar 2022 15:37:35 UTC (2,817 KB)
 [v2] Mon, 28 Mar 2022 08:12:14 UTC (2,818 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.LG
 < prev | next >
 new | recent | 2203
 Change to browse by:
 cond-mat
 cond-mat.dis-nn
 cs
 cs.NE

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 IArxiv recommender toggle
 IArxiv Recommender (What is IArxiv?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Date Published: 2022-03-28
Authors:
Greg  Yang, gregyang@microsoft.com
Edward J Hu, edwardhu@microsoft.com
Igor  Babuschkin
Szymon  Sidor
Xiaodong  Liu
David  Farhi
Nick  Ryder
Jakub  Pachocki
Weizhu  Chen
Jianfeng  Gao
× × Microsoft
Corporation • Openai
Abstract
Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (µP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call µTransfer: parametrize the target model in µP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify µTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at github.com/microsoft/mup and installable via pip install mup.
Conventionally and in contrast with our technique, different widths do not share the same optimal hyperparameter; wider networks do not always perform better than narrower ones; in fact they underperform the same-width networks in our technique even after tuning learning rate (see dashed line). See Sections 3 and 4 for experimental setup.
Hyperparameter (HP) tuning is critical to deep learning. Poorly chosen HPs result in subpar performance and training instability. Many published baselines are hard to compare to one another due to varying degrees of HP tuning. These issues are exacerbated when training extremely large deep learning models, since stateof-the-art networks with billions of parameters become prohibitively expensive to tune.
Recently, [57] showed that different neural network parametrizations induce different infinitewidth limits and proposed the Maximal Update Parametrization (abbreviated µP) (summarized in Table 3 ) that enables "maximal" feature learning in the limit. Intuitively, it ensures that each layer is updated on the same order during training regardless of width. 2 In contrast, while the standard parametrization (SP) ensures activations are of unit order at initialization, it actually causes them to blow up in wide models during training [57] essentially due to an imbalance of per-layer learning rate (also see Fig. 5 ). We leverage µP to zero-shot transfer HPs from small models to large models in this work -that is, we obtain near optimal HPs on a large model without directly tuning it at all! While practitioners have always guessed HPs of large models from those of small models, the results are hit-or-miss at best because of incorrect parametrization. For example, as shown in Fig. 1 , in a Transformer, the optimal learning rate is stable with width in µP (right) but far from so in standard parametrization (left). In addition to width, we empirically verify that, with a few caveats, HPs can also be transferred across depth (in Section 6.1) as well as batch size, language model sequence length, and training time (in Appendix G.2.1). This reduces the tuning problem of an (arbitrarily) large model to that of a (fixed-sized) small model. Our overall procedure, which we call µTransfer, is summarized in Algorithm 1 and Fig. 2 , and the HPs we cover are summarized in Tables 1 and 2 . There are several benefits to our approach: 1. Better Performance: µTransfer is not just about predicting how the optimal learning rate scales in SP. In general, we expect the µTransferred model to outperform its SP counterpart with learning rate optimally tuned. For example, this is the case in Fig. 1 with the width-8192 Transformer. We discuss the reason for this in Section 5 and Appendix C. 2. Speedup: It provides massive speedup to the tuning of large models. For example, we are able to outperform published numbers of (350M) BERT-large [11] purely by zero-shot HP transfer, with tuning cost approximately equal to 1 BERT-large pretraining. Likewise, we outperform the published numbers of the 6.7B GPT-3 model [7] with tuning cost being only 7% of total pretraining cost. For models on this scale, HP tuning is not feasible at all without our approach. 3. Tune Once for Whole Family: For any fixed family of models with varying width and depth (such as the BERT family or the GPT-3 family), we only need to tune a single small model and can reuse its HPs for all models in the family. 3 For example, we will use this technique to tune BERT-base (110M parameters) and BERT-large (350M parameters) simultaneously by transferring from a 13M model. 4 . Better Compute Utilization: While large model training needs to be distributed across many GPUs, the small model tuning can happen on individual GPUs, greatly increasing the level of parallelism for tuning (and in the context of organizational compute clusters, better scheduling and utilization ratio). 5. Painless Transition from Exploration to Scaling Up: Often, researchers explore new ideas on small models but, when scaling up, find their HPs optimized during exploration work poorly on large models. µTransfer would solve this problem.
In addition to the HP stability property, we find that wider is better throughout training in µP, in contrast to SP (Section 8). This increases the reliability of model scaling in deep learning.
In this work, we primarily focus on hyperparameter transfer with respect to training loss. In settings where regularization is not the bottleneck to test performance, as in all of our experiments here, this also translates to efficacy in terms of test loss. In other settings, such as finetuning of models on small datasets, µTransfer may not be sufficient, as we discuss in Section 6.1. • We demonstrate it is possible to zero-shot transfer near optimal HPs to a large model from a small version via the Maximal Update Parametrization (µP) from [57] . • While [57] only covered SGD, here we derive µP for Adam as well (Table 3 ).
• We propose a new HP tuning technique, µTransfer, for large neural networks based on this observation that provides massive speedup over conventional methods and covers both SGD and Adam training; • We thoroughly verify our method on machine translation and large language model pretraining (in Section 7.3) as well as image classification (in Appendix G.1); • We release a PyTorch [35] package for implementing µTransfer painlessly. A sketch of this package is given in Appendix H.
Terminologies Sometimes, to be less ambiguous, we often refer to the "large model" as the target model, as it is the model we wish to ultimately tune, while we refer to the "small model" as the proxy model, as it proxies the HP tuning process. We follow standard notation d model , d head = d k , d v , n head , d f f n regarding dimensions in a Transformer; one can see Fig. 11 for a refresher.
Tensor Programs Series This paper is the 5th installment of the Tensor Programs series. While it is self-contained with the target audience being practitioners and empirical researchers, this paper presents the first major practical payoff of the theoretical foundation built in previous works [53] [54] [55] [56] [57] [58] .
2 Parametrization Matters: A Primer
In this section, we give a very basic primer on why the correct parametrization can allow HP transfer across width, but see Appendices J.1 to J.3 for more (mathematical) details.
The Central Limit Theorem (CLT) says that, if x 1 , . . . , x n are iid samples from a zero-mean, unitvariance distribution, then 1 √ n (x 1 + • • • + x n ) converges to a standard Gaussian N (0, 1) as n → ∞. Therefore, we can say that 1 √ n is the right order of scaling factor c n such that c n (x 1 + • • • + x n ) converges to something nontrivial. In contrast, if we set c n = 1/n, then c n (x
1 + • • • + x n ) → 0; or if c n = 1, then c n (x 1 + • • • + x n ) blows up in variance as n → ∞.
Now suppose we would like to minimize the function Then for sufficiently large n, the optimal α * n def = arg min α G n (α) should be close to α * N for any N > n, and indeed, for N = ∞ -this precisely means we can transfer the optimal c * n or α * n for a smaller problem (say F n ) to a larger problem (say F N ): G N is approximately minimized by α * n and F N is approximately minimized by c * n n/N . Because the transfer algorithm is simply copying α, we say the parametrization c = α/ √ n is the correct parametrization for this problem.
F n (c) def = E x1,...,xn f (c(x 1 + • • • + x n ))In the scenario studied in this paper, x 1 , . . . , x n are akin to randomly initialized parameters of a width-n neural network, c is akin to a HP such as learning rate, and f is the test-set performance of the network after training, so that F n gives its expectation over random initializations. Just as in this example, if we parametrize the learning rate and other HPs correctly, then we can directly copy the optimal HPs for a narrower network into a wide network and expect approximately optimal performance -this is the (zero-shot) hyperparameter transfer we propose here. It turns out the Maximal Update Parametrization (µP) introduced in [57] is correct (akin to the parametrization in α above), while the standard parametrization (SP) is incorrect (akin to the parametrization in c). We will review both parametrizations shortly. Theoretically, a µP network has a well-defined infinite-width limit -akin to (x 1 + • • • + x n )/ √ n having a N (0, 1) limit by CLT -while a SP network does not (the limit will blow up) [57] . 4 In fact, based on the theoretical foundation laid in [57] , we argue in Appendix J.3 that µP should also be the unique parametrization that allows HP transfer across width. For a more formal discussion of the terminologies parametrization and transfer, see Appendix A We emphasize that, to ensure transferability of any hyperparameter (such as learning rate), it's not sufficient to reparametrize only that hyperparameter, but rather, we need to identify and correctly reparametrize all hyperparameters in Table 2 . For example, in Fig. 1 , the wide models in SP still underperform their counterparts in µP, even with learning rate tuned optimally. This is precisely because SP does not scale parameter multipliers and input/output layer learning rates correctly in contrast to µP (see Table 3 ). See Appendix C for more intuition via a continuation of our example here. We shall also explain this more concretely in the context of neural networks in Section 5.
3 Hyperparameters Don't Transfer Conventionally
In the community there seem to be conflicting assumptions about HP stability. A priori, models of different sizes don't have any reason to share the optimal HPs. Indeed, papers aiming for stateof-the-art results often tune them separately. On the other hand, a nontrivial fraction of papers in deep learning fixes all HPs when comparing against baselines, which reflects an assumption that the optimal HPs should be stable -not only among the same model of different sizes but also among models of different designs -therefore, such comparisons are fair. Here, we demonstrate HP instability across width explicitly in MLP and Transformers in the standard parametrization. We will only look at training loss to exclude the effect of regularization.
MLP with Standard Parametrization
We start with a 2-hidden-layer MLP with activation function φ, using the standard parametrization 5 with LeCun initialization 6 akin to the default in PyTorch: where As shown on the left in Fig. 3 , the optimal learning rate shifts by roughly an order of magnitude as the width increases from 256 to 8192; using the optimal learning of the smallest model on the largest model gives very bad performance, if not divergence.
f (ξ) = W 3 φ(W 2 φ(W 1 ξ + b 1 ) + b 2 ) with init. W 1 ∼ N (0, 1 /din), W {2,3} ∼ N (0, 1 /n), b {1,2} = 0,W 1 ∈ R din×n , b 1 ∈ R n , W 2 ∈ R n×n , b 2 ∈ R n , W 3 ∈ R n×dout and
Transformer with Standard Parametrization This perhaps unsurprising observation holds for more complex architectures such as Transformer as well, as shown in Fig. 1 (left). We define width Table 3 : µP [57] and SP for General Neural Networks. Here, we emphasize the scaling with width (fan_in or fan_out); in practice, we may insert tunable multipliers in front of fan_in and fan_out as in Eq. ( 4). The fan_out of a bias vector is its dimension (whereas fan_in is 1). Purple text highlights key differences from standard parametrization (SP); Gray text recalls the corresponding SP. SGD (resp. Adam) here can be replaced by variants such as SGD with momentum (resp. Adagrad, etc); see Appendix B.3 for other optimizers. In general, the three columns here can be interpreted as linear layers that have {finite, infinite, infinite} input dimension and {infinite, finite, infinite} output dimension in an infinite-width network; this description generalizes more readily to other parameters such as those of layernorm. Transformer µP requires one more modification (1/d attention instead of 1/ √ d); see Definition 4.1. This version of µP gets rid of parameter multipliers; for the version similar to that in [57] , see Table 9 . Also see Table 8 for a µP formulation that is easier to implement (and compatible with input/output weight sharing). Further explanation of this table can be found in Appendix B. Its derivation can be found in Appendix J.
Input weights & all biases Output weights Hidden weights
Init. Var. 1 /fan_in 1 /fan_in 2 ( 1 /fan_in) 1 /fan_in SGD LR fan_out (1) 1 /fan_in (1) 1 Adam LR 1 1 /fan_in (1) 1 /fan_in (1)
as d model , with
d k = d q = d v = d model/n head and d f f n = 4d model .
The models are trained on wikitext-2 for 5 epochs. In Fig. 18 in the appendix we also show the instability of initialization scale and other HPs.
4 Unlocking Zero-Shot Hyperparameter Transfer with µP
We show that µP solves the problems we see in Section 3.
MLP with µP For the MLP in Section 3, to switch to µP, we just need to modify Eq. ( 2)'s initialization of the last layer and its learning rates of the first and last layer as well as of the biases. The basic form is 7initialize
W 1 ∼ N (0, 1 /din), W 2 ∼ N (0, 1 /n), W 3 ∼ N (0, 1 /n 2 ), b {1,2} = 0 with SGD learning rates η W 1 = η b 1 = η b 2 = ηn, η W 2 = η, η W 3 = ηn -1 .Here, η specifies the "master" learning rate, and we highlighted in purple the differences in the two parametrizations. This basic form makes clear the scaling with width n of the parametrization, but in practice we will often insert (possibly tune-able) multiplicative constants in front of each appearance of n. For example, this is useful when we would like to be consistent with a SP MLP at a base width n 0 . Then we may insert constants as follows: For
ñ def = n/n 0 , initialize W 1 ∼ N (0, 1 /din), W 2 ∼ N (0, 1 /n), W 3 ∼ N (0, 1 /n•ñ), b {1,2} = 0 with SGD learning rates η W 1 = η b 1 = η b 2 = ηñ, η W 2 = η, η W 3 = ηñ -1 .Then at width n = n 0 , all purple factors above are 1, and the parametrization is identical to SP (Eq. ( 2)) at width n 0 . Of course, as n increases from n 0 , then Eq. ( 4) quickly deviates from Eq. ( 2). In other words, for a particular n, µP and SP can be identical up to the choice of some constants (in this case n 0 ), but µP determines a different "set" of networks and optimization trajectory than SP as one varies n. As we will see empirically in the next section, this deviation is crucial for HP transfer.
Indeed, in Fig. 3 (right), we plot the CIFAR10 performances, over various learning rates and widths, of µP MLPs with n 0 = 128. In contrast to SP, the optimal learning rate under µP is stable. This means that, the best learning rate for a width-128 network is also best for a width-8192 network in µP -i.e. HP transfer works -but not for SP. In addition, we observe performance for a fixed learning rate always weakly improves with width in µP , but not in SP.
This MLP µP example can be generalized easily to general neural networks trained under SGD or Adam, as summarized in Table 3 , which is derived in Appendix J. When not specified in the legend, the width used is 256, depth 2, batch size 20, sequence length 256, and LR schedule constant. We sweep a particular HP, corresponding to each column, while fixing all others constant. See Section 6.1 for discussion of these results.
Transformers with µP We repeat the experiments with base width n 0 = 128 for Transformers: Definition 4.1. The Maximal Update Parametrization (µP) for a Transformer is given by Table 3 and 1/d attention instead of 1/ √ d, i.e. the attention logit is calculated as q k/d instead of q k/ √ d where query q and key k have dimension d. 8The results are shown on the right in Fig. 1 , where the optimal learning rate is stable, and the performance improves monotonically with width. See Appendix B for further explanation of µP.
5 The Defects of SP and How µP Fixes Them
The question of SP vs µP has already been studied at length in [57] . Here we aim to recapitulate the key insights, with more explanations given in Appendix J.3.
An Instructive Example
As shown in [57] and Appendix J.3, in SP, the network output will blow up with width after 1 step of SGD. It's instructive to consider a 1-hidden-layer linear perceptron f (x) = V U x with scalar inputs and outputs, as well as weights
V, U ∈ R n×1 . In SP, V α ∼ N (0, 1/n) ad U α ∼ N (0, 1) for each α ∈ [n]
. This sampling ensures that f (x) = Θ(|x|) at initialization. After 1 step of SGD with learning rate 1, the new weights are V ← V + θU, U ← U + θV , where θ is some scalar of size Θ(1) depending on the inputs, labels, and loss function. But now
f (x) = V U x = (V U + θU U + θV V + θ 2 U V )xblows up with width n because U U = Θ(n) by Law of Large Numbers. Now consider the same network in µP. According to Table 3 , we now have V α ∼ N (0, 1/n 2 ) in contrast to SP, but U α ∼ N (0, 1) as before, with learning rates η V = 1/n, η U = n. After 1 step of SGD, we now have and one can verify this is Θ(1) and thus does not blow up with width. 9Some Layers Update Too Fast, Others Too Slow One can observe the same behavior in more advanced architectures like Transformers and optimizers like Adam; in fact, in SP, other hidden quantities like attention logits will also blow up with width after 1 step, but in µP still remain bounded, as shown in Fig. 5 (middle).
f (x) = (V U + θn -1 U U + θnV V + θ 2 U V )x,
One might think scaling down the learning rate with width can solve this problem in SP. However, other hidden activations like the word embedding (Fig. 5 (right)) in a Transformer update by a widthindependent amount for each step of training, so scaling down the learning rate will effectively mean the word embeddings are not learned in large width models. Similar conclusions apply to other models like ResNet (in fact, one can observe in the SP linear MLP example above, the input layer is updated much more slowly than the output layer). On the other hand, µP is designed so that all hidden activations update with the same speed in terms of width (see Appendix J.2 for why).
Performance Advantage of µP This is why a wide model tuned with µTransfer should in general outperform its SP counterpart with (global) learning rate tuned. For example, this is the case for the width-8192 Transformer in Fig. 1 , where, in SP, the optimal learning rate needs to mollify the blow-up in quantities like logits and attention logits, but this implies others like word embeddings do not learn appreciably. This performance advantage means µTransfer does more than just predicting the optimal learning rate of wide SP models. Relatedly, we observe, for any fixed HP combination, training performance never decreases with width in µP, in contrast to SP (e.g., the µP curves in Figs. 1, 3 and 16 do not cross, but the SP curves do; see also Section 8).
6 Which Hyperparameters Can Be µTransferred?
In this section, we explore how common HPs fit into our framework. In general, they can be divided into three kinds, summarized in Table 1: 1. those that can transfer from the small to the large model, such as learning rate (Table 2 ); 2. those that primarily control regularization and don't work well with our technique; and 3. those that define training scale, such as width as discussed above as well as others like depth and batch size, across which we transfer other HPs.
Those in the first category transfer across width, as theoretically justified above in Section 2. To push the practicality and generality of our technique, we empirically explore the transfer across the other dimensions in the third category. Note that µTransfer across width is quite general, e.g. it allows varying width ratio of different layers or number of attention heads in a Transformer; see Appendix E.2. This will be very useful in practice. For the second category, the amount of regularization (for the purpose of controlling overfitting) naturally depends on both the model size and data size, so we should not expect transfer to work if the parametrization only depends on model size. We discuss these HPs in more detail in Appendix E.1.
6.1 Empirical Validation and Limitations
Our empirical investigations focus on Transformers (here) and ResNet (in Appendix G.1.1), the most popular backbones of deep learning models today. We train a 2-layer pre-layernorm µP 10Transformer with 4 attention heads on Wikitext-2. We sweep one of four HPs (learning rate, output weight multiplier, initialization standard deviation, and learning rate schedule) while fixing the others and sweeping along width and depth (with additional results in Fig. 19 on transfer across batch size, sequence length, and training time). Fig. 4 shows the results averaged over 5 random seeds.
Empirically, we find that for language modeling on Transformers, HPs generally transfer across scale dimensions if some minimum width (e.g. 256), depth (e.g., 4), batch size (e.g., 32), sequence length (e.g., 128), and training steps (e.g., 5000) are met, and the target scale is within the "reasonable range" as in our experiments. Now, there are some caveats. While the exact optimum can shift slightly with increasing scale, this shift usually has very small impact on the loss, compared to SP (Figs. 1 and 3(left)). However, there are some caveats. For example, the best initialization standard deviation does not seem to transfer well across depth (2nd row, 3rd column), despite having a stabler optimum across width. In addition, while our results on width, batch size, sequence length, and training time still hold for post-layernorm (Fig. 17 ), 11 the transfer across depth only works for pre-layernorm Transformer. Nevertheless, in practice (e.g. our results in Section 7.3) we find that fixing initialization standard deviation while tuning other HPs works well when transferring across depth.
7 Efficiency and Performance of µTransfer
Now that the plausibility of µTransfer has been established in toy settings, we turn to more realistic scenarios to see if one can achieve tangible gains. Specifically, we perform HP tuning only on a smaller proxy model, test the obtained HPs on the large target model directly, and compare against baselines tuned using the target model. We seek to answer the question: Can µTransfer make HP tuning more efficient while achieving performance on par with traditional tuning? As we shall see by the end of the section, the answer is positive. We focus on Transformers here, while experiments on ResNets on CIFAR10 and Imagenet can be found as well in Appendix G.1. All of our experiments are run on V100 GPUs.
7.1 Transformer on IWSLT14 De-En
Setup IWSLT14 De-En is a well-known machine translation benchmark. We use the default IWSLT (post-layernorm) Transformer implemented in fairseq [33] with 40M parameters, which we denote as the 1x model. 12 For µTransfer, we tune on a 0.25x model with 1/4 of the width, amounting to 4M parameters. For this experiment, we tune via random search the learning rate η, the output layer parameter multiplier α output , and the attention key-projection weight multiplier α attn . See the grid and other experimental details in Appendix F.1.
We compare transferring from the 0.25x model with tuning the 1x model while controlling the total tuning budget in FLOPs. 13 To improve the reproducibility of our result: 1) we repeat the entire HP search process (a trial) 25 times for each setup, with number of samples as indicated in Table 4 , and report the 25th, 50th, 75th, and 100th percentiles in BLEU score; 2) we evaluate each selected HP combination using 5 random initializations and report the mean performance. We pick the HP combination that achieves the lowest validation loss 15 for each trial. The reported best outcome is chosen according to the validation loss during tuning. We compare against the default in fairseq, which is presumably heavily tuned. The result is shown in Table 4 . 
Sample Quality of Proxy Model vs Target
Model The Pareto frontier in Fig. 6 (right) suggests that, given a fixed number of random samples from the HP space, 1) tuning the target model directly yields slightly better results than tuning the proxy model (while taking much more compute of course), but 2) this performance gap seems to vanish as more samples are taken. This can be explained by the intuition that the narrower proxy model is a "noisy estimator" of the wide target model [57] .With few samples, this noise can distort the random HP search, but with more samples, this noise is suppressed.
7.2 Transformer on WMT14 En-De
We scale up to WMT14 En-De using the large (post-layernorm) Transformer from [50] with 211M parameters. We tune on a proxy model with 15M parameters by shrinking d model , d f f n , and n head .
For this experiment, we tune via random search the learning rate η, the output layer parameter multiplier α output , and the attention key-projection weight multiplier α attn following the grid in Appendix F.2. The result is shown in Table 5 : While random search with 3 HP samples far underperforms the fairseq default, we are able to match it via transfer using the same tuning budget.
7.3 BERT
Finally, we consider large-scale language model pretraining where HP tuning is known to be challenging. Using Megatron (pre-layernorm) BERT [43] as a baseline, we hope to recover the performance of the published HPs by only tuning a proxy model that has roughly 13M parameters, which we call BERT-prototype. While previous experiments scaled only width, here we will also scale depth, as discussed in Section 6 and validated in Fig. 4 . We use a batch size of 256 for all runs and follow the During HP tuning, we sample 256 combinations from the search space and train each combination on BERT-prototype for 10 5 steps. The total tuning cost measured in FLOPs is roughly the same as training 1 BERT-large for the full 10 6 steps; the exact calculation is shown in Appendix F.3. The results are shown in Table 6 . Notice that on BERT-large, we obtain sizeable improvement over the well-tuned Megatron BERT-large baseline. 
7.4 GPT-3
In order to further verify µTransfer at scale, we applied it to GPT-3 6.7B [7] with relative attention. This target model consists of 32 residual blocks with width 4096. We form the small proxy model by shrinking width to 256, resulting in roughly 40 million trainable parameters, 168 times smaller than the target model. HPs were then determined by a random search on the proxy model. The total tuning cost was only 7% of total pretraining cost. Details of the HP sweep can be found in Appendix F. 4 .
In order to exclude code difference as a possible confounder, we also re-trained GPT-3 6.7B from scratch using the original HPs from [7] . Unfortunately, after we have finished all experiments, we found this baseline mistakenly used absolute attention (like models in [7] ) when it was supposed to use relative attention like the target model. In addition, during training of the µTransfer model we encountered numerical issues that lead to frequent divergences. In order to avoid them, the model was trained using FP32 precision, even though the original 6.7B model and our re-run were trained using FP16. 16 17 The resulting µTransfer model outperforms the 6.7B from [7] , and is in fact comparable to the twice-as-large 13B model across our evaluation suite (see Table 11 ). Selected evaluation results can be found in Table 7 and further details are given in Table 10 and Appendix F.4. 16 While we are mainly focused on the efficacy of µTransfer regardless of precision, it would be interesting to ablate the effect of precision in our results, but we did not have enough resources to rerun the baseline in FP32 17 It is quite interesting that µTransfer identified a useful region of hyperparameters leading to much improved performance, which probably would be difficult to discover normally because 1) researchers usually change hyperparameters to accomodate precision and 2) there was no precise enough justification to go against this judgment until µTransfer.
Table 7 : GPT-3 6.7B Pretraining. Selected evaluation results for the GPT-3 6.7B model tuned with µTransfer (transfered from a small proxy model of 40M parameters), compared to the results published in [7] and a re-run with original HPs, as well as the 13B model in [7] for reference. Note that the perplexities in this table are based on a custom tokenization and are not comparable to the literature. The validation loss refers to the loss achieved on a random held-out part of our dataset. Zero-shot, One-Shot and Few-Shot refer to the number of additional query and answer pairs passed in the context when performing the sampling-based evaluations. See Appendix F.4 for full evaluation. Task Metric 6.7B+µP 6.7B re-run 6.7B [7] 13B [7] Validation loss cross-entropy Here we trained a GPT-3 transformer with 4 layers and widths from 256 to 32,768. Modulo a brief period around 1e8 training tokens, wider is better throughout training.
In earlier plots like Figs. 1 and 3 , we saw that at the end of training, wider is always better in µP but not in SP. In fact, we find this to be true throughout training, as seen in Fig. 7 , modulo noise from random initialization and/or data ordering, and assuming the output layer is zero-initialized (which has no impact on performance as discussed in Appendix D.2). We then stress-tested this on a µP GPT-3 Transformer (on the GPT-3 training data) by scaling width from 256 to 32,768 using a fixed set of HPs (Fig. 8 ). Wider models consistently match or outperform narrower models at each point in training (except a brief period around 1e8 training tokens, likely due to noise because we ran only 1 seed due to computational cost). Our observation suggests that wider models are strictly more data-efficient if scaled appropriately. By checking "wider-is-better" early in training, one can also cheaply debug a µP implementation.
9 Useful Hyperparameter Transfer: A Theoretical Puzzle
We want to tune HPs on a small model with width N such that its HP landscape looks like that of a large model with width N . Our intuition in Section 2 and Appendices C and J leads us to µP. However, for this to be useful, we do not want the small model (as a function) after training to be close to that of the large model -otherwise there is no point in training the large model to begin with. So N 1) must be large enough so that the HP optimum converges, but 2) cannot be so large that the functional dynamics (and the loss) converges. The fact that such N exists, as demonstrated by our experiments, shows that: In some sense, the HP optimum is a "macroscopic" or "coarse" variable which converges quickly with width, while the neural network function (and its loss) is a very "microscopic" or "fine" detail that converges much more slowly with width. However, theoretically, it is unclear why this should happen, and where else we should expect such useful HP transfer. We leave an explanation to future work.
10 Related Works
10.1 Hyperparameter Tuning
Many have sought to speedup HP tuning beyond the simple grid or random search. Snoek et al. [45] treated HP tuning as an optimization process and used Bayesian optimization by treating the performance of each HP combination as a sample from a Gaussian process (GP). Snoek et al. [46] further improved the runtime by swapping the GP with a neural network. Another thread of work investigated how massively parallel infrasture can be used for efficient tuning under the multi-arm bandit problem [18, 22] . There are also dedicated tools such as Optuna [4] and Talos [3] which integrate with existing deep learning frameworks and provide an easy way to apply more advanced tuning techniques.
Our approach is distinct from all of the above in that it does not work on the HP optimization process itself. Instead, it decouples the size of the target model from the tuning cost, which was not feasible prior to this work. This means that no matter how large the target model is, we can always use a fixed-sized proxy model to probe its HP landscape. Nevertheless, our method is complementary, as the above approaches can naturally be applied to the tuning of the proxy model; it is only for scientific reasons that we use either grid search or random search throughout this work.
10.2 Hyperparameter Transfer
Many previous works explored transfer learning of HP tuning (e.g. [15, 36, 47, 62] ). However, to the best of our knowledge, our work is the first to explore zero-shot HP transfer. In addition, we focus on transferring across model scale rather than between different tasks or datasets. Some algorithms like Hyperband [23] can leverage cheap estimates of HP evaluations (like using a small model to proxy a large model) but they are not zero-shot algorithms, so would still be very expensive to apply to large model training. Nevertheless, all of the above methods are complementary to ours as they can be applied to the tuning of our proxy model.
10.3 Previously Proposed Scaling Rules of Hyperparameters
(Learning Rate, Batch Size) Scaling [44] proposed to scale learning rate with batch size while fixing the total epochs of training; [14] proposed to scale learning rate as √ batchsize while fixing the total number of steps of training. However, [41] showed that there's no consistent (learning rate, batch size) scaling law across a range of dataset and models. Later, [30] studied the trade-off of training steps vs computation as a result of changing batch size. They proposed an equation of a/(1 + b/batchsize), where a and b are task-and model-specific constants, for the optimal learning rate (see their fig 3  and fig 5 ). This law suggests that for sufficiently large batch size, the optimal learning rate is roughly constant. 18 This supports our results here as well as the empirical results in [41, fig 8] .
Learning Rate Scaling with Width Assuming that the optimal learning rate should scale with batch size following [44] , [34] empirically investigated how the optimal "noise ratio" LR/batchsize scales with width for MLP and CNNs in NTK parametrization (NTP) or standard parametrization (SP) trained with SGD. They in particular focus on test loss in the regime of small batch size and training to convergence. In this regime, they claimed that in networks without batch normalization, the optimal noise ratio is constant in SP but scales like 1/width for NTP. However, they found this law breaks down for networks with normalization.
In contrast, here we focus on training loss, without training to convergence and with a range of batch sizes from small to very large (as is typical in large scale pretraining). Additionally, our work applies universally to 1) networks with normalization, along with 2) Adam and other adaptive optimizers; furthermore 3) we empirically validate transfer across depth and sequence length, and 4) explicitly validate tuning via µTransfer on large models like BERT-large and GPT-3.
Finally, as argued in [57] and Appendix J.3, SP and NTP lead to bad infinite-width limits in contrast to µP and hence are suboptimal for wide neural networks. For example, sufficiently wide neural networks in SP and NTP would lose the ability to learn features, as concretely demonstrated on word2vec in [57] .
Input Layer Parametrization The original formulation of µP in [57] (see Table 9 , which is equivalent to Table 3 ) uses a fan-out initialization for the input layer. This is atypical in vision models, but in language models where the input and output layers are shared (corresponding to word embeddings), it can actually be more natural to use a fan-out initialization (corresponding to fan-in initialization of the output layer). In fact, we found that fairseq [33] by default actually implements both the fan-out initialization and the √ fan_out multiplier.
Other Scaling Rules Many previous works proposed different initialization or parametrizations with favorable properties, such as better stability for training deep neural networks [5, 13, 16, 26, 40, 59, 60, 66] . Our work differs from these in that we focus on the transferability of optimal HPs from small models to large models in the same parametrization.
10.4 Infinite-Width Neural Networks: From Theory to Practice and Back [57] introduced µP as the unique parametrization that enables all layers of a neural network to learn features in the infinite-width limit, especially in contrast to the NTK parametrization [17] (which gives rise to the NTK limit) that does not learn features in the limit. Based on this theoretical insight, in Appendix J.3, we argue that µP should also be the unique parametrization (in the sense of [57] ) that allows HP transfer across width; in short this is because it both 1) preserves feature learning, so that performance on feature learning tasks (such as language model pretraining) does not become trivial in the limit, and 2) ensures each parameter tensor is not stuck at initialization in the large width limit, so that its learning rate does not become meaningless. At the same time, our results here suggest that µP is indeed the correct parametrization for wide neural networks and thus provide empirical motivation for the theoretical study of the infinite-width µP limit. Note, parametrization here refers to a rule to scale hyperparameters with width ("how should my initialization and learning rate change when my width doubles?"), which is coarser than a prescription for setting hyperparameters at any particular width ("how should I set my initialization and learning rate at width 1024?").
11 Conclusion
Leveraging the discovery of a feature learning neural network infinite-width limit, we hypothesized and verified that the HP landscape across NNs of different width is reasonably stable if parametrized according to Maximal Update Parametrization (µP). We further empirically showed that it's possible to transfer across depth, batch size, sequence length, and training time, with a few caveats. This allowed us to indirectly tune a very large network by tuning its smaller counterparts and transferring the HPs to the full model. Our results raise an interesting new theoretical question of how useful HP transfer is possible in neural networks in the first place.
Venues of Improvement Nevertheless, our method has plenty of room to improve. For example, initialization does not transfer well across depth, and depth transfer generally still does not work for post-layernorm Transformers. This begs the question whether a more principled parametrization in depth could solve these problems. Additionally, Fig. 4 shows that the optimal HP still shifts slightly for smaller models. Perhaps by considering finite-width corrections to µP one can fix this shift. Finally, it will be interesting to study if there's a way to transfer regularization HPs as a function of both the model size and data size, especially in the context of finetuning of pretrained models. 
A Parametrization Terminologies
This section seeks to make formal and clarify some of the notions regarding parametrization discussed informally in the main text. Definition A.1 (Multiplier and Parameter Multiplier). In a neural network, one may insert a "multiply by c" operation anywhere, where c is a non-learnable scalar hyperparameter. If c = 1, then this operation is a no-op. This c is called a multiplier.
Relatedly, for any parameter tensor W in a neural network, we may replace W with cW for some non-learnable scalar hyperparameter c. When c = 1, we recover the original formulation. This c is referred to as a parameter multiplier.
For example, in the attention logit calculation k, q / √ d head where q = W x, the 1/ √ d head factor is a multiplier. It may also be thought of as the parameter multiplier of W if we rewrite the attention logit as k, (W/ √ d head )x .
Note parameter multipliers cannot be absorbed into the initialization in general, since they affect backpropagation. Nevertheless, after training is done, parameter multipliers can always be absorbed into the weight. Definition A.2 (Parametrization). In this work, a parametrization is a rule for how to change hyperparameters when the widths of a neural network change, but note that it does not necessarily prescribes how to set the hyperparameters for any specific width. In particular, for any neural network, an abc-parametrization is a rule for how to scale a) the parameter multiplier, b) the initialization, and c) the learning rate individually for each parameter tensor as the widths of the network change, as well as any other multiplier in the network; all other hyperparameters are kept fixed with width.
For example, SP and µP are both abc-parametrizations. Again, we note that, in this sense, a parametrization does not prescribe, for example, that the initialization variance be 1/fan_in, but rather that it be halved when fan_in doubles. Definition A. Throughout this paper, we take L to be the training loss, but because regularization is not the bottleneck in our experiments (especially large scale pretraining with BERT and GPT-3), we nevertheless see high quality test performance in all of our results. We also remark that empirically, using training loss as the metric can be more robust to random seed compared to validation loss and especially BLEU score. See Table 1 (left) for H. By our arguments in Appendix J.3 and our empirical results, µP is the unique abc-parametrization admitting zero-shot transfer for such H and L in this sense.
More generally, one may define a K-shot transfer algorithm of a set of hyperparameters H w.r.t. a metric L as one that 1) takes width values n and n and an approximately optimal combination of values of H w.r.t. L at a width n and 2) returns an approximately optimal combination of values of H w.r.t. L at width n , given 3) a budget of K evaluations of candidate hyperparameter combinations on models of width n . However, we will have no use for this definition in this paper.
B Further Explanations of the µP Tables
In addition to Table 3 , we provide Table 8 as an equivalent µP formulation that is easier to implement, as well as Table 9 for those more familiar with the original µP formulation in [57] . Below, we provide some commentary on corner cases not well specified by the tables. Ultimately, by understanding Appendix J, one can derive µP for any architecture, new or old.
Matrix-Like, Vector-Like, Scalar-Like Parameters We can classify any dimension in a neural network as "infinite" if it scales with width, or "finite" otherwise. For example, in a Transformer, d model , d f f n , d head , n head are all infinite, but vocab size and context size are finite. Then we can categorize parameter tensors by how many infinite dimensions they have. If there are two such dimensions, then we say the parameter is matrix-like; if there is only one, then we say it is vector-like; if there is none, we say it is scalar-like. Then in Tables 3, 8 and 9, "input weights & all biases" and "output weights" are all vector-like parameters, while hidden weights are matrix-like parameters. An Table 8 : Alternative (Equivalent) µP Formulation for Easier Implementation. Same format as in Table 3 . In contrast to the formulation in Table 3 , here all "vector-like" parameters (i.e. those that have only one dimension tending to infinity), including input and output weights and biases, have the same width scaling for initialization variance and SGD/Adam LR (note the 1/fan_in for input weight/bias init. var. is Θ(1) in width). This has two benefits in practice: 1) implementation is unified and simplified for all "vector-like" parameters; 2) input and output weights can now be tied, in contrast to 
/ √ fan_in (1) 1 SGD LR 1 1 1 Adam LR 1 / √ fan_out√ fan_in
(1)
1 /fan_in (1)
advantage of Table 8 is that it gives a uniform scaling rule of initialization and learning rate for all vector-like parameters. The multiplier rule in Table 8 can be more interpreted more generally as the following: a multiplier of order 1/fan_in should accompany any weight that maps an infinite dimension to a finite one. This interpretation then nicely covers both the output logits and the attention logits (i.e. 1/d attention).
Scalar-like parameters are not as common as matrix-like and vector-like ones, but we will mention a few examples in Appendix B.2. The scaling rule for their initialization, learning rate (for both SGD and Adam), and multiplier is very simple: hold them constant with width.
Initialization Mean We did not specify the initialization mean in the tables, since most commonly the mean is just set to 0, but it can be nonzero for vector-like parameters (e.g., layernorm weights) and scalar-like parameters but must be 0 for matrix-like parameters.
Zero Initialization Variance
The initialization scaling rules in our tables can all be trivially satisfied if the initialization variance is set to 0. This can be useful in some settings (e.g., Appendix D.2) but detrimental in other settings (e.g., hidden weights).
What Are Considered Input Weights? Output Weights? Here, input weights very specifically refer to weights that map from an infinite dimension to a finite dimension. As a counterexample, in some architectures, the first layer can actually map from a finite dimension to another finite dimension, e.g., a PCA layer. Then this is not an "input weight"; if the next layer maps into an infinite dimension, then that's the input weight. A similar, symmetric discussion applies to output weights.
What Counts As a "Model"? Does the MLP in a Transformer Count As a "Model"? For our tables, a model is specifically a function that maps a finite dimension to another finite dimension, consistent with the discussion above. For example, for an image model on CIFAR10, it maps from 3 × 32 × 32 = 3072 dimensions to 10 dimensions, and these numbers are fixed regardless of the width of the model. Likewise, for an autoregressive Transformer model, the input and output dimension are both the vocab size, which is independent of the width. In contrast, an MLP inside a Transformer is not a "model" in this sense because its input and output dimension are both equal to the width of the Transformer.
B.1 Walkthrough of µP Implementation in a Transformer
To ground the abstract description in Tables 3, 8 and 9, we walk through the parameters of a typical Transformer and discuss concretely how to parametrize each.
We assume that the user wants to replicate SP when the model widths are equal to some base widths, for example, when Below, we introduce hyperparameters σ • , η • for each parameter tensor, as well as a few multipliers α • . One may always tie σ • (resp. η • ) across all parameter tensors, but in our experiments, we found it beneficial to at least distinguish the input and output layer initialization and learning rates.
d model = d model,0 = 128, d f f n = d f f n,0 = 512,
Input Word Embeddings
The input word embedding matrix W wordemb has size d model × vocabsize, where vocabsize is the fan-in and d model is the fan-out. Follow the "input weight & all biases" column in Tables 3, 8 and 9. For example, for Tables 3 and 8 ,
W wordemb ∼ N (0, σwordemb ), with Adam LR η wordemb Note here, because fan-in (vocabsize) here is independent of width (d model ), the "1/fan_in" for the initialization variance in these tables is equivalent to "1", i.e. the initialization variance can be anything fixed with width. In this case of the word embedding, setting the variance to 1, for example, is more natural than setting the variance to 1/fan_in, because the embedding is one-hot (1/fan_in would be more natural for image inputs).
Positional Embeddings The (absolute or relative) positional embedding matrix W posemb has size d model × contextsize, where contextsize is the fan-in and d model is the fan-out. With the same discussion as above for input word embeddings, follow the "input weight & all biases" column in Tables 3, 8 and 9. For example, for Tables 3 and 8 ,
W posemb ∼ N (0, σ 2 posemb ), with Adam LR η posemb
Layernorm Weights and Biases Layernorm weights w LN and biases b LN both have shape d model and can be thought of "input weights" to the scalar input of 1. Hence one should follow the "input weight & all biases" column in Tables 3, 8 and 9. In particular, the usual initialization of layernorm weights as all 1s and biases as all 0s suffice (where the initialization variance is 0). For example, for Tables 3 and 8 , Self-Attention There are 4 matrices, W q , W k ∈ R (d k n head )×d model , W v ∈ R (dvn head )×d model , and W o ∈ R d model ×(dvn head ) (where the shapes are R fan_out×fan_in ). Since d model , (d k n head ), and (d v n head ) all scale with width (where the latter two are commonly just set to d model ), all 4 matrices should be parametrized according to the "hidden weights" column in Tables 3, 8 and 9. For example, for Tables 3 and 8 ,
w LN ← 1,
W q ∼ N (0, σ 2 q /d model ), with Adam LR η q / dmodel W k ∼ N (0, σ 2 k /d model ), with Adam LR η k / dmodel W v ∼ N (0, σ 2 v /d model ), with Adam LR η v / dmodel W o ∼ N (0, σ 2 o /(d v n head )),
with Adam LR η o /( dv ñhead ).
Attention Logit Scaling
We use 1/d instead of 1/ √ d attention. To be compatible with 1/ √ d attention when at a particular base d head = d head,0 , we set
AttnLogit = α attn d head,0 d head q k,
where α attn is a tunable multiplier.
MLP There are 2 matrices, W 1 ∈ R d f f n ×d model , W 2 ∈ R d model ×d f f n (where the shapes are R fan_out×fan_in ), where d f f n is commonly set to 4d model . Since both d model , d f f n scale with width, both matrices are considered "hidden weights." For example, for Tables 3 and 8 ,
W 1 ∼ N (0, σ 2 q /d model ), with Adam LR η q / dmodel W 2 ∼ N (0, σ 2 k /d f f n ), with Adam LR η k / dffn
Word Unembeddings Symmetric to the discussion on input word embeddings, the output word unembeddings should be parametrized according to the "output weights" column of Tables 3, 8 and 9 .
Often, the unembeddings are tied with the embeddings, and Tables 8 and 9 allow for this as their initialization schemes are symmetric between input and output weights.
For example, for Table 3 , we'd set
W unemb ∼ N (0, σ 2 unemb /(d model dmodel )), with Adam LR η unemb / dmodel .
For Table 8 , we would instead have
W unemb ∼ N (0, σ 2 unemb /d model,0
), with Adam LR η unemb , (note d model,0 here is the base width and therefore is a constant) and the output is computed as
logits = α output dmodel W unemb z
where z is the final layer embedding of a token, and α output is a tunable multiplier.
B.2 Other Parameters
Learnable scalar multipliers For learnable scalar multipliers (e.g., softmax inverse temperature), one can initialize them to 1 and use a constant (in width) learning rate for both SGD and Adam. This is compatible with Tables 3, 8 and 9.
Positional Bias Some Transformers use positional bias (of size contextsize × contextsize, which are added to the attention logits). They are considered "scalar-like" in that it has no width dimension. One can initialize them to 0 and use a constant (in width) learning rate for both SGD and Adam. This is compatible with Tables 3, 8 and 9.
Spatial MLPs Recent works [12, 24, 31, 48, 49] on MLP-only architectures in NLP and CV replace the self-attention layer in Transformers with MLPs across tokens or spatial locations. In our language here, such MLPs have finite input and output dimensions (the context size) and infinite hidden dimensions, so their input, output, and hidden weights should be parametrized via the corresponding columns in Tables 3, 8 and 9.
B.3 Optimizer Variants and Hyperparameters
AdamW Exactly the same as Adam in all of our tables, with the added benefit that weight decay is automatically scaled correctly in AdamW (but is incompatible with µP Adam). For this reason, we recommend using AdamW when weight decay is desired (which is consistent with current standard practice).
Frobenius Normalization LARS [63] , Adafactor [42] , Lamb [64] , Layca [8] , Fromage [6] , Nero [28] all involve a normalization step in which the update g (which may be obtained from SGD, Adam, or other optimzers) is normalized to have Frobenius norm equal to that of the parameter w: g ← w F g F g. They can be made compatible with µP in Table 8 by scaling their learning rate for hidden weights like 1/ √ fan_in (for Table 3 , the output weight learning rate should be likewise scaled). The intuitive reasoning (which can be formalized straightforwardly using Tensor Programs) is as follows.
This normalization implicitly encodes a width scaling: If one initializes a weight matrix with variance 1/fan_in, then an n × n matrix (e.g., a hidden weight matrix) has Frobenius norm √ n at initialization. Thus, in the first step and, by induction, in any step t, the normalized update to this n × n weight also has Frobenius norm Θ( √ n) (for any fixed t, as n → ∞). Heuristically, this means each entry of g is approximately of size Θ(1/ √ n). But, by the derivation of Appendix J, we want Θ(1/n) and this is Θ( √ n) too large! Thus, in wide enough networks, one should see a network blowup after one update, like demonstrated in Fig. 5 .
However, note that the Θ(1/ √ n) coordinate size induced by the normalization here is closer to the right size Θ(1/n) than Adam, whose update have coordinate size Θ(1). This may partially explain the apparent benefit of these optimizers. In particular, this may explain the observation that T5 [38] , using Adafactor, was able to train its entire range of models from 220 million to 11 billion parameters with a fixed set of hyperparameters, while GPT-3 [7] , using Adam, needed to decrease its learning rate with model size.
RAdam RAdam [25] is a variant of Adam that uses SGD with momentum in an initial stage with learning rate warmup, followed by a second stage of Adam with a particular setting of learning rate with time. Thus, one can adapt RAdam to µP by individually scaling the learning rates of the initial SGD stage and the final Adam stage according to Table 3 , Table 8 , or Table 9 .
Adagrad and RMSProp Exactly the same as Adam in all of our tables.
in Adam and Its Variants All of our derivations here assume is negligible in Adam. If it is set to a non-negligible number, then it needs to be scaled, for all parameters, like 1/fan_in 2 if it is added before the square root, or like 1/fan_in if it is added after the square root.
Gradient Clipping Gradient ( 2 -norm-wise) clipping is compatible with Table 3 (as well as Tables 8 and 9 ), for either SGD or Adam, if the clip value is held constant with respect to width.
Weight Decay Weight decay should be scaled independently of width in SGD and AdamW, for all of our tables. However, note it's not compatible with µP Adam.
Momentum Momentum should be scaled independently of width for all of our tables.
C Parametrization Matters: A Primer for Multiple Hyperparameters
Here we give more intuition why we need to reparametrize all hyperparameters. In practice, neural networks have multitudes of hyperparameters all interacting together. In our example of Section 2, hyperparameter optimization would be akin to minimizing the function19 
F n (c 1 , . . . , c k ) def = E x1,...,xn f ((c 1 + • • • + c k )(x 1 + • • • + x n )).
where x 1 , . . . , x n are as in Eq. ( 1) and c 1 , . . . , c k are analogous to k hyperparameters. For the same reasoning in Section 2, the correct parametrization is in (α 1 , . . . , α k ) where
α i = c i √ n.
While this is straightforward, in practice, researchers often fix some hyperparameters (e.g., they tune only learning rate but neglects to scale parameter multipliers or initialization correctly). For example, if we only partially reparametrize and optimize in α 1 while fixing c 2 , . . . , c k , then the optimal α 1 is
(α 1 ) * = α * -(c 1 + . . . + c k ) √
n where α * is the optimal α for Eq. ( 1). Thus, as n → ∞, (α 1 ) * still blows up even though we parametrized α 1 correctly. More generally, the incorrect parametrization of some hyperparameters forces other hyperparameters to increasingly compensate for it as width grows, distorting their optima, even if the latter are correctly parametrized.
D Practical Considerations
In this section, we outline several useful tips and tricks that can improve the quality of hyperparameter transfer in practice.
D.1 Verifying µP Implementation via Coordinate Checking
Even though µP is neatly encapsulated by Table 3 , implementing it correctly can in practice be error-prone, just like how implementing autograd by hand can be error-prone even though the math behind is just chain-rule. In the case of autograd, gradient checking is a simple way of verifying implementation correctness; similarly, we propose coordinate checking to verify the correctness of µP implementation: Exemplified by Fig. 5 , one calculates the average coordinate size of every (pre)activation vector in the network over a few steps of training, as width is varied over a large range. An incorrect implementation will see some activation vector blow up or shrink to zero with width (like in the top row of Fig. 5 ). In the mup package we release with this paper, we include an easy-to-use method for coordinate checking.
D.2 Zero Initialization for Output Layers and Query Layers in Attention
We find that the optimal hyperparameters of small and large width models match more closely when we initialize output layers at 0 (i.e. with variance σ 2 /fan_in where σ = 0 instead of positive σ). This is because the neural network in µP is approximately a Gaussian process (GP) at initialization with variance on the order Θ(σ 2 /width) (contrast this with SP networks, which approximates a GP with Θ(σ 2 ) variance) [21, 29, 53, 57] . Of course, when width is large, this variance vanishes, but this can be far from so in the small proxy model. This discrepancy in the initial GP can cause the training trajectory of the proxy model to be very different from the trajectory of the large target model, causing a mismatch in the optimal hyperparameters. By initializing the output layer at 0, we remove this mismatch in the initial GP. Empirically we do not find this modification to be detrimental to performance.
A similar consideration applies to the query layer in self-attention: At initialization, the attention logit q k/d head looks like a Gaussian with variance Θ(1/d head ) because q and k are almost independent and zero-mean. In the limit d head → ∞, the logit is exactly 0, which can be a large discrepancy compared to when d head is small in the small proxy model we want to tune. By initializing the query projection matrix W q to 0, q will also be 0, and hence the attention logit is always 0 at initialization regardless of width (but will generally become nonzero after a gradient step), resolving this discrepancy.
More generally, any layer or computation that goes from an "infinite" dimension (i.e. width) to a "finite" dimension (e.g. output dimension or sequence length) can exhibit this kind of discrepancy due to the initial GP. When d head → ∞ and n head is fixed, attention logit calculation can be viewed in the same vein as a function R seqlen×d model → R n head ×seqlen×seqlen , which "reduces to" R ∞ → R 1 . When the network is narrow, its approximation to the infinite-width behavior becomes crude, which is manifested as large fluctuations in preactivation coordinates. When using a squashing activation functions like softmax or tanh, this causes narrower networks to saturate the activation more than wider ones, which results in a systematic bias toward small gradients and therefore distorting the hyperparameter landscape. This can be seen in Fig. 9 , where we use tanh as the network activation function. Therefore, we recommend replacing non-essential squashing activation functions with ReLU, whose derivative depends only on the sign of the pre-activation. A similar reasoning can be applied to superlinear activation functions, where the distribution of activation values can have heavy tails, leading to slow convergence to the infinite-width limit. However, such activations are rarely used in practice.
D.4 Enlarge d k
We find that small d head = d k can lead to a highly noisy HP landscape, as shown in Fig. 10 . This can significiantly decrease the quality of random HP search on the small proxy model. To solve this, we find it useful to decouple d 
D.5 Non-Gaussian vs Gaussian Initialization
We find non-Gaussian (e.g. uniform) initialization can sometimes cause wider models to perform worse than narrower models, whereas we do not find this behavior for Gaussian initialization. This is consistent with theory, since in the large width limit, one should expect non-Gaussian initialization to behave like Gaussian initializations anyway (essentially due to Central Limit Theorem, or more precisely, universality), but the non-Gaussianity slows down the convergence to this limit.
D.6 Using a Larger Sequence Length
For Transformers, we empirically find that we can better transfer initialization standard deviation from a narrower model (to a wide model) if we use a larger sequence length. It is not clear why this is the case. We leave an explanation to future work.
D.7 Tuning Per-Layer Hyperparameters
The techniques in this paper allow the transfer across width of (learning rate, initialization, multipliers) simultaneously for all parameter tensors. Thus, to get the best results, one should ideally tune all such hyperparameters. In practice, we find that just tuning the global learning rate and initialization, along with input, output, and attention multipliers, yield good results.
E Which Hyperparameters Can Be Transferred? (Continued) E.1 Further Discussions on Hyperparameter Categories
Below, we discuss the reasoning behind each kind, which are supported by our empirical evidence collected in Fig. 4 on Transformers as well as those in Appendix G.1 on ResNet.
Transferable Hyperparameters In Table 2 , we summarize which HPs can be transferred across training scale. The transfer across width, as explained in Section 2, is theoretically justified, while we present the transfer across the other dimensions as empirical results.
These cover most of the well-known and important HPs when the need for regularization is not paramount, e.g., during large scale language model pretraining. Parameter Multipliers are not wellknown HPs, yet we include them here as they serve a bridge between SP and µP and can impact model performance in practice. Concretely, any SP and µP neural networks of the same width can have their Parameter Multipliers tuned so that their training dynamics become identical.
Hyperparameters That Don't Transfer Well Not all HPs transfer well even if we use µP. In particular, those whose primary function is to regularize training to mitigate "overfitting" tend not to transfer well. Intuitively, regularization needs to be applied more heavily in larger models and when data is scarce, but µP does not know the data size so cannot adjust the regularization accordingly.
To the best of our knowledge, there is no strict separation between HPs that regularize and those that don't. However, conventional wisdom tells us that there exists a spectrum of how much regularizing effect a HP has. For example, dropout probability and weight decay are among those whose primary function is to regularize, whereas batch size and learning rate might regularize training in some cases but affect the dynamics more so in other ways. Our empirical exploration tells us that the former do not transfer well, while the latter do. Our subsequent discussion will focus on the latter; we leave to future works the expansion to the former.
Hyperparameters Transfered Across
We have left out a category of HPs that defines the training scale, or in practical terms, training cost. This includes 1) those that define how many operations a model's forward/backward pass takes, such as the model's width, depth, and in the case of language modeling, sequence length; and 2) those that define how many such passes are performed, such as batch size and number of training steps.
As recent works have shown [7, 19, 39] , improvements along any of these scale dimensions lead to apparently sustainable gain in performance; as a result, we are primarily interested in transferring other HPs across these dimensions that define scale, rather than finding the optimal scale. 20 This category of HPs is particularly crucial as one can speedup training by downsizing in one or multiple such dimensions. Indeed, it's very common for practitioners to implicitly transfer HPs across the number of training samples by tuning on only a subset of the full training data.
Our insights from the infinite-width limit inspired us to explore HP tranfer across width, which does not work under SP as we have shown earlier. Building upon our success with width, which is well explained theoretically, we hope to push the limit of compute-saving by investigating the other dimensions empirically. To the best of our knowledge, the transferability of optimal HPs across depth, batch size, sequence length, and training time has not been rigorously investigated previously, with the main exception of the literature on (learning rate, batch size) scaling [41, 44] where our transferability result of learning rate across batch size recapitulates [30] . 21 See Section 10.3 on how our results relate to prior works. We will primarily focus on the Transformer architecture in the main text with evidence for ResNet in Appendix G.1.
E.2 On the Definitions of Width
Our theory allows more general notions of width. This is especially relevant in Transformers, where d model , d head = d k , d v , n head , d f f n (see Fig. 11 ) can all be construed as measures of width. We briefly discuss these here, with more theoretical justification in Appendix J.2.1 and empirical validation below.
Varying Width Ratio So far we have assumed that every hidden layer is widened by the same factor. But in fact we can widen different hidden layers differently. This is useful, for example, in a Transformer where we may want to use a smaller d f f n during tuning. If we are using Adam, as long as the width of every layer still tends to infinity, we still obtain approximately the same limit 22 , so the µTransfer remains theoretically justified.
See Fig. 12 for an empirical validation on IWSLT-14 using a Transformer.
Number of Attention Heads
In attention-based models, one typically splits hidden size into multiple attention heads following d model = d head × n head . So far we have assumed d head and d model to be width, but it's possible and potentially advantageous to fix d head and treat n head as the width, or increasing both simultaneously. This allows our technique to handle many popular models, including GPT-3 [7] , which scale up by fixing d head and increasing n head . See Fig. 13 for an empirical validation on Wikitext-2.
Varying Just the Width of Attention Heads A specific useful instance of varying width ratio is decoupling the key and value dimensions d k and d v and scaling d k differently from (typically larger √ d as is done commonly). When tuning on the small proxy model, if d k is too small, the HP landscape can be quite noisy. Keeping d k relatively large while shrinking all other dimensions solves this problem, while still obtaining significant speedup.
F Experimental Details
F.1 IWSLT IWSLT14 De-En is a well-known machine translation benchmark. We use a Transformer implemented in fairseq [33] with a default
d model = 1 /4d f f n = 512 and d k = d q = d v = d model/n head = 128
(amounting to 40M parameters), which we denote as the 1x model. For transfer, we tune on a proxy model with the same n head but with d model and other dimensions 4 times smaller; we will call this the 0.25x model (but it has 4M parameters). All models are trained with Adam for 100 epochs and validated at the end of every epoch. We tune via random search the learning rate η, the output layer parameter multiplier α output , and the attention key-projection weight multiplier α attn following the grid • η: 5 × 10 -4 × 2 z , where z ∈ {-1.5, -1.25, -1, ..., 1.25}
• α output : 2 z , where z ∈ {-8, -7, -6, ..., 7}
• α attn : 2 z , where z ∈ {-3, -2, -1, ..., 8}
F.2 WMT
We scale up to WMT14 En-De using the large Transformer from [50] , with a d model = 1 /4d f f n = 1024 and d q = d k = d v = d model/n head = 64. We use the exact same setup and reproduce their result as our baseline. Then, we build the proxy model by shrinking the target model's d model from the original 1024 to 256, d f f n from 4096 to 256 and n head from 16 to 4. This reduces the total parameter count from 211M to 15M. We then perform the HP search on the proxy model and take the best according to validation loss, before testing on the target model. We tune via random search the learning rate η, the output layer parameter multiplier α output , and the attention key-projection weight multiplier α attn following the grid
• η: 6 × 10 -4 × 2 z , where z ∈ {-1.5, -1.25, -1, ..., 1.25}
• α output : 2 z , where z ∈ {-8, -7, -6, ..., 7}
• α attn : 2 z , where z ∈ {-3, -2, -1, ..., 8} F.3 BERT Details of BERT Prototype Our proxy model has 10 Transformer layers with d model = d f f n = 256. We also reduce the number of attention heads to 8 with a d head of 32. We call it BERT Prototype since we can increase its width and depth according to our definitions to recover both BERT Base and BERT Large, which enables us to sweep HPs once and use for both models. Overall, BERT Prototype has 13M trainable parameters, a fraction of the 110M in BERT Base and the 350M in BERT Large.
Hyperparameters Tuned for Pretraining We tune the following HPs for pretraining: Adam learning rate η, embedding learning rate η emb , output weight multiplier α output , attention logits multiplier α attn , layernorm gain multiplier α LNgain , and bias multiplier α bias .
We sample 256 combinations from the follow grid:
• η: 1 × 10 -4 × 2 z , where z ∈ {1.5, 2, 2.5, 3, 3.5}
• η emb : 1 × 10 -4 × 2 z , where z ∈ {-1, -0.5, 0, 0.5, 1} • α output : 2 z , where z ∈ {2, 4, 6} • α attn : 2 z , where z ∈ {3, 3.5, 4, ..., 7} • α LNgain : 2 z , where z ∈ {8.5, 9, 9.5, 10, 10.5}
• α bias : 2 z , where z ∈ {8.5, 9, 9.5, 10, 10.5}
The ranges are chosen to include the implicit choices of these HPs in SP BERT Large.
Finetuning Procedure and Hyperparameters We hand-pick the finetuning HPs after training the full-sized model. As regularization is an essential ingredient in successful finetuning, we do not transfer such HPs (at least via the suite of techniques presented in this work) (see Table 1 ). We focus on MNLI [52] and QQP, which are two representative tasks from GLUE [51] . Following [27] , we used Adam [20] with a learning rate of 5 × 10 -5 and a batch size of 64. The maximum number of epochs was set to 5. A linear learning rate decay schedule with warm-up of 0.1 was used. All the texts were tokenized using wordpieces and were chopped to spans no longer than 128 tokens.
F.4 GPT-3
Baseline 6.7B GPT-3 Transformer As the GPT-3 codebase has evolved since the publication of [7] , we re-trained the 6.7B model from scratch to remove changes in our codebase as a possible confounder. The main differences to [7] are 1) a modified learning rate decay schedule, where the learning rate is decayed to zero at the end of training rather than being decayed to 0.1 of the initial value, and 2) use of relative attention in place of absolute attention. Unfortunately, after all experiments were finished, we found this re-run baseline used absolute attention instead of relative attention, while the µTransfer model still used relative attention.
Random Search using Reduced-Width Proxy Model In order to find a good set of hyperparameters for the µTransfer version of the 6.7B model, we performed a hyperparameter search over a reduced version of the model (i.e., the proxy model), where the width is set to 256 hidden units. This proxy model inherits changes from the evolved GPT-3 codebase: it uses relative [10] (instead of absolute) position encoding. Early on, we noted that on the proxy model, linear learning rate decay outperformed the default cosine schedule, so all subsequent experiments for the proxy models use a linear decay schedule. By Fig. 4 , µTransferring this linear decay schedule to the full model should maintain such a performance advantage over the cosine schedule.
The hyperparameter search space consists of the following hyperparameters:
• learning rate: Sampled from 10 Uniform(-4,-1)
• initialization scale: All the parameters are multiplied -sampled from 10 Uniform(-1,1) • attention temperature: Reciprocal of the multiplier applied to the input to attention softmax. Sampled from 4 Uniform(-1,1) . • output temperature: Reciprocal of the multiplier applied to the input to softmax that produces the distribution over output tokens. Sampled from 4 Uniform(-1,1) . • embedding multiplier: Scalar by which we multiply the output of the embedding layer.
Sampled from 10 Uniform(-1,1) . • relative position embedding multiplier: Scalar by which we multiply vectors representing relative position. Sampled from 10 Uniform(-1,1) .
In order to make the search more efficient we reduced the total number of training tokens. We hypothesized that tuning hyperparameters on a reduced total number of tokens does not significantly affect optimal hyperparameters. To verify, we trained two different horizons and compared the results. As suspected, we observed that the results are well-aligned for both 4 and 16 billion tokens versions. We observe learning rate and initialization scale impact the results the most. Based on the results we chose 0.006 for the former and 2.5 for the latter. Since most other hyperparameters appear to have negligible effect on performance, they were kept at their default values of 1, the only exception being the embedding scale, where higher values seem to perform better and it was therefore set to 10.
Training the µTransfer Model We encountered frequent divergences in our initial attempt to train the µTransfer model. We traced the issue back to underflow of FP16 tensors in the backwards pass and therefore switched to training the model in FP32. This allowed us to finish the training run without divergences. We hypothesize that the divergence issue is related to µTransfer picking more aggressive hyperparameters, for example a higher learning rate on linear weight tensors compared to the original model. In order to exclude code differences as a possible confounder, we re-trained GPT-3 6.7B from scratch using the original hyperparameters. The only difference compared to the version published in [7] is that the learning rate was decayed fully, whereas the learning rate of the model from [7] was only decayed to 10% of its starting value. The retrained model performs slightly worse than the original published in [7] . We suspect that this is because it made less progress during the last phase of training where the learning rate is close to zero. The training curves of the µTransfer model and the re-run of the original 6.7B can be seen in Fig. 15 . Detailed evaluation results can be found in Table 10 and Table 11 . The training curves of the GPT-3 6.7B model with µTransfer (orange) and a re-run with the original settings from [7] (blue). The µTransfer model uses relative attention while the re-run uses absolute attention. In addition, the former was trained using FP32 activations and weights after initially encountering stability issues with the hyperparameters computed using µP, while the re-run used the original FP16 training. The µTransfer model seems to underperform in the middle of training, but achieves a much better final validation loss once the learning rate is fully decayed. While the original model uses a cosine schedule, the µTransfer model uses a linear learning rate decay schedule transferred from the proxy model.
Ratio of Tuning Cost to Pretraining Cost in FLOPs can be approximated as
s(t 1 N 1 + t 2 N 2 ) ST ≈ 0.
• T = 300 Billion is the number of training tokens for the 6.7B target model.
Here we are using the fact that the training FLOPs of a Transformer per token is roughly proportional to its number of parameters. Setup For this case we use Davidnet [2] , a ResNet variant that trains quickly on CIFAR-10, so as to efficiently investigate its HP landscape. We train with SGD on CIFAR-10 for 10 epochs; all results are averaged over 15 random seeds. We use a width multiplier to identify models of different width, and a multiplier of 1 corresponds to the original model in [2] . We look at validation accuracy here as the model barely overfits, and our observations will hold for the training accuracy as well. We first conduct a learning rate sweep for models of different widths using SP; the result is shown in Fig. 16 , on the left.
G Additional Experiments
Hyperparameter Stability Note that the best model with a width multiplier of 8 under-performs that with a multiplier of 4. We run the same sweep with µP, along with a sweep of the output multiplier (α output ); the result is shown in Fig. 16 , on the right. We notice that wider models always perform better under µP and that the optimal learning rate η and α output are stable across width.
Table 10 : Full evaluation results of our GPT-3 6.7B models: The new model tuned with µTransfer (marked µP), the original model from [7] , and a re-training of this model from scratch with the original hyperparameter settings (marked re-run). The sampling-based evaluations shown here are a subset of the ones from [7] . Since the sampling-based evaluations are subject to high variance, Wikitext 103 and the LM1B benchmark have been added to help distinguish the relative performance of the µP and non-µP model. Note that Wikitext-103 [32] and the LM1B [9] Max Update Parametrization ( P)
Figure 16 : ResNet on CIFAR-10 for different widths (compared to a base network). On the left, the widest network SP underperforms; on the right, the µP network has a more consistent HP landscape and performs better. Both networks are tuned at the smallest width for the HP (η or α output ) not in the x-axis.
Hyperparameter Transfer Next, we perform a grid search for learning rate (η) and α output on the 0.5x model for both SP and µP. 23 Then, we take the best combination and test on the 8x model, simulating how a practitioner might use µTransfer. The result is shown in Table 12 , where µP outperforms SP by 0.43% ± .001%. Setup For this case we use Wide-Resnet, or WRN [65] , a ResNet variant with more channels per layer, to further showcase µTransfer across width, i.e., number of channels. We train with SGD on ImageNet for 50 epochs following standard data augmentation procedures. We use a width multiplier to identify models of different width, and a multiplier of 1 corresponds to the original WRN-50-2-bottleneck in [65] .
Hyperparameter Transfer We start with a proxy model with a width multiplier of 0.125 and tune several HPs using the following grid:
• η: 1 × 2.048 × 2 z , where z ∈ {-5, -4, -3, ..., 4}
• α output : 10 × 2 z , where z ∈ {-5, -4, -3, ..., 4}
• weight decay co-efficient γ: 3.05 × 10 -5 × 2 z , where z ∈ {-2, -1.5, -1, ..., 1.5}
• SGD momentum β: 0.875 × 2 z , where z ∈ {-2, -1.5, -1, ..., 1.5}
The grid is centered around the default HPs used by [1] for ResNet-50; while not expected to be competitive for WRN, they represent a reasonable starting point for our experiment.
We randomly sample 64 HP combinations from the grid and train for 50 epochs, before selecting the one with the highest top-1 validation accuracy. Then, we scale up the model following both µP and SP and run with the same HPs we just selected. The result is shown in Table 13 , where µP outperforms SP by 0.41% in terms of top-1 validation accuracy. 
G.2.2 Post-Layernorm Transformers
Fig. 17 shows the transferability of learning rate, α output , initialization standard deviation, and Adam β 2 across width, batch size, sequence length, and training steps for post-layernorm Transformers. However, in general, we find transfer across depth to be fragile. 
G.2.3 Hyperparameter Instability of SP Transformers
Fig. 18 and Fig. 20 show the HP instability inherent in SP Transformers.
H Implementing µTransfer in a Jiffy
As we have shown, one can enable µTransfer by just reparametrizing the desired model in Maximal Update Parametrization (µP). While conceptually simple, switching from Standard Parametrization (SP) to µP can be error-prone, as popular deep learning frameworks are built around SP. We strive to build a tool that fulfills two goals:
1. Minimize code changes when switching to µP;
2. Keep model behavior invariant, under this switch, at a given base model shape.
By model shape, we mean the collection of dimensions of all parameters of the model. The latter goal, which we call parametrization backward compatibility, ensures that any code base works exactly as before at the base model shape, similar to Eq. ( 4), e.g. the loss at any time step remains exactly the same before and after the switch to µP. Of course, when widths start to differ from the base model shape, the model behavior necessarily changes so that HPs can be transferred. What Happens in the mup Package Under the hood, mup implements the µP formulation in Table 8 . By invoking set_base_shape(model, base_model), each parameter tensor p of model gets a p.infshape attribute that stores, for each of its dimensions, the corresponding base dimension and whether that dimension should be considered "infinite" (i.e. will be scaled up/down, e.g., d model of a Transformer) or "finite" (i.e. will be fixed, e.g., vocabulary size). This information is used in the initializers and optimizers to automatically scale the parameters or learning rates to be compliant with µP. For example, by Table 8 , the Adam learning rate of hidden weights p is calculated as η/p.infshape.width_mult(), where p.infshape.width_mult() essentially calculates fan_in base_fan_in .
I Reverse-µTransfer for Diagnosing Training Instability in Large Models
Large Transformers are famously fickle to train [25, 37] . We note that a possible source of this instability for larger transformers is the failure of naive hyperparameter transfer via the standard parametrization. This is certainly consistent with Fig. 1 , which shows that the optimal learning rate for small Transformers can lead to trivial performance in large Transformers. We support this hypothesis further by reverse-µTransferring the instability-inducing HPs from a large Transformer to a small one and replicating the training instability. This is shown in Fig. 21 .
Practically, this reverse-µTransfer technique can be used to diagnose or debug training instability problems of large models. We offer two case studies toward this claim.
1) When training transformers of width 8192 on Wikitext-2, we found certain HP combinations caused divergence in the middle of training. We reverse-µTransferred one such HP combination to a model of width 256 and replicated this divergence. By analyzing this small model's activations right before this divergence, we found that the cause is due to attention logits blowing up. Note this debugging session proceeded much more quickly than if we directly worked with the large model. Later we confirmed this is indeed the same cause of the width-8192 model's divergence.
2) A 6B-parameter language model (in standard parametrization) in a separate project experienced repeated blow-up in the middle of training. We reverse-µTransferred its hyperparameters to a smaller, 100M-parameter model and replicated the training instability. This was solved by a retuning of the small model via random search. Transformer) . Takeaway: The overall shapes of the curves are identical between the left and right plots 24 ; in particular, a learning rate leads to instability in a wide model iff it does so when transferred back to a narrow model.
J An Intuitive Introduction to the Theory of Maximal Update Parametrization
In what follows, we seek to describe useful intuitions and rules of thumb that would be helpful to practitioners and empirical researchers alike in figuring out what is the right neural network parametrization. The intuitions we shall describe regarding SGD can be made rigorous as in [56, 57] ; those regarding Adam are new, and their formalization will be done in an upcoming paper.
First, we write down the most basic intuition regarding sums of many random elements, which will underlie all of the calculations that follow.
Law of Large Numbers (LLN) If x 1 , . . . , x n , . . . "look like" random independent samples of a random variable X, then
1 n n i=1 x i → E[X], as n → ∞.
Central Limit Theorem (CLT) In the same scenario as above,
1 √ n n i=1 (x i -E[X]) → N (0, σ(X)), as n → ∞,
where σ(X) is the standard deviation of the random variable X.
Of course, there are many subtleties one must resolve to make the statements above truly rigorous (e.g., what is the meaning of "look like"?), but as rules of thumb, they typically give the correct prediction. 
A ∈ R n×n A ∈ R n×n A ∈ R 1×n Entry size of Av Θ( √ n) Θ(n) Θ(n)
In particular, here we want to note the following basic intuition regarding the size of a sum of x i :
when n is large,
n i=1 x i has typical size Θ(n) if E[X] = 0 Θ( √ n) otherwise
Here, "typical size" can be taken to mean the size 99% of time. Again, we stress that this is a good rule of thumb that yields the correct prediction in the cases we are concerned with here; the rigorous versions of this will come from the Tensor Programs framework (e.g., [56] ).
J.1 Behaviors of Gaussian Matrices vs Tensor Product Matrices
Central to the derivation of µP for any architecture are key insights on the behaviors of two kinds of random matrices: 1) iid Gaussian random matrix and 2) tensor product matrix (by which we mean a sum of outer products) and more generally what we call nonlinear tensor product matrix (see Eq. ( 7)). For example, a neural network, randomly initialized in the typical way, will have each weight matrix look like the former. However, every step of training by gradient descent adds a sum of outer products to this initial matrix, so that the change in weights constitute a tensor product matrix. For Adam, the change in weights is not a tensor product but a more general nonlinear tensor product matrix (see Eq. ( 7)). In this section, we will particularly focus on the right scaling for the entries of such matrices, leading to a discussion of the right neural network parametrization in the next section. We concentrate on the key heuristics but eschew burdensome rigor.
Key Insights Consider a random vector v ∈ R n with approximately iid entries and a random matrix A of either size n × n or 1 × n, both having entries of size Θ(1). 25 In the context of deep learning, v for example can be an activation vector in an MLP, a Gaussian A the hidden weights at initialization, a (nonlinear) tensor product A the change in hidden weights due to training, and a vector A the readout layer weights. Then Av corresponds to a part of the next layer preactivation or the network output. To make sure the preactivations and the output don't blow up, we thus need to understand the scale of Av, especially in the general case where A is correlated with v. 26 This is summarized in Table 14 , with the derivations below. Intuitively, a (nonlinear) tensor product or vector A will interact with a correlated v via Law of Large Numbers, hence the n-scaling, while a Gaussian A interacts with v via Central Limit Theorem, hence the √ n-scaling.
In the derivations below, we answer a slightly different but equivalent question of "how to scale A such that Av has entry size Θ(1)?"
J.1.1 Preparation for the Derivations
By the results of [57] , each (pre-)activation vector and its gradient vector in a multi-layer perceptron, at any time during training, have approximately iid coordinates in the large width limit, 27 and something similar can be said for more advanced networks such as ResNet and Transformers28 . Definition J.1. We say any such vector v ∈ R n has Θ(n a )-sized coordinates, or just Θ(n a )coordinates for short, if v 2 /n = Θ(n 2a ) as n → ∞. Because, by the above discussion, the coordinates are roughly iid when n is large, this intuitively means that each entry of v has "typical size" Θ(n a ). We make similar definitions with Θ replaced by O and Ω.
Furthermore, to each such vector v with Θ(1)-sized coordinates, we can associate a random variable Z v , independent of n, that represents the coordinate distribution of v, in such a way that: If vector u is correlated with v, then Z u will also be correlated with Z v , and lim n→∞ v u/n = E Z u Z v .
J.1.2 Linear Tensor Product Matrix (e.g. SGD Updates)
The case of (linear) tensor product matrix can be reduced to the outer product case by linearity. Given u, v, x ∈ R n having approximately iid coordinates (of size Θ(1)) like discussed above, we can form the outer product
A def = u ⊗ v/n = uv /n,which is the form of a single (batch size 1) gradient update to a weight matrix. Then, by Law of Large Numbers,
Ax = u v x n ≈ cu, where c = E Z v Z x .
So Ax also has approximately iid coordinates, distributed like
Z Ax def = Z u E Z v Z x . Likewise, if A is a sum of outer products A = k i=1 u i ⊗ v i /n, then Ax = k i=1 u i v i x n , with coordinates distributed as Z Ax = k i=1 Z u i E Z v i Z x .
Notice that each coordinate of A has size Θ(1/n). The above reasoning shows that, in order for Ax to have coordinate size Θ(1) (assuming x does), then Θ(1/n) is the right coordinate size for A, in the general case that v i and x are correlated (as is generically the case during gradient descent, with A = ∆W for some weights W and x being the previous activations). 29J.1.3 Nonlinear Tensor Product Matrix (e.g. Adam Updates)
When using Adam or another adaptive optimizer that normalizes the gradient coordinatewise before applying them, we need to modify our argument slightly to obtain the right coordinate size scaling of the matrix. The gradient update A, after such normalization, will take the form of A αβ = ψ(u 1 α , . . . , u k α , v 1 β , . . . , v k β ), for some ψ : R 2k → R and vectors u i , v j ∈ R n .
We say a matrix of this form is a nonlinear tensor product matrix.
First, note the tensor product matrices (e.g. the form of SGD update) discussed previously (Eq. ( 6)) already takes this form, with ψ(u 1 α , . . . , u k α , v 1 β , . . . , v k β ) = n -1 (u
1 α v 1 β + • • • + u k α v k β )
, so Eq. ( 7) is a strict generalization of linear tensor products. Next, for the example of Adam, each gradient update is µ/σ where µ (resp. σ 2 ) is the moving average of previous (unnormalized) gradients (resp. the coordinatewise square of the same). 30 If these unnormalized gradients are the outer products u 1 ⊗ v 1 , . . . , u k ⊗ v k , then the update has coordinates
(µ/σ) αβ = ψ(u 1 α , . . . , u k α , v 1 β , . . . , v k β ) def = i γ i u i α v i β / i ω i (u i α v i β ) 2 ,where γ i and ω i are the weights involved in the moving averages. Now suppose we have some A ∈ R n×n of the form Eq. ( 7), where u i , v i ∈ R n have approximately iid coordinates (of size Θ(1)), and ψ = n -1 ψ where ψ doesn't depend on n (in terms of Adam where ψ corresponds to the ψ of Eq. ( 8), this corresponds to using a learning rate of 1/n). Then for x ∈ R n having approximately iid coordinates of size Θ(1), by Law of Large Numbers,
(Ax) α = 1 n n β=1
ψ(u 1 α , . . . , u k α , v 1 β , . . . , v k β )x β ≈ E ψ(u 1 α , . . . , u k α , Z v 1 , . . . , Z v k )Z x def = Ψ(u 1 α , . . . , u k α ).
2. Neural network output should be O(1).
3. All parameters should be updated as much as possible (in terms of scaling in width) without leading to divergence.
Let's briefly justify these desiderata. For the desideratum 1, if the coordinates are ω(1) or o(1), then for sufficiently wide networks their values will go out of floating point range. This problem is particularly acute for low-precision formats that are essential for training large models such as BERT or GPT. Moreover, a general nonlinearity is only well-behaved if its input is in a fixed range (although this is not a problem for homogeneous nonlinearities like relu). For example, for tanh nonlinearity, if the preactivation is vanishing o(1), then tanh is essentially linear; if the preactivation is exploding ω(1), then the tanh gradient vanishes.
For the desideratum 2, a similar justification applies to the numerical fidelity of the loss function and loss derivative. Note that, with desideratum 3, this means the network output should be Θ(1) after training (but it can go to zero at initialization).
Finally, desideratum 3 means that 1) we are doing "maximal feature learning" [57] and 2) every parameter contribute meaningfully in the infinite-width limit. This ensures that learning rate "plays the same role" in the finite-width case as in the infinite-width limit. For example, it prevents the scenario where a weight matrix gets stuck at initialization in the limit for any learning rate (so learning rate does not matter) but evolves nontrivially in any finite-width network (so learning rate does matter).
These desiderata will essentially uniquely single out µP. More formally, µP is the unique parametrization that admits feature learning in all parameters of the neural network [57] , and this property theoretically guarantees HP transfer across width (for sufficiently large width). However, for the sake of reaching a broader audience, we will focus more on the intuitive derivations from the desiderata rather than on this formal aspect.
Below, we first assume for simplicity that the width of every layer is n, and we focus only on dense weights. Later, we will discuss convolutions and varying the widths between layers.
J.2.1 µP Derivation From the Desiderata
Below, we will derive the µP formulation in Table 3 . Tables 8 and 9 can be derived from Table 3 via the following equivalences, which can be easily derived via some simple calculations. Lemma J.1. Let f t (ξ) denote the neural network function after t steps of training (using any fixed sequence of batches), evaluated on input ξ. Consider a parameter tensor W with learning rate C, initialized as W ∼ N (0, B 2 ), and with a multiplier A. Then for any θ > 0, f t (ξ) stays fixed for all t and ξ if we set
• when the optimizer is SGD
A ← Aθ, B ← B/θ, C ← C/θ 2
• when the optimizer is Adam,
A ← Aθ, B ← B/θ, C ← C/θ;
For example, for output weights, Table 3 has A = 1, B = 1/fan_in, C = η/fan_in for SGD and Adam. Then taking θ = 1/fan_in, we get the entries in Table 8 , with A = 1/fan_in, B = 1, C = η • fan_in for SGD and C = η for Adam. Taking θ = 1/ √ fan_in instead, we get the entries in Table 9 , with A = 1/ √ fan_in, B = 1/fan_in, C = η for SGD and η/ √ fan_in for Adam. Similar calculations hold for the input weights scaling in those tables, after taking into consideration that fan_in is considered a constant in terms of width for the input layer.
We proceed with the derivation of Table 3 below. Recall the definitions of Θ(n a )-sized coordinates or Θ(n a )-coordinates from Definition J.1.
Output Weights Suppose W ∈ R 1×n is an output weight. By desideratum 1, the input x to W has Θ(1)-sized coordinates. Thus W should have Θ(1/n)-coordinates so that |W x| = O(1). We can initialize W with Θ(1/n)-coordinates and scale its (per-layer) LR so that ∆W has Θ(1/n)coordinates as well. This means initializing W αβ ∼ N (0, Θ(1/n 2 )) and use Θ(1/n) learning rate for both SGD and Adam. but this induces a non-maximal feature learning limit, which, as we argue below, cannot transfer hyperparameters in all situations. 2. For SGD, the gradient of R n×n weight has Θ(1/ √ n)-coordinates, so Θ(1) learning rate would make preactivation scale like Θ( √ n) and hence blow up. If we use Θ(1/width) learning rate, then blow-up does not occur. However, this infinite-width limit is in the kernel regime [57] and thus does not allow HP transfer for the same reason that NTP below does not.
Neural Tangent Parametrization (NTP) We have concrete examples, e.g. Word2Vec in [57] , where the NTK limit has trivial performance -so HPs have no effect at all -vastly outperformed by finite-width networks -where HPs matter. More importantly, wider does not always do better in NTP, especially in tasks where feature learning is crucial [57, 61] . So in the context of modern deep learning e.g. large language model pretraining, NTP (or SP with Θ(1/width) LR) does not make sense for wide neural networks.
Other Parametrizations Recall the Dynamical Dichotomy Theorem proven in [57] , which says that any nontrivial stable "natural parametrization" (formally, "abc-parametrization," [57] ) either admits a feature learning limit or a kernel limit, but not both.
Our argument above against SP and NTP will also work against any parametrization inducing a kernel limit. Therefore, it remains to ask, can other feature learning parametrizations transfer HPs?
We argue no. As shown in [57] , any other feature learning parametrization differs from µP essentially only in that some parameters are not updated maximally. By [57, Sec 6.4] , in the infinite-width limit, such parameters can be thought of as being fixed at initialization. Therefore, in such infinite-width limits, the learning rate of such parameters becomes useless. As such, we cannot hope for the HP landscape of the limit to reflect the HP landscape of finite-width neural networks. µP is the unique feature learning parametrization that updates all parameters maximally, so that the learning rate of each parameter plays approximately the same role in finite-width neural networks as in the infinite-width limit. Consequently, the HP landscape of the µP limit should reflect the HP landscape of finite-width neural networks.
but possibly not for different data and/or tasks.
The more theoretically astute reader may observe that SP with a Θ(1/width) learning rate induces a well-defined infinite-width limit exists as well. Nevertheless, this does not allow HP transfer because this limit is in kernel regime as shown in[57]. See Appendix J.3 for more discussions.
i.e. the default parametrization offered by common deep learning frameworks. See Table3for a review.
The key here is that the init. variance ∝ 1/fan_in, so the same insights here apply with e.g. He initialization.
While superficially different, this parametrization is equivalent to the µP defined in[57].
This is roughly because during training, q and k will be correlated so q k actually scales like d due to Law of Large Numbers, in contrast to the original motivation that q, k are uncorrelated at initialization so Central Limit applies instead. See Appendix J.2.1 for a more in-depth discussion.
Note in this example, Glorot initialization[13] (i.e. with variance 1/(fan_in + fan_out)) would scale asymptotically the same as µP and thus is similarly well-behaved. However, if one adds layernorm or batchnorm, then Glorot will cause logit blowup like SP, but µP still will not.
"2 layers" means the model has 2 self-attention blocks. To compare with SP Transformer, see Fig.18.
in fact, post-layernorm Transformers are much more sensitive to HPs than pre-layernorm, so our technique is more crucial for them, especially for transfer across width. Fig.1uses post-layernorm.
https://github.com/pytorch/fairseq/blob/master/examples/translation/README.md.
Ideally we would like to measure the wall clock time used for tuning. However, smaller models such as the proxy Transformer used for IWSLT are not efficient on GPUs, so wall clock time would not reflect the speedup for larger models like GPT-3. Thus, we measure in FLOPs, which is less dependent on hardware optimization.
We do not report the standard deviation over random initializations to avoid confusion.
We find this provides more reliable result than selecting for the best BLEU score.
while the optimal learning is roughly linear in batch size when the latter is small
Here, for simplicity of the example, we model the interaction between "hyperparameters" c 1 , . . . , c k as additive, but in real neural networks such interactions are usually much more complicated.
In particular, we are not fixing the total training FLOPs when we scale, which requires understanding the tradeoff of different scale HPs. For example, when we transfer across batch size, we fix the number of steps of training (not the number of epochs), so that the total FLOPs scales linearly.
There's also a literature on the proper initialization for training deep networks effectively (e.g.[5,16,26,40,59,60,66]), but they do not study the transferability per se. SeeSection 10.3
This also applies for SGD, but we need more involved scaling to keep the limit approximately the same.
Here we tune the 0.5x model instead of the 1x model to simulate the situation that one does "exploratory work" on the 1x model but, when scaling up, would like to tune faster by using a smaller proxy model.
Note that the curves on the left are "lower" than curves on the right. This just reflects the increasing capacity of wider models able to fit the training data better, so is orthogonal to our point.
in the sense that the the variance of the entries are Θ(1)
Here "correlated" formally means v depends on W in a Tensor Program. This essentially captures all scenarios of "v correlated with W " that occurs in deep learning.
Our intuition here is derived from the assumption that width is much larger than training time; of course, as illustrated by our myriad experiments, these intuition are very useful even when this is not the case, such as when training to convergence.
E.g. in a convnet, the (pre-)activations are iid across channels, but correlated across pixels
In some corner cases when x is uncorrelated with v, then v x = Θ( √ n) by Central Limit, so actually Ax has Θ(1/ √ n)-coordinates. However, this case does not come up much in the context of training neural networks.
Adam also has bias correction for the moving averages which can be accomodated easily, but for simplicity we omit them here.
This is because every "reasonable" deep learning computation can be expressed in a Tensor Program.
In a convnet, a (pre-)activation vector corresponds to a single pixel across all channels; in general , we expect (pre-)activations are iid across channels, but correlated across pixels
References
[1]. Davidnet, mit license.  2019;:.
[2]. Autonomio talos, mit license.  2019;:.
[3]Akiba Takuya, Sano Shotaro, Yanase Toshihiko, Ohta Takeru, Koyama Masanori. Optuna: A next-generation hyperparameter optimization framework.  2019;:.
[4]Bachlechner Thomas, Bodhisattwa Prasad, Majumder , Huanru Henry, Mao , Garrison W, et al.. ReZero is All You Need: Fast Convergence at Large Depth.  2020;:.
[5]Bernstein Jeremy, Vahdat Arash, Yue Yisong, Liu Ming-Yu. On the distance between two neural networks and the stability of learning.  2021;:.
[6]Tom B, Brown Benjamin, Mann Nick, Ryder Melanie, Subbiah Jared, Kaplan Prafulla, et al.. Language models are few-shot learners.  2020;:.
[7]Carbonnelle Simon, Vleeschouwer ChristopheDe. Layer rotation: a surprisingly powerful indicator of generalization in deep networks?.  2019;:.
[8]Chelba Ciprian, Mikolov Tomas, Schuster Mike, Ge Qi, Brants Thorsten, Koehn Phillipp, et al.. One billion word benchmark for measuring progress in statistical language modeling.  2014;:.
[9]Dai Zihang, Yang Zhilin, Yang Yiming, Carbonell Jaime, Le QuocV, Salakhutdinov Ruslan. Transformer-xl: Attentive language models beyond a fixed-length context.  2019;:.
[10]Devlin Jacob, Chang Ming-Wei, Lee Kenton, Toutanova Kristina. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  2019;:.
[11]Ding Xiaohan, Xia Chunlong, Zhang Xiangyu, Chu Xiaojie, Han Jungong, Ding Guiguang. RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition.  2021;:.
[12]Glorot Xavier, Bengio Yoshua. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics 2010;9:249--256.
[13]Hoffer Elad, Hubara Itay, Soudry Daniel. Train longer, generalize better: closing the generalization gap in large batch training of neural networks.  2017;:.
[14]Horváth Samuel, Klein Aaron, Richtárik Peter, Archambeau Cédric. Hyperparameter transfer learning with adaptive complexity.  2021;:.
[15]Shi Xiao,  Huang, Pérez Felipe. Improving Transformer Optimization Through Better Initialization.  None;:.
[16]Jacot Arthur, Gabriel Franck, Hongler Clément. Neural Tangent Kernel: Convergence and Generalization in Neural Networks.  2018;:.
[17]Jamieson Kevin, Talwalkar Ameet. Non-stochastic best arm identification and hyperparameter optimization.  2015;:.
[18]Kaplan Jared, Mccandlish Sam, Henighan Tom, Brown TomB, Chess Benjamin, Child Rewon, et al.. Scaling Laws for Neural Language Models.  2020;:.
[19]Kingma Diederik, Ba Jimmy. Adam: A method for stochastic optimization.  2014;:.
[20]Lee Jaehoon, Bahri Yasaman, Novak Roman, Schoenholz Sam, Pennington Jeffrey, Sohl-Dickstein Jascha. Deep Neural Networks as Gaussian Processes. International Conference on Learning Representations 2018;:.
[21]Li Liam, Jamieson Kevin, Rostamizadeh Afshin, Gonina Ekaterina, Hardt Moritz, Recht Benjamin, et al.. A system for massively parallel hyperparameter tuning.  2020;:.
[22]Li Lisha, Jamieson Kevin, Desalvo Giulia, Rostamizadeh Afshin, Talwalkar Ameet. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. JMLR None;18:.
[23]Liu Hanxiao, Dai Zihang, So DavidR, Quoc V, Le . Pay Attention to MLPs.  2021;:.
[24]Liu Liyuan, Liu Xiaodong, Gao Jianfeng, Chen Weizhu, Han Jiawei. Understanding the difficulty of training transformers. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020;:5747--5763.
[25]Liu Liyuan, Liu Xiaodong, Gao Jianfeng, Chen Weizhu, Han Jiawei. Understanding the Difficulty of Training Transformers.  2020;:.
[26]Liu Xiaodong, He Pengcheng, Chen Weizhu, Gao Jianfeng. Multi-task deep neural networks for natural language understanding. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics 2019;:4487--4496.
[27]Liu Yang, Bernstein Jeremy, Meister Markus, Yue Yisong. Learning by Turning: Neural Architecture Aware Optimisation.  2021;:.
[28]De AlexanderG, Matthews G, Rowland Mark, Hron Jiri, Turner RichardE, Ghahramani Zoubin. Gaussian Process Behaviour in Wide Deep Neural Networks.  2018;:.
[29]Mccandlish Sam, Kaplan Jared, Amodei Dario, Dota Openai,  Team. An Empirical Model of Large-Batch Training.  2018;:.
[30]Melas-Kyriazi Luke. Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet.  2021;:.
[31]Merity Stephen, Xiong Caiming, Bradbury James, Socher Richard. Pointer sentinel mixture models.  2016;:.
[32]Ott Myle, Edunov Sergey, Baevski Alexei, Fan Angela, Gross Sam, Ng Nathan, et al.. fairseq: A fast, extensible toolkit for sequence modeling, mit license. Proceedings of NAACL-HLT 2019;:.
[33]Park DanielS, Sohl-Dickstein Jascha, Le QuocV, Smith SamuelL. The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study.  2019;:.
[34]Paszke Adam, Gross Sam, Massa Francisco, Lerer Adam, Bradbury James, Chanan Gregory, et al.. Pytorch: An imperative style, high-performance deep learning library.  None;:.
[35]Wallach InH, Larochelle H, Beygelzimer A, Buc F, Fox E, Garnett R. Advances in Neural Information Processing Systems.  2019;32:8024--8035.
[36]Popel Martin, Bojar Ondřej. Training Tips for the Transformer Model. The Prague Bulletin of Mathematical Linguistics 2018;110:43--70.
[37]Raffel Colin, Shazeer Noam, Roberts Adam, Lee Katherine, Narang Sharan, Matena Michael, et al.. Exploring the Limits of Transfer Learning with a.  2020;:.
[38]Rosenfeld JonathanS, Rosenfeld Amir, Belinkov Yonatan, Shavit Nir. A Constructive Prediction of the Generalization Error Across Scales.  2019;:.
[39]Samuel S, Schoenholz Justin, Gilmer Surya, Ganguli Jascha, Sohl-Dickstein . Deep Information Propagation.  2016;:.
[40]Shallue ChristopherJ, Lee Jaehoon, Antognini Joseph, Sohl-Dickstein Jascha, Frostig Roy, Dahl GeorgeE. Measuring the Effects of Data Parallelism on Neural Network Training.  2018;:.
[41]Shazeer Noam, Stern Mitchell. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost.  2018;:.
[42]Shoeybi Mohammad, Patwary Mostofa, Puri Raul, Legresley Patrick, Casper Jared, Catanzaro Bryan. Megatron-lm: Training multi-billion parameter language models using model parallelism.  2019;:.
[43]Samuel L, Smith Pieter-Jan, Kindermans , Quoc V, Le . Don't Decay the Learning Rate. Increase the Batch Size 2017;:.
[44]Snoek Jasper, Larochelle Hugo, Adams RyanP. Practical bayesian optimization of machine learning algorithms.  2012;:.
[45]Snoek Jasper, Rippel Oren, Swersky Kevin, Kiros Ryan, Satish Nadathur, Sundaram Narayanan, et al.. Scalable bayesian optimization using deep neural networks.  2015;:.
[46]Stoll Danny, Jörg KH, Franke Diane, Wagner Simon, Selg Frank, Hutter . Hyperparameter transfer across developer adjustments.  2021;:.
[47]Tolstikhin Ilya, Houlsby Neil, Kolesnikov Alexander, Beyer Lucas, Zhai Xiaohua, Unterthiner Thomas, et al.. An all-MLP Architecture for Vision.  2021;:.
[48]Touvron Hugo, Bojanowski Piotr, Caron Mathilde, Cord Matthieu, El-Nouby Alaaeldin, Grave Edouard, et al.. Feedforward networks for image classification with data-efficient training.  2021;:.
[49]Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez AidanN, et al.. Attention is all you need.  2017;:.
[50]Wang Alex, Singh Amanpreet, Michael Julian, Hill Felix, Levy Omer, Samuel R Bowman . Glue: A multi-task benchmark and analysis platform for natural language understanding.  2018;:.
[51]Williams Adina, Nangia Nikita, Bowman Samuel. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2018;1:1112--1122.
[52]Yang Greg. Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes.  2019;:.
[53]Yang Greg. Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation.  2019;:.
[54]Yang Greg. Tensor Programs II: Neural Tangent Kernel for Any Architecture.  2020;:.
[55]Yang Greg. Tensor Programs III: Neural Matrix Laws.  2020;:.
[56]Yang Greg, Hu EdwardJ. Feature learning in infinite-width neural networks. arXiv 2020;:.
[57]Yang Greg, Littwin Etai. Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics.  2021;:.
[58]Yang Greg, Schoenholz SamS. Deep Mean Field Theory: Layerwise Variance and Width Variation as Methods to Control Gradient Explosion.  2018;:.
[59]Yang Greg, Schoenholz SamuelS. Mean Field Residual Networks: On the Edge of Chaos.  2017;:.
[60]Yang Greg, Santacroce Michael, Hu EdwardJ. Efficient computation of deep nonlinear infinite-width neural networks that learn features. International Conference on Learning Representations 2022;:.
[61]Yogatama Dani, Mann Gideon. Efficient Transfer Learning Method for Automatic Hyperparameter Tuning. Artificial Intelligence and Statistics 2014;:1077--1085.
[62]You Yang, Gitman Igor, Ginsburg Boris. Large Batch Training of Convolutional Networks.  2017;:.
[63]You Yang, Li Jing, Reddi Sashank, Hseu Jonathan, Kumar Sanjiv, Bhojanapalli Srinadh, et al.. Large Batch Optimization for Deep Learning: Training BERT in 76 minutes.  2020;:.
[64]Zagoruyko Sergey, Komodakis Nikos. Wide residual networks.  2017;:.
[65]Zhang Hongyi, Dauphin YannN, Ma Tengyu. Residual Learning Without Normalization via Better Initialization. International Conference on Learning Representations 2019;:.
[66]. G.2.1 Verifying Transfer across Batch Size, Sequence Length, and.  None;:.
[67]. H Implementing µTransfer in a Jiffy I Reverse-µTransfer for Diagnosing Training Instability in Large Models J An Intuitive Introduction to the Theory of Maximal Update Parametrization J.1 Behaviors of Gaussian Matrices vs.  None;:.
[68]. Nonlinear Tensor Product Matrix.  None;:.
[69] J. Why Other Parametrizations Cannot Admit Hyperparameter Transfer.  None;:.
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
https://arxiv.org/abs/2203.03466
<div><div>
    
    
                
    <p><a href="https://arxiv.org/pdf/1707.04674">View PDF</a></p><blockquote>
            <span>Abstract:</span>Model-free policy learning has enabled robust performance of complex tasks with relatively simple algorithms. However, this simplicity comes at the cost of requiring an Oracle and arguably very poor sample complexity. This renders such methods unsuitable for physical systems. Variants of model-based methods address this problem through the use of simulators, however, this gives rise to the problem of policy transfer from simulated to the physical system. Model mismatch due to systematic parameter shift and unmodelled dynamics error may cause sub-optimal or unsafe behavior upon direct transfer. We introduce the Adaptive Policy Transfer for Stochastic Dynamics (ADAPT) algorithm that achieves provably safe and robust, dynamically-feasible zero-shot transfer of RL-policies to new domains with dynamics error. ADAPT combines the strengths of offline policy learning in a black-box source simulator with online tube-based MPC to attenuate bounded model mismatch between the source and target dynamics. ADAPT allows online transfer of policy, trained solely in a simulation offline, to a family of unknown targets without fine-tuning. We also formally show that (i) ADAPT guarantees state and control safety through state-action tubes under the assumption of Lipschitz continuity of the divergence in dynamics and, (ii) ADAPT results in a bounded loss of reward accumulation relative to a policy trained and evaluated in the source environment. We evaluate ADAPT on 2 continuous, non-holonomic simulated dynamical systems with 4 different disturbance models, and find that ADAPT performs between 50%-300% better on mean reward accrual than direct policy transfer.
    </blockquote>

    
    
  </div><div>
      <h2>Submission history</h2><p> From: James Harrison [<a href="https://arxiv.org/show-email/a61f66d4/1707.04674">view email</a>]      <br/>            <strong><a href="https://arxiv.org/abs/1707.04674v1">[v1]</a></strong>
        Sat, 15 Jul 2017 01:16:44 UTC (3,064 KB)<br/>
    <strong>[v2]</strong>
        Wed, 8 Nov 2017 22:41:31 UTC (3,240 KB)<br/>
</p></div></div>||||I||||  Skip to main content
      We gratefully acknowledge support from the Simons Foundation, member institutions , and all contributors. Donate
      > cs > arXiv:1707.04674

          Help | Advanced Search

            All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
        Search
            GO

          quick links

            * Login
            * Help Pages
            * About

          Computer Science > Robotics

          arXiv:1707.04674 (cs)
              [Submitted on 15 Jul 2017 (v1), last revised 8 Nov 2017 (this version, v2)]

            Title: ADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems

              Authors: James Harrison, Animesh Garg, Boris Ivanovic, Yuke Zhu, Silvio Savarese, Li Fei-Fei, Marco Pavone
              View a PDF of the paper titled ADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems, by James Harrison and 6 other authors
            View PDF
            Abstract: Model-free policy learning has enabled robust performance of complex tasks with relatively simple algorithms. However, this simplicity comes at the cost of requiring an Oracle and arguably very poor sample complexity. This renders such methods unsuitable for physical systems. Variants of model-based methods address this problem through the use of simulators, however, this gives rise to the problem of policy transfer from simulated to the physical system. Model mismatch due to systematic parameter shift and unmodelled dynamics error may cause sub-optimal or unsafe behavior upon direct transfer. We introduce the Adaptive Policy Transfer for Stochastic Dynamics (ADAPT) algorithm that achieves provably safe and robust, dynamically-feasible zero-shot transfer of RL-policies to new domains with dynamics error. ADAPT combines the strengths of offline policy learning in a black-box source simulator with online tube-based MPC to attenuate bounded model mismatch between the source and target dynamics. ADAPT allows online transfer of policy, trained solely in a simulation offline, to a family of unknown targets without fine-tuning. We also formally show that (i) ADAPT guarantees state and control safety through state-action tubes under the assumption of Lipschitz continuity of the divergence in dynamics and, (ii) ADAPT results in a bounded loss of reward accumulation relative to a policy trained and evaluated in the source environment. We evaluate ADAPT on 2 continuous, non-holonomic simulated dynamical systems with 4 different disturbance models, and find that ADAPT performs between 50%-300% better on mean reward accrual than direct policy transfer.
                Comments:  International Symposium on Robotics Research (ISRR), 2017
                Subjects:  Robotics (cs.RO)                                         
                Cite as:   arXiv:1707.04674 [cs.RO]                                 
                           (or arXiv:1707.04674v2 [cs.RO] for this version)         
                           https://doi.org/10.48550/arXiv.1707.04674                
                             Focus to learn more                                    
                               arXiv-issued DOI via DataCite                        
                

            Submission history

            From: James Harrison [view email]
            [v1] Sat, 15 Jul 2017 01:16:44 UTC (3,064 KB)
            [v2] Wed, 8 Nov 2017 22:41:31 UTC (3,240 KB)
            Full-text links:

            Access Paper:

                  View a PDF of the paper titled ADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems, by James Harrison and 6 other authors
              * View PDF
              * TeX Source
              * Other Formats
              view license
            Current browse context:
              cs.RO
              < prev | next >
              new | recent | 2017-07
              Change to browse by:
                cs

            References & Citations

              * NASA ADS
              * Google Scholar
              * Semantic Scholar

            DBLP - CS Bibliography

              listing | bibtex
              James Harrison
              Animesh Garg
              Boris Ivanovic
              Yuke Zhu
              Silvio Savarese
                …
            a export BibTeX citation Loading...

                BibTeX formatted citation

                ×
                loading...
                Data provided by:

              Bookmark

            Bibliographic Tools

              Bibliographic and Citation Tools

                    Bibliographic Explorer Toggle
                    Bibliographic Explorer (What is the Explorer?)
                    Litmaps Toggle
                    Litmaps (What is Litmaps?)
                    scite.ai Toggle
                    scite Smart Citations (What are Smart Citations?)
            Code, Data, Media

              Code, Data and Media Associated with this Article

                    Links to Code Toggle
                    CatalyzeX Code Finder for Papers (What is CatalyzeX?)
                    DagsHub Toggle
                    DagsHub (What is DagsHub?)
                    GotitPub Toggle
                    Gotit.pub (What is GotitPub?)
                    Links to Code Toggle
                    Papers with Code (What is Papers with Code?)
                    ScienceCast Toggle
                    ScienceCast (What is ScienceCast?)
            Demos

              Demos

                    Replicate Toggle
                    Replicate (What is Replicate?)
                    Spaces Toggle
                    Hugging Face Spaces (What is Spaces?)
                    Spaces Toggle
                    TXYZ.AI (What is TXYZ.AI?)
            Related Papers

              Recommenders and Search Tools

                    Link to Influence Flower
                    Influence Flower (What are Influence Flowers?)
                    Connected Papers Toggle
                    Connected Papers (What is Connected Papers?)
                    Core recommender toggle
                    CORE Recommender (What is CORE?)
            About arXivLabs

                  arXivLabs: experimental projects with community collaborators

                  arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

                  Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

                  Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

          Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
            * About
            * Help
            * Click here to contact arXiv Contact
            * Click here to subscribe Subscribe
            * Copyright
            * Privacy Policy
            * Web Accessibility Assistance

            * arXiv Operational Status
              Get status notifications via email or slack

View PDF  
            Abstract:Model-free policy learning has enabled robust performance of complex tasks with relatively simple algorithms. However, this simplicity comes at the cost of requiring an Oracle and arguably very poor sample complexity. This renders such methods unsuitable for physical systems. Variants of model-based methods address this problem through the use of simulators, however, this gives rise to the problem of policy transfer from simulated to the physical system. Model mismatch due to systematic parameter shift and unmodelled dynamics error may cause sub-optimal or unsafe behavior upon direct transfer. We introduce the Adaptive Policy Transfer for Stochastic Dynamics (ADAPT) algorithm that achieves provably safe and robust, dynamically-feasible zero-shot transfer of RL-policies to new domains with dynamics error. ADAPT combines the strengths of offline policy learning in a black-box source simulator with online tube-based MPC to attenuate bounded model mismatch between the source and target dynamics. ADAPT allows online transfer of policy, trained solely in a simulation offline, to a family of unknown targets without fine-tuning. We also formally show that (i) ADAPT guarantees state and control safety through state-action tubes under the assumption of Lipschitz continuity of the divergence in dynamics and, (ii) ADAPT results in a bounded loss of reward accumulation relative to a policy trained and evaluated in the source environment. We evaluate ADAPT on 2 continuous, non-holonomic simulated dynamical systems with 4 different disturbance models, and find that ADAPT performs between 50%-300% better on mean reward accrual than direct policy transfer.
    
    
    
   
      Submission history  From: James Harrison [view email]                   [v1] 
        Sat, 15 Jul 2017 01:16:44 UTC (3,064 KB)
    [v2]
        Wed, 8 Nov 2017 22:41:31 UTC (3,240 KB)
  ||||I||||  Skip to main content
      We gratefully acknowledge support from the Simons Foundation, member institutions , and all contributors. Donate
      > cs > arXiv:1707.04674
          Help | Advanced Search
            All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
        Search
            GO
          quick links
            * Login
            * Help Pages
            * About
          Computer Science > Robotics
          arXiv:1707.04674 (cs)
              [Submitted on 15 Jul 2017 (v1), last revised 8 Nov 2017 (this version, v2)]
            Title: ADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems
              Authors: James Harrison, Animesh Garg, Boris Ivanovic, Yuke Zhu, Silvio Savarese, Li Fei-Fei, Marco Pavone
              View a PDF of the paper titled ADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems, by James Harrison and 6 other authors
            View PDF
            Abstract: Model-free policy learning has enabled robust performance of complex tasks with relatively simple algorithms. However, this simplicity comes at the cost of requiring an Oracle and arguably very poor sample complexity. This renders such methods unsuitable for physical systems. Variants of model-based methods address this problem through the use of simulators, however, this gives rise to the problem of policy transfer from simulated to the physical system. Model mismatch due to systematic parameter shift and unmodelled dynamics error may cause sub-optimal or unsafe behavior upon direct transfer. We introduce the Adaptive Policy Transfer for Stochastic Dynamics (ADAPT) algorithm that achieves provably safe and robust, dynamically-feasible zero-shot transfer of RL-policies to new domains with dynamics error. ADAPT combines the strengths of offline policy learning in a black-box source simulator with online tube-based MPC to attenuate bounded model mismatch between the source and target dynamics. ADAPT allows online transfer of policy, trained solely in a simulation offline, to a family of unknown targets without fine-tuning. We also formally show that (i) ADAPT guarantees state and control safety through state-action tubes under the assumption of Lipschitz continuity of the divergence in dynamics and, (ii) ADAPT results in a bounded loss of reward accumulation relative to a policy trained and evaluated in the source environment. We evaluate ADAPT on 2 continuous, non-holonomic simulated dynamical systems with 4 different disturbance models, and find that ADAPT performs between 50%-300% better on mean reward accrual than direct policy transfer.
                Comments:  International Symposium on Robotics Research (ISRR), 2017
                Subjects:  Robotics (cs.RO)                                         
                Cite as:   arXiv:1707.04674 [cs.RO]                                 
                           (or arXiv:1707.04674v2 [cs.RO] for this version)         
                           https://doi.org/10.48550/arXiv.1707.04674                
                             Focus to learn more                                    
                               arXiv-issued DOI via DataCite                        
                
            Submission history
            From: James Harrison [view email]
            [v1] Sat, 15 Jul 2017 01:16:44 UTC (3,064 KB)
            [v2] Wed, 8 Nov 2017 22:41:31 UTC (3,240 KB)
            Full-text links:
            Access Paper:
                  View a PDF of the paper titled ADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems, by James Harrison and 6 other authors
              * View PDF
              * TeX Source
              * Other Formats
              view license
            Current browse context:
              cs.RO
              
              new | recent | 2017-07
              Change to browse by:
                cs
            References & Citations
              * NASA ADS
              * Google Scholar
              * Semantic Scholar
            DBLP - CS Bibliography
              listing | bibtex
              James Harrison
              Animesh Garg
              Boris Ivanovic
              Yuke Zhu
              Silvio Savarese
                …
            a export BibTeX citation Loading...
                BibTeX formatted citation
                ×
                loading...
                Data provided by:
              Bookmark
            Bibliographic Tools
              Bibliographic and Citation Tools
                    Bibliographic Explorer Toggle
                    Bibliographic Explorer (What is the Explorer?)
                    Litmaps Toggle
                    Litmaps (What is Litmaps?)
                    scite.ai Toggle
                    scite Smart Citations (What are Smart Citations?)
            Code, Data, Media
              Code, Data and Media Associated with this Article
                    Links to Code Toggle
                    CatalyzeX Code Finder for Papers (What is CatalyzeX?)
                    DagsHub Toggle
                    DagsHub (What is DagsHub?)
                    GotitPub Toggle
                    Gotit.pub (What is GotitPub?)
                    Links to Code Toggle
                    Papers with Code (What is Papers with Code?)
                    ScienceCast Toggle
                    ScienceCast (What is ScienceCast?)
            Demos
              Demos
                    Replicate Toggle
                    Replicate (What is Replicate?)
                    Spaces Toggle
                    Hugging Face Spaces (What is Spaces?)
                    Spaces Toggle
                    TXYZ.AI (What is TXYZ.AI?)
            Related Papers
              Recommenders and Search Tools
                    Link to Influence Flower
                    Influence Flower (What are Influence Flowers?)
                    Connected Papers Toggle
                    Connected Papers (What is Connected Papers?)
                    Core recommender toggle
                    CORE Recommender (What is CORE?)
            About arXivLabs
                  arXivLabs: experimental projects with community collaborators
                  arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
                  Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
                  Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
          Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
            * About
            * Help
            * Click here to contact arXiv Contact
            * Click here to subscribe Subscribe
            * Copyright
            * Privacy Policy
            * Web Accessibility Assistance
            * arXiv Operational Status
              Get status notifications via email or slack
ADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems
https://arxiv.org/abs/1707.04674?utm_content=bufferf9385&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
https://arxiv.org/abs/2102.12375
Transfer of Fully Convolutional Policy-Value Networks Between Games and Game Variants
2021-02-24
Soemers; Dennis J N J; Mella; Vegard; Piette; Eric; Stephenson; Matthew; Browne; Cameron; Teytaud; Olivier
<div><div>
 
 
 
 
 
 
 <p><a href="/pdf/2102.12375">Download PDF</a></p><blockquote>
 Abstract: In this paper, we use fully convolutional architectures in AlphaZero-like
self-play training setups to facilitate transfer between variants of board
games as well as distinct games. We explore how to transfer trained parameters
of these architectures based on shared semantics of channels in the state and
action representations of the Ludii general game system. We use Ludii's large
library of games and game variants for extensive transfer learning evaluations,
in zero-shot transfer experiments as well as experiments with additional
fine-tuning time.
 </blockquote>
 
 
 </div><div>
 <h2>Submission history</h2><p> From: Olivier Teytaud [<a href="/show-email/5296dc8d/2102.12375">view email</a>]
 <br /><strong>[v1]</strong>
Wed, 24 Feb 2021 15:59:58 UTC (1,553 KB)<br /></p></div>||||I|||| Skip to main content
 We gratefully acknowledge support from
 the Simons Foundation and member institutions.
 > cs > arXiv:2102.12375

 Help | Advanced Search

 All fields Title Author Abstract Comments Journal reference ACM classification MSC classification Report number arXiv identifier DOI ORCID arXiv author ID Help pages Full text
 Search
 GO

 quick links

 * Login
 * Help Pages
 * About

 Computer Science > Machine Learning

 arXiv:2102.12375 (cs)
 [Submitted on 24 Feb 2021]

 Title: Transfer of Fully Convolutional Policy-Value Networks Between Games and Game Variants

 Authors: Dennis J.N.J. Soemers, Vegard Mella, Eric Piette, Matthew Stephenson, Cameron Browne, Olivier Teytaud
 Download PDF
 Abstract: In this paper, we use fully convolutional architectures in AlphaZero-like self-play training setups to facilitate transfer between variants of board games as well as distinct games. We explore how to transfer trained parameters of these architectures based on shared semantics of channels in the state and action representations of the Ludii general game system. We use Ludii's large library of games and game variants for extensive transfer learning evaluations, in zero-shot transfer experiments as well as experiments with additional fine-tuning time.
 Subjects: Machine Learning (cs.LG) 
 Cite as: arXiv:2102.12375 [cs.LG] 
 (or arXiv:2102.12375v1 [cs.LG] for this version)
 https://doi.org/10.48550/arXiv.2102.12375 
 Focus to learn more 
 arXiv-issued DOI via DataCite 
 

 Submission history

 From: Olivier Teytaud [view email]
 [v1] Wed, 24 Feb 2021 15:59:58 UTC (1,553 KB)
 Full-text links:

 Download:

 * PDF
 * Other formats
 (license)
 Current browse context:
 cs.LG
 < prev | next >
 new | recent | 2102
 Change to browse by:
 cs

 References & Citations

 * NASA ADS
 * Google Scholar
 * Semantic Scholar

 DBLP - CS Bibliography

 listing | bibtex
 Dennis J. N. J. Soemers
 Vegard Mella
 Éric Piette
 Matthew Stephenson
 Cameron Browne
 …
 a export bibtex citation Loading...

 Bibtex formatted citation

 ×
 loading...
 Data provided by:

 Bookmark

 Bibliographic Tools

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle
 Bibliographic Explorer (What is the Explorer?)
 Litmaps Toggle
 Litmaps (What is Litmaps?)
 scite.ai Toggle
 scite Smart Citations (What are Smart Citations?)
 Code, Data, Media

 Code, Data and Media Associated with this Article

 Links to Code Toggle
 Papers with Code (What is Papers with Code?)
 ScienceCast Toggle
 ScienceCast (What is ScienceCast?)
 Demos

 Demos

 Replicate Toggle
 Replicate (What is Replicate?)
 Spaces Toggle
 Hugging Face Spaces (What is Spaces?)
 Related Papers

 Recommenders and Search Tools

 Connected Papers Toggle
 Connected Papers (What is Connected Papers?)
 Core recommender toggle
 CORE Recommender (What is CORE?)
 IArxiv recommender toggle
 IArxiv Recommender (What is IArxiv?)
 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

 Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)
 * About
 * Help
 * Click here to contact arXiv Contact
 * Click here to subscribe Subscribe
 * Copyright
 * Privacy Policy
 * Web Accessibility Assistance

 * arXiv Operational Status
 Get status notifications via email or slack

Transfer of Fully Convolutional Policy-Value Networks Between Games and Game Variants
Date Published: 2021-02-24
Authors:
Dennis J N J Soemers, Maas-tricht University, Maastricht, the Netherlands, nis.soemers@maastrichtuniversity.nl>
Vegard  Mella
Éric  Piette, Maas-tricht University, Maastricht, the Netherlands
Matthew  Stephenson, Maas-tricht University, Maastricht, the Netherlands
Cameron  Browne, Maas-tricht University, Maastricht, the Netherlands
Olivier  Teytaud
Abstract
In this paper, we use fully convolutional architectures in AlphaZero-like self-play training setups to facilitate transfer between variants of board games as well as distinct games. We explore how to transfer trained parameters of these architectures based on shared semantics of channels in the state and action representations of the Ludii general game system. We use Ludii's large library of games and game variants for extensive transfer learning evaluations, in zero-shot transfer experiments as well as experiments with additional fine-tuning time.
1. Introduction
AlphaGo (Silver et al., 2016) and its successors (Silver et al., 2017; 2018) have inspired a significant amount of research (Anthony et al., 2017; Tian et al., 2019; Morandin et al., 2019; Wu, 2019; Cazenave et al., 2020; Cazenave, 2020) on combinations of self-play, Monte-Carlo Tree Search (MCTS) (Kocsis & Szepesvári, 2006; Coulom, 2007; Browne et al., 2012) and Deep Learning (LeCun et al., 2015) for automated game-playing. Originally, AlphaGo used distinct value and policy networks, each of which have convolutional layers (LeCun et al., 1989) followed by fullyconnected layers. Silver et al. (2017) demonstrated that the use of residual blocks (He et al., 2016) , alongside merging the policy and value networks into a single network with two output heads, significantly improved playing strength in AlphaGo Zero. Other modifications to neural network architectures were also explored in subsequent research (Wu, 2019; Cazenave, 2020) .
In the majority of prior research, spatial structures present in the state-based inputs for board games are exploited by the inductive bias of convolutional layers, but the policy headwhich has one output for every distinct possible move in a board game -is preceded by one or more fully-connected layers which do not leverage any spatial semantics. Various architectures that also account for spatial semantics in outputs have been proposed in computer vision literature (Ronneberger et al., 2015; Shelhamer et al., 2017) , and in the context of games can also handle changes in board size (Lin et al., 2014; Wu, 2019; Cazenave et al., 2020) .
The primary contribution of this paper is an approach for transfer learning between variants of games, as well as distinct games. We use fully convolutional networks with global pooling from the Polygames framework (Cazenave et al., 2020) for their ability to handle changes in spatial dimensions during transfer. We focus on transfer between games implemented in the Ludii general game system (Browne et al., 2020; Piette et al., 2020) . Its consistent state and action representations (Piette et al., 2021) and gameindependent manner of constructing tensor representations for states and actions (Soemers et al., 2021) enables the identification of shared semantics between the non-spatial dimensions (channels) of inputs and outputs for different (variants of) games. This facilitates transfer of trained parameters of fully convolutional networks. Despite previous publications of benchmarks (Nichol et al., 2018) , transfer learning in games has remained a challenging problem with limited successes. We propose that Ludii's large library of board games can be used as a new benchmark for transfer learning in games, and provide extensive baseline results that include various cases of successful zero-shot transfer and transfer with fine-tuning, for transfer between variants of board games as well as distinct board games. Our experiments include transfer between domains with differences in board sizes, board shapes, victory conditions, piece types, and other aspects -many of these have been recognised as important challenges for learning in games (Marcus, 2018) .
2. Background
This section first provides background information on the implementation of AlphaZero-like (Silver et al., 2018) selfplay training processes for games in Polygames (Cazenave et al., 2020) , and the architectures of neural networks used for this purpose. Secondly, we briefly describe tensor representations in the Ludii general game system (Browne et al., 2020; Piette et al., 2020; Soemers et al., 2021) , which we use for all transfer learning experiments. Finally, we discuss background information on transfer learning in games.
2.1. Learning to Play Games in Polygames
Similar to AlphaZero (Silver et al., 2018) , game-playing agents in Polygames (Cazenave et al., 2020) use a combination of MCTS and deep neural networks (DNNs). Experience for training is generated in the form of selfplay games between MCTS agents that are guided by the DNN. Given a tensor representation of an input state s, the DNN outputs a value estimate V (s) of the value of that state, as well as a discrete probability distribution P(s) = [P (s, a 1 ), P (s, a 2 ), . . . , P (s, a n )] over an action space of n distinct actions. Both of these outputs are used to guide the MCTS-based tree search. The outcomes (typically losses, draws, or wins) of self-play games are used as training targets for the value head (which produces V (s) outputs), and the distribution of visit counts to children of the root node by the tree search process is used as a training target for the policy head (which produces P(s) outputs).
For board games, input states s are customarily represented as three-dimensional tensors of shape (C, H, W ), where C denotes a number of channels, H denotes the height of a 2D playable area (e.g., a game board), and W denotes the width. The latter two are interpreted as spatial dimensions by convolutional neural networks. It is typically assumed that the complete action space can be feasibly enumerated in advance, which means that the shape of P(s) output tensors can be constructed such that every possibly distinct action a has a unique, matching scalar P (s, a) for any possible state s in the policy head. A DNN first produces logits L(s, a) for all actions a, which are transformed into probabilities using a softmax after masking out any actions that are illegal in s.
In some general game systems, it can be difficult or impossible to guarantee that there will never be multiple different actions that share a single output in the policy head (Soemers et al., 2021) without manually incorporating additional game-specific domain knowledge. We say that distinct actions are aliased if they are represented by a single, shared position in the policy head's output tensor. In Polygames, the MCTS visit counts of aliased actions are summed up to produce a single shared training target for the corresponding position in the policy head. In the denominator of the softmax, we only sum over the distinct logits that correspond to legal actions (i.e., logits for aliased actions are not counted more than once). All aliased actions a receive the same prior probability P (s, a) to bias the tree search -because the DNN cannot distinguish between them -but the tree search itself can still distinguish between them.
2.2. The Ludii General Game System
Ludii (Browne et al., 2020; Piette et al., 2020) is a general game system with over 500 built-in games, many of which support multiple variants with different board sizes, board shapes, rulesets, etc. It automatically constructs suitable object-oriented state and action representations for any game described in its game description language, and these can be converted into tensor representations in a consistent manner without the need for additional game-specific engineering effort (Soemers et al., 2021) . All games in Ludii are modelled as having one or more "containers", which can be viewed as areas with spatial semantics (such as boards) that contain relevant elements of game states and positions that are affected by actions. This means that all games in Ludii are compatible with fully convolutional architectures.
2.3. Transfer Learning in Games
AlphaZero-like training approaches have produced superhuman agents for a variety of board games (Silver et al., 2018; Cazenave et al., 2020) , and hence been shown to have fairly general applicability, but models are generally trained from scratch for every distinct game. Transfer learning (Taylor & Stone, 2009; Lazaric, 2012; Zhu et al., 2020) may allow for significant savings in computation costs by transferring trained parameters from a source domain (i.e., a game that we train on first) to one or more target domains (i.e., variants of the source game or different games altogether).
To the best of our knowledge, research on transfer learning between distinct games has been fairly limited. Kuhlmann & Stone (2007) investigated the automated discovery of certain types of relations between source and target games, and transferred trained value functions in specific ways for specific relations. Banerjee & Stone (2007) proposed to transfer value functions based on game-independent features of a game tree's shape. Both approaches were evaluated, and the former is also restricted to, games defined in the Stanford Game Description Language (Love et al., 2008) .
In this paper, we focus on transferring complete policyvalue networks -with policy as well as value heads -as they are commonly used in modern game AI literature, using the Ludii general game system to provide easy access to a large and diverse number of games and game variants. Crucially, the transfer of trained parameters for a network with a policy head requires the ability to construct a mapping between action spaces of source and target domains, which we propose a method for. This is in contrast to approaches that only transfer value functions, which also only require the ability to create a mapping between state spaces. The policy output P(s) is a tensor of shape (Caction, H, W ). The value output V (s) is a scalar. Global pooling is used to reduce the dimensionality for the V (s) output in a manner that can transfer to different board sizes (Wu, 2019; Cazenave et al., 2020) .
2.4. Fully Convolutional Architectures
Following Subsection 2.1, we assume that game states s are represented as tensors of shape (C state , H, W ), and that the action space can be represented by a shape (C action , H, W ) -we may think of actions as being represented by an action channel, and coordinates in the 2D space of H rows and W columns. A network's policy head therefore also has a shape of (C action , H, W ). We focus on 2-player zero-sum games, which means that a single scalar suffices as output for the value head. Note that different games may have different numbers of channels and different values for H and W .
It is common to use architectures that first process input states s using convolutional layers, but at the end use one or more non-convolutional layers (such as fully-connected layers) preceding the outputs (Anthony et al., 2017; Silver et al., 2017; 2018) . This leads to networks that cannot handle changes in the spatial dimensions (i.e., changes in H or W ), and have no inductive biases that leverage any spatial structure that may be present in the action representation. Fully convolutional architectures with global pooling, such as those implemented in Polygames (Cazenave et al., 2020) , can address both of those concerns. In particular the ability to handle changes in spatial dimensions is crucial for transfer between distinct games or game variants, which is why we restrict our attention to them in the majority of this paper.
A simplified version of such an architecture is depicted in Figure 1 . Note that some details, such as the use of ReLU activations (Nair & Hinton, 2010) , batch normalization (Ioffe & Szegedy, 2015) , and residual connections (He et al., 2016) have been omitted for brevity. Full source code for Polygames (Cazenave et al., 2020) , including its architectures, is available online.1 
3. Transferring Parameters Between Games
The fully convolutional architectures as described in the previous section allow for DNNs trained in a source task S to be directly used in any target task T if it only has different values for one or both of the spatial dimensions H and W . However, it is also possible that different tasks have different numbers of channels C state or C action for state or action representations (or the same number of channels, but significantly different semantics for those channels). This section describes how we identify channels -for actions as well as states -that are likely to be semantically "equivalent" for any pair of games in Ludii's tensor representations (Soemers et al., 2021) , and how to transfer trained parameters accordingly. We use a relatively simplistic, binary notion of equivalence; a pair of a channel in S and a channel in T will either be considered to have identical semantics, or completely different semantics. Furthermore, we only consider the raw data that is encoded by a channel, and do not account for any differences in game rules in this notion of equivalence. For example, the two channels that encode presence of the two players' pieces in many games such as Hex, Go, Tic-Tac-Toe, etc., are considered to have identical semantics. A link to all the source code for identifying these mappings will be provided after double-blind peer review.
3.1. Mapping State Channels
For the majority of channels in Ludii's state tensor representations (Soemers et al., 2021) , semantically equivalent channels are straightforward to identify. For example, a channel that encodes whether player p is the current player, or whether a position was the destination of the previous move in a source domain S, will always have a semantically equivalent channel that encodes identical data and can easily be identified as such in a target domain T . These cases are listed in detail in Appendix A of the supplementary material. The first non-trivial case is that of binary channels which encode, for every pair of spatial coordinates, whether or not the corresponding position exists in a container represented by that channel. For example, the game of Shogi in Ludii has three separate containers; a large container for the game board, and two smaller containers representing player "hands", which hold captured pieces. These different containers are assigned different sections of the 2D space, and every container has a binary channel that tells the DNN which positions exist in which containers. In all the built-in games available in Ludii, containers are ordered in a "semantically consistent" manner; the first container is always the main game board, always followed by player hands (if there are any), etc. Therefore, we use the straightforward approach of mapping these container-based channels simply in the order in which they appear. It may be possible to remove the reliance on such domain knowledge with techniques that automatically analyse the rules of a game in more detail (Kuhlmann & Stone, 2007; Bou Ammar, 2013; Bou Ammar et al., 2014) , but is outside the scope of this work.
The second case that warrants additional explanation is that of binary channels that encode the presence of pieces; for every distinct piece type that is defined in a game, Ludii creates a binary channel that indicates for every possible position whether or not a piece of that type is placed in that position. Many games (such as Hex, Go, Tic-Tac-Toe, etc.) only have a single piece type per player, and for these we could easily decide that a channel indicating presence of pieces of Player p in one game is semantically equivalent to a channel indicating presence of pieces of the same player in another game. For cases with more than a single piece type per player, we partially rely on an unenforced convention that pieces in built-in games of Ludii tend to be named consistently across closely-related games (e.g., similar piece type names are used in many variants of Chess). If we find an exact name match for piece types between S and T , we treat the corresponding channels as semantically equivalent. Otherwise, for any piece type j in T , we loop over all piece types in S, and compute the Zhang-Shasha tree edit distances (Zhang & Shasha, 1989) between the trees of "ludemes" that describe the rules for the piece types in Ludii's game description language (Piette et al., 2020) .
3.2. Transferring State Channel Parameters
For the purpose of determining how to transfer parameters that were trained in S to a network that can play T , we make the assumption that S and T are exactly the same game, but with different state tensor representations; some state channels may have been added, removed, or shuffled around. We make this assumption because it enables us to build a more rigorous notion of what it means to "correctly" transfer parameters without accounting for differences in rules, optimal strategies, or value functions. In practice, S and T can end up being different games, and we intuitively still expect this transfer to be potentially beneficial if the games are sufficiently similar, but the notion of "correct" transfer cannot otherwise be made concrete without detailed domain knowledge of the specific games involved.
Let s denote the tensor representation, of shape (C S state , H, W ), for any arbitrary state in S. Let s denote the tensor representation, of shape (C T state , H, W ), for the same state represented in T -this must exist by the assumption that S and T are the same game, modelled in different ways. Let h S 1 (s) denote the hidden representation obtained by the application of the first convolutional operation on s, in a network trained on S. For brevity we focus on the case used throughout all our experiments, but most if not all of these assumptions can likely be relaxed; a nn.Conv2d layer as implemented in PyTorch (Paszke et al., 2019) , with 3×3 filters, a stride and dilation of 1, and a padding of 1 (such that the spatial dimensions do not change). Let Θ S denote this layer's tensor of parameters trained in S, of shape (k out , C S state , 3, 3), and B S -of shape (k out ) -the corresponding bias. This leads to h S 1 having a shape of (k out , H, W ). Similarly, let h T 1 (s ) denote the first hidden representation in the network after transfer, for the matching state s in the new target domain's representation, with weight and bias tensors Θ T and B T .
Under the assumption of source and target games being identical, we could obtain correct transfer by ensuring that h S 1 (s) = h T 1 (s ). Achieving this would mean that the first convolutional layer would handle any changes in the state representation, and the remainder of the network could be transferred in its entirety and behave as it learned to do in the source domain. B T is simply initialised by copying B S . Let i ∼ = j denote that the i th channel in a source domain S has been determined (as described in Subsection 3.1) to be semantically equivalent to the j th channel in a target domain T . Channels on the left-hand side are always source domain channels, and channels on the right-hand side are always target domain channels. For any channel j in T , if there exists a channel i in S such that i ∼ = j, we initialise
Θ T (k, j, •, •) := Θ S (k, i, •, •) for all k.
If there is no such channel i, we initialise these parameters using the default approach for initialising untrained parameters (or initialise them to 0 for zero-shot evaluations, where these parameters do not need to remain trainable through backpropagation).
If T contains channels j such that there are no equivalent channels i in S, i.e. {i | i ∼ = j} = ∅, we have no transfer to the Θ T (k, j, •, •) parameters. This can be the case if T involves new data for which there was no equivalent in the representation of S, which means that there was also no opportunity to learn about this data in S.
If S contained channels i such that there are no equivalent channels j in T , i.e. j (i ∼ = j), we have no transfer from the Θ S (k, i, •, •) parameters. This can be the case if S involved data that is no longer relevant or accessible in T . Not using them for transfer is equivalent to pretending that these channels still are present, but always filled with 0 values. If S contained a channel i such that there are multiple equivalent channels j in T , i.e. |{j | i ∼ = j}| > 1, we copy a single set of parameters multiple times. This can be the case if T uses multiple channels to encode data that was encoded by just a single channel in S (possibly with a loss of information). In the case of Ludii's tensor representations, this only happens when transferring channels representing the presence of piece types from a game S with fewer types of pieces, to a game T with more distinct piece types. In the majority of games in Ludii, such channels are "mutually exclusive" in the sense that if a position contains a 1 entry in one of these channels, all other channels in the set are guaranteed to have a 0 entry in the same position. This means that copying the same parameters multiple times can still be viewed as "correct"; for any given position in a state, they are guaranteed to be multiplied by a non-zero value at most once. The only exceptions are games T that allow for multiple pieces of distinct types to be stacked on top of each other in a single position, but these games are rare and not included in any of the experiments described in this paper.
If T contains a single channel j such that there are multiple equivalent channels i in S, i.e. |{i | i ∼ = j}| > 1, there is no clear way to correctly transfer parameters without incorporating additional domain knowledge on how the single target channel summarises -likely with a loss of information -multiple source channels. This case never occurs when mapping channels as described in Subsection 3.1.
3.3. Mapping Action Channels
Channels in Ludii's action tensor representations (Soemers et al., 2021) have three broad categories of channels; a channel for pass moves, a channel for swap moves, and one or more channels for all other moves. Channels for pass or swap moves in one domain can easily be classified as being semantically equivalent only to channels for the same type of moves in another domain.
We refer to games where, in Ludii's internal move representation (Piette et al., 2021) , some moves have separate "from" and "to" (or source and destination) positions as movement games (e.g. Amazons, Chess, Shogi, etc.), and games where all moves only have a "to" position as placement games (e.g. Go, Hex, Tic-Tac-Toe, etc.). In placement games, there is only one more channel to encode all moves that are not pass or swap moves. In movement games, there are 49 additional channels, which can distinguish moves based on any differences in x and y coordinates between "from" and "to" positions in {≤ -3, -2, -1, 0, 1, 2, ≥ 3}.
If both S and T are placement games, or if both are movement games, we can trivially obtain one-to-one mappings between all move channels. If S is a movement game, but T is a placement game, we only treat the source channel that encodes moves with equal "from" and "to" positions as semantically equivalent (in practice, this channel remains unused in the vast majority of movement games, which means that we effectively get no meaningful transfer for moves due to the large discrepancy in movement mechanisms). If S is a placement game, and T is a movement game, we treat the sole movement channel from S as being semantically equivalent to all the movement channels in T .
3.4. Transferring Action Channel Parameters
Similar to Subsection 3.2, we make the assumption that source and target games S and T are identical games, but with different action tensor representations, such that we can define a clear notion of correctness for transfer. Let s and a denote any arbitrary state and action in S, such that a is legal in s, and let s and a denote the corresponding representations in T . We assume that the state representations have been made equivalent through transfer of parameters for the first convolutional layer, as described in Subsection 3.2. Let h n (s) be the hidden representation that, in a fully convolutional architecture trained in S, is transformed into a tensor L(h n (s)) S of shape (C S action , H, W ) of logits. Similarly, let L(h n (s )) T of shape (C T action , H, W ) denote such a tensor of logits in the target domain. By assumption, we have that h n (s) = h n (s ).
If the action representations in S and T are equally powerful in their ability to distinguish actions, we can define a notion of correct transfer of parameters by requiring the transfer to ensure that L(h n (s)) S = L(h n (s )) T after accounting for any shuffling of channel indices. If we have one-to-one mappings for all action channels, this can be easily achieved by copying parameters of the final convolutional operation in a similar way as for state channels (see Subsection 3.2).
If the action representation of T can distinguish actions from each other that cannot be distinguished in S, we have a reduction in move aliasing (see Subsection 2.1). This happens when transferring from placement games to movement games. Since these actions were treated as identical when training in S, it is sensible to continue treating them as identical and give them equal probabilities in T . This is achieved by mapping a single source channel to multiple target channels, and copying trained parameters accordingly.
If the action representation of S could distinguish actions from each other that can no longer be distinguished in T . This happens when transferring from movement games to placement games. As described in the previous subsection, we handle this case conservatively by only allowing transfer from a single source channel -the one that is arguably the "most similar" -and discarding all other parameters.
4. Experiments
This section discusses experiments used to evaluate the performance of fully convolutional architectures, as well as several transfer learning experiments between variants of games and between distinct games. We used the training code from Polygames (Cazenave et al., 2020) . For transfer learning experiments, we used games as implemented in Ludii v1.1.6 (Browne et al., 2020) . Appendix B of the supplementary material provides details on hyperparameters. (Ronneberger et al., 2015) . Deep is better than shallow, U-nets are slightly better than their classical counterparts, and deep nets are greatly improved by (i) using fully convolutional policy heads (ii) using global pooling.
4.1. Evaluation of Fully Convolutional Architectures
We selected a variety of board games as implemented in Polygames (Cazenave et al., 2020) , and trained networks of various sizes and architectures, using 24 hours on 8 GPUs and 80 CPU cores per model. Models of various sizes -measured by the number of trainable parameters -have been constructed by randomly drawing choices for hyperparameters such as the number of layers, blocks, and channels for hidden layers. After training, we evaluated the performance of every model by recording the win percentage of an MCTS agent using 40 iterations per move with the model, versus a standard untrained UCT (Browne et al., 2012) agent with 800 iterations per move. These win percentages are depicted in Figure 2 . In the majority of cases, ResConvConvLogitPoolModel -a fully convolutional model with global pooling -is among the strongest architectures. Fully convolutional models generally outperform ones with dense layers, and models with global pooling generally outperform those without global pooling. This suggests that using such architectures can be beneficial in and of itself, and their use to facilitate transfer learning does not lead to a sacrifice in baseline performance.
4.2. Evaluation of Transfer Learning
All transfer learning experiments discussed below used the ResConvConvLogitPoolModelV2 architecture from Polygames (Cazenave et al., 2020) . All models were trained for 20 hours on 8 GPUs and 80 CPU cores, using 1 server for training and 7 clients for the generation of self-play games.
4.2.1. TRANSFER BETWEEN GAME VARIANTS
We selected a set of nine different board games, as implemented in Ludii, and for each of them consider a few different variants. The smallest number of variants for a single game is 2, and the largest number of variants for a single game is 6. In most cases, the different variants are simply different board sizes. For example, we consider Gomoku played on 9×9, 13×13, 15×15, and 19×19 boards as four different variants of Gomoku. We also include some cases where board shapes change (i.e., Breakthrough played on square boards as well as hexagonal boards), "small" changes in rules (i.e., Broken Line played with a goal line length of 3, 4, 5, or 6), and "large" changes in rules (i.e., Hex with the standard win condition and Misère Hex with an inverted win condition). Details on all the games and game variants used are provided in Appendix C of the supplementary material.
We trained a separate model for every variant of each of these games, and within each game, transferred models from all variants to all other variants. We evaluate zeroshot transfer performance for a source domain S and target domain T by reporting the win percentage of the model trained in S against the model that was trained in T , over 300 evaluation games per (S, T ) tuple running in T . the neural network passes are more expensive, the game logic in Ludii is more expensive, and episodes often tend to last for a higher number of turns when played on larger boards. Hence, all data points in Figure 3 have a ratio ≤ 1.0, and all data points in Figure 4 have a ratio ≥ 1.0.
When transferring a model that was trained on a large board to a small board (Figure 3 ), zero-shot win percentages tend to be below 50%, but frequently still above 0%. This suggests that training on a larger board than the one we intend to play on does not outperform simply training on the correct board directly, but it often still produces a capable model that can win a non-trivial number of games. When transferring a model that was trained on a small board to a large board (Figure 4 ), we also frequently obtain win percentages above 50%, even reaching up to 100%, against models that were trained directly on the board used for evaluation.
Zero-shot transfer between variants with larger differences, such as modified board shapes or changes in win conditions, only leads to win percentages significantly above 0% in a few cases. These results, as well as more detailed tables, are presented in Appendix D of the supplementary material.
For every model transferred from a source domain S to a target domain T as described above, we train it under identical conditions as the initial training runs -for an additional 20 hours -to evaluate the effect of using transfer for initialisation of the network. Figure 5 depicts scatterplots of win percentages for four distinct cases; transfer to variants with smaller board sizes, with larger board sizes, with different board shapes, and with different win conditions. There are many cases of win percentages close to 50%, which can be interpreted as cases where transfer neither helped nor hurt final performance, and many cases with higher win percentages -which can be interpreted as cases where transfer increased final performance in comparison to training from scratch. We observe a small number of cases, especially when S has a larger board than T , or has different win conditions, in which there is clear negative transfer (Zhang et al., 2020) and the final performance is still closer to 0% even after fine-tuning on T . More detailed results are provided in Appendix E of the supplementary material.
4.2.2. TRANSFER BETWEEN DIFFERENT GAMES
For our final set of experiments, we collected four sets of games, and within each set carried out similar experiments as described above -this time transferring models between distinct games, rather than game variants. The first set consists of six different Line Completion Games; in each of these games the win condition is to create a line of n pieces, but the games differ in aspects such as the value of n, board sizes and shapes, move rules, loss conditions, etc. We evaluate transfer from each of those games, to each of these games. The second set consists of four Shogi Variants: we include Hasami Shogi, Kyoto Shogi, Minishogi, and Shogi, and evaluate transfer from and to each of them. In the third set we evaluate transfer from each of four variants of Broken Line, to each of the six line completion games. Broken Line is a custom-made line completion game where only diagonal lines count towards the win condition, whereas the standard line completion games allow for orthogonal lines. In the fourth set, we evaluate transfer from each of five variants of Diagonal Hex, to each of six variants of Hex. Diagonal Hex only considers diagonal connections for the win condition of Hex, whereas the Hex variants only consider orthogonal connections. Appendix C of the supplementary material provides more details on all the games and variants.
In most cases, zero-shot win percentages for models transferred between distinct games are close to 0%. We observe some success with zero-shot win percentages greater than 30% for transfer from several different line completion games to Connect 6, zero-shot win percentages between 20% and 50% for transfer from three different Shogi vari- ants to Hasami Shogi, as well as a win percentage of 97% for zero-shot transfer from Minishogi to Shogi. Appendix F of the supplementary material contains more detailed results.
Figure 6 depicts win percentages for transferred models after they received an additional 20 hours of fine-tuning time on T . Most notably for various cases of transfer from Diagonal Hex to Hex, there is a high degree of negative transfer, with many win percentages far below 50% even after finetuning. It may be that the differences in connectivity rules are too big to allow for consistently successful transfer. The difficulties in transfer may also be due to large differences in the distributions of outcomes, resulting in large mismatches for the value head; ties are a common result in some variants of Diagonal Hex, but impossible in Hex. In particular for line completion games, transfer appears to be generally successful; there are no severe cases of negative transfer, and a significant amount with strong positive transfer.
5. Conclusions
In this paper, we explored the ability to transfer fully convolutional networks with global pooling, trained using an AlphaZero-like approach, between variants of board games, and distinct board games. Firstly, we compared the performance of such architectures to various others outside of a transfer learning setting, and demonstrated them to be among the top-performing architectures in a variety of board games: fully convolutional nets bring a strong improvement in particular for large board games, and global pooling and U-nets provide slight improvements (Fig. 1 ). Secondly, we explored how to transfer parameters of such networks between different games with different state and action representations in the Ludii general game system. We evaluated zero-shot transfer performance, as well as the performance of transferred models after additional fine-tuning in the target domain, for a wide variety of source and target games and game variants in Ludii. We find several cases where even zero-shot transfer is highly successful -especially when transferring from smaller games to larger ones. We also observe a significant number of cases of beneficial transfer after fine-tuning, even when the source and target domains have more significant differences than just changes in board size or shape (see Figure 6 ). Finally, we find some cases with clear negative transfer, even after fine-tuning, which point to avenues for future research.
A. Directly Transferable Ludii State Channels
Most state channels in Ludii's tensor representations (Soemers et al., 2021) are directly transferable between games, in the sense that they encode semantically similar data for any game in which they are present, and can be modelled as channels that always have only values of 0 in any game that they are not present in. These channels are briefly described here:
• A channel encoding the height of a stack of pieces for every position (only present in games that allow for pieces of more than a single piece type to stack on the same position).
• A channel encoding the number of pieces per position (only present in games that allow multiple pieces of the same type to form a pile on the same position).
• Channels encoding the (typically monetary) "amount" value per player.
• Binary channels encoding whether a given player is the current player to move.
• Channels encoding "local state" values per position (for instance used to memorise whether pieces moved to determine the legality of castling in Chess).
• A channel encoding whether or not players have swapped roles.
• Channels encoding "from" and "to" positions of the last and second-to-last moves.
B. Details on Experimental Setup
For all training runs for transfer learning experiments, the following command-line arguments were supplied to the train command of Polygames (Cazenave et al., 2020 ):
• --num_game 2: Affects the number of threads used to run games per self-play client process.
• --epoch_len 256: Number of training batches per epoch.
• --batchsize 128: Batch size for model training.
• --sync_period 32: Affects how often models are synced.
• --num_rollouts 400: Number of MCTS iterations per move during self-play training.
• --replay_capacity 100000: Capacity of replay buffer.
• --replay_warmup 9000: Minimum size of replay buffer before training starts.
• --model_name "ResConvConvLogitPoolModelV2": Type of architecture to use (a fully convolutional architecture with global pooling).
• --bn: Use of batch normalization (Ioffe & Szegedy, 2015) .
• --nnsize 2: A value of 2 means that hidden convolutional layers each have twice as many channels as the number of channels for the state input tensors.
• --nb_layers_per_net 6: Number of convolutional layers per residual block.
• --nb_nets 10: Number of residual blocks.
• --tournament_mode=true: Use the tournament mode of Polygames to select checkpoints to play against in self-play.
• --bsfinder_max_bs=800: Upper bound on number of neural network queries batched together during inference (we used a lower value of 400 to reduce memory usage in Breakthrough, Hasami Shogi, Kyoto Shogi, Minishogi, Shogi, and Tobi Shogi). (b) A "diagonal" line of 4 on the hexagonal board is a win for the red player in Broken Line.
(c) A chain of "diagonally" connected pieces on the hexagonal board is a win for the red player in Diagonal Hex. 
D. Detailed Results -Zero-shot Transfer Between Game Variants
Tables 13-21 provide detailed results for all evaluations of zero-shot transfer between variants within each out of nine different games. 
E. Detailed Results -Transfer Between Game Variants With Fine-tuning
Tables 22-30 provide detailed results for all evaluations of transfer performance after fine-tuning, for transfer between variants within each out of nine different games. Models are trained for 20 hours on the source domain, followed by 20 hours on the target domain, and evaluated against models trained for 20 hours only on the target domain. Tables 31 32 33 34 35 36 37 38 39 provide additional results for a similar evaluation where we reinitialised all the parameters of the final convolutional layers before policy and value heads prior to fine-tuning. The basic idea behind this experiment was that it would lead to a more random, less biased policy generating experience from self-play at the start of a fine-tuning process, and hence may improve fine-tuning transfer in cases where full transfer produces a poor initial policy. Overall we did not observe many major changes in transfer performance. 
G. Detailed Results -Transfer Between Games With Fine-tuning
Tables 44-47 provide detailed results for evaluations of transfer performance after fine-tuning, where source domains are different games from target domains (not just different variants). Note that in these cases, the two models that play against each other do not always have exactly the same number of trainable parameters. For hidden convolutional layers, we always use twice as many channels as the number of channels in a game's state tensor representation, and this is not modified when transferring to a new domain. This means that if a source domain has a greater number of channels in its state tensor representation than the target domain, the transferred model will also still use more channels in its hidden convolutional layers than the baseline model, and vice versa when the source domain has fewer state channels. Tables 48-51 provide additional results where we adjust the number of channels of hidden convolutional layers when transferring models, prior to fine-tuning, for a more "fair" evaluation in terms of network size. 
https://github.com/facebookincubator/ Polygames
References
[1]Anthony T, Tian Z, Barber D, Guyon I, Luxburg UV, Bengio S, et al.. Thinking fast and slow with deep learning and tree search. Advances in Neural Information Processing Systems 2017;30:5360--5370.
[2]Banerjee B, Stone P. General game learning using knowledge transfer. The 20th International Joint Conference on Artificial Intelligence 2007;:672--677.
[3]Bou Ammar H. Automated transfer in reinforcement learning.  2013;:.
[4]Bou Ammar H, Eaton E, Taylor ME, Mocanu DC, Driessens K, Weiss G, et al.. An automated measure of mdp similarity for transfer in reinforcement learning. Proceedings of the Interactive Systems Workshop at the American Association of Artificial Intelligence (AAAI) 2014;:31--37.
[5]Browne C, Powley E, Whitehouse D, Lucas S, Cowling PI, Rohlfshagen P, et al.. A survey of Monte Carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games 2012;4:1--49.
[6]Browne C, Stephenson M, Piette É, Soemers DJ N J. A practical introduction to the ludii general game system. Advances in Computer Games 2019;12516:.
[7]Cazenave T. Mobile networks for computer go. IEEE Transactions on Computational Intelligence and AI in Games 2020;:.
[8]Cazenave T, Chen Y.-C, Chen G, Chen S.-Y, Chiu X.-D, Dehos J, et al.. Improved zero learning. ICGA Journal 2020;:.
[9]Coulom R. Efficient selectivity and backup operators in Monte-Carlo tree search. Computers and Games 2007;4630:72--83.
[10]He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016;:770--778.
[11]Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning 2015;:448--456.
[12]Kocsis L, Szepesvári C. Bandit based Monte-Carlo planning. Machine Learning: ECML 2006 None;4212:282--293.
[13]Kuhlmann G, Stone P. Graph-based domain mapping for transfer learning in general games. Machine Learning: ECML 2007 2007;:188--200.
[14]Lazaric A. Transfer in reinforcement learning: a framework and a survey. Reinforcement Learning 2012;12:143--173.
[15]Lecun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, et al.. Backpropagation applied to handwritten zip code recognition. Neural Computation 1989;1:541--551.
[16]Lecun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521:436--444.
[17]Love N, Hinrichs T, Haley D, Schkufza E, Genesereth M. General game playing: Game description language specification.  2008;:.
[18]Marcus G. Innateness, alphazero, and artificial intelligence.  2018;:.
[19]Morandin F, Amato G, Gini R, Metta C, Parton M, Pascutto G.-C. SAI: a sensible artificial intelligence that plays Go. Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN) 2019;:.
[20]Nair V, Hinton G. Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference on Machine Learning 2010;:807--814.
[21]Nichol A, Pfau V, Hesse C, Klimov O, Schulman J. Gotta learn fast: A new benchmark for generalization in rl.  2018;:.
[22]Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, et al.. An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems 2019;32:8024--8035.
[23]Piette É, Soemers DJ N J, Stephenson M, Sironi CF, Winands MH M, Browne C. Ludii -the ludemic general game system. Proceedings of the 24th European Conference on Artificial Intelligence (ECAI 2020 2020;325:411--418.
[24]Piette É, Browne C, Soemers DJ N J. Ludii game logic guide.  2021;:.
[25]Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention -MICCAI 2015 2015;:234--241.
[26]Shelhamer E, Long J, Darrell T. Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 2017;39:640--651.
[27]Silver D, Huang A, Maddison C, Guez A, Sifre L, Van Den Driessche G, et al.. Mastering the game of Go with deep neural networks and tree search. Nature 2016;529:484--489.
[28]Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, et al.. Mastering the game of Go without human knowledge. Nature 2017;550:354--359.
[29]Silver D, Hubert T, Schrittwieser J, Antonoglou I, Lai M, Guez A, et al.. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 2018;362:1140--1144.
[30]Soemers DJ N J, Mella V, Browne C, Teytaud O. Deep learning for general game playing with ludii and polygames.  2021;:.
[31]Taylor ME, Stone P. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research 2009;10:1633--1685.
[32]Tian Y, Ma J, Gong Q, Sengupta S, Chen Z, Pinkerton J, et al.. ELF OpenGo: An analysis and open reimplementation of AlphaZero. Proc. 36th Int. Conf. Mach. Learn. (ICML) 2019;:6244--6253.
[33]Wu DJ. Accelerating self-play learning in go.  2019;:.
[34]Zhang K, Shasha D. Simple fast algorithms for the editing distance between trees and related problems. In SIAM Journal on Computing 1989;18:1245--1262.
[35]Zhang W, Deng L, Zhang L, Wu D. Overcoming negative transfer: A survey.  2020;:.
[36]Zhu Z, Lin K, Zhou J. Transfer learning in deep reinforcement learning: A survey.  2020;:.
Transfer of Fully Convolutional Policy-Value Networks Between Games and Game Variants
https://arxiv.org/abs/2102.12375