Accelerated Training via Incrementally Growing Neural Networks using
  Variance Transfer and Learning Rate Adaptation
We develop an approach to efficiently grow neural networks, within which
parameterization and optimization strategies are designed by considering their
effects on the training dynamics. Unlike existing growing methods, which follow
simple replication heuristics or utilize auxiliary gradient-based local
optimization, we craft a parameterization scheme which dynamically stabilizes
weight, activation, and gradient scaling as the architecture evolves, and
maintains the inference functionality of the network. To address the
optimization difficulty resulting from imbalanced training effort distributed
to subnetworks fading in at different growth phases, we propose a learning rate
adaption mechanism that rebalances the gradient contribution of these separate
subcomponents. Experimental results show that our method achieves comparable or
better accuracy than training large fixed-size models, while saving a
substantial portion of the original computation budget for training. We
demonstrate that these gains translate into real wall-clock training speedups.
Xin Yuan, Pedro Savarese, Michael Maire
# Accelerated Training via Incrementally

###### Abstract

We develop an approach to efficiently grow neural networks, within which parameterization and optimization strategies are designed by considering their effects on the training dynamics. Unlike existing growing methods, which follow simple replication heuristics or utilize auxiliary gradient-based local optimization, we craft a parameterization scheme which dynamically stabilizes weight, activation, and gradient scaling as the architecture evolves, and maintains the inference functionality of the network. To address the optimization difficulty resulting from imbalanced training effort distributed to subnetworks fading in at different growth phases, we propose a learning rate adaption mechanism that rebalances the gradient contribution of these separate subcomponents. Experimental results show that our method achieves comparable or better accuracy than training large fixed-size models, while saving a substantial portion of the original computation budget for training. We demonstrate that these gains translate into real wall-clock training speedups.

## 1 Introduction

Modern neural network design typically follows a "larger is better" rule of thumb, with models consisting of millions of parameters achieving impressive generalization performance across many tasks, including image classification [23; 33; 31; 47], object detection [13; 27; 11], semantic segmentation [28; 3; 25] and machine translation [35; 7]. Within a class of network architecture, deeper or wider variants of a base model typically yield further improvements to accuracy. Residual networks (ResNets) [15] and wide residual networks [46] illustrate this trend in convolutional neural network (CNN) architectures. Dramatically scaling up network size into the billions of parameter regime has recently revolutionized transformer-based language modeling [35; 7; 1].

The size of these models imposes prohibitive training costs and motivates techniques that offer cheaper alternatives to select and deploy networks. For example, hyperparameter tuning is notoriously expensive as it commonly relies on training the network multiple times, and recent techniques aim to circumvent this by making hyperparameters transferable between models of different sizes, allowing them to be tuned on a small network prior to training the original model once [42].

Our approach incorporates these ideas, but extends the scope of transferability to include the parameters of the model itself. Rather than view training small and large models as separate events, we grow a small model into a large one through many intermediate steps, each of which introduces additional parameters to the network. Our contribution is to do so in a manner that preserves the function computed by the model at each growth step (functional continuity) and offers stable training dynamics, while also saving compute by leveraging intermediate solutions. More specifically, we
use partially trained subnetworks as scaffolding that accelerates training of newly added parameters, yielding greater overall efficiency than training a large static model from scratch.

Competing recent efforts to grow deep models from simple architectures [4; 24; 5; 26; 40; 38; 39; 45; 10] draw inspiration from other sources, such as the progressive development processes of biological brains. In particular, Net2Net [4] grows the network by randomly splitting learned neurons from previous phases. This replication scheme, shown in Figure 1(a) is a common paradigm for most existing methods. Gradient-based methods [39; 40] determine which neurons to split and how to split them by solving a combinatorial optimization problem with auxiliary variables.

At each growth step, naive random initialization of new weights destroys network functionality and may overwhelm any training progress. Weight rescaling with a static constant from a previous step is not guaranteed to be maintained as the network architecture evolves. Gradient-based methods outperform these simple heuristics but require additional training effort in their parameterization schemes. Furthermore, all existing methods use a global LR scheduler to govern weight updates, ignoring the discrepancy among subnetworks introduced in different growth phases. The gradient itself and other parameterization choices may influence the correct design for scaling weight updates.

We develop a growing framework around the principles of enforcing transferability of parameter settings from smaller to larger models (extending [42]), offering functional continuity, smoothing optimization dynamics, and rebalancing learning rates between older and newer subnetworks. Figure 1(b) illustrates key differences with prior work. Our core contributions are:

* **Parameterization using Variance Transfer:** We propose a parameterization scheme accounting for the variance transition among networks of smaller and larger width in a single training process. Initialization of new weights is gradient-free and requires neither additional memory nor training.
* **Improved Optimization with Learning Rate Adaptation:** Subnetworks trained for different lengths have distinct learning rate schedules, with dynamic relative scaling driven by weight norm statistics.
* **Better Performance and Broad Applicability:** Our method not only trains networks fast, but also yields excellent generalization accuracy, even outperforming the original fixed-size models. Flexibility in designing a network growth curve allows choosing different trade-offs between training resources and accuracy. Furthermore, adopting an adaptive batch size schedule provides acceleration in terms of wall-clock training time. We demonstrate results on image classification and machine translation tasks, across various network architectures.

## 2 Related Work

**Network Growing.** A diverse range of techniques train models by progressively expanding the network architecture [37; 9; 5; 38; 45]. Within this space, the methods of [4; 26; 40; 39; 10] are most relevant to our focus - incrementally growing network width across multiple training stages. Net2Net [4] proposes a gradient-free neuron splitting scheme via replication, enabling knowledge transfer from previous training phases; initialization of new weights follows simple heuristics. [26]'s Splitting approach derives a gradient-based scheme for duplicating neurons by formulating a combinatorial optimization problem. FireFly [39] gains flexibility by also incorporating brand new neurons. Both methods improve Net2Net's initialization scheme by solving an optimization problem

Figure 1: Dynamic network growth strategies. Different from (a) which rely on either splitting [4; 26; 40] or adding neurons with auxiliary local optimization [39; 10], our initialization (b) of new neurons is random but function-preserving. Additionally, our separate learning rate scheduler governs weight updating to address the discrepancy in total accumulated training between different growth stages.
with auxiliary variables, at the cost of extra training effort. GradMax [10], in consideration of training dynamics, performs initialization via solving a singular value decomposition (SVD) problem.

**Neural Architecture Search (NAS) and Pruning.** Another subset of methods mix growth with dynamic reconfiguration aimed at discovering or pruning task-optimized architectures. Network Morphism [37] searches for efficient networks by extending layers while preserving the parameters. Autogrow [38] takes an AutoML approach governed by heuristic growing and stopping policies. [45] combine learned pruning with a sampling strategy that dynamically increases or decreases network size. Unlike these methods, we focus on the mechanics of growth when the target architecture is known, addressing the question of how to best transition weight and optimizer state to continue training an incrementally larger model. NAS and pruning are orthogonal to, though potentially compatible with, the technical approach we develop.

**Hyperparameter Transfer.**[43; 30; 17] explore transferrable hyperparameter (HP) tuning. The recent Tensor Program (TP) work of [41] and [42] focuses on zero-shot HP transfer across model scale and establishes a principled network parameterization scheme to facilitate HP transfer. This serves as an anchor for our strategy, though, as Section 3 details, modifications are required to account for dynamic growth.

**Learning Rate Adaptation.** Surprisingly, the existing spectrum of network growing techniques utilize relatively standard learning rate schedules and do not address potential discrepancy among subcomponents added at different phases. While general-purpose adaptive optimizers, _e.g._, AdaGrad [8], RMSProp [34], Adam [21], or AvaGrad [32], might ameliorate this issue, we choose to explicitly account for the discrepancy. As layer-adaptive learning rates (LARS) [12; 44] benefit in some contexts, we explore further learning rate adaption specific to both layer and growth stage.

## 3 Method

### Parameterization and Optimization with Growing Dynamics

**Functionality Preservation**. We grow a network's capacity by expanding the width of computational units (_e.g.,_ hidden dimensions in linear layers, filters in convolutional layers). To illustrate our scheme, consider a 3-layer fully-connected network with ReLU activations \(\phi\) at a growing stage \(t\):

\[\mathbf{u}_{t}=\phi(\mathbf{W}_{t}^{x}\mathbf{x})\quad\mathbf{h}_{t}=\phi(\mathbf{W}_{t}^{u}\mathbf{u} _{t})\quad\quad\mathbf{y}_{t}=\mathbf{W}_{t}^{h}\mathbf{h}_{t}\,, \tag{1}\]

where \(\mathbf{x}\in\mathbb{R}^{C^{x}}\) is the network input, \(\mathbf{y}_{t}\in\mathbb{R}^{C^{y}}\) is the output, and \(\mathbf{u}_{t}\in\mathbb{R}^{C_{t}^{u}},\mathbf{h}_{t}\in\mathbb{R}^{C_{t}^{h}}\) are the hidden activations. In this case, \(\mathbf{W}_{t}^{x}\) is a \(C_{t}^{u}\times C^{x}\) matrix, while \(\mathbf{W}_{t}^{u}\) is \(C_{t}^{h}\times C_{t}^{u}\) and \(\mathbf{W}_{t}^{h}\) is \(C^{y}\times C_{t}^{h}\). Our growing process operates by increasing the dimensionality of each hidden state, _i.e._, from \(C_{t}^{u}\) and \(C_{t}^{h}\) to \(C_{t+1}^{u}\) and \(C_{t+1}^{h}\), effectively expanding the size of the parameter tensors for the next growing stage \(t+1\). The layer parameter matrices \(\mathbf{W}_{t}\) have their shapes changed accordingly and become \(\mathbf{W}_{t+1}\). Figure 2 illustrates the process for initializing \(\mathbf{W}_{t+1}\) from \(\mathbf{W}_{t}\) at a growing step.1

Footnote 1: We defer the transformation between \(\mathbf{W}_{t}\) and \(\mathbf{W}_{t}^{{}^{\prime}}\) to the next subsection. It involves rescaling by constant factors, does not affect network functionality, and is omitted in Eq. 1- 4 for simplicity.

Following Figure 2(a), we first expand \(\mathbf{W}_{t}^{x}\) along the output dimension by adding two copies of new weights \(\mathbf{V}_{t}^{x}\) of shape \(\frac{C_{t+1}^{u}-C_{t}^{u}}{2}\times C^{x}\), generating new features \(\phi(\mathbf{V}_{t}^{x}\mathbf{x})\). The first set of activations become

\[\mathbf{u}_{t+1}=\operatorname{concat}\left(\mathbf{u}_{t},\phi(\mathbf{V}_{t}^{x}\mathbf{x}),\phi(\mathbf{V}_{t}^{x}\mathbf{x})\right)\,, \tag{2}\]

where \(\operatorname{concat}\) denotes the concatenation operation. Next, we expand \(\mathbf{W}_{t}^{u}\) across both input and output dimensions, as shown in Figure 2(b). We initialize new weights \(\mathbf{Z}_{t}^{h}\) of shape \(C_{t}^{h}\times\frac{C_{t+1}^{u}-C_{t}^{u}}{2}\) and add to \(\mathbf{W}_{t}^{u}\) two copies of it with different signs: \(+\mathbf{Z}_{t}^{u}\) and \(-\mathbf{Z}_{t}^{u}\). This preserves the output of the layer since \(\phi(\mathbf{W}_{t}^{u}\mathbf{u}_{t}+\mathbf{Z}_{t}^{u}\phi(\mathbf{V}_{t}^{x}\mathbf{x})+(-\mathbf{Z} _{t}^{u})\phi(\mathbf{V}_{t}^{x}\mathbf{x}))=\phi(\mathbf{W}_{t}^{u}\mathbf{u}_{t})=\mathbf{h}_{t}\,\). We then add two copies of new weights \(\mathbf{V}_{t}^{u}\), which has shape \(\frac{C_{t+1}^{u}-C_{t}^{h}}{2}\times C_{t+1}^{u}\), yielding activations

\[\mathbf{h}_{t+1}=\operatorname{concat}(\mathbf{h}_{t},\phi(\mathbf{V}_{t}^{u}\mathbf{u}_{t+1} ),\phi(\mathbf{V}_{t}^{u}\mathbf{u}_{t+1}))\,. \tag{3}\]

We similarity expand \(\mathbf{W}_{t}^{h}\) with new weights \(\mathbf{Z}_{t}^{h}\) to match the dimension of \(\mathbf{h}_{t+1}\), as shown in Figure 2(c). The network's output after the growing step is:

\[\mathbf{y}_{t+1} =\mathbf{W}_{t}^{h}\mathbf{h}_{t}+\mathbf{Z}_{t}^{h}\phi(\mathbf{V}_{t}^{u}\mathbf{u}_ {t+1})+(-\mathbf{Z}_{t}^{h})\phi(\mathbf{V}_{t}^{u}\mathbf{u}_{t+1}) \tag{4}\] \[=\mathbf{W}_{t}^{h}\mathbf{h}_{t}=\mathbf{y}_{t}\,,\]
which preserves the original output features in Eq. 1. Appendix provides illustrations for more layers.

**Weights Initialization with Variance Transfer (VT).**[42] investigate weight scaling with width at initialization, allowing hyperparameter transfer by calibrating variance across model size. They modify the variance of output layer weights from the commonly used \(\frac{1}{\text{fan}_{in}}\) to \(\frac{1}{\text{fan}_{in}^{2}}\). We adopt this same correction for the added weights with new width: \(\mathbf{W}^{h}\) and \(\mathbf{Z}^{h}\) are initialized with variances of \(\frac{1}{C_{t}^{\text{n}^{2}}}\) and \(\frac{1}{C_{t+1}^{h^{2}}}\).

However, this correction considers training differently-sized models separately, which fails to accommodate the training dynamics in which width grows incrementally. Assuming that the weights of the old subnetwork follow \(\mathbf{W}_{t}^{h}\sim\mathcal{N}(0,\frac{1}{C_{t}^{h^{2}}})\) (which holds at initialization), we make them compatible with new weight tensor parameterization by rescaling it with the \(\text{fan}_{in}\) ratio as \(\mathbf{W}_{t}^{h^{\prime}}=\mathbf{W}_{t}^{h}\cdot\frac{C_{t}^{h}}{C_{t+1}^{h^{2}}}\). See Table 1 (top). Appendix provides detailed analysis.

This parameterization rule transfers to modern CNNs with batch normalization (BN). Given a weight scaling ratio of \(c\), the running mean \(\mu\) and variance \(\sigma\) of BN layers are modified as \(c\mu\) and \(c^{2}\sigma\), respectively.

**Stage-wise Learning Rate Adaptation (LRA).** Following [42], we employ a learning rate scaling factor of \(\propto\frac{1}{\text{fan}_{in}}\) on the output layer when using SGD, compensating for the initialization scheme. However, subnetworks from different growth stages still share a global learning rate, though they have trained for different lengths. This may cause divergent behavior among the corresponding weights, making the training iterations after growing sensitive to the scale of the newly-initialized weights. Instead of adjusting newly added parameters via local optimization [39; 10], we govern the update of each subnetwork in a stage-wise manner.

Let \(\mathcal{W}_{t}\) denote the parameter variables of a layer at a growth stage \(t\), where we let \(\mathbf{W}_{t}\) and \(\mathbf{W}_{t}^{\prime}\) correspond to the same set of variables such that \(\mathcal{W}_{t+1}\setminus\mathcal{W}_{t}\) denotes the new parameter variables whose values are initialized with \(\mathbf{Z}_{t}\) and \(\mathbf{V}_{t}\). Moreover, let \(\mathbf{W}_{\Delta k}\) and \(\mathbf{G}_{\Delta k}\) denote the values and gradients of \(\mathcal{W}_{k}\setminus\mathcal{W}_{k-1}\). We adapt the learning rate used to update each sub-weight \(\mathbf{W}_{\Delta k}\), for \(0\leq k\leq t\), as follows:

\[\eta_{k}=\eta_{0}\cdot\frac{f(\mathbf{W}_{\Delta k})}{f(\mathbf{W}_{\Delta 0})}\quad\mathbf{W}_{ \Delta k}\leftarrow\mathbf{W}_{\Delta k}-\eta_{k}\mathbf{G}_{\Delta k}\,, \tag{5}\]

where \(\eta_{0}\) is the base learning rate, \(f\) is a function that maps subnetworks of different stages to corresponding train-time statistics, and \(\mathbf{W}_{\Delta 0}\) are the layer's parameter variables at the first growth stage. Table 1 (bottom) summarizes our LR adaptation rule for SGD when \(f\) is instantiated as weight norm, providing an stage-wise extension to the layer-wise adaptation method LARS [12], i.e., \(LR\propto||\mathbf{W}||\). Alternative heuristics are possible; see Appendix.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & & Input Layer & Hidden Layer & Output Layer \\ \hline \multirow{2}{*}{Init.} & Old Re-scaling & 1 & \(\sqrt{C_{t}^{u}/C_{t+1}^{u}}\) & \(C_{t}^{h}/C_{t+1}^{h}\) \\  & New Init. & \(1/C_{t}^{x}\) & \(1/C_{t+1}^{u}\) & \(1/(C_{t+1}^{h})^{2}\) \\ \hline \multirow{2}{*}{Adapt.} & 0-th Stage & 1 & 1 & \(1/C_{0}\) \\  & \(t\)-th Stage & \(\frac{||\mathbf{W}_{\Delta k}^{\star}||}{\|\mathbf{W}_{\Delta 0}^{\star}\|}\) & \(\frac{||\mathbf{W}_{\Delta k}^{\star}||}{\|\mathbf{W}_{\Delta 0}^{\star}\|}\) & \(\frac{||\mathbf{W}_{\Delta k}^{\star}||}{\|\mathbf{W}_{\Delta 0}^{\star}\|}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Parameterization and optimization transition for different layers during growing. \(C_{t}\) and \(C_{t+1}\) denote the input dimension before and after a growth step.

Figure 2: Initialization scheme. In practice, we also add noise to the expanded parameter sets for symmetry breaking.
### Flexible and Efficient Growth Scheduler

We train the model for \(T_{total}\) epochs by expanding the channel number of each layer to \(C_{final}\) across \(N\) growth phases. Existing methods [26, 39] fail to derive a systemic way for distributing training resources across a growth trajectory. Toward maximizing efficiency, we experiment with a coupling between model size and training epoch allocation.

**Architectural Scheduler.** We denote initial channel width as \(C_{0}\) and expand exponentially:

\[C_{t}=\begin{cases}C_{t-1}+\lfloor p_{c}C_{t-1}\rceil_{2}&\text{if}\quad t<N-1 \\ C_{final}&\text{if}\quad t=N-1\end{cases} \tag{6}\]

where \(\lfloor\cdot\rceil_{2}\) rounds to the nearest even number and \(p_{c}\) is the growth rate between stages.

**Epoch Scheduler.** We denote number of epochs assigned to \(t\)-th training stage as \(T_{t}\), with \(\sum_{t=0}^{N-1}T_{t}=T_{total}\). We similarly adapt \(T_{t}\) via an exponential growing scheduler:

\[T_{t}=\begin{cases}T_{t-1}+\lfloor p_{t}T_{t-1}\rceil\quad\text{if}\quad t<N-1 \\ T_{total}-\sum_{i=0}^{N-2}T_{i}\quad\text{if}\quad t=N-1\end{cases} \tag{7}\]

**Wall-clock Speedup via Batch Size Adaptation.** Though the smaller architectures in early growth stages require fewer FLOPs, hardware capabilities may still restrict practical gains. When growing width, in order to ensure that small models fully utilize the benefits of GPU parallelism, we adapt the batch size along with the exponentially-growing architecture in a reverse order:

\[B_{t-1}=\begin{cases}B_{base}&\text{if}\quad t=N\\ B_{t}+\lfloor p_{b}B_{t}\rceil\quad\text{if}\quad t<N\end{cases} \tag{8}\]

where \(B_{base}\) is the batch size of the large baseline model. Algorithm 1 summarizes our full method.

```
Input: Data \(\mathbf{X}\), labels \(\mathbf{Y}\), task loss \(L\) Output: Grown model \(\mathcal{W}\)  Initialize: \(\mathcal{W}_{0}\) with \(C_{0}\), \(T_{0}\), \(B_{0}\), \(\eta_{0}\) for\(\mathbf{t}=0\)to \(N-1\)do if\(t>0\)then  Init. \(S_{n}\) from \(S_{n-1}\) using VT in Table 1.  Update \(C_{t}\) and \(T_{t}\) using Eq. 6 and Eq. 7.  Update \(B_{t}\) using Eq. 8 (optional) \(\text{Iter}_{total}=T_{t}*len(X)//B_{t}\) endif for\(\mathbf{iter}=1\)to\(\mathbf{Iter}_{total}\)do  Forward and calculate \(l=L(\mathcal{W}_{t}(\mathbf{x}),\mathbf{y}))\).  Back propagation with \(l\).  Update each sub-component using Eq. 5. endfor endfor  return \(\mathcal{W}_{N-1}\)
```

**Algorithm 1** : Growing using Var. Transfer and Learning Rate Adapt. with Flexible Scheduler

## 4 Experiments

We evaluate on image classification and machine translation tasks. For image classification, we use CIFAR-10 [22], CIFAR-100 [22] and ImageNet [6]. For the neural machine translation, we use the IWSLT'14 dataset [2] and report the BLEU [29] score on German to English (De-En) translation.

**Large Baselines via Fixed-size Training.** We use VGG-11 [33] with BatchNorm [20], ResNet-20 [16], MobileNetV1 [18] for CIFAR-10 and VGG-19 with BatchNorm, ResNet-18, MobileNetV1 for CIFAR-100. We follow [19] for data augmentation and processing, adopting random shifts/mirroring and channel-wise normalization. CIFAR-10 and CIFAR-100 models are trained for 160 and 200 epochs respectively, with a batch size of 128 and initial learning rate (LR) of 0.1 using SGD. We adopt a cosine LR schedule and set the weights decay and momentum as \(5\)e-\(4\) and \(0.9\). For ImageNet, we train the baseline ResNet-50 and MobileNetV1 [18] using SGD with batch sizes of 256 and 512, respectively. We adopt the same data augmentation scheme as [14], the cosine LR scheduler with initial LR of 0.1, weight decay of 1e-\(4\) and momentum of \(0.9\).

For IWSLT'14, we train an Encoder-Decoder Transformer (6 attention blocks each) [35]. We set width as \(d_{model}=1/4d_{ffn}=512\), the number of heads \(n_{head}=8\) and \(d_{k}=d_{q}=d_{v}=d_{model}/n_{head}=64\). We train the model using Adam for 20 epochs with learning rate 1e-\(3\) and \((\beta_{1},\beta_{2})=(0.9,0.98)\). Batch size is 1500 and we use 4000 warm-up iterations.

**Implementation Details.** We compare with the growing methods Net2Net [4], Splitting [26], FireFly split, FireFly [39] and GradMax [10]. In our method, noise for symmetry breaking is 0.001 to the
norm of the initialization. We re-initialize the momentum buffer at each growing step when using SGD while preserving it for adaptive optimizers (e.g., Adam, AvaGrad).

For image classification, we run the comparison methods except GradMax alongside our algorithm for all architectures under the same growing scheduler. For the architecture scheduler, we set \(p_{c}\) as 0.2 and \(C_{0}\) as 1/4 of large baselines in Eq. 6 for all layers and grow the seed architecture within \(N=9\) stages towards the large ones. For epoch scheduler, we set \(p_{t}\) as \(0.2\), \(T_{0}\) as \(8\), \(10\), and \(4\) in Eq. 7 on CIAFR-10, CIFAR-100, and ImageNet respectively. Total training epochs \(T_{total}\) are the same as the respective large fixed-size models. We train the models and report the results averaging over 3 random seeds.

For machine translation, we grow the encoder and decoder layers' widths while fixing the embedding layer dimension for a consistent positional encoding table. The total number of growing stages is 4, each trained for 5 epochs. The initial width is 1/8 of the large baseline (i.e. \(d_{model}=64\) and \(d_{k,q,v}=8\)). We set the growing ratio \(p_{c}\) as 1.0 so that \(d_{model}\) evolves as 64, 128, 256 and 512.

We train all the models on an NVIDIA 2080Ti 12GB GPU for CIFAR-10, CIFAR-100, and IWSLT'14, and two NVIDIA A40 48GB GPUs for ImageNet.

### CIFAR Results

All models grow from a small seed architecture to the full-sized one in 9 stages, each trained for \(\{8,9,11,13,16,19,23,28,33\}\) epochs (\(160\) total) in CIFAR-10, and \(\{10,12,14,17,20,24,29,35,39\}\) (\(200\) total) in CIFAR-100. Net2Net follows the design of growing by splitting via simple neuron replication, hence achieving the same training efficiency as our gradient-free method under the same growing schedule. Splitting and Firely require additional training effort for their neuron selection schemes and allocate extra GPU memory for auxiliary variables during the local optimization stage. This is computationally expensive, especially when growing a large model.

**ResNet-20, VGG-11, and MobileNetV1 on CIFAR-10.** Table 2 shows results in terms of test accuracy and training cost calculated based on overall FLOPs. For ResNet-20, Splitting and Firefly achieve better test accuracy than Net2Net, which suggests the additional optimization benefits neuron selection at the cost of training efficiency. Our method requires only \(54.9\%\) of the baseline training cost and outperforms all competing methods, while yielding only \(0.09p.p\) (percentage points) performance degradation compared to the static baseline. Moreover, our method even outperforms the large fixed-size VGG-11 by \(0.20p.p\) test accuracy, while taking only \(52.91\%\) of its training cost.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{3}{*}{Method} & \multicolumn{3}{c}{ResNet-20} & \multicolumn{3}{c}{VGG-11} & \multicolumn{3}{c}{MobileNetv1} \\ \cline{2-7}  & \multicolumn{1}{c}{\begin{tabular}{c} Train \\ Cost\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Test \\ Accuracy\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Train \\ Cost\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Test \\ Accuracy\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Train \\ Cost\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{
\begin{tabular}{c} Test \\ Accuracy\((\%)\) \\ \end{tabular} } \\ \hline Large Baseline & \(100\) & \(92.62\pm 0.15\) & \(100\) & \(92.14\pm 0.22\) & \(100\) & \(92.27\pm 0.11\) \\ Net2Net & \(\mathbf{54.90}\) & \(91.60\pm 0.21\) & \(\mathbf{52.91}\) & \(91.78\pm 0.27\) & \(\mathbf{53.80}\) & \(90.34\pm 0.20\) \\ Splitting & \(70.69\) & \(91.80\pm 0.10\) & \(63.76\) & \(91.88\pm 0.15\) & \(65.92\) & \(91.50\pm 0.06\) \\ FireFly-split & \(58.47\) & \(91.78\pm 0.11\) & \(56.18\) & \(91.91\pm 0.15\) & \(56.37\) & \(91.56\pm 0.06\) \\ FireFly & \(68.96\) & \(92.10\pm 0.13\) & \(60.24\) & \(92.08\pm 0.16\) & \(62.12\) & \(91.69\pm 0.07\) \\ Ours & \(\mathbf{54.90}\) & \(\mathbf{92.53\pm 0.11}\) & \(\mathbf{52.91}\) & \(\mathbf{92.34\pm 0.15}\) & \(\mathbf{53.80}\) & \(\mathbf{92.01\pm 0.10}\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Growing ResNet-20, VGG-11, and MobileNetV1 on CIFAR-10.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{3}{*}{Method} & \multicolumn{3}{c}{ResNet-18} & \multicolumn{3}{c}{VGG-19} & \multicolumn{3}{c}{MobileNetv1} \\ \cline{2-7}  & \multicolumn{1}{c}{\begin{tabular}{c} Train \\ Cost\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Test \\ Accuracy\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Train \\ Cost\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Test \\ Accuracy\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Train \\ Cost\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Test \\ Accuracy\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Train \\ Cost\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Test \\ Accuracy\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Train \\ Cost\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Test \\ Accuracy\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{\begin{tabular}{c} Test \\ Accuracy\((\%)\) \\ \end{tabular} } & \multicolumn{1}{c}{
\begin{tabular}{c} Test \\ Accuracy\((\%)\) \\ \end{tabular} } \\ \hline Large Baseline & \(100\) & \(78.36\pm 0.12\) & \(100\) & \(72.59\pm 0.23\) & \(100\) & \(72.13\pm 0.13\) \\ Net2Net & \(\mathbf{52.63}\) & \(76.48\pm 0.20\) & \(\mathbf{52.08}\) & \(71.88\pm 0.24\) & \(\mathbf{52.90}\) & \(70.01\pm 0.20\) \\ Splitting & \(68.01\) & \(77.01\pm 0.12\) & \(60.12\) & \(71.96\pm 0.12\) & \(58.39\) & \(70.45\pm 0.10\) \\ FireFly-split & \(56.11\) & \(77.22\pm 0.11\) & \(54.64\) & \(72.19\pm 0.14\) & \(54.36\) & \(70.69\pm 0.11\) \\ FireFly & \(65.77\) & \(77.25\pm 0.12\) & \(57.48\) & \(72.79\pm 0.13\) & \(56.49\) & \(70.99\pm 0.10\) \\ \hline Ours & \(\mathbf{52.63}\) & \(\mathbf{78.12\pm 0.15}\) & \(\mathbf{52.08}\) & \(\mathbf{73.26\pm 0.14}\) & \(\mathbf{52.90}\) & \(\mathbf{71.53\pm 0.13}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Growing ResNet-18, VGG-19, and MobileNetV1 on CIFAR-100.
For MobileNetV1, our method also achieves the best trade-off between training efficiency and test accuracy among all competitors.

**ResNet-18, VGG-19, and MobileNetV1 on CIFAR-100.** We also evaluate all methods on CIFAR-100 using different network architectures. Results in Table 3 show that Firely consistently achieves better test accuracy than Firefly-split, suggesting that adding new neurons provides more flexibility for exploration than merely splitting. Both Firely and our method achieve better performance than the original VGG-19, suggesting that network growing might have an additional regularizing effect. Our method yields the best accuracy and largest training cost reduction.

### ImageNet Results

We first grow ResNet-50 on ImageNet and compare the performance of our method to Net2Net and FireFly under the same epoch schedule: \(\{4,4,5,6,8,9,11,14,29\}\) (\(90\) total) with 9 growing phases. We also grow MobileNetV1 from a small seed architecture, which is more challenging than ResNet-50. We train Net2Net and our method uses the same scheduler as for ResNet-50. We also compare with Firefly-Opt (a variant of FireFly) and GradMax and report their best results from [10]. Note that both methods not only adopt additional local optimization but also train with a less efficient growing scheduler: the final full-sized architecture needs to be trained for a \(75\%\) fraction while ours only requires \(32.2\%\). Table 4 shows that our method dominates all competing approaches.

### IWSLT14 De-En Results

We grow a Transformer from \(d_{model}=64\) to \(d_{model}=512\) within 4 stages, each trained with 5 epochs using Adam. Applying gradient-based growing methods to the Transformer architecture is non-trivial due to their domain specific design of local optimization. As such, we only compare with Net2Net. We also design variants of our method for self-comparison, based on the adaptation rules for Adam in Appendix. As shown in Table 5, our method generalizes well to the Transformer architecture.

### Analysis

**Ablation Study.** We show the effects of turning on/off each of our modifications to the baseline optimization process of Net2Net (1) Growing: adding neurons with functionality preservation. (2) Growing+VT: only applies variance transfer. (3) Growing+RA: only applies LR rate adaptation. (4) Full method. We conduct experiments using both ResNet-20 on CIFAR-10 and ResNet-18 on CIFAR-100. As shown in Table 6, different variants of our growing method not only outperform Net2Net consistently but also reduce the randomness (std. over 3 runs) caused by random replication. We also see that, both RA and VT boost the baseline growing method. Both components are designed and woven to accomplish the empirical leap. Our full method bests the test accuracy.

**Justification for Variance Transfer.** We train a simple neural network with 4 convolutional layers on CIFAR-10. The network consists of 4 resolution-preserving convolutional layers; each convolution has 64, 128, 256 and 512 channels, a \(3\times 3\) kernel, and is followed by BatchNorm and ReLU activations. Max-pooling is applied to each layer for a resolution-downsampling of 4, 2, 2, and 2. These CNN layers are then followed by a linear layer for classification. We first alternate this network into four variants, given by combinations of training epochs \(\in\{13(1\times),30(2.3\times)\}\) and initialization methods \(\in\) {standard, \(\mu\)transfer [42]}. We also grow from a thin architecture within

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{ResNet-50} & \multicolumn{2}{c}{MobileNet-v1} \\ \cline{2-5} Method & \begin{tabular}{c} Train \\ Cost(\(\%\)) \\ \end{tabular} & \begin{tabular}{c} Test \\ Acc.(\(\%\)) \\ \end{tabular} & \begin{tabular}{c} Train \\ Cost(\(\%\)) \\ \end{tabular} & \begin{tabular}{c} Test \\ Acc.(\(\%\)) \\ \end{tabular} \\ \hline \begin{tabular}{l} Large \\ Net2Net \\ Firefly \\ GradMax \\ \end{tabular} & \begin{tabular}{c} 100 \\ \(60.12\) \\ \end{tabular} & \begin{tabular}{c} 76.72 \\ \(74.89\) \\ \(74.0\) \\ \end{tabular} & \begin{tabular}{c} 100 \\ \(63.72\) \\ \(66.60\) \\ \end{tabular} & \begin{tabular}{c} 70.80 \\ \(66.19\) \\ \(66.40\) \\ \end{tabular} \\ \begin{tabular}{l} Ours \\ \end{tabular} & \begin{tabular}{c} **60.12** \\ \end{tabular} & \begin{tabular}{c} **75.90** \(\pm\) **0.14** \\ \end{tabular} & 
\begin{tabular}{c} **63.72** \\ \(69.91\) \(\pm\) **0.16** \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table}
Table 4: ResNet-50 and MobileNetV1 on ImageNet.

\begin{table}
\begin{tabular}{l c c} \hline \hline  & \multicolumn{2}{c}{Transformer} \\ \cline{2-3} Method & \begin{tabular}{c} Train Cost(\(\%\)) \\ \end{tabular} & \begin{tabular}{c} BLEU\(\uparrow\) \\ \end{tabular} \\ \hline \begin{tabular}{l} Large \\ Net2Net \\ \end{tabular} & \begin{tabular}{c} 100 \\ \(64.64\) \\ \end{tabular} & \begin{tabular}{c} \(32.82\pm 0.21\) \\ \(30.97\pm 0.35\) \\ \(31.44\pm 0.18\) \\ \(31.62\pm 0.16\) \\ \end{tabular} \\ \begin{tabular}{l} Ours-w/o buffer \\ \end{tabular} & \begin{tabular}{c} \(64.64\) \\ \(64.64\) \\ \end{tabular} & 
\begin{tabular}{c} \(31.44\pm 0.18\) \\ \(31.62\pm 0.16\) \\ \(32.01\pm 0.16\) \\ \(32.01\pm 0.16\) \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table}
Table 5: Transformer on IWSLT’14.
3 stages, where the channel number of each layer starts with only 1/4 of the original one, _i.e.,_\(\{16,32,64,128\}\rightarrow\{32,64,128,256\}\rightarrow\{64,128,256,512\}\), each stage is trained for 10 epochs.

For network growing, we compare the baselines with standard initialization and variance transfer. We train all baselines using SGD, with weight decay set as 0 and learning rates sweeping over \(\{0.01,0.02,0.05,0.1,0.2,0.5,0.8,1.0,1.2,1.5,2.0\}\). In Figure 3(b), growing with Var. Transfer (blue) achieves overall better test accuracy than standard initialization (orange). Large baselines with merely \(\mu\)transfer in initialization consistently underperform standard ones, which validate that the compensation from the LR re-scaling is necessary in [42]. We also observe, in both Figure 3(a) and 3(b), all growing baselines outperform fixed-size ones under the same training cost, demonstrating positive regularization effects. We also show the effect of our initialization scheme by comparing test performance on standard ResNet-20 on CIFAR-10. As shown in Figure 4(a), compared with standard initialization, our variance transfer not only achieves better final test accuracy but also appears more stable. See Appendix for a fully-connected network example.

**Justification for Learning Rate Adaptation.** We investigate the value of our proposed stage-wise learning rate adaptation as an optimizer for growing networks. As shown in the red curve in Figure 3, rate adaptation not only bests the train loss and test accuracy among all baselines, but also appears to be more robust over different learning rates. In Figure 4(a), rate adaptation further improves final test accuracy for ResNet-20 on CIFAR-10, under the same initialization scheme.

Figure 4(b) and 4(c) visualize the gradients of different sub-components for the 17-th convolutional layer of ResNet-20 during last two growing phases of standard SGD and rate adaptation, respectively. Our rate adaptation mechanism rebalances subcomponents' gradient contributions to appear in lower divergence than global LR, when components are added at different stages and trained for different durations. In Figure 5, we observe that the LR for newly added Subnet-8 (red) in last stage starts around \(1.8\times\) the base LR, then quick adapts to a smoother level. This demonstrates that our method is able to automatically adapt the updates applied to new weights, without any additional local optimization costs [40; 10]. All above show our method has a positive effect in terms of stabilizing training dynamics, which is lost if one attempts to train different subcomponents using a global LR scheduler. Appendix provides more analysis.

**Flexible Growing Scheduler.** Our growing scheduler gains the flexibility to explore the best trade-offs between training budgets and test performance in a unified configuration scheme (Eq. 6 and Eq. 7). We compare the exponential epoch scheduler (\(p_{t}\in\{0.2,0.25,0.3,0.35\}\)) to a linear one (\(p_{t}=0\))
in ResNet-20 growing on CIFAR-10, denoted as 'Exp.' and 'Linear' baselines in Figure 6. Both baselines use the architectural schedulers with \(p_{c}\in\{0.2,0.25,0.3,0.35\}\), each generates trade-offs between train costs and test accuracy by alternating \(T_{0}\). The exponential scheduler yields better overall trade-offs than the linear one with the same \(p_{c}\). In addition to different growing schedulers, we also plot a baseline for fixed-size training with different models. Growing methods with both schedulers consistently outperforms the fixed-size baselines, demonstrating that the regularization effect of network growth benefits generalization performance.

**Wall-clock Training Speedup.** We benchmark GPU memory consumption and wall-clock training time on CIFAR-100 for each stage during training on single NVIDIA 2080Ti GPU. The large baseline ResNet-18 trains for 140 minutes to achieve 78.36\(\%\) accuracy. As shown in the green bar of Figure 7(b), the growing method only achieves marginal wall-clock acceleration, under the same fixed batch size. As such, the growing ResNet-18 takes 120 minutes to achieve \(78.12\%\) accuracy. The low GPU utilization in the green bar in Figure 7(a) hinders FLOPs savings from translating into real-world training acceleration. In contrast, the red bar of Figure 7 shows our batch size adaptation results in a large proportion of wall clock acceleration by filling the GPU memory, and corresponding parallel execution resources, while maintaining test accuracy. ResNet-18 trains for \(84\) minutes (\(1.7\times\) speedup) and achieves \(78.01\%\) accuracy.

## 5 Conclusion

We tackle a set of optimization challenges in network growing and invent a corresponding set of techniques, including initialization with functionality preservation, variance transfer and learning rate adaptation to address these challenges. Each of these techniques can be viewed as 'upgrading' an original part for training static networks into a corresponding one that accounts for dynamic growing. There is a one-to-one mapping of these replacements and a guiding principle governing the formulation of each replacement. Together, they accelerate training without impairing model accuracy - a result that uniquely separates our approach from competitors. Applications to widely-used architectures on image classification and machine translation tasks demonstrate that our method bests the accuracy of competitors while saving considerable training cost.