Learn Faster and Forget Slower via Fast and Stable Task Adaptation

Farshid Varno<EMAIL_ADDRESS>
Department of Computer Science

Dalhousie University

6050 University Ave, Halifax, NS B3H 1W5, Canada
Lucas May Petry<EMAIL_ADDRESS>
Programa de Pós-Graduação em Ciência da Computação

Universidade Federal de Santa Catarina

Florianópolis, SC, Brazil
Lisa Di Jorio<EMAIL_ADDRESS>
Imagia Cybernetics Inc.

6650 St Urbain St #100, Montreal, Quebec H2S 3G9, Canada
Stan Matwin<EMAIL_ADDRESS>
Department of Computer Science

Dalhousie University

6050 University Ave, Halifax, NS B3H 1W5, Canada

Francis Bach and David Blei

and Bernhard Schölkopf

Training Deep Neural Networks (DNNs) is still highly time-consuming and compute-intensive.
It has been shown that adapting a pretrained model may significantly accelerate this process. With a focus on classification, we show that current fine-tuning techniques make the pretrained models catastrophically forget the transferred knowledge even before anything about the new task is learned. Such rapid knowledge loss undermines the merits of transfer learning and may result in a much slower convergence rate compared to when the maximum amount of knowledge is exploited. We investigate the source of this problem from different perspectives and to alleviate it, introduce Fast And Stable Task-adaptation (FAST), an easy to apply fine-tuning algorithm. The paper provides a novel geometric perspective on how the loss landscape of source and target tasks are linked in different transfer learning strategies. We empirically show that compared to prevailing fine-tuning practices, FAST learns the target task faster and forgets the source task slower.

Transfer Learning, Catastrophic Forgetting, Deep Neural Networks, Classification, Optimization

§ INTRODUCTION

§ BACKGROUND AND NOTATION

§ PROPOSED GEOMETRIC INTERPRETATION OF TRANSFER LEARNING

§ FAST

§ RELATED WORKS AND DISCUSSION

Initialization and optimization are closely-related subjects in deep learning [Sutskever et al., 2013], although they have often been studied individually. In fact, the former may help the latter to speed up the convergence. Optimization algorithms are usually used to search through the space that is formed by the model's parameters. GD, in particular, surfs this space by moving through it and looking at the objective of the search (minimizing the loss) after each step. Undoubtedly, the number of steps to reach the objective is affected by where the first step is taken from. Transferring knowledge in form of pretrained parameters from one neural network to another can be viewed as a kind of parameter initialization. Furthermore, we already have emphasized the  importance of a  customized optimization procedure toward the efficiency of fine-tuning the pretrained models.
In the following, we take a closer look at how previous works are related to ours in respect of initialization, optimization, and transfer learning concepts.

§.§ Initialization

Most widely known studies on parameter initialization in DNNs focus on preserving the variance of the flowing data along the depth [Glorot and Bengio, 2010, He et al., 2015]. This strategy prevents the activations from vanishing/exploding and allows training faster and deeper networks.
Specifically, for a parameterized linear layer with $D$ input and $N$ output neurons, Xavier's initialization method [Glorot et al., 2011] recommends initializing the parameters from a symmetric uniform distribution with variance equal to $\frac{1}{6D}$ to preserve the variance in the forwarding flow (fan-in mode), and a variance equal to $\frac{1}{6N}$ to preserve the variance in the backwarding flow (fan-out mode). Kaiming initialization [He et al., 2015] takes ReLU activations into account and relaxes the distribution to be symmetric and independent and identically distributed (i.i.d.). They provide variances $\frac{2}{D}$ and $\frac{2}{N}$ for the mentioned directions respectively. [Arpit and Bengio, 2019] recently acknowledged that the initialization introduced by He et al., 2015 is the optimal one for a ReLU network trained from scratch.
They recommended the use of the fan-out mode.
In line with the mentioned initialization methods, to avoid the exploding or vanishing problem,  [Saxe et al., 2013, Mishkin and Matas, 2015, Xiao et al., 2018] initialize the weights with orthogonal matrices. Similarly, [Balduzzi et al., 2017] proposed look linear initialization which prevents gradients in deep feed-forward networks to further shatter along the depth.[Gradient shattering is defined by Balduzzi et al., 2017 to indicate the phenomenon in which the back-propagating error may further go toward resembling white noise as the depth increases.]

§.§.§ Initialization for the Classifier

So far, the research community has paid little attention to the initialization for task-adaptation. In many recent studies, the classifier's parameters that are appended to a pretrained model are initialized using variance preserving methods  [Li and Hoiem, 2018, Shermin et al., 2018, Wang et al., 2019]. Included in this group, EfficientNet [Tan and Le, 2019, Xie et al., 2019], which shows state-of-the-art classification performance on ImageNet's ILSVRC-2012 data set [Russakovsky et al., 2015], uses classifiers initialized with Xavier's method [Glorot et al., 2011] for fine-tuning on new tasks [Abadi et al., 2017]. Some works do not adjust the initialization based on the embedding size or the number of classes; for instance, Wang et al., 2019 and Krizhevsky et al., 2012 always choose 0.000025 and 0.001 as the variance of elements of $\mW^{(0}$ respectively.[[Krizhevsky et al., 2012] also use an intercept (bias) vector that is initialized to all ones.]
He et al. [He et al., 2015], exempted the classification layer of the models used in their experiments from the distribution for initializing weights that they have recommended. This layer's distribution is stated to be found experimentally. Such a strategy could be traced down to even earlier practices in constructing deep neural networks  [Simonyan and Zisserman, 2015]. Instead of speculation, we investigate the effect of the initialization of the classifier on the training procedure. However, we narrow down our focus to an optimal initialization for adapting a pretrained model to a new task. Our previous work on classifier initialization [Varno et al., 2019] and a recently published paper by Dodge et al., 2020 have evidently enlightened the importance of the initialization for task-adaptation.
FAST uses similar initialization technique as of our previous work [Varno et al., 2019] but it highly improves the way its merits are exploited and comprehended.

§.§.§ Deterministic Initialization for the Classifier

Looking into special learning scenarios, two papers [Qi et al., 2018, Gidaris and Komodakis, 2018] concurrently proposed to initialize the rows of $\mW$ used in cosine classifier by averaging the features of the support set for their corresponding class. These average vectors are similar to those Snell et al., 2017 called prototypes. This kind of label-space weight initialization, known as weight imprinting, was originally proposed for a joint-training.[We use joint-training to indicate the learning scenario studied by Qi et al., 2018 according to the terminology used by Li] and Hoiem, 2018. The models employed by Dhillon et al., 2019 use a similar technique but only to classify the novel target classes. Our method also initializes the weights deterministically (all zeros) but in contrast to that of Dhillon et al., 2019, our initialization maximizes the uncertainty at the beginning of the training which leads to smoothly guiding the label-space mapping toward an efficient direction. Classifiers that use FAST promote confidence in results with a gradual and rational pace that is correlated with the knowledge they obtain from the learning task. Additionally, unlike that of Dhillon et al., 2019 our method does not need explicit calculations to find the proper initial values which makes it much easier to apply.
A method more similar to our initialization is presented by [Zhang et al., 2019]—who generalized Balduzzi et al., 2017 for residual networks.
However, no analytical or even empirical reasoning is provided to support their choice for initializing the classifier's weights.

§.§ Optimization

Due to being computationally affordable, first order approximation of GD is the preferred optimization algorithm for training deep learning models. Based on the size of memory required for loading the training data set and the model's parameters, in each step, GD may only be applied on a portion of the data (mini-batch of data). This makes the behaviour of the algorithm dependent on the sequence of the observed mini-batches [McCoy et al., 2020, Dodge et al., 2020]; thus, the algorithm is often called Stochastic Gradient Descent (SGD) in the context of DNNs. Its simplest form known as vanilla SGD applies

\begin{equation*}
\vtheta^{(t+1)} = \vtheta^{(t)} - \beta \frac{\partial l^{(t)}}{\partial \vtheta^{(t)}},
\end{equation*}

for any set of parameters $\vtheta$.

§.§.§ Speed of the Convergence

A line of research focuses on reducing the number of required steps to reach a minimum in the loss landscape [Qian, 1999, Botev et al., 2017, Duchi et al., 2011, Zeiler, 2012, Tieleman and Hinton, 2012, Kingma and Ba, 2015]. Adding momentum to SGD [Qian, 1999] decreases its dependency on the noise in individual mini-batches. It simply adds a moving average to the gradient of each parameter (look at line 10 of Algorithm <ref>). From a rather different point of view, Duchi et al., 2011 introduced AdaGrad to dampen the oscillation along very steep dimensions in the loss landscape.[Steepest dimension is the dimension that determines the degree of Lipschitz continuity and could vary from one neighborhood to another in the loss landscape.]
It normalizes the gradient of the loss with respect to each parameter with its L$_2$-norm that is summed over all the previous steps. It could be thought as modifying the vanilla SGD by scaling the learning rate of each parameter $\evtheta_i$ at any step $t$ by $g^{(t)}_{\evtheta_i}$ defined as

\begin{equation}
\label{eq:adagrad}
g^{(t)}_{\evtheta_i} = g^{(t-1)}_{\evtheta_i} + \left(\frac{\partial \ell^{(t)}}{\partial \evtheta_i^{(t-1)}}\right)^2.
\end{equation}

Since elements in this summation are always positive, the learning rate keeps scaling down as the training goes on until when the optimization algorithm becomes unable to modify any parameter. To address this issue, Adadelta [Zeiler, 2012] and RMSProp [Tieleman and Hinton, 2012] were introduced with a slight difference. They altered eq:adagrad such that instead of taking all of the previous optimization steps into account only a weighted average over a window of the steps is considered.
Taking advantage of a hybrid strategy, Adam [Kingma and Ba, 2015] integrated momentum with RMSProp and is known for notably speeding-up the convergence compared to SGD that only works with with momentum.

§.§.§ Performance Gap

Despite its merits, Adam has been largely criticized for asymptotically being outperformed by SGD with momentum [Keskar and Socher, 2017, Reddi et al., 2018].
Dampening the gradients along the dimensions with a high frequency  [Zeiler, 2012, Tieleman and Hinton, 2012, Kingma and Ba, 2015] is found to gain over SGD only in the beginning of the training [Keskar and Socher, 2017] which corresponds to moving across regions of the loss landscape with high altitudes. These regions are often more chaotic compared to regions closer to the minima [Li et al., 2018]. Motivated by these facts, we will empirically show that in the case of fine-tuning on a similar task, the gap between the performance of Adam and SGD with momentum could be significantly reduced. This mainly comes from the fact that, unlike the traditional fine-tuning, optimizing the classification objective with FAST does not make parameters to step out of the proximity of $\vtheta_{\gT_2}^*$ (the minimum in the loss landscape of the target task) far onto the high altitudes.

§.§.§ Initial Variance of the Error

[Vaswani et al., 2017] and [Popel and Bojar, 2018] showed that a warmup phase for adaptive learning rate can significantly accelerate the training convergence.[This is different from the classifier warmup phase in which only the classifier is updated.] It is an initial training phase in which a very small learning rate is applied. [Liu et al., 2019] found the source of this phenomenon in the large initial variance of the gradients. They demonstrate this by showing that the distribution of the gradients has momentous changes in the first few optimization steps and addressed the issue by an optimization algorithm that dampens the variance accordingly. Similarly, [Luo et al., 2019] and [Zhang et al., 2019] proposed to set dynamic boundaries on the magnitude of the updates. [Keskar and Socher, 2017] proposed to start training with Adam to take advantage of the convergence speed and then switch to SGD with momentum to better generalize when converged. Our proposed optimization algorithm for fine-tuning also decreases the initial variance of the gradients and similarly accelerates the training convergence.
Using FAST, neither the speed of convergence nor the generalizable performance at convergence are scarified.
FAST finds its merits in controlling the velocity of applying updates
only via the first gate that gradients back-propagate through (e.g., the classifier layer). Although our work focuses only on classification, the analysis we provide suggests that in an already stable model (e.g., pretrained on a similar task, so at comparably low altitude regions of the loss landscape), SGD with momentum is able to show a competent convergence speed as long as the pretrained parameters are not disrupted by sudden and large-variance gradients, which can cause overshooting the minimum of the loss landscape that they aim to converge. It is worth mentioning that label smoothing [Szegedy et al., 2016] is also relevant in the sense that it reduces the initial variance of the gradients by decreasing the L$_2$-norm of the true labels, while our method focuses on the norm of the predicted labels for which the magnitude could be automatically adapted during the course of training (unlike the norm of the true labels).

§.§.§ Other Related Works on Optimization

Loshchilov and Hutter, 2016 aim to improve the anytime performance while training DNN models
with a scheduling the value of the learning rate. Although using learning rate schedulers have been shown to be a promising direction, we choose not to experiment with it to avoid introducing conflating factors into our study.
Moreover, depicting the loss landscape in the other spaces rather than the space of all parameter have not been a common strategy in the deep learning community. Among the particular works that have employed this strategy, Yu et al., 2019 found visualizing the loss landscape in the input space to be more inline with their findings about adversarial vulnerability and robustness of neural networks.
Furthermore, although our analysis
about the connection between the minima in loss landscapes of two similar tasks is novel, depicting such connection between minima in the loss landscape of a single task has been already investigated. In this line of research,  Becker et al., 2020 found the depth (number of layers) to be an indicator of how strided the minima are spread on the loss surface.  Draxler et al., 2018 disputed the common perception about the location of minima of the loss landscape of CNN models at the bottom of distinct valleys and suggested that they are connected in a common low-altitude valley.

§.§ Transfer Learning

Although the literature is not unanimous about the exact concept for which transfer learning should refer to, in its most widely known meaning it indicates borrowing knowledge from a learned task to learn a new one [Pan and Yang, 2009]. It usually is categorized into feature extraction and fine-tuning [Li and Hoiem, 2018], where the former means only the appended parameters are altered during the training on the target task, while the latter also considers adapting the pretrained parameters.
It is well-accepted among the community that fine-tuning could accelerate the training procedure compared to learning from scratch;[Learning from scratch refers to a setting in which the parameters of the model are randomly initialized. In deep learning literature, learning from scratch sometimes, comes as the opposite case of transfer learning.] however, whether it also improves the outcomes [Kornblith et al., 2019, Raghu et al., 2019] or they results asymptotically emerge [He et al., 2019] is still an open debate. More recently, the latter hypothesis got back the attention of the community since the models that exhibit state-of-the-art classification performance on many machine vision benchmarks are those that are pretrained on larger source data sets [Wang et al., 2019, Xie et al., 2019].

§.§.§ Catastrophic Forgetting

Fine-tuning a pretrained neural network on a new task, greatly degrades the performance of the model on the source task. This phenomenon is widely known as catastrophic forgetting  [McCloskey and Cohen, 1989]. [Goodfellow et al., 2013] empirically found that dropout [Srivastava et al., 2014] reduces catastrophic forgetting although at the cost of performing sub-optimal on the new task.
Kirkpatrick et al., 2017 introduced a way to spot the parameters that are more important for the source task and to assign a smaller learning rate to them.
The practical importance of less forgetting is highlighted in a generic learning scenario so called continual learning [van de Ven and Tolias, 2019, Hsu et al., 2018]. It consists of several learning sub-scenarios with real life applications that take into account incrementally learning new tasks [Li and Hoiem, 2018], domains [van de Ven and Tolias, 2019] or classes [Rebuffi et al., 2017]. Due to its practicality, continual learning has recently become more popular and is even studied combined with other learning scenarios [Lao et al., 2020]. We consider exploiting the knowledge preserving aspect of FAST in continual learning as a possible direction for the future works. However, in this paper we retain our focus on the classic transfer learning where interestingly, in contrast to the previous studies, our method does not
make an explicit effort to retain the performance of the source task and does not compromise the performance on the target task.
Our main goal is to accelerate the fine-tuning process on the target task without compromising its convergence performance and as a bi-product the source task is also forgotten less than in a typical fine-tuning procedure.

§.§.§ Meta-learning

Meta-learning or learning to learn [Schmidhuber, 1993] is a generalization of the classic transfer learning, where the transferred knowledge is obtained from a collection of tasks instead of only one. It has been a while since meta-learning has become the leading research direction to tackle few-shot learning [Finn et al., 2017, Nichol et al., 2018, Li et al., 2017, Mishra et al., 2018]. However, recently, it has been shown that state-of-the-art meta-learning methods are not much superior compared to a more complex fine-tuning [Dhillon et al., 2019] or even feature extraction [Chen et al., 2019].

§.§.§ Classifier Warmup

Li and Hoiem, 2018 considered a classification warmup phase before fine-tuning on a new target task. In this phase, the feature extractor works in its inference mode and only the parameters of the classifier are updated. This is similar to finding a better initialization for the classifier though not only the representation learning is postponed, the number of required updates in the classification warmup phase is not known. The optimal number depends on many factors, including the values drawn for $\mW^{(0)}$. Li and Hoiem, 2018 used a validation set to determine when the classifier warmup phase should be ended. Our experiments suggest that, the transition from feature extraction to fine-tuning in this form causes a minimum overshooting, which immediately damages the model's performance (see Section <ref> for a more complete explanation). In contrast, our method does not demand manually switch between training phases. Instead, it chooses the initialization and the learning rate of $\mW$ such that the magnitude of the updates on the feature extractor's parameters become proportional to both the error and its certainty (through gradual entropy minimization).

§ EXPERIMENTS

§ CONCLUSION AND FUTURE WORKS

[Abadi et al., 2017]
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado,
A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving,
M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg,
D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens,
B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan,
F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and
X. Zheng.

[Abnar et al., 2020]
S. Abnar, M. Dehghani, and W. Zuidema.
Transferring inductive biases through knowledge distillation.
arXiv preprint arXiv:2006.00555, 2020.

[Agrawal et al., 2014]
P. Agrawal, R. Girshick, and J. Malik.
Analyzing the performance of multilayer neural networks for object
In European conference on computer vision, pages 329–344.
Springer, 2014.

[Amthor et al., 2016]
M. Amthor, E. Rodner, and J. Denzler.
Impatient dnns - deep neural networks with dynamic time budgets.
Procedings of the British Machine Vision Conference 2016,
URL <http://dx.doi.org/10.5244/C.30.116>.

[Arkes et al., 1988]
H. R. Arkes, D. Faust, T. J. Guilmette, and K. Hart.
Eliminating the hindsight bias.
Journal of applied psychology, 730 (2):0 305,

[Arpit and Bengio, 2019]
D. Arpit and Y. Bengio.
The benefits of over-parameterization at initialization in deep relu
arXiv preprint arXiv:1901.03611, 2019.

[Ashmore et al., 2019]
R. Ashmore, R. Calinescu, and C. Paterson.
Assuring the machine learning lifecycle: Desiderata, methods, and
challenges, 2019.

[Ba et al., 2016]
J. L. Ba, J. R. Kiros, and G. E. Hinton.
Layer normalization.
arXiv preprint arXiv:1607.06450, 2016.

[Balduzzi et al., 2017]
D. Balduzzi, M. Frean, L. Leary, J. Lewis, K. W.-D. Ma, and B. McWilliams.
The shattered gradients problem: If resnets are the answer, then what
is the question?
In Proceedings of the 34th International Conference on Machine
Learning-Volume 70, pages 342–350. JMLR. org, 2017.

[Bandura and Walters, 1977]
A. Bandura and R. H. Walters.
Social learning theory, volume 1.
Prentice-hall Englewood Cliffs, NJ, 1977.

[Becker et al., 2020]
S. Becker, Y. Zhang, et al.
Geometry of energy landscapes and the optimizability of deep neural
Physical Review Letters, 1240 (10):0 108301,

[Bengio, 2012]
Y. Bengio.
Deep learning of representations for unsupervised and transfer
In Proceedings of ICML workshop on unsupervised and transfer
learning, pages 17–36, 2012.

[Boddy and Dean, 1989]
M. Boddy and T. Dean.
Solving time-dependent planning problems.
In IJCAI'89 Proceedings of the 11th international joint
conference on Artificial intelligence - Volume 2, pages 979–984, 1989.
URL <https://academic.microsoft.com/paper/1601612555>.

[Botev et al., 2017]
A. Botev, G. Lever, and D. Barber.
Nesterov's accelerated gradient and momentum as approximations to
regularised update descent.
In 2017 International Joint Conference on Neural Networks
(IJCNN), pages 1899–1903. IEEE, 2017.

[Byrnes, 1995]
D. A. Byrnes.
The title of the work.
Anti-Defamation League of B'nai B'rith, 1995.

[Chen et al., 2019]
W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, and J.-B. Huang.
A closer look at few-shot classification.
In International Conference on Learning Representations, 2019.
URL <https://openreview.net/forum?id=HkxLXnAcFQ>.

[Chu et al., 2016]
B. Chu, V. Madhavan, O. Beijbom, J. Hoffman, and T. Darrell.
Best practices for fine-tuning visual classifiers to new domains.
In European conference on computer vision, pages 435–442.
Springer, 2016.

[Czarnecki et al., 2019]
W. M. Czarnecki, S. Osindero, R. Pascanu, and M. Jaderberg.
A deep neural network's loss surface contains every low-dimensional
arXiv preprint arXiv:1912.07559, 2019.

[Dhillon et al., 2019]
G. S. Dhillon, P. Chaudhari, A. Ravichandran, and S. Soatto.
A baseline for few-shot image classification, 2019.

[Dodge et al., 2020]
J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith.
Fine-tuning pretrained language models: Weight initializations, data
orders, and early stopping.
arXiv preprint arXiv:2002.06305, 2020.

[Donahue et al., 2014]
J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell.
Decaf: A deep convolutional activation feature for generic visual
In International conference on machine learning, pages
647–655, 2014.

[Draxler et al., 2018]
F. Draxler, K. Veschgini, M. Salmhofer, and F. A. Hamprecht.
Essentially no barriers in neural network energy landscape.
arXiv preprint arXiv:1803.00885, 2018.

[Duchi et al., 2011]
J. Duchi, E. Hazan, and Y. Singer.
Adaptive subgradient methods for online learning and stochastic
Journal of machine learning research, 120
(Jul):0 2121–2159, 2011.

[Finn et al., 2017]
C. Finn, P. Abbeel, and S. Levine.
Model-agnostic meta-learning for fast adaptation of deep networks.
In Proceedings of the 34th International Conference on Machine
Learning-Volume 70, pages 1126–1135. JMLR. org, 2017.

[Gidaris and Komodakis, 2018]
S. Gidaris and N. Komodakis.
Dynamic few-shot visual learning without forgetting.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 4367–4375, 2018.

[Girshick et al., 2014]
R. Girshick, J. Donahue, T. Darrell, and J. Malik.
Rich feature hierarchies for accurate object detection and semantic
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2014.

[Glorot and Bengio, 2010]
X. Glorot and Y. Bengio.
Understanding the difficulty of training deep feedforward neural
In Y. W. Teh and M. Titterington, editors, Proceedings of the
Thirteenth International Conference on Artificial Intelligence and
Statistics, volume 9 of Proceedings of Machine Learning Research,
pages 249–256, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR.
URL <http://proceedings.mlr.press/v9/glorot10a.html>.

[Glorot et al., 2011]
X. Glorot, A. Bordes, and Y. Bengio.
Deep sparse rectifier neural networks.
In Proceedings of the fourteenth international conference on
artificial intelligence and statistics, pages 315–323, 2011.

[Goodfellow et al., 2016]
I. Goodfellow, Y. Bengio, and A. Courville.
Deep Learning.
MIT Press, 2016.

[Goodfellow et al., 2013]
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio.
An empirical investigation of catastrophic forgetting in
gradient-based neural networks, 2013.

[Griffin et al., 2007]
G. Griffin, A. Holub, and P. Perona.
Caltech-256 object category dataset, 2007.

[He et al., 2015]
K. He, X. Zhang, S. Ren, and J. Sun.
Delving deep into rectifiers: Surpassing human-level performance on
imagenet classification.
In The IEEE International Conference on Computer Vision
(ICCV), December 2015.

[He et al., 2016]
K. He, X. Zhang, S. Ren, and J. Sun.
Deep residual learning for image recognition.
2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), Jun 2016.
URL <http://dx.doi.org/10.1109/CVPR.2016.90>.

[He et al., 2019]
K. He, R. Girshick, and P. Dollar.
Rethinking imagenet pre-training.
In The IEEE International Conference on Computer Vision
(ICCV), October 2019a.

[He et al., 2019]
T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie, and M. Li.
Bag of tricks for image classification with convolutional neural
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2019b.

[Hinton et al., 2015]
G. Hinton, O. Vinyals, and J. Dean.
Distilling the knowledge in a neural network.
arXiv preprint arXiv:1503.02531, 2015.

[Hsu et al., 2018]
Y.-C. Hsu, Y.-C. Liu, A. Ramasamy, and Z. Kira.
Re-evaluating continual learning scenarios: A categorization and case
for strong baselines.
arXiv preprint arXiv:1810.12488, 2018.

[Hu et al., 2019]
H. Hu, D. Dey, M. Hebert, and J. A. Bagnell.
Learning anytime predictions in neural networks via adaptive loss
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 33, pages 3812–3821, 2019.

[Huang et al., 2017]
G. Huang, D. Chen, T. Li, F. Wu, L. van der Maaten, and K. Q. Weinberger.
Multi-scale dense networks for resource efficient image
classification, 2017a.

[Huang et al., 2017]
G. Huang, Z. Liu, L. v. d. Maaten, and K. Q. Weinberger.
Densely connected convolutional networks.
2017 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), Jul 2017b.
URL <http://dx.doi.org/10.1109/cvpr.2017.243>.

[Ioffe and Szegedy, 2015]
S. Ioffe and C. Szegedy.
Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
In International Conference on Machine Learning, pages
448–456, 2015.

[Keskar and Socher, 2017]
N. S. Keskar and R. Socher.
Improving generalization performance by switching from adam to sgd.
arXiv preprint arXiv:1712.07628, 2017.

[Kingma and Ba, 2015]
D. P. Kingma and J. Ba.
Adam: A method for stochastic optimization.
In 3rd International Conference on Learning Representations,
ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track
Proceedings, 2015.
URL <http://arxiv.org/abs/1412.6980>.

[Kirkpatrick et al., 2017]
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A.
Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al.
Overcoming catastrophic forgetting in neural networks.
Proceedings of the national academy of sciences, 1140
(13):0 3521–3526, 2017.

[Korf, 1998]
R. E. Korf.
A complete anytime algorithm for number partitioning.
Artificial Intelligence, 1060 (2):0 181–203,
URL <https://academic.microsoft.com/paper/2090865599>.

[Kornblith et al., 2019]
S. Kornblith, J. Shlens, and Q. V. Le.
Do better imagenet models transfer better?
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2019.

[Krizhevsky et al., 2009]
A. Krizhevsky, G. Hinton, et al.
Learning multiple layers of features from tiny images.
Technical report, Citeseer, 2009.

[Krizhevsky et al., 2012]
A. Krizhevsky, I. Sutskever, and G. E. Hinton.
Imagenet classification with deep convolutional neural networks.
In Advances in neural information processing systems, pages
1097–1105, 2012.

[Kruger and Dunning, 1999]
J. Kruger and D. Dunning.
Unskilled and unaware of it: how difficulties in recognizing one's
own incompetence lead to inflated self-assessments.
Journal of personality and social psychology, 770
(6):0 1121, 1999.

[Lao et al., 2020]
Q. Lao, X. Jiang, M. Havaei, and Y. Bengio.
Continuous domain adaptation with variational domain-agnostic feature
arXiv preprint arXiv:2003.04382, 2020.

[LeCun et al., 1998]
Y. LeCun, C. Cortes, and C. J. Burges.
The mnist database of handwritten digits, 1998.
URL http://yann. lecun. com/exdb/mnist, 10:0 34, 1998.

[Li et al., 2016]
D. Li, X. Chen, M. Becchi, and Z. Zong.
Evaluating the energy efficiency of deep convolutional neural
networks on cpus and gpus.
In 2016 IEEE International Conferences on Big Data and Cloud
Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable
Computing and Communications (SustainCom)(BDCloud-SocialCom-SustainCom),
pages 477–484. IEEE, 2016.

[Li et al., 2018]
H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein.
Visualizing the loss landscape of neural nets.
In Advances in Neural Information Processing Systems, pages
6389–6399, 2018.

[Li and Hoiem, 2018]
Z. Li and D. Hoiem.
Learning without forgetting.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 400 (12):0 2935–2947, Dec 2018.

[Li et al., 2017]
Z. Li, F. Zhou, F. Chen, and H. Li.
Meta-sgd: Learning to learn quickly for few-shot learning.
arXiv preprint arXiv:1707.09835, 2017.

[Lifchitz et al., 2019]
Y. Lifchitz, Y. Avrithis, S. Picard, and A. Bursuc.
Dense classification and implanting for few-shot learning.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2019.

[Liu et al., 2019]
L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han.
On the variance of the adaptive learning rate and beyond.
arXiv preprint arXiv:1908.03265, 2019.

[Loshchilov and Hutter, 2016]
I. Loshchilov and F. Hutter.
Sgdr: Stochastic gradient descent with warm restarts.
arXiv preprint arXiv:1608.03983, 2016.

[Lu et al., 2019]
L. Lu, Y. Shin, Y. Su, and G. E. Karniadakis.
Dying relu and initialization: Theory and numerical examples.
arXiv preprint arXiv:1903.06733, 2019.

[Luo et al., 2018]
C. Luo, J. Zhan, X. Xue, L. Wang, R. Ren, and Q. Yang.
Cosine normalization: Using cosine similarity instead of dot product
in neural networks.
In V. Kůrková, Y. Manolopoulos, B. Hammer, L. Iliadis, and
I. Maglogiannis, editors, Artificial Neural Networks and Machine
Learning – ICANN 2018, pages 382–391, Cham, 2018. Springer International
ISBN 978-3-030-01418-6.

[Luo et al., 2019]
L. Luo, Y. Xiong, Y. Liu, and X. Sun.
Adaptive gradient methods with dynamic bound of learning rate.
arXiv preprint arXiv:1902.09843, 2019.

[Ma et al., 2018]
N. Ma, X. Zhang, H.-T. Zheng, and J. Sun.
Shufflenet v2: Practical guidelines for efficient cnn architecture
In The European Conference on Computer Vision (ECCV),
September 2018.

[McCloskey and Cohen, 1989]
M. McCloskey and N. J. Cohen.
Catastrophic interference in connectionist networks: The sequential
learning problem.
In Psychology of learning and motivation, volume 24, pages
109–165. Elsevier, 1989.

[McCoy et al., 2020]
R. T. McCoy, R. Frank, and T. Linzen.
Does syntax need to grow on trees? sources of hierarchical inductive
bias in sequence-to-sequence networks.
Transactions of the Association for Computational Linguistics,
8:0 125–140, 2020.

[Meyniel et al., 2015]
F. Meyniel, D. Schlunegger, and S. Dehaene.
The sense of confidence during probabilistic learning: A normative
PLoS computational biology, 110 (6), 2015.

[Mishkin and Matas, 2015]
D. Mishkin and J. Matas.
All you need is a good init.
arXiv preprint arXiv:1511.06422, 2015.

[Mishra et al., 2018]
N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel.
A simple neural attentive meta-learner.
In International Conference on Learning Representations, 2018.

[Nesterov, 1983]
Y. E. Nesterov.
A method for solving the convex programming problem with convergence
rate o (1/k 2).
In Dokl. akad. nauk Sssr, volume 269, pages 543–547, 1983.

[Nichol et al., 2018]
A. Nichol, J. Achiam, and J. Schulman.
On first-order meta-learning algorithms.
arXiv preprint arXiv:1803.02999, 2018.

[Pallier et al., 2002]
G. Pallier, R. Wilkinson, V. Danthiir, S. Kleitman, G. Knezevic, L. Stankov,
and R. D. Roberts.
The role of individual differences in the accuracy of confidence
The Journal of general psychology, 1290 (3):0
257–299, 2002.

[Pan and Yang, 2009]
S. J. Pan and Q. Yang.
A survey on transfer learning.
IEEE Transactions on knowledge and data engineering,
220 (10):0 1345–1359, 2009.

[Popel and Bojar, 2018]
M. Popel and O. Bojar.
Training tips for the transformer model.
The Prague Bulletin of Mathematical Linguistics, 1100
(1):0 43–70, 2018.

[Qi et al., 2018]
H. Qi, M. Brown, and D. G. Lowe.
Low-shot learning with imprinted weights.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2018.

[Qian, 1999]
N. Qian.
On the momentum term in gradient descent learning algorithms.
Neural networks, 120 (1):0 145–151, 1999.

[Qiao et al., 2019]
S. Qiao, H. Wang, C. Liu, W. Shen, and A. Yuille.
Weight standardization.
arXiv preprint arXiv:1903.10520, 2019.

[Raghu et al., 2019]
M. Raghu, C. Zhang, J. Kleinberg, and S. Bengio.
Transfusion: Understanding transfer learning for medical imaging.
In Advances in Neural Information Processing Systems, pages
3342–3352, 2019.

[Rebuffi et al., 2017]
S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert.
icarl: Incremental classifier and representation learning.
In Proceedings of the IEEE conference on Computer Vision and
Pattern Recognition, pages 2001–2010, 2017.

[Reddi et al., 2018]
S. J. Reddi, S. Kale, and S. Kumar.
On the convergence of adam and beyond.
In International Conference on Learning Representations, 2018.
URL <https://openreview.net/forum?id=ryQu7f-RZ>.

[Russakovsky et al., 2015]
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang,
A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei.
ImageNet Large Scale Visual Recognition Challenge.
International Journal of Computer Vision (IJCV), 1150
(3):0 211–252, 2015.

[Russo and Schoemaker, 1992]
J. E. Russo and P. J. Schoemaker.
Managing overconfidence.
Sloan management review, 330 (2):0 7–17,

[Sanchez and Dunning, 2018]
C. Sanchez and D. Dunning.
Overconfidence among beginners: Is a little learning a dangerous
Journal of personality and Social Psychology, 1140
(1):0 10, 2018.

[Sandler et al., 2018]
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.
Mobilenetv2: Inverted residuals and linear bottlenecks.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2018.

[Saxe et al., 2013]
A. M. Saxe, J. L. McClelland, and S. Ganguli.
Exact solutions to the nonlinear dynamics of learning in deep linear
neural networks.
arXiv preprint arXiv:1312.6120, 2013.

[Schmidhuber, 1993]
J. Schmidhuber.
A neural network that embeds its own meta-levels.
In IEEE International Conference on Neural Networks, pages
407–412. IEEE, 1993.

[Shermin et al., 2018]
T. Shermin, M. Murshed, G. Lu, and S. W. Teng.
Transfer learning using classification layer features of cnn, 2018.

[Simonyan and Zisserman, 2015]
K. Simonyan and A. Zisserman.
Very deep convolutional networks for large-scale image recognition.
In 3rd International Conference on Learning Representations,
ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track
Proceedings, 2015.
URL <http://arxiv.org/abs/1409.1556>.

[Skorokhodov and Burtsev, 2019]
I. Skorokhodov and M. Burtsev.
Loss surface sightseeing by multi-point optimization.
arXiv preprint arXiv:1910.03867, 2019.

[Snell et al., 2017]
J. Snell, K. Swersky, and R. Zemel.
Prototypical networks for few-shot learning.
In Advances in Neural Information Processing Systems, pages
4077–4087, 2017.

[Speirs-Bridge et al., 2010]
A. Speirs-Bridge, F. Fidler, M. McBride, L. Flander, G. Cumming, and
M. Burgman.
Reducing overconfidence in the interval judgments of experts.
Risk Analysis: An International Journal, 300
(3):0 512–523, 2010.

[Srivastava et al., 2014]
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.
Dropout: A simple way to prevent neural networks from overfitting.
Journal of Machine Learning Research, 15:0 1929–1958,
URL <http://jmlr.org/papers/v15/srivastava14a.html>.

[Strubell et al., 2019]
E. Strubell, A. Ganesh, and A. McCallum.
Energy and policy considerations for deep learning in nlp.
arXiv preprint arXiv:1906.02243, 2019.

[Sutskever et al., 2013]
I. Sutskever, J. Martens, G. Dahl, and G. Hinton.
On the importance of initialization and momentum in deep learning.
In International conference on machine learning, pages
1139–1147, 2013.

[Szegedy et al., 2016]
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.
Rethinking the inception architecture for computer vision.
2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), Jun 2016.
URL <http://dx.doi.org/10.1109/CVPR.2016.308>.

[Szegedy et al., 2017]
C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi.
Inception-v4, inception-resnet and the impact of residual connections
on learning.
In Thirty-First AAAI Conference on Artificial Intelligence,

[Tan and Le, 2019]
M. Tan and Q. V. Le.
Efficientnet: Rethinking model scaling for convolutional neural
In Proceedings of the 36th International Conference on Machine
Learning, pages 6105–6114, 2019.

[Tieleman and Hinton, 2012]
T. Tieleman and G. Hinton.
Lecture 6.5-rmsprop: Divide the gradient by a running average of its
recent magnitude.
COURSERA: Neural networks for machine learning, 40
(2):0 26–31, 2012.

[van de Ven and Tolias, 2019]
G. M. van de Ven and A. S. Tolias.
Three scenarios for continual learning, 2019.

[Varno et al., 2019]
F. Varno, B. H. Soleimani, M. Saghayi, L. Di Jorio, and S. Matwin.
Efficient neural task adaptation by maximum entropy initialization.
arXiv preprint arXiv:1905.10698, 2019.

[Vaswani et al., 2017]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
Ł. Kaiser, and I. Polosukhin.
Attention is all you need.
In Advances in neural information processing systems, pages
5998–6008, 2017.

[Vinyals et al., 2016]
O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al.
Matching networks for one shot learning.
In Advances in neural information processing systems, pages
3630–3638, 2016.

[Wan et al., 2019]
C. Wan, M. Santriaji, E. Rogers, H. Hoffmann, M. Maire, and S. Lu.
Alert: Accurate anytime learning for energy and timeliness.
arXiv preprint arXiv:1911.00119, 2019.

[Wang et al., 2019]
J. Wang, Y. Chen, H. Yu, M. Huang, and Q. Yang.
Easy transfer learning by exploiting intra-domain structures.
In 2019 IEEE International Conference on Multimedia and Expo
(ICME), pages 1210–1215. IEEE, 2019a.

[Wang et al., 2019]
L. Wang, Y. Zhao, Y. Jinnai, Y. Tian, and R. Fonseca.
Alphax: exploring neural architectures with deep neural networks and
monte carlo tree search.
arXiv preprint arXiv:1903.11059, 2019b.

[Winman et al., 2004]
A. Winman, P. Hansson, and P. Juslin.
Subjective probability intervals: how to reduce overconfidence by
interval evaluation.
Journal of Experimental Psychology: Learning, Memory, and
Cognition, 300 (6):0 1167, 2004.

[Wu and He, 2018]
Y. Wu and K. He.
Group normalization.
In The European Conference on Computer Vision (ECCV),
September 2018.

[Xiao et al., 2018]
L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington.
Dynamical isometry and a mean field theory of cnns: How to train
10,000-layer vanilla convolutional neural networks.
In International Conference on Machine Learning, pages
5393–5402, 2018.

[Xie et al., 2019]
Q. Xie, E. Hovy, M.-T. Luong, and Q. V. Le.
Self-training with noisy student improves imagenet classification.
arXiv preprint arXiv:1911.04252, 2019.

[Xie et al., 2017]
S. Xie, R. Girshick, P. Dollar, Z. Tu, and K. He.
Aggregated residual transformations for deep neural networks.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), July 2017a.

[Xie et al., 2017]
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He.
Aggregated residual transformations for deep neural networks.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 1492–1500, 2017b.

[Yang et al., 2019]
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le.
Xlnet: Generalized autoregressive pretraining for language
In Advances in neural information processing systems, pages
5754–5764, 2019.

[Yu et al., 2019]
F. Yu, Z. Qin, C. Liu, L. Zhao, Y. Wang, and X. Chen.
Interpreting and evaluating neural network robustness.
arXiv preprint arXiv:1905.04270, 2019.

[Zeiler, 2012]
M. D. Zeiler.
Adadelta: an adaptive learning rate method.
arXiv preprint arXiv:1212.5701, 2012.

[Zeiler and Fergus, 2013]
M. D. Zeiler and R. Fergus.
Stochastic pooling for regularization of deep convolutional neural
arXiv preprint arXiv:1301.3557, 2013.

[Zela et al., 2018]
A. Zela, A. Klein, S. Falkner, and F. Hutter.
Towards automated deep learning: Efficient joint neural architecture
and hyperparameter search.
arXiv preprint arXiv:1807.06906, 2018.

[Zhang et al., 2019]
H. Zhang, Y. N. Dauphin, and T. Ma.
Fixup initialization: Residual learning without normalization.
In International Conference on Learning Representations,

[Zhang et al., 2019]
J. Zhang, S. P. Karimireddy, A. Veit, S. Kim, S. J. Reddi, S. Kumar, and
S. Sra.
Why adam beats sgd for attention models.
arXiv preprint arXiv:1912.03194, 2019b.

[Zhong et al., 2016]
Y. Zhong, J. Sullivan, and H. Li.
Face attribute prediction using off-the-shelf cnn features.
In 2016 International Conference on Biometrics (ICB), pages
1–7. IEEE, 2016.