Unsupervised Learning of Initialization in Deep Neural Networks via
  Maximum Mean Discrepancy
Despite the recent success of stochastic gradient descent in deep learning,
it is often difficult to train a deep neural network with an inappropriate
choice of its initial parameters. Even if training is successful, it has been
known that the initial parameter configuration may negatively impact
generalization. In this paper, we propose an unsupervised algorithm to find
good initialization for input data, given that a downstream task is d-way
classification. We first notice that each parameter configuration in the
parameter space corresponds to one particular downstream task of d-way
classification. We then conjecture that the success of learning is directly
related to how diverse downstream tasks are in the vicinity of the initial
parameters. We thus design an algorithm that encourages small perturbation to
the initial parameter configuration leads to a diverse set of d-way
classification tasks. In other words, the proposed algorithm ensures a solution
to any downstream task to be near the initial parameter configuration. We
empirically evaluate the proposed algorithm on various tasks derived from MNIST
with a fully connected network. In these experiments, we observe that our
algorithm improves average test accuracy across most of these tasks, and that
such improvement is greater when the number of labelled examples is small.
Cheolhyoung Lee, Kyunghyun Cho
# Unsupervised Learning of Initialization in Deep Neural Networks via Maximum Mean Discrepancy

###### Abstract

Despite the recent success of stochastic gradient descent in deep learning, it is often difficult to train a deep neural network with an inappropriate choice of its initial parameters. Even if training is successful, it has been known that the initial parameter configuration may negatively impact generalization. In this paper, we propose an unsupervised algorithm to find good initialization for input data, given that a downstream task is \(d\)-way classification. We first notice that each parameter configuration in the parameter space corresponds to one particular downstream task of \(d\)-way classification. We then conjecture that the success of learning is directly related to how diverse downstream tasks are in the vicinity of the initial parameters. We thus design an algorithm that encourages small perturbation to the initial parameter configuration leads to a diverse set of \(d\)-way classification tasks. In other words, the proposed algorithm ensures a solution to _any_ downstream task to be near the initial parameter configuration. We empirically evaluate the proposed algorithm on various tasks derived from MNIST with a fully connected network. In these experiments, we observe that our algorithm improves average test accuracy across most of these tasks, and that such improvement is greater when the number of labelled examples is small.

## 1 Introduction

Initialization of parameters has long been identified as playing a critical role in improving both the convergence and generalization performance of deep neural networks (Glorot & Bengio, 2010; Erhan et al., 2010; He et al., 2015). In recent years, however, various normalization techniques, such as batch normalization (Ioffe & Szegedy, 2015), layer normalization (Ba et al., 2016), and weight normalization (Salimans & Kingma, 2016), have been found to somewhat reduce this heavy reliance on the initialization of parameters. The normalization techniques have done so by preserving some of the conditions that motivated various initialization schemes throughout training more explicitly. For instance, batch normalization normalizes each neuron to have zero mean and unit variance across examples within a minibatch, which is what Xavier initialization (Glorot & Bengio, 2010) and He initialization (He et al., 2015) aim to achieve in an ideal situation.

Although batch normalization has been widely used for training deep neural networks (He et al., 2016; Tan & Le, 2019), there are a small number of studies about why it helps training (Santurkar et al., 2018). Rather than revealing its theoretical effect, several researchers studied whether batch normalization is really necessary by training deep neural networks without batch normalization. Zhang et al. (2019) have proposed Fixup initialization replacing batch normalization in ResNet (He et al., 2016) by adding additional parameters to each residual block. Brock et al. (2021) have also succeeded to train ResNet with adaptive gradient clipping that adjusts unit-wise ratio of gradient norms to parameter norms during training. The similarity among their algorithms and batch normalization is that they add their own schemes to adaptively supervise optimization of the deep neural networks.
We suspect that the necessity for such adaptation comes from some neighborhood properties of an initial parameter configuration. Training is an optimization process finding an optimal parameter configuration which well-approximates a particular task derived from input data in the parameter space. It means that each parameter configuration corresponds to each task but this is not necessarily one-to-one. We hypothesize that training encourages the current parameter configuration to converge to the nearest optimal parameter configuration from the initial one. If there is no optimal solution near the initial parameter configuration, then the current parameter configuration either deviates from the initial parameter configuration (exploding gradient) or stays around it (vanishing gradient). We thus propose an algorithm to find a initial parameter configuration that can solve various tasks in their neighborhood.

Before finding such initial parameter configuration, we first need to check whether a given network solves any task derived from the input data. Zhang et al. (2016) empirically showed that over-parametrization of deep neural networks enables them to memorize the entire dataset so that they can be fitted to its arbitrary target task. Based on this, Pondenkandath et al. (2018) have empirically demonstrated that pre-training on random labels can accelerate training on downstream tasks. However, Maennel et al. (2020) has shown that the random label pre-training sometimes hurts the convergence of fine-tuning on the downstream tasks. They also presented that the pre-trained model generalizes worse than randomly initialized networks even if the random label pre-training promotes learning on the downstream task. For these studies, we further conjecture that a given over-parametrized network can solve any task in its parameter space, but it cannot do this at a single parameter configuration.

We therefore decide to utilize a set of parameter configurations, where we can find an optimal parameter configuration for any target task. If this set can be accumulated to the vicinity of one parameter configuration, we view this configuration as a good initial parameter configuration. To do this, we first restrict possible downstream tasks to \(d\)-way classification to make the model output domain be the same as a \((d-1)\)-dimensional unit simplex defined in equation 1. We then define a neighbor of the initial parameter configuration as small perturbation to this. Our unsupervised algorithm encourages each neighbor to solve a different task so that optimizers based on stochastic gradient descent such as Adam (Kingma & Ba, 2014) can easily find a solution near our initial parameter configuration.

We offer the mathematical statement for our conjecture in SS3.1, and propose an optimization problem to satisfy our claim for a given input. In doing so, we observe two possible degenerate cases to achieve our goal. In SS3.2.1 and SS3.2.2, we present how to avoid these unwanted situations. We validate our algorithm by various binary tasks derived from MNIST (LeCun et al., 1998) in SS5. From these experiments, we observe that fine-tuning deep neural networks from our initial parameters improves average test accuracy across the various binary tasks, and this gain is greater when the number of labelled examples is small.

## 2 Preliminaries and Notations

NormsUnless explicitly stated, a norm \(\|\cdot\|\) refers to \(L^{2}\) norm. We denote the Frobenius norm of a matrix \(\mathbf{A}\in\mathbb{R}^{m\times n}\) by \(\|\mathbf{A}\|_{F}=\sqrt{\sum_{i=1}^{m}\sum_{j=1}^{n}A_{ij}^{2}}\), where \(A_{ij}\) is the \((i,j)\)-th entry of \(\mathbf{A}\). We write the \(L^{2}\) operator norm of \(\mathbf{A}\) as \(\|\mathbf{A}\|^{*}=\sup_{\|\mathbf{x}\|=1}\|\mathbf{A}\mathbf{x}\|\), where \(\mathbf{x}\in\mathbb{R}^{n}\).

SupportsFor a distribution \(p(\mathbf{x})\), we write its support as \(\texttt{supp}(p(\mathbf{x}))=\{\mathbf{x}\in\mathbb{R}^{n}\mid p(\mathbf{x})>0\}\).

Model predictionA model prediction for \(d\)-way classification is a point in the \((d-1)\)-dimensional unit simplex \(\Delta^{d-1}\subset\mathbb{R}^{d}\) defined by

\[\Delta^{d-1}=\left\{(p_{1},p_{2},\cdots,p_{d})\in\mathbb{R}^{d}_{\geq 0}: \sum_{i=1}^{d}p_{i}=1\right\}, \tag{1}\]

where \(\mathbb{R}_{\geq 0}\) is the set of non-negative real numbers. We refer to a prediction of the model parametrized by \(\mathbf{\theta}\) for an input \(\mathbf{x}\), as \(\mathbf{f}_{\mathbf{\theta}}(\mathbf{x})\in\Delta^{d-1}\).
Uniform distribution over \(\Delta^{d-1}\)In this paper, we mainly deal with the uniform distribution over \(\Delta^{d-1},\mathcal{U}(\Delta^{d-1})\). We can generate its random sample \(\mathbf{u}\) by

\[\mathbf{u}=\left(\frac{\mathbf{e}_{1}}{\sum_{i=1}^{d}\mathbf{e}_{i}},\frac{\mathbf{e}_{2}}{\sum_{ i=1}^{d}\mathbf{e}_{i}},\cdots,\frac{\mathbf{e}_{d}}{\sum_{i=1}^{d}\mathbf{e}_{i}}\right), \tag{2}\]

where each \(\mathbf{e}_{i}\) is independently drawn from \(\texttt{exponential}(1)\)(Marsaglia, 1961).

Maximum mean discrepancy (MMD)The MMD (Gretton et al., 2012) is a framework for comparing two distributions \(p(\mathbf{x})\) and \(q(\mathbf{y})\) when we have samples from both distributions. The kernel MMD is defined by

\[\texttt{MMD}(p(\mathbf{x}),q(\mathbf{y});\gamma)= \mathbb{E}_{\mathbf{x}\sim p(\mathbf{x})}\mathbb{E}_{\mathbf{x}^{\prime}\sim p (\mathbf{x})}[k_{\gamma}(\mathbf{x},\mathbf{x}^{\prime})] \tag{3}\] \[-2\mathbb{E}_{\mathbf{x}\sim p(\mathbf{x})}\mathbb{E}_{\mathbf{y}\sim q(\mathbf{ y})}[k_{\gamma}(\mathbf{x},\mathbf{y})]\] \[+\mathbb{E}_{\mathbf{y}\sim q(\mathbf{y})}\mathbb{E}_{\mathbf{y}^{\prime} \sim q(\mathbf{y})}[k_{\gamma}(\mathbf{y},\mathbf{y}^{\prime})],\]

where \(k\) is a kernel function. A Gaussian kernel is often used, i.e., \(k_{\gamma}(\mathbf{x},\mathbf{y})=\exp\left(-\frac{\|\mathbf{x}-\mathbf{y}\|^{2}}{2\gamma^{2}}\right)\). Gretton et al. (2012) showed that \(p(\mathbf{x})=q(\mathbf{y})\) in distribution if and only if \(\texttt{MMD}(p(\mathbf{x}),q(\mathbf{y});\gamma)=0\).

## 3 Unsupervised learning of initialization

We start by conjecturing that the parameter configuration for any \(d\)-way classification must be in the vicinity of _good_ initial parameters. In other words, a parameter configuration, that solves any \(d\)-way classification task, is near the initial parameter configuration, so that such configuration can be readily found by stochastic gradient descent using labelled examples. The question we answer here is then how we can identify such an initial parameter configuration given a set of unlabelled examples.

### Uniformity over all mappings

Let \(\mathbf{f}_{\mathbf{\theta}}(\mathbf{x})\in\mathbb{R}^{d}\) be an output of a deep neural network parametrized by \(\mathbf{\theta}\in\mathbb{R}^{m}\) given an input \(\mathbf{x}\in\mathbb{R}^{n}\) sampled from an input distribution \(p(\mathbf{x})\). In supervised learning, there is a target mapping \(\mathbf{f}^{*}\) defined on \(\texttt{supp}(p(\mathbf{x}))\), and we want to find \(\mathbf{\theta}^{*}\) that

\[\min_{\mathbf{\theta}\in\mathbb{R}^{m}}l(\mathbf{f}_{\mathbf{\theta}},\mathbf{f}^{*}), \tag{4}\]

for a given loss function \(l\). For example, we often use \(l(\mathbf{f}_{\mathbf{\theta}},\mathbf{f}^{*})=\mathbb{E}_{\mathbf{x}\sim p(\mathbf{x})}[\|\mathbf{f}_ {\mathbf{\theta}}(\mathbf{x})-\mathbf{f}^{*}(\mathbf{x})\|^{2}]\) for regression and \(l(\mathbf{f}_{\mathbf{\theta}},\mathbf{f}^{*})=\mathbb{E}_{\mathbf{x}\sim p(\mathbf{x})}[\mathbb{ KL}(\mathbf{f}^{*}(\mathbf{x})\|\mathbf{f}_{\mathbf{\theta}}(\mathbf{x}))]\) for classification task, where \(\mathbb{KL}(\mathbf{f}^{*}(\mathbf{x})\|\mathbf{f}_{\mathbf{\theta}}(\mathbf{x}))\) is the Kullback-Leibler (KL) divergence from \(\mathbf{f}_{\mathbf{\theta}}(\mathbf{x})\) to \(\mathbf{f}^{*}(\mathbf{x})\).

In deep learning, it is usual to search for an optimal solution \(\mathbf{\theta}^{*}\) from equation 4 in the full parameter space \(\mathbb{R}^{m}\) by using a first-order optimizer, such as SGD and Adam (Kingma & Ba, 2014). In this process, Hoffer et al. (2017) have demonstrated however that

\[\|\mathbf{\theta}_{t}-\mathbf{\theta}_{0}\|\sim\log t, \tag{5}\]

where \(\mathbf{\theta}_{t}\) is a vector of parameters at the \(t\)-th optimization step and \(\mathbf{\theta}_{0}\) is that of initial parameters. In other words, the rate of deviation from \(\mathbf{\theta}_{0}\) decreases as training progresses. It means that the first order optimizer tends to find an optimal solution near the initial point. We thus rewrite equation 4 as

\[\mathbf{\theta}^{*}=\arg\min_{\mathbf{\theta}\in\mathbb{B}_{r}(\mathbf{\theta}_{0})}l(\mathbf{ f}_{\mathbf{\theta}},\mathbf{f}^{*}), \tag{6}\]

where \(\mathbb{B}_{r}(\mathbf{\theta}_{0})\) is a \(r\)-ball centered at \(\mathbf{\theta}_{0}\), \(\mathbb{B}_{r}(\mathbf{\theta}_{0})=\{\mathbf{\theta}\in\mathbb{R}^{d}:\|\mathbf{\theta}- \mathbf{\theta}_{0}\|<r\}\).

With this in our mind, what is the good initialization \(\mathbf{\theta}_{0}\) for equation 6? To answer this question, we look at what kind of classifiers we have within \(\mathbb{B}_{r}(\mathbf{\theta}_{0})\). If \(\mathbf{x}\) is an example randomly drawn from the input distribution \(p(\mathbf{x})\), the set of all possible model outputs from \(\mathbf{x}\) in \(\mathbb{B}_{r}(\mathbf{\theta}_{0})\) is

\[\mathbb{F}(\mathbf{x};\mathbf{\theta}_{0})=\{\mathbf{f}_{\mathbf{\theta}}(\mathbf{x}):\mathbf{\theta} \in\mathbb{B}_{r}(\mathbf{\theta}_{0})\}.\]

We define the collection of all possible target mappings from the input space into the \((d-1)\)-dimensional unit simplex \(\Delta^{d-1}\) defined in equation 1 as

\[\mathcal{F}=\{\mathbf{f}^{*}\mid\mathbf{f}^{*}:\texttt{supp}(p(\mathbf{x}))\to\Delta^{d-1} \subset\mathbb{R}^{d}\}.\]
If \(\mathbf{\theta}_{0}\) is a good initial configuration, \(\mathbb{F}(\mathbf{x};\mathbf{\theta}_{0})\) has to be \(\Delta^{d-1}\). Otherwise, our model cannot approximate \(\mathbf{f}^{*}\in\mathcal{F}\) such that \(\mathbf{f}^{*}(\mathbf{x})\in\Delta^{d-1}\setminus\mathbb{F}(\mathbf{x};\mathbf{\theta}_{0})\) in \(\mathbb{B}_{r}(\mathbf{\theta}_{0})\).

To approximate all target mappings in \(\mathcal{F}\) by \(\mathbf{f}_{\mathbf{\theta}}\) near \(\mathbf{\theta}_{0}\) for \(\mathbf{x}\), there must be \(\mathbf{\theta}\in\mathbb{B}_{r}(\mathbf{\theta}_{0})\) satisfying \(\mathbf{f}_{\mathbf{\theta}}(\mathbf{x})=\mathbf{s}\) for arbitrary \(\mathbf{s}\in\Delta^{d-1}\). In other words, if we randomly pick \(\mathbf{\theta}\) in \(\mathbb{B}_{r}(\mathbf{\theta}_{0})\), the probability density of \(\mathbf{f}_{\mathbf{\theta}}(\mathbf{x})=\mathbf{s}\) for any \(\mathbf{s}\in\Delta^{d-1}\) is positive and should be the same over \(\Delta^{d-1}\) without prior knowledge of target mappings.

**Claim 1**.: _Denote the distribution of \(\mathbf{y}=\mathbf{f}_{\mathbf{\theta}}(\mathbf{x})\) given \(\mathbf{x}\sim p(\mathbf{x})\) over \(\mathbf{\theta}\sim\mathcal{U}(\mathbb{B}_{r}(\mathbf{\theta}_{0}))\) as \(q_{\mathbf{x}}(\mathbf{y};\mathbf{\theta}_{0},r)\).1 Then, \(\mathbf{\theta}_{0}\) is a good initialization if and only if \(\texttt{supp}(q_{\mathbf{x}}(\mathbf{y};\mathbf{\theta}_{0},r))=\Delta^{d-1}\) and \(q_{\mathbf{x}}(\mathbf{y};\mathbf{\theta}_{0},r)\) is equal to \(\mathcal{U}(\Delta^{d-1})\) in distribution, because we do not know which \(\mathbf{s}\in\Delta^{d-1}\) is more likely._

Footnote 1: Although \(\mathbf{x}\) is given, \(\mathbf{f}_{\mathbf{\theta}}(\mathbf{x})\) is random due to the randomness of \(\mathbf{\theta}\).

To obtain \(\mathbf{\theta}_{0}\) satisfying Claim 1, we build an optimization problem that makes \(q_{\mathbf{x}}(\mathbf{y};\mathbf{\theta}_{0},r)\) converge to \(\mathcal{U}(\Delta^{d-1})\) in distribution for a given \(\mathbf{x}\sim p(\mathbf{x})\). The first step toward this goal is to use the maximum mean discrepancy (MMD) (Gretton et al., 2012) from equation 3. We define an example specific loss as

\[\mathcal{L}_{\mathbf{x}}^{uni}(\mathbf{\theta}_{0};r,\Delta^{d-1},\gamma)=\texttt{MMD} (q_{\mathbf{x}}(\mathbf{y};\mathbf{\theta}_{0},r),\mathcal{U}(\Delta^{d-1});\gamma). \tag{7}\]

According to Gretton et al. (2012), equation 7 is equal to \(0\) if and only if \(q_{\mathbf{x}}(\mathbf{y};\mathbf{\theta}_{0},r)\) is equal to \(\mathcal{U}(\Delta^{d-1})\) in distribution. We can therefore find \(\mathbf{\theta}_{0}\) that satisfies Claim 1, by minimizing equation 7 with respect to \(\mathbf{\theta}_{0}\).

The minimization of equation 7 with respect to \(\mathbf{\theta}_{0}\) needs samples from both \(\mathcal{U}(\Delta^{d-1})\) and \(\mathcal{U}(\mathbb{B}_{r}(\mathbf{\theta}_{0}))\). In the case of \(\mathcal{U}(\Delta^{d-1})\), we draw samples using equation 2. For \(\mathcal{U}(\mathbb{B}_{r}(\mathbf{\theta}_{0}))\), we relax it to \(\mathcal{N}(\mathbf{\theta}_{0},\mathbf{\Sigma})\) where \(\mathbf{\Sigma}=\texttt{diag}(\sigma_{1}^{2},\sigma_{2}^{2},\cdots,\sigma_{m}^{2})\) for two reasons: i) this applies the same with uniform, since we can change the value range for each parameter separately; ii) the normal distribution allows us to use the reparametrization trick to compute \(\nabla_{\mathbf{\theta}_{0}}\mathcal{L}_{\mathbf{x}}^{uni}(\mathbf{\theta}_{0};r,\Delta^{ d-1},\gamma)\) from equation 7 (Kingma & Welling, 2013). Furthermore, as shown in Theorem 1 below, a proper choice of the covariance matrix makes Gaussian perturbation have similar effect as uniform perturbation:

**Theorem 1**.: _Let \(\mathbf{\theta}\sim\mathcal{N}(\mathbf{\theta}_{0},\texttt{diag}(\sigma_{1}^{2}, \sigma_{2}^{2},\cdots,\sigma_{m}^{2}))\) and \(\alpha_{*}=\max_{i=1,2,\cdots,m}\sigma_{i}^{2}\). If \(r^{2}\) is greater than \(m\alpha_{*}\), then we have_

\[\mathbb{P}\left(\left\|\mathbf{\theta}-\mathbf{\theta}_{0}\right\|\geq\ r \right)\leq\exp\left(-\frac{1}{8}\min\left\{\eta^{2},m\eta\right\}\right), \tag{8}\]

_where \(\eta=\frac{r^{2}}{m\alpha_{*}}-1\) (proved in SSA.1)._

Theorem 1 implies that if we add a Gaussian perturbation \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})\) to \(\mathbf{\theta}_{0}\), then the perturbed parameter configuration, \(\mathbf{\theta}=\mathbf{\theta}_{0}+\mathbf{\epsilon}\), is enough closed to \(\mathbf{\theta}_{0}\) with a high probability, when \(\alpha^{*}=\max_{i}\sigma_{i}^{2}\) is sufficiently small. In other words, although \(\mathcal{N}(\mathbf{\theta}_{0},\mathbf{\Sigma})\) is not exactly equivalent to \(\mathcal{U}(\mathbb{B}_{r}(\mathbf{\theta}_{0}))\) in distribution, these two distributions play a similar role in the view of generating random parameter configurations near \(\mathbf{\theta}_{0}\). We therefore rewrite equation 7 to enable reparametrization trick, as below:

\[\mathcal{L}_{\mathbf{x}}^{uni}(\mathbf{\theta}_{0};\mathbf{\Sigma},\Delta^{d -1},\gamma)= \mathbb{E}_{\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})}\mathbb{E}_ {\mathbf{\epsilon}^{\prime}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})}[k_{\gamma}(\mathbf{f}_{ \mathbf{\theta}_{0}+\mathbf{\epsilon}}(\mathbf{x}),\mathbf{f}_{\mathbf{\theta}_{0}+\mathbf{\epsilon} ^{\prime}}(\mathbf{x}))] \tag{9}\] \[-2\mathbb{E}_{\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})} \mathbb{E}_{\mathbf{u}\sim\mathcal{U}(\Delta^{d-1})}[k_{\gamma}(\mathbf{f}_{\mathbf{ \theta}_{0}+\mathbf{\epsilon}}(\mathbf{x}),\mathbf{u})]\] \[+\mathbb{E}_{\mathbf{u}\sim\mathcal{U}(\Delta^{d-1})}\mathbb{E}_{\mathbf{u} ^{\prime}\sim\mathcal{U}(\Delta^{d-1})}[k_{\gamma}(\mathbf{u},\mathbf{u}^{\prime})],\]

where \(q_{\mathbf{x}}(\mathbf{y};\mathbf{\theta}_{0},\mathbf{\Sigma})\) is the distribution of \(\mathbf{f}_{\mathbf{\theta}}(\mathbf{x})\) given \(\mathbf{x}\) with \(\mathbf{\theta}\sim\mathcal{N}(\mathbf{\theta}_{0},\mathbf{\Sigma})\). In other words, we add Gaussian noise to each parameter and encourage prediction for \(\mathbf{x}\) based on such perturbed parameter configuration to be well spread out over \(\Delta^{d-1}\). From now on, we use \(\mathbf{\theta}_{0}+\mathbf{\epsilon}\) to denote the perturbed parameter configuration, with \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})\), to be more explicit about our use of reparametrization trick.

Equation 9 is an example specific loss, and minimizing this with respect to \(\mathbf{\theta}_{0}\) only guarantees the existence of \(\mathbf{\theta}^{*}\) near \(\mathbf{\theta}_{0}\) satisfying \(\mathbf{f}^{*}=\mathbf{f}_{\mathbf{\theta}^{*}}\) for a single \(\mathbf{x}\). Hence, we take the expectation of equation 9 over the input distribution \(p(\mathbf{x})\):

\[\mathcal{L}^{uni}(\mathbf{\theta}_{0};\mathbf{\Sigma},\Delta^{d-1},\gamma,p(\mathbf{x}))= \mathbb{E}_{\mathbf{x}\sim p(\mathbf{x})}[\mathcal{L}_{\mathbf{x}}^{uni}(\mathbf{\theta}_{0}; \mathbf{\Sigma},\Delta^{d-1},\gamma)]. \tag{10}\]
We minimize this expected loss to find an initial parameter configuration \(\mathbf{\theta}_{0}^{*}\) that satisfies Claim 1 for the input data on average. When done so, we can find \(\mathbf{f}_{\mathbf{\theta}}\) within the close proximity of \(\mathbf{\theta}_{0}^{*}\) that approximates any \(d\)-way target mapping \(\mathbf{f}^{*}\), given \(p(\mathbf{x})\).

### Degeneracies and remedies

Let \(\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{M}\) be random samples drawn from \(p(\mathbf{x})\), \(\mathbf{\epsilon}_{1},\mathbf{\epsilon}_{2},\cdots,\mathbf{\epsilon}_{N}\) be random perturbations from \(\mathcal{N}(\mathbf{0},\mathbf{\Sigma})\), and \(\mathbf{c}_{1},\mathbf{c}_{2},\cdots,\mathbf{c}_{N}\) from \(\mathcal{U}(\Delta^{d-1})\). If \(\mathbf{\theta}_{0}^{1}\) satisfies

\[\mathbf{c}_{j}=\mathbf{f}_{\mathbf{\theta}_{0}^{1}+\mathbf{c}_{j}}(\mathbf{x}_{1})=\mathbf{f}_{\mathbf{ \theta}_{0}^{1}+\mathbf{c}_{j}}(\mathbf{x}_{2})=\cdots=\mathbf{f}_{\mathbf{\theta}_{0}^{1}+ \mathbf{c}_{j}}(\mathbf{x}_{M}), \tag{11}\]

for each \(j\), then \(\mathcal{L}_{\mathbf{x}_{i}}^{uni}(\mathbf{\theta}_{0}^{1};\mathbf{\Sigma},\Delta^{d-1}, \gamma)=0\) for all \(i\). Hence, \(\mathbf{\theta}_{0}^{1}\) is one of the optimal solutions for equation 10. In the case of \(\mathbf{\theta}_{0}^{1}\), each perturbed model near \(\mathbf{\theta}_{0}^{1}\) is a constant function, to which we refer as _input-output detachment_. Furthermore, each of these constant functions may output a _degenerate_ categorical distribution whose support does not cover all \(d\) classes, for which we refer to this phenomenon as _degenerate softmax_. We empirically demonstrate that both degeneracies indeed occur when we train a fully connected network by minimizing \(\mathcal{L}^{uni}\) in SSB.1. In this section, we present two regularization terms, to be added to equation 10, to avoid these two unwanted cases, respectively.

#### 3.2.1 Degenerate softmax

We first address the latter issue of degenerate softmax. Since we have specified that the task of our interest is \(d\)-way classification, we prefer models that can classify inputs into all \(d\) classes in the neighborhood of \(\mathbf{\theta}_{0}^{*}\). We thus impose a condition that there exists at least one example categorized into each and every class. We first define a set of the points \(\mathbb{A}_{i}\) classified into the \(i\)-th class as

\[\mathbb{A}_{i}=\left\{\mathbf{a}=(a_{1},a_{2},\cdots,a_{d})\in\Delta^{d-1}:a_{i} \geq a_{j}\text{ for all }j=1,2,\cdots,d\right\}. \tag{12}\]

Given \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})\), the probability of _'the model at \(\mathbf{\theta}_{0}^{*}+\mathbf{\epsilon}\) classifies \(\mathbf{x}\) into the \(i\)-th class'_ is \(\mathbb{P}_{\mathbf{x}\sim p(\mathbf{x})}(\mathbf{f}_{\mathbf{\theta}_{0}^{*}+\mathbf{\epsilon}}( \mathbf{x})\in\mathbb{A}_{i})\). This probability should be positive for all \(i=1,2,\cdots,d\) to avoid degenerate softmax at \(\mathbf{\theta}_{0}^{*}\). To satisfy this, we use Theorem 2 which offers a lower bound of \(\mathbb{P}_{\mathbf{x}\sim p(\mathbf{x})}(\mathbf{f}_{\mathbf{\theta}_{0}^{*}+\mathbf{\epsilon}}( \mathbf{x})\in\mathbb{A}_{i})\) using the distance from the \(i\)-th vertex \(\mathbf{v}^{(i)}\):

**Theorem 2**.: _Let \(\mathbf{v}^{(i)}=\left(v_{1}^{(i)},v_{2}^{(i)},\cdots,v_{d}^{(i)}\right)\in\Delta ^{d-1}\), where \(v_{i}^{(i)}=1\) and \(\mathbb{A}_{i}\) be a subset of \(\Delta^{d-1}\), as defined in equation 12. Then,_

\[\mathbb{P}_{\mathbf{x}\sim p(\mathbf{x})}(\mathbf{f}_{\mathbf{\theta}_{0}^{*}+\mathbf{\epsilon}}( \mathbf{x})\in\mathbb{A}_{i})\geq 1-\sqrt{d}\mathbb{E}_{\mathbf{x}\sim p(\mathbf{x})}[\|\mathbf{v}^{ (i)}-\mathbf{f}_{\mathbf{\theta}_{0}^{*}+\mathbf{\epsilon}}(\mathbf{x})\|], \tag{13}\]

_for a given \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})\) (proved in SSA.2)._

According to equation 13, \(\mathbb{E}_{\mathbf{x}\sim p(\mathbf{x})}[\|\mathbf{v}^{(i)}-\mathbf{f}_{\mathbf{\theta}_{0}^{*}+ \mathbf{\epsilon}}(\mathbf{x})\|]<\frac{1}{\sqrt{d}}\) implies \(\mathbb{P}_{\mathbf{x}\sim p(\mathbf{x})}(\mathbf{f}_{\mathbf{\theta}_{0}^{*}+\mathbf{\epsilon}}( \mathbf{x})\in\mathbb{A}_{i})>0\) for each \(i\), given \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})\). This means that we can avoid degenerate softmax by minimizing

\[\mathcal{L}^{sd}(\mathbf{\theta}_{0};\mathbf{\Sigma},d,p(\mathbf{x}))=\mathbb{E}_{\mathbf{ \epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})}\left[\max\left\{\max_{i=1,2, \cdots,d}\mathbb{E}_{\mathbf{x}\sim p(\mathbf{x})}[\|\mathbf{v}^{(i)}-\mathbf{f}_{\mathbf{\theta} _{0}^{*}+\mathbf{\epsilon}}(\mathbf{x})\|],\frac{1}{\sqrt{d}}\right\}-\frac{1}{\sqrt{d }}\right]. \tag{14}\]

This minimization pulls the softmax output toward the furthest vertex for each \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})\), eventually avoiding the issue of degenerate softmax.

#### 3.2.2 Input-output detachment

Here, let us go back to the first issue of input-output detachment we identified in equation 11. This issue happens when each perturbed model near \(\mathbf{\theta}_{0}^{1}\) is a constant function. In other words, the Jacobian of the model's output with respect to the input is zero, and in the case of multi-layered neural networks, the Jacobian of the model's output with respect to one of the intermediate layers is zero. This largely prevents learning from \(\mathbf{\theta}_{0}^{1}\), because \(\mathbf{\theta}_{0}^{1}\) is surrounded by the parameter configurations from which learning cannot happen. We thus design an additional loss that regularizes the Jacobian of model prediction with respect to its input and hidden neurons to prevent the input-output detachment.
In the rest of this section, we consider \(\mathbf{f}\) as the logits instead of the values after applying softmax, in order to avoid an issue of saturation caused by softmax (Varga et al., 2017). Let \(\mathbf{x}_{l}\in\mathbb{R}^{n_{l}}\), for \(l\in\{0,1,\cdots,L\}\), be a vector of pre-activated neurons at the \((l+1)\)-th layer parametrized by \(\mathbf{\theta}_{0}^{(l+1)}\), where \(\mathbf{x}_{0}\in\mathbb{R}^{n_{0}}\) and \(\mathbf{x}_{L}\in\mathbb{R}^{n_{L}}=\mathbb{R}^{d}\) are an input vector and its corresponding output vector, respectively. \(\mathbf{f}_{\mathbf{\theta}_{0}^{(l:L)}}\) is the function from \(\mathbb{R}^{n_{l}}\) to \(\mathbb{R}^{d}\), parametrized by \(\mathbf{\theta}_{0}^{(l+1)},\mathbf{\theta}_{0}^{(l+2)},\cdots,\mathbf{\theta}_{0}^{(L)}\). Let us now consider the effect of perturbing the input to such a function:

\[\mathbf{f}_{\mathbf{\theta}_{0}^{(l:L)}}(\mathbf{x}_{l}+\mathbf{\xi}_{l})\approx\mathbf{f}_{\mathbf{ \theta}_{0}^{(l:L)}}(\mathbf{x}_{l})+\mathbf{J}_{\mathbf{\theta}_{0}^{(l:L)}}(\mathbf{x}_{l}) \mathbf{\xi}_{l}, \tag{15}\]

where \(\mathbf{J}_{\mathbf{\theta}_{0}^{(l:L)}}(\mathbf{x}_{l})\in\mathbb{R}^{d\times n_{l}}\) is the Jacobian matrix of \(\mathbf{f}_{\mathbf{\theta}_{0}^{(l:L)}}\) with respect to \(\mathbf{x}_{l}\).

We then look at equation 15 entry-wise:

\[f_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l}+\mathbf{\xi}_{l})\approx f_{\mathbf{ \theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l})+J_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{ x}_{l})\mathbf{\xi}_{l}, \tag{16}\]

where \(f_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}\) is the \(i\)-th entry of \(\mathbf{f}_{\mathbf{\theta}_{0}^{(l:L)}}\), and \(J_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}\) is the \(i\)-th row of \(\mathbf{J}_{\mathbf{\theta}_{0}^{(l:L)}}\) for \(i=1,2,\cdots,d\). From equation 16, we can see that the absolute difference between \(f_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l})\) and \(f_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l}+\mathbf{\xi}_{l})\) can be well approximated by the absolute value of the gradient-perturbation product:

\[\left|f_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l}+\mathbf{\xi}_{l})-f_{\mathbf{ \theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l})\right|\approx\left|J_{\mathbf{\theta}_{0}^{ (l:L)}}^{(i)}(\mathbf{x}_{l})\mathbf{\xi}_{l}\right|. \tag{17}\]

Assuming the perturbation's norm to be unit, we can bound this quantity by the operator norm of the \(i\)-th row of Jacobian:

\[\sup_{\|\mathbf{\xi}_{l}\|=1}\left|f_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l}+ \mathbf{\xi}_{l})-f_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l})\right|\approx\sup_ {\|\mathbf{\xi}_{l}\|_{2}=1}\left|J_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l}) \mathbf{\xi}_{l}\right|=\left\|J_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l}) \right\|^{*}. \tag{18}\]

Since \(J_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l})\) is a row vector, i.e., a matrix of rank 1, the Frobenius norm \(\|\cdot\|_{F}\) is equivalent to the operator norm \(\|\cdot\|^{*}\). This allows us to rewrite equation 18 as

\[\sup_{\|\mathbf{\xi}_{l}\|_{2}=1}\left|f_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l }+\mathbf{\xi}_{l})-f_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l})\right|\approx \left\|J_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l})\right\|_{F}. \tag{19}\]

According to Equation 19, if \(\|J_{\mathbf{\theta}_{0}^{(l:L)}}^{(i)}(\mathbf{x}_{l})\|_{F}\) is positive, our initial model \(\mathbf{f}_{\mathbf{\theta}_{0}}\) is sensitive to the change in \(\mathbf{x}_{l}\). That is, it is not a constant function.

Per the derivation above, in order to avoid the input-output detachment, we can for instance impose that, for all \(i=1,2,\cdots,d\),

\[c=\left\|J_{\mathbf{\theta}_{0}^{(0:L)}}^{(i)}(\mathbf{x}_{0})\right\|_{F}=\left\|J_{ \mathbf{\theta}_{0}^{(1:L)}}^{(i)}(\mathbf{x}_{1})\right\|_{F}=\cdots=\left\|J_{\mathbf{ \theta}_{0}^{(L-1:L)}}^{(i)}(\mathbf{x}_{L-1})\right\|_{F}, \tag{20}\]

where \(c>0\) is a constant. Here, we set \(c=1\) which has an equivalent effect of setting the parameters using the so-called He initialization (He et al., 2015), as shown in the following theorem:

**Theorem 3**.: _Let \(\mathbf{f}_{\mathbf{\theta}_{0}}\) be a fully connected network with ReLU (Nair & Hinton, 2010) non-linearity. We write the layerwise non-linear transformation from \(\mathbf{x}_{l}\) to \(\mathbf{x}_{l+1}\) for \(l\neq 0\) as_

\[\mathbf{\hat{f}}_{\mathbf{\theta}_{0}^{(l:L)}}(\mathbf{x}_{l})=\mathbf{W}^{(l+1)}\textsc{Re} \textsc{L}\textsc{U}(\mathbf{x}_{l})+\mathbf{b}^{(l+1)},\]

_where \(\mathbf{W}^{(l+1)}\in\mathbb{R}^{n_{l+1}\times n_{l}}\) is the weight matrix and \(\mathbf{b}\in\mathbb{R}^{n_{l+1}}\) is the bias vector. Assume that each element of \(\mathbf{x}_{l}\) has a symmetric distribution at \(0\) and all elements of \(\mathbf{x}_{l}\) are mutually independent. If the \((i,j)\)-th entry of \(\mathbf{W}^{(l+1)}\), \(W_{ij}^{(l+1)}\), is a random sample from \(\mathcal{N}(0,\sigma_{l}^{2})\) and \(\mathbf{b}^{(l+1)}\) is \(\mathbf{0}\), then the following equality holds for all \(k=1,2,\cdots,n_{l+1}\) when \(\sigma_{l}=\sqrt{\frac{2}{n_{l}}}\) with sufficiently large \(n_{l}\):_

\[1\approx\left\|J_{\mathbf{\theta}_{0}^{(l+1)}}^{(k)}(\mathbf{x}_{l})\right\|_{F}=\|\mathbf{W} ^{(l+1)}\mathds{1}(\mathbf{x}_{l}>0)\|_{F}, \tag{21}\]

_where \(\mathds{1}(\mathbf{x}_{l}>0)\) turns each positive entry in \(\mathbf{x}_{l}\) to \(1\) and \(0\) otherwise (proved in SSA.3)._
In order to prevent input-output detachment, we thus introduce an additional regularization term:

\[\mathcal{L}^{iod}(\mathbf{\theta}_{0};p(\mathbf{x}))=\mathbb{E}_{\mathbf{x}\sim p(\mathbf{x})} \left[\frac{1}{d}\sum_{i=1}^{d}\left\{\max_{l\in\{0,1,\cdots,L-1\}}\left(1- \left\|J^{(i)}_{\mathbf{\theta}_{0}^{(L,i)}}(\mathbf{x}_{l})\right\|_{F}\right)^{2} \right\}\right], \tag{22}\]

where \(\mathbf{x}_{l}\) is a vector of pre-activated neurons at the \(l\)-th layer and \(\mathbf{x}_{0}\) is an input vector. By minimizing equation 22 with respect to \(\mathbf{\theta}_{0}\), we prevent \(\mathbf{\theta}_{0}\) from being constant, and consequently all nearby models as well, which we demonstrate empirically in SSB.2.

### Hyperparameters and our recommendation

We designed three loss functions to find a good initial parameter configuration \(\mathbf{\theta}_{0}^{*}\) for \(d\)-way classification, using only unlabelled examples; i) \(\mathcal{L}^{uni}(\mathbf{\theta}_{0};\mathbf{\Sigma},\Delta^{d-1},\gamma)\) in SS3.1; ii) \(\mathcal{L}^{sd}(\mathbf{\theta}_{0};\mathbf{\Sigma},d)\) in SS3.2.1; iii) \(\mathcal{L}^{iod}(\mathbf{\theta}_{0})\) in SS3.2.2. \(\mathcal{L}^{uni}(\mathbf{\theta}_{0};\mathbf{\Sigma},\Delta^{d-1},\gamma)\) makes our model predictions be evenly spread over \(\Delta^{d-1}\) centered on \(\mathbf{\theta}_{0}\). \(\mathcal{L}^{sd}(\mathbf{\theta}_{0};\mathbf{\Sigma},d)\) encourages the neighborhood of \(\mathbf{\theta}_{0}\) to have solutions specialized for \(d\)-way classification by preventing _degenerate softmax_. \(\mathcal{L}^{iod}(\mathbf{\theta}_{0})\) avoids the issue of _input-output detachment_. We additively combine all these to form the final loss function:

\[\mathcal{L}(\mathbf{\theta}_{0};\mathbf{\Sigma},\Delta^{d-1},\gamma,p( \mathbf{x}),\lambda,\xi)= \mathcal{L}^{uni}(\mathbf{\theta}_{0};\mathbf{\Sigma},\Delta^{d-1},\gamma,p (\mathbf{x})) \tag{23}\] \[+\lambda\mathcal{L}^{sd}(\mathbf{\theta}_{0};\mathbf{\Sigma},d,p(\mathbf{x}))\] \[+\xi\mathcal{L}^{iod}(\mathbf{\theta}_{0};p(\mathbf{x})).\]

In SSB.3, we empirically present that \(\mathcal{L}^{sd}\) and \(\mathcal{L}^{iod}\) indeed prevent the degenerate softmax and the input-output detachment, and all these three loss functions in equation 23 are necessary to find a good initial parameter configuration. In the rest of this section, we provide guidelines on how to choose some of the hyperparameters.

We select the bandwidth of MMD in \(\mathcal{L}^{uni}\), \(\gamma\), based on the median heuristic (Smola & Scholkopf, 1998). It uses the median of all pairwise distances for the Gaussian kernel in equation 9. This technique is commonly used in many unsupervised learning based on the Gaussian kernel Garreau et al. (2017) such as kernel CCA (Bach & Jordan, 2002) and kernel two-sample test (Gretton et al., 2012). For more detailed description of the median heuristic in our experiments, see SSC.1.

For \(\mathbf{\Sigma}=\texttt{diag}(\sigma_{1}^{2},\sigma_{2}^{2},\cdots,\sigma_{m}^{2})\) of both \(\mathcal{L}^{uni}\) and \(\mathcal{L}^{sd}\), each \(\sigma_{i}^{2}\) corresponding to \(\theta_{0,i}\) is set based on the number of neurons connected to \(\theta_{0,i}\). For instance, if \(\theta_{0,i}\) is the entry of either \(\mathbf{W}\in\mathbb{R}^{n_{out}\times n_{in}}\) or \(\mathbf{b}\in\mathbb{R}^{n_{out}}\) (i.e., a parameter in a fully-connected layer), we set \(\sigma_{i}\) to \(\sqrt{s^{2}/n_{in}}\) for \(\mathbf{W}\) and \(\sqrt{s^{2}/n_{out}}\) for \(\mathbf{b}\) where \(s\) is a hyperparameter shared across all \(i\)'s. For all the experiments in SS5, we set \(s=\sqrt{0.5}\), based on the preliminary experiments in SSC.2.

In the cases of \(\lambda\) and \(\xi\), we mainly focus on selecting \(\lambda\) while fixing \(\xi\) to \(1\), because these two loss functions, \(\mathcal{L}^{uni}\) and \(\mathcal{L}^{iod}\), are intertwined. We use \(\lambda=0.4\) for all the experiments in SS5. With \(\lambda=0.4\), we observed in the preliminary experiments that both \(\mathcal{L}^{uni}\) and \(\mathcal{L}^{sd}\) decrease. See SSC.3 for more details.

## 4 Experimental Settings

To evaluate our algorithm, we fine-tune deep neural networks on the various binary downstream tasks synthetically created out of existing dataset. Here, we describe the experimental setup.

Datasets and tasksWe derive binary tasks from MNIST (LeCun et al., 1998), using the original labels. For example, we can create a binary classification problem, distinguishing odd and even numbers from MNIST which originally has 10 classes (0-9 digits). In this way, we can create \(2^{10}-2\) tasks from MNIST. After we define how to convert the original labels to either 0 or 1, we randomly select \(N\) (for training) + \(0.2N\) (for validation) instances, which allows us to test the impact of the size of labelled set. We standardize each image to have zero-mean and unit variance across all the examples. We do not use any data augmentation.

ModelsWe train a multi-layer perceptron with fully-connected layers, FCN, on MNIST. FCN has three hidden layers with ReLU (Nair & Hinton, 2010) nonlinearity. +BN refers to the addition of
batch normalization (Ioffe & Szegedy, 2015) to all hidden layers before ReLU. Additional details about the network architectures are included in SSD.1.

BaselinesIn order to assess the effectiveness of the proposed approach, we compare it against more conventional approaches to initialization. First, we compare our approach against data-agnostic initialization schemes, including Xavier initialization (Glorot & Bengio, 2010) and He initialization (He et al., 2015). We also compare it to _R.label_ which refers to a data-dependent initialization scheme proposed by Pondenkandath et al. (2018). In the case of R.label, we randomly assign labels to the examples in each mini-batch and minimize the cross entropy loss. Both our initial parameter configuration and R.label's initial parameter configuration are pre-trained on the same number of unlabelled examples for the same maximum number of epochs. For each pre-training run, we choose the parameter configuration based on the pre-training loss. See SSD.2 for more details about the baselines and our pre-training setup.

Orthogonal to these initialization schemes, we also test adding batch normalization to these baseline approaches. It has been observed by some that batch normalization makes learning less sensitive to initialization (Ioffe & Szegedy, 2015).

Training and evaluationFor each initialization scheme, we fine-tune the network by minimizing the cross entropy loss, using Adam (Kingma & Ba, 2014) with a fixed learning rate of \(10^{-3}\) and momentum parameters set to \((\beta_{1},\beta_{2})=(0.9,0.999)\). We use mini-batches of size 50 and train the network for up to 10 epochs without any regularization. For each binary task, we monitor the validation loss over the epochs and calculate the test accuracy (%) on 10,000 test examples when the validation loss is at its minimum. We then report the mean and standard deviation of the test accuracy (%) across 20 random binary tasks. We repeat this whole set of experiments four times, for each setup.

## 5 Results

Table 1 shows that the average test scores on 20 random binary tasks across 4 random runs. The 20 binary tasks for each run is the same regardless of model, initialization, and pre-training. Pre-training FCN with 60,000 unlabelled examples by our algorithm improves average test accuracy across 20 random tasks compared to that of training FCN from scratch, and this improvement is greater than the number of labelled instances is small. Furthermore, our test scores are better than all the schemes

\begin{table}
\begin{tabular}{c c c|c c c c} \hline
**Model** & **Init** & **Pre-trained** & **N=5** & **N=10** & **N=20** & **N=40** \\ \hline \hline FCN & Xavier & Ours & **82.42**\(\pm\)0.72 & 85.98\(\pm\)0.65 & **90.07**\(\pm\)0.17 & 92.48\(\pm\)0.57 \\ FCN & Xavier & - & 79.63\(\pm\)0.78 & 83.70\(\pm\)0.59 & 87.54\(\pm\)0.67 & 90.91\(\pm\)0.53 \\ FCN & Xavier & R.label & 76.81\(\pm\)2.13 & 83.34\(\pm\)0.79 & 87.53\(\pm\)0.91 & 90.88\(\pm\)0.52 \\ FCN+BN & Xavier & - & 77.09\(\pm\)1.22 & 83.50\(\pm\)0.44 & 88.00\(\pm\)0.60 & 91.48\(\pm\)0.53 \\ FCN+BN & Xavier & R.label & 78.87\(\pm\)1.75 & 84.38\(\pm\)0.97 & 88.71\(\pm\)0.53 & 91.57\(\pm\)0.59 \\ \hline FCN & He & Ours & 82.27\(\pm\)0.78 & **86.46**\(\pm\)0.37 & 89.69\(\pm\)0.28 & **92.61**\(\pm\)0.51 \\ FCN & He & - & 79.17\(\pm\)1.21 & 83.41\(\pm\)0.92 & 87.96\(\pm\)0.64 & 91.34\(\pm\)0.37 \\ FCN & He & R.label & 77.41\(\pm\)2.09 & 83.52\(\pm\)0.77 & 87.31\(\pm\)0.68 & 90.66\(\pm\)0.41 \\ FCN+BN & He & - & 76.89\(\pm\)1.48 & 83.01\(\pm\)0.98 & 88.01\(\pm\)0.66 & 91.55\(\pm\)0.57 \\ FCN+BN & He & R.label & 78.82\(\pm\)0.78 & 85.33\(\pm\)0.62 & 89.15\(\pm\)0.68 & 92.14\(\pm\)0.67 \\ \hline \end{tabular}
\end{table}
Table 1: We present the average (\(\pm\)stdev) test scores on MNIST across four random experiments by varying the number of labelled examples (\(10N\) for training and \(2N\) for validation). We denote the random label pre-training by _R.label_. **Bold** marks the best score within each column. For all \(N\), our initialization approximates various tasks better than the others do. Especially, when the number of labelled examples is small, the improvement is significant. Although both R.label and our initialization use 60,000 unlabelled data, our pre-training is superior to R.label. The positive effect of batch normalization (+BN) can be observed with \(N=40\), but its effect does not match that of our approach. Compared to FCN trained from scratch, we observe that +BN negatively impacts on the test score when the number of labelled instances is small (\(N=5\)) while our initialization improves the test score regardless of \(N\).
applied to FCN + BN which has more parameters than FCN. Both R.label and +BN bring the positive effect when the number of labelled examples is sufficient (N=40). However, for \(N=5\), both hurt the test performance of the randomly initialized plain network.

We also present the standard deviation of test scores across 20 binary random tasks created from MNIST in Table 2. Similar to Table 1, our initialization improves the ability to solve most of downstream tasks, and this improvement is greater when the number of labelled instances is small. We also observe R.label and +BN can hurt this ability in terms of the standard deviation for \(N=5\).

## 6 Conclusion

In this paper we proposed a novel criterion for identifying good initialization of parameters in deep neural networks. This criterion looks at the distribution over models derived from parameter configurations in the vicinity of an initial parameter configuration. If this distribution is close to a uniform distribution, the initial parameters are considered good, since we can easily reach any possible solution rapidly from there on.

We then derived an unsupervised initialization algorithm based on this criterion. In addition to maximizing this uniformity, our algorithm prevents two degenerate cases; (1) degenerate softmax and (2) input-output detachment. Our experiments reveal that the model initialized by our algorithm can be trained better than the one trained from scratch, in terms of average test accuracy across a diverse set of tasks. This improvement was found to be comparable to or better than random label pre-training (Pondenkandath et al., 2018; Maennel et al., 2020) and batch normalization (Ioffe and Szegedy, 2015) combined with typical initialization strategies.

The effectiveness of the proposed approach leaves us with one puzzling question. The proposed algorithm does not take into account the use of gradient-based optimization, unlike model-agnostic meta-learning (Finn et al., 2017), and it could still find initial parameters that were amenable to gradient-based fine-tuning. This raises a question on the relative importance between initialization and the choice of optimizer in deep learning. We leave this question for the future.

#### Acknowledgments

This work was supported by 42dot, Hyundai Motor Company (under the project Uncertainty in Neural Sequence Modeling), Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI), and NSF Award 1922658 NRT-HDR:

\begin{table}
\begin{tabular}{c c c|c c c c} \hline
**Model** & **Init** & **Pre-trained** & **N=5** & **N=10** & **N=20** & **N=40** \\ \hline \hline FCN & Xavier & Ours & **4.76\(\pm\)**0.88 & 4.54\(\pm\)0.52 & **3.01\(\pm\)**0.71 & 2.26\(\pm\)0.40 \\ FCN & Xavier & - & 6.62\(\pm\)1.29 & 5.55\(\pm\)0.62 & 3.54\(\pm\)0.27 & 2.65\(\pm\)0.53 \\ FCN & Xavier & R.label & 6.08\(\pm\)0.92 & 5.02\(\pm\)1.16 & 3.82\(\pm\)0.23 & 2.78\(\pm\)0.49 \\ FCN+BN & Xavier & - & 7.47\(\pm\)1.70 & 5.53\(\pm\)0.69 & 3.72\(\pm\)0.78 & 2.72\(\pm\)0.32 \\ FCN+BN & Xavier & R.label & 6.62\(\pm\)1.50 & 5.44\(\pm\)0.33 & 3.20\(\pm\)0.41 & 2.40\(\pm\)0.37 \\ \hline FCN & He & Ours & 5.26\(\pm\)0.87 & **4.04\(\pm\)**0.80 & 3.25\(\pm\)0.42 & **2.16\(\pm\)**0.40 \\ FCN & He & - & 5.74\(\pm\)0.81 & 5.32\(\pm\)0.45 & 3.31\(\pm\)0.48 & 2.47\(\pm\)0.31 \\ FCN & He & R.label & 6.37\(\pm\)1.10 & 4.84\(\pm\)1.02 & 3.98\(\pm\)0.44 & 3.03\(\pm\)0.85 \\ FCN+BN & He & - & 7.52\(\pm\)0.76 & 6.50\(\pm\)1.76 & 3.59\(\pm\)0.80 & 2.74\(\pm\)0.41 \\ FCN+BN & He & R.label & 7.33\(\pm\)1.10 & 4.95\(\pm\)1.08 & 3.18\(\pm\)0.55 & 2.30\(\pm\)0.28 \\ \hline \end{tabular}
\end{table}
Table 2: We additionally demonstrate the standard deviation of test scores across 20 binary random tasks derived from MNIST by varying the number of labelled examples (\(10N\) for training and \(2N\) for validation). This metric measures the ability to solve most of tasks well (lower is better). We perform four random runs and report the average standard deviation. Here, (\(\pm\)stddev) means the standard deviation across four random experiments. We denote the random label pre-training by _R.label_. **Bold** marks the best score within each column. Similar to Table 1, our initialization solves most of tasks well even if there are a small number of labelled examples. Both +BN and R.label can hurts the performance to approximate various tasks when the number of labelled instances is small (N=5).
FUTURE Foundations, Translation, and Responsibility for Data Science. This work was supported in part through the NYU IT High Performance Computing resources, services, and staff expertise.
