Transferring Learning Trajectories of Neural Networks
Training deep neural networks (DNNs) is computationally expensive, which is
problematic especially when performing duplicated or similar training runs in
model ensemble or fine-tuning pre-trained models, for example. Once we have
trained one DNN on some dataset, we have its learning trajectory (i.e., a
sequence of intermediate parameters during training) which may potentially
contain useful information for learning the dataset. However, there has been no
attempt to utilize such information of a given learning trajectory for another
training. In this paper, we formulate the problem of "transferring" a given
learning trajectory from one initial parameter to another one (learning
transfer problem) and derive the first algorithm to approximately solve it by
matching gradients successively along the trajectory via permutation symmetry.
We empirically show that the transferred parameters achieve non-trivial
accuracy before any direct training, and can be trained significantly faster
than training from scratch.
Daiki Chijiwa
# Transferring Learning Trajectories

###### Abstract

Training deep neural networks (DNNs) is computationally expensive, which is problematic especially when performing duplicated training runs, such as model ensemble or knowledge distillation. Once we have trained one DNN on some dataset, we have its learning trajectory (i.e., a sequence of intermediate parameters during training) which may potentially contain useful information for learning the dataset. However, there has been no attempt to utilize such information of a given learning trajectory for another training. In this paper, we formulate the problem of "transferring" a given learning trajectory from one initial parameter to another one, called _learning transfer problem_, and derive the first algorithm to approximately solve it by matching gradients successively along the trajectory via permutation symmetry. We empirically show that the transferred parameters achieve non-trivial accuracy before any direct training. Also, we analyze the loss landscape property of the transferred parameters, especially from a viewpoint of mode connectivity.

## 1 Introduction

Enormous computational cost is a major issue in deep learning, especially in training large-scale neural networks (NNs). Their highly non-convex objective and high-dimensional parameters make their training difficult and inefficient. Toward a better understanding of training processes of NNs, their loss landscapes [21; 7] have been actively studied from viewpoints of optimization [17; 38; 57] and geometry [14; 47]. One of the geometric approaches to loss landscapes is mode connectivity [15; 9], which shows the existence of low-loss curves between any two optimal solutions trained with different random initializations or data ordering. This indicates a surprising connection between seemingly different independent trainings.

Linear mode connectivity (LMC), a special case of mode connectivity, focuses on whether or not two optimal solutions are connected by a low-loss linear path, which is originally studied in relation to neural network pruning [13]. It is known that the solutions trained from the same initialization (and data ordering in the early phase) tend to be linearly mode connected [45; 13], but otherwise they cannot be linearly connected in general. However, Entezari et al. [10] observed that even two solutions trained from different random initializations can be linearly connected by an appropriate permutation symmetry. Ainsworth et al. [3] developed an efficient method to find such permutations and confirmed the same phenomena with modern NN architectures. These observations strength the expectation on some sort of similarity between two independent training runs even from different random initializations, via permutation symmetry.

In this paper, motivated by these observations, we make the first attempt to leverage such similarity between independent training processes for efficient training. In particular, we introduce a novel problem called _learning transfer problem_, which aims to reduce training costs for seemingly duplicated training runs on the same dataset, such as model ensemble or knowledge distillation, by transferring
a learning trajectory for one initial parameter to another one without actual training. The problem statement is informally stated as follows:

Learning transfer problem (informal)._Suppose that a learning trajectory \((\theta^{0}_{1},\cdots,\theta^{T}_{1})\) is given for an initial parameter \(\theta^{0}_{1}\). Given another initial parameter \(\theta^{0}_{2}\), how can we synthesize the learning trajectory \((\theta^{0}_{2},\cdots,\theta^{T}_{2})\) for the given initialization \(\theta^{0}_{2}\)?_

To tackle this problem, as illustrated in Figure 2, we take an approach to transform the source trajectory \((\theta^{0}_{1},\cdots,\theta^{T}_{1})\) by an appropriate permutation symmetry like the previous works in LMC. In Section 3, we formulate the learning transfer problem in a mathematically rigorous way, and then we derive a theoretically-grounded algorithm to approximately solve it. We also develop practical techniques to reduce the storage and computational cost in our derived method. In Section 4, first we empirically demonstrated that learning trajectories can be successfully transferred between different random or pre-trained initial parameters (Section 4.1). Next we further confirmed that the transferred parameters can indeed accelerate the convergence in the subsequent training (Section 4.2). Finally, we empirically analyze the mode connectivity properties of the transferred parameters. We observed that, while the transferred parameter tends to be linearly mode connected to the source parameter \(\theta^{T}_{1}\), the transferred parameter and the one actually trained from \(\theta^{0}_{2}\) are linearly connected only when the permutation is obtained by matching the source trajectory and the actually trained trajectory (Section 4.3).

## 2 Background

### Neural networks

Let \(L,N\in\mathbb{N}\). Let \(f(x;\theta)\) be an \(L\)-layered neural network (NN) parameterized by \(\theta\in\mathbb{R}^{N}\) with a non-linear activation function \(\sigma:\mathbb{R}\rightarrow\mathbb{R}\) and intermediate dimensions \((d_{0},\cdots,d_{L})\in\mathbb{N}^{L+1}\). Given an input \(x\in\mathbb{R}^{d_{0}}\), the output \(f(x;\theta):=x_{L}\in\mathbb{R}^{d_{L}}\) is computed inductively as follows:

\[x_{i}:=\begin{cases}x,&(i=0)\\ \sigma(W_{i}x_{i-1}+b_{i}),&(1\leq i\leq L-1)\\ W_{L}x_{L-1}+b_{L-1},&(i=L)\end{cases}\]

where \(W_{i}\in\mathbb{R}^{d_{i}\times d_{i-1}},b_{i}\in\mathbb{R}^{d_{i}}\) are weight matrices and bias vectors. Under these notation, the parameter vector \(\theta\) is described as \(\theta=(W_{1},b_{1},\cdots,W_{L},b_{L})\in\mathbb{R}^{N}\).

Stochastic gradient descent (SGD) is a widely used approach for training neural networks. Let \(\mathcal{X}\) be the input space \(\mathbb{R}^{d_{0}}\) and \(\mathcal{Y}\) be the output space \(\mathbb{R}^{d_{L}}\). Let \(\mathcal{D}\) be a probabilistic distribution over the input-output space \(\mathcal{X}\times\mathcal{Y}\), and \(\mathcal{L}:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}\) be a differentiable loss function. SGD trains a neural network \(f(x;\theta)\) by iterating the following steps: (1) Sampling a mini-batch \(B=((x_{1},y_{1}),\cdots,(x_{b},y_{b}))\sim\mathcal{D}^{b}\) of size \(b\in\mathbb{N}\), (2) computing an estimated gradient \(g_{B}:=\frac{1}{b}\sum_{i=1}^{b}\nabla_{\theta}\mathcal{L}(f(x_{i};\theta),y_ {i})\) for the mini-batch and (3) updating the model parameter \(\theta\) by \(\theta-\alpha g_{B}+\text{(momentum)}\) where \(\alpha\in\mathbb{R}_{>0}\) is a fixed or scheduled step size.

### Permutation symmetry of NNs

For simplicity, we assume that all bias vectors \(b_{i}\) are zero by viewing them as a part of the weight matrices. Let \(\Theta\) be the parameter space \(\{\theta=(W_{1},\cdots,W_{L})\in\mathbb{R}^{N}\}\) for the \(L\)-layered neural
network \(f(x;\theta)\). Now we introduce a permutation group action on the parameter space \(\Theta\). For \(n\in\mathbb{N}\), let \(S_{n}\) denotes the symmetric group over \(\{1,\cdots,n\}\subset\mathbb{N}\), which is the set of all bijective mapping \(\sigma:\{1,\cdots,n\}\rightarrow\{1,\cdots,n\}\). We define our permutation group \(G\) by \(G:=S_{d_{1}}\times\cdots\times S_{d_{L-1}}\). The group action \(G\times\Theta\rightarrow\Theta\) is defined as follows: For \(\pi=(\sigma_{1},\cdots,\sigma_{L-1})\in G\) and \(\theta\in\Theta\), the action \(\pi\theta\) is defined by

\[\pi\theta:=(\sigma_{1}W_{1},\cdots,\sigma_{i}W_{i}\sigma_{i-1}^{-1},\cdots,W_ {L}\sigma_{L-1}^{-1})\in\Theta, \tag{1}\]

where each \(\sigma_{i}\) is viewed as the corresponding permutation matrix of size \(d_{i}\times d_{i}\). We call this group action the permutation symmetry of \(L\)-layered neural networks.

Simply speaking, the action \(\pi\theta\) just interchanges the axes of the intermediate vector \(x_{i}\) of the neural network \(f(x;\theta)\) with the corresponding base change of the weight matrices and bias vectors (Figure 1). Thus we can see that this action does not change the output of the neural network, i.e., \(f(x;\pi\theta)=f(x;\theta)\) for every \(x\in\mathcal{X},\theta\in\Theta,\pi\in G\). In other words, the two parameters \(\theta\) and \(\pi\theta\) can be identified from the functional perspective of neural networks.

### Parameter alignment by permutation symmetry

Previous work by Ainsworth et al. [3] attempts to merge given two NN models into one model by leveraging their permutation symmetry. They reduced the merge problem into the parameter alignment problem:

\[\min_{\pi\in G}\lVert\pi\theta_{1}-\theta_{2}\rVert_{2}^{2}=\min_{\pi=(\sigma _{1},\cdots,\sigma_{L-1})}\sum_{1\leq i\leq L}\lVert\sigma_{i}W_{i}\sigma_{i- 1}^{-1}-Z_{i}\rVert_{F}^{2}, \tag{2}\]

where \(\theta_{1}=(W_{1},\cdots,W_{L})\) and \(\theta_{2}=(Z_{1},\cdots,Z_{L})\) are the parameters to be merged. To solve this, they also proposed a coordinate descent algorithm by iteratively solving the following linear optimizations regarding to each \(\sigma_{i}\)'s:

\[\max_{\sigma_{i}\in S_{i}}\langle\sigma_{i},Z_{i}\sigma_{i-1}W_{i}^{\top}+Z_{ i+1}^{\top}\sigma_{i+1}W_{i+1}\rangle \tag{3}\]

The form of this problem has been well-studied as a linear assignment problem, and we can solve it in a very efficient way [33]. Although the coordinate descent algorithm was originally proposed for model merging, we can also use it for other problems involving the parameter alignment problem (Eq. 2).

## 3 Learning Transfer

In this section, first we formulate the problem of transferring learning trajectories (which we call _learning transfer problem_) as a non-linear optimization problem. Next, we derive an algorithm to solve it by reducing the non-linear optimization problem to a sequence of linear optimization problems. Finally, we introduce additional techniques for reducing the storage and computation cost of the derived algorithm.

### Problem formulation

Let \(f(x;\theta)\) be some NN model with an \(N\)-dimensional parameter \(\theta\in\mathbb{R}^{N}\). A sequence of \(N\)-dimensional parameters \((\theta^{0},\cdots,\theta^{T})\in\mathbb{R}^{N\times(T+1)}\) is called a _learning trajectory_ of length \(T\) for the neural network \(f(x;\theta)\) when each \(\theta^{t}\) is the \(t\)-th intermediate parameter during training of \(f\) with the initial parameter \(\theta^{0}\) and the convergent one \(\theta^{T}\). In other words, \(\theta^{t}\) is a result of some iterations of SGD from \(\theta^{t-1}\). Note that here we do not specify what \(t\) is, it could be iteration, epoch or other notion of training time. Now we can state our main problem:

Learning transfer problem (informal).Suppose that a learning trajectory \((\theta^{0}_{1},\cdots,\theta^{T}_{1})\) is given for an initial parameter \(\theta^{0}_{1}\), which we call a _source trajectory_. Given another initial parameter \(\theta^{0}_{2}\) "similar" to \(\theta^{0}_{1}\) in some sense, how can we synthesize the learning trajectory \((\theta^{0}_{2},\cdots,\theta^{T}_{2})\), which we call a _transferted trajectory_, for the given initialization \(\theta^{0}_{2}\)? (Figure 2)

To convert this informal problem into a formal and computable one, we need to formalize the notion of "similarity" between two initial parameters. For this, we assume that the learning trajectories for the two initial parameters are indistinguishable up to permutation sym
(Section 2.2). In other words, for the two learning trajectories \((\theta_{1}^{0},\cdots,\theta_{1}^{T})\) and \((\theta_{2}^{0},\cdots,\theta_{2}^{T})\), we consider the following assumption:

Assumption (P).There exists a permutation \(\pi\) satisfying \(\pi(\theta_{1}^{t}-\theta_{1}^{t-1})\approx\theta_{2}^{t}-\theta_{2}^{t-1}\) for \(t=1,\cdots,T\), where the transformation \(\pi(\theta_{1}^{t}-\theta_{1}^{t-1})\) is as defined in Section 2.2.

Under this assumption, if we know the permutation \(\pi\) providing the equivalence between the source and transferred trajectories, we can recover the latter one \((\theta_{2}^{0},\cdots,\theta_{2}^{T})\) from the former one \((\theta_{1}^{0},\cdots,\theta_{1}^{T})\) and the permutation \(\pi\), by setting \(\theta_{2}^{t}:=\theta_{2}^{t-1}+\pi(\theta_{1}^{t}-\theta_{1}^{t-1})\) inductively on \(t\) (Figure 2). Therefore, the learning-trajectory problem can be reduced to estimating the permutation \(\pi\) from the source trajectory \((\theta_{1}^{0},\cdots,\theta_{1}^{T})\) and the given initialization \(\theta_{2}^{0}\).

Naively, to estimate the permutation \(\pi\), we want to consider the following optimization problem:

\[\min_{\pi}\sum_{t=1}^{T}\left\|\pi(\theta_{1}^{t}-\theta_{1}^{t-1})-(\theta_{2 }^{t}-\theta_{2}^{t-1})\right\|_{2}^{2} \tag{4}\]

However, this problem is ill-defined in our setting since each \(\theta_{2}^{t}\) is not available for \(1\leq t\leq T\) in advance. Even if we defined \(\theta_{2}^{t}:=\theta_{2}^{t-1}+\pi(\theta_{1}^{t}-\theta_{1}^{t-1})\) in the equation (4) as discussed above, the optimization problem became trivial since any permutation \(\pi\) makes the \(L^{2}\) norm to be zero.

Thus we need to fix the optimization problem (Eq. 4) not to involve unavailable terms. We notice that the difference \(\theta_{2}^{t}-\theta_{2}^{t-1}\) can be approximated by a negative gradient at \(\theta_{2}^{t-1}\) averaged over a mini-batch if the trajectory is enough fine-grained. Therefore, we can consider the approximated version of equation (4) as follows:

\[\mathcal{P}_{T}:\ \min_{\pi}\sum_{t=0}^{T-1}\left\|\pi\nabla_{\theta_{1}^{t} }\mathcal{L}-\nabla_{\theta_{2,\pi}^{t}}\mathcal{L}\right\|_{2}^{2},\ \text{where}\ \theta_{2,\pi}^{t}:=\theta_{2,\pi}^{t-1}+\pi(\theta_{1}^{t}-\theta_{1}^{t-1}). \tag{5}\]

In contrast to the equation (4), this optimization problem is well-defined even in our setting because each \(\theta_{2,\pi}^{t}\) is defined by using the previous parameter \(\theta_{2,\pi}^{t-1}\) inductively.

### Algorithm: gradient matching along trajectory

Now our goal is to solve the optimization problem \(\mathcal{P}_{T}\) (Eq. 5). However, the problem \(\mathcal{P}_{T}\) seems hard to solve directly because the variable \(\pi\) appears non-linearly in the second term \(\nabla_{\theta_{2,\pi}^{t}}\mathcal{L}\). To avoid the non-linearity, we introduce a sequence of linear sub-problems \(\{\mathcal{P}_{s}^{\prime}\}_{1\leq s\leq T}\) whose solution converges to the solution for \(\mathcal{P}_{T}\). For each \(s\in\{1,\cdots,T\}\), we consider the following problem:

\[\mathcal{P}_{s}^{\prime}:\ \min_{\pi_{s}}\sum_{t=0}^{s-1}\left\|\pi_{s} \nabla_{\theta_{1}^{t}}\mathcal{L}-\nabla_{\theta_{2,\pi_{s-1}}^{t}}\mathcal{ L}\right\|_{2}^{2} \tag{6}\]

Since the second term in \(\mathcal{P}_{s}^{\prime}\) uses the solution \(\pi_{s-1}\) for the previous sub-problem \(\mathcal{P}_{s-1}^{\prime}\), the unknown variable \(\pi_{s}\) appears only in the first term \(\pi_{s}\nabla_{\theta_{1}^{t}}\mathcal{L}\) in a linear way. Moreover, the following lemma implies that the final solution \(\pi_{T}\) from the sequence \(\{\mathcal{P}_{s}^{\prime}\}_{1\leq s\leq T}\) approximates the solution for the original problem \(\mathcal{P}_{T}\):

**Lemma 3.1**: _Under some regularity assumption, we have \(\theta_{2,\pi_{s}}^{t}\approx\theta_{2,\pi_{s}}^{t}\), for \(0\leq t\leq s<s^{\prime}\)._

The proof will be given in Appendix. Indeed, by this approximation, we find out that the solution \(\pi_{T}\) for the last sub-problem \(\mathcal{P}_{T}^{\prime}\) minimizes

\[\sum_{t=0}^{T-1}\left\|\pi_{T}\nabla_{\theta_{1}^{t}}\mathcal{L}-\nabla_{ \theta_{2,\pi_{T-1}}^{t}}\mathcal{L}\right\|_{2}^{2}\approx\sum_{t=0}^{T-1} \left\|\pi_{T}\nabla_{\theta_{1}^{t}}\mathcal{L}-\nabla_{\theta_{2,\pi_{T}}^{t} }\mathcal{L}\right\|_{2}^{2}, \tag{7}\]

where the right-hand side is nothing but the objective of the original problem \(\mathcal{P}_{T}\).

Algorithm 1 gives a step-by-step procedure to obtain the transferred learning trajectory \((\theta_{2}^{1},\cdots,\theta_{2}^{T})\) by solving the sub-problems \(\{\mathcal{P}_{s}^{\prime}\}_{0\leq s\leq T}\) sequentially. In lines 2-6, it computes an average of gradients \(\nabla_{\theta}\mathcal{L}\) over a single mini-batch for each \(\theta=\theta_{1}^{t-1},\theta_{2}^{t-1},(1\leq t\leq s)\), which is required in the \(s\)-th sub-problem \(\mathcal{P}_{s}^{\prime}\) (Eq. 6). In line 7, the \(s\)-th permutation \(\pi_{s}\) is obtained as a solution of the sub-problem \(\mathcal{P}_{s}^{\prime}\), which can be solved as a linear optimization (Eq. 3) using the coordinate descent algorithm proposed in [3]. Then we update the transferred parameter \(\theta_{2}^{t}\) for \(t=1,\cdots,s\) in line 8.
### Additional techniques

While Algorithm 1 solves the learning transfer problem (Eq. 5) approximately, it still has some issues in terms of storage and computation cost. Here we explain two practical techniques to resolve them.

Linear trajectory.In terms of the storage cost, Algorithm 1 requires a capacity of \(T+1\) times the model size to keep a learning trajectory of length \(T\), which will be a more substantial issue as model size increases or the trajectory becomes fine-grained. To reduce the required storage capacity, instead of keeping the entire trajectory, we propose to imitate it by linearly interpolating the end points. In other words, given an initial parameter \(\theta_{1}^{0}\) and the final \(\theta_{1}^{T}\), we define a new trajectory \([\theta_{0}^{0}:\theta_{1}^{T}]:=(\theta_{1}^{0},\cdots,\theta_{1}^{0},\cdots,\theta_{1}^{T})\) with \(\theta_{1}^{t}:=(1-\lambda_{t})\theta_{1}^{0}+\lambda_{t}\theta_{1}^{T}\) and \(0=\lambda_{0}\leq\cdots\leq\lambda_{t}\leq\cdots\leq\lambda_{T}=1\). This idea is motivated by the previous findings on monotonic linear interpolation [16; 12]. For our experiments, we consider two strategies for scheduling \(\lambda_{t}\): (1) uniform scheduling by \(\lambda_{t+1}-\lambda_{t}:=1/T\), and (2) cosine scheduling by \(\lambda_{t+1}-\lambda_{t}:=\alpha_{t}/Z\) with \(\alpha_{t}:=1+\cos(\pi t/T)\) and the normalizing factor \(Z:=\sum_{t=0}^{T}\alpha_{t}\). We found that the cosine scheduling is better than the uniform one (Fig. 2(a)) probably because the learning rate during the training of \(\theta_{1}^{T}\) is also cosine annealed in our setting.

Next, using the cosine scheduling, we compare the transferred results between the linear trajectory \([\theta_{1}^{0}:\theta_{1}^{T}]\) and the actual trajectory \((\theta_{1}^{0},\cdots,\theta_{1}^{T})\) where each \(\theta_{1}^{t}\) is a checkpoint at the \(t\)-th training epoch on CIFAR-10 with \(T=60\). Interestingly, the transfer of the linear trajectory is more stable and has less variance than the transfer of the actual one. This may be because the actual trajectory contains noisy information while the linear trajectory is directed towards the optimal solution \(\theta_{1}^{T}\). Due to its storage efficiency and stability in accuracy, we employ the linear trajectory with \(T=30\) (except for MNIST where \(T=10\) is used) throughout our experiments in Section 4.

Gradient caching.In terms of the computation cost, Algorithm 1 requires \(O(T^{2})\) times gradient computation. To reduce the number of gradient computation, we propose to cache the gradients once computed instead of re-computing them for every \(s=1,\cdots,T\). In fact, the cached gradients \(\nabla_{\theta_{2,\pi_{s}}^{t-1}}\mathcal{L}\) and the re-computed gradients \(\nabla_{\theta_{2,\pi_{s}}^{t-1}}\mathcal{L}\) are not the same quantity exactly since the intermediate parameter \(\theta_{2,\pi_{s}}^{t-1}=\theta_{2}^{0}+\pi_{s}(\theta_{1}^{t-1}-\theta_{1}^ {0})\) takes different values for each \(s\). However, they can be treated as approximately equal by Lemma 3.1 if we assume the continuity of the gradients. Now we can reduce the number of gradient computation from \(O(T^{2})\) to \(O(T)\) by caching the gradients once computed. We describe this computationally efficient version in Algorithm 2.

Figure 3: Transfer experiments on CIFAR-10 with Conv8.
## 4 Experiments

In this section, we empirically evaluate how learning transfer works on standard vision datasets. First, we compare our proposed methods (**GMT**, **FGMT**) and two baselines (**Naive**, **Oracle**), which are explained below, under the following two scenarios: (1) transferring learning trajectories starting from randomly initialized parameters and (2) transferring learning trajectories starting from pre-trained parameters (Section 4.1). Next, we evaluate how efficiently the transferred parameters can be trained in their subsequent training. (Section 4.2). Finally, we investigate the loss landscape properties of the transferred parameters from a viewpoint of mode connectivity (Section 4.3). The details on the datasets, NN architectures and training settings used in our experiments are provided in Appendix.

Baselines.As baselines for learning transfer, we introduce two natural methods: **Naive** and **Oracle**. Both in the two baselines, we transfer a given learning trajectory \((\theta^{0}_{1},\cdots,\theta^{T}_{1})\) by a single permutation \(\pi_{\texttt{naive}}\) or \(\pi_{\texttt{oracle}}\), according to the problem formulation in Section 3.1. In the Naive baseline, we define \(\pi_{\texttt{naive}}\) as the identity permutation, which satisfies \(\pi_{\texttt{naive}}\theta=\theta\). In other words, the transferred parameter by Naive is simply obtained as \(\theta^{t}_{2,\pi_{\texttt{naive}}}=\theta^{0}_{2}+(\theta^{t}_{1}-\theta^{0} _{1})\). On the other hand, in the Oracle baseline, we first obtain a true parameter \(\theta^{T}_{2}\) by actually training the given initial parameter \(\theta^{0}_{2}\) with the same optimizer as training of \(\theta^{T}_{1}\). Then we define \(\pi_{\texttt{oracle}}\) by minimizing the layer-wise \(L^{2}\) distance between the actually trained trajectories \(\theta^{T}_{2}-\theta^{0}_{2}\) and \(\pi_{\texttt{oracle}}(\theta^{T}_{1}-\theta^{0}_{1})\), where we simply apply the coordinate descent [3] explained in Section 2.3. The Oracle baseline is expected to be close to the optimal solution for the learning transfer problem via permutation symmetry.

Source trajectories.In our experiments, as discussed in Section 3.3, we consider to transfer linear trajectories \([\theta^{0}_{1}:\theta^{T}_{1}]\) of length \(T\) rather than actual trajectories for \(\theta^{T}_{1}\) due to the storage cost and instability emerging from noise. For the fixed permutation \(\pi\) (i.e., in the case of the baselines), the final transferred parameter \(\theta^{T}_{2,\pi}\) does not depend on which of the two trajectories we use. Therefore we note that the choice of the two types of trajectory does not lead to unfair evaluation for the baselines. Nevertheless, it is an interesting direction for future work to explore more elaborated methods that can transfer actual trajectories effectively.

Figure 4: We plot the validation accuracies of the transferred parameter \(\theta^{t}_{2,\pi_{t}}\) for each \(t=1,\cdots,T\) with various datasets and NN architectures. We also provide the standard deviation over three runs for each experiment. (Upper) Transfer of a learning trajectory on a single dataset between random initial parameters. (Lower) Transfer of a fine-tuning trajectory between pre-trained parameters. For example, “ImageNet \(\rightarrow\) Cars” means the transfer of the fine-tuning trajectory on the Cars dataset between the parameters pre-trained on ImageNet.
### Learning transfer experiments

Figure 4 shows the validation accuracies of the transferred parameters \(\theta^{t}_{2,\pi_{t}}\) for each timestep \(t=1,\cdots,T\) during the transfer. For the baselines (Naive and Oracle), we set the \(t\)-th permutation \(\pi_{t}\) by the fixed \(\pi_{\mathtt{naive}}\) and \(\pi_{\mathtt{oracle}}\) for every \(t\). For our algorithms (GMT and FGMT), the \(t\)-th permutation \(\pi_{t}\) corresponds to \(\pi_{s}\) in Algorithm 1 and 2.

In the upper figures 3(a)-3(c), we transfer a learning trajectory trained with a random initial parameter on a single dataset (MNIST [34], CIFAR-10 [32] and ImageNet [8]) to another random initial parameter. We will refer to this experimental setting as the random initialization scenario. We can see that our methods successfully approximate the Oracle baseline and outperform the Naive baseline on each dataset. Also, we can see that FGMT, the fast approximation version of GMT, performs very similarly to or even outperforms GMT. This is probably because the update of \(\pi_{t}\) affects the previously computed gradients in GMT, but not in FGMT, resulting in the stable behavior of FGMT.

In the lower figures 3(d)-3(f), we transfer a learning trajectory of fine-tuning on a specialized dataset (a 10-classes subset of CIFAR-100 [32], Stanford Cars [31] and CUB-200-2011 [52]) from an initial parameter that is pre-trained on ImageNet to another pre-trained one. We refer to this experimental setting as the pre-trained initialization scenario. This scenario seems to be more difficult to transfer the learning trajectories than the random initialization scenario shown in the upper figures, since the Naive baseline always fails to transfer the trajectories. We can see that, while our methods behave closely to the Oracle baseline up to the middle of the timestep, the accuracy deteriorates immediately after that. Nevertheless, the peak accuracies of our methods largely outperform those of the Naive baseline. By stopping the transfer at the peak points (i.e., so-called early stopping), we can take an advantage of the transferred parameters as we will see in the next section.

Figure 5: Subsequent training of the transferred parameters. In each figure, we plot the validation accuracies during the training of the transferred parameters for \(10\) epochs, on the same dataset as the source trajectory being trained on. The transferred parameters obtained by solving the equation (5) can be trained faster than the Naive baseline especially in the pre-trained initialization scenario.

Figure 6: Mode connectivity analysis on CIFAR-10 with Conv8. In Figure 5(a) and 5(b) we plot the validation losses along the linear path between two parameters. The label Source* refers to the appropriately permuted source parameter corresponding to each transfer method, and the label Trained refers to the parameter actually trained from the initial parameter \(\theta^{0}_{2}\). In Figure 5(c), we plot the validation losses over the \(uv\)-plane following the same protocol as Garipov et al. [15].
### Efficient training of transferred parameters

In the previous section, we obtained the transferred parameters that achieve non-trivial accuracy without any direct training. Here we evaluate how efficiently the transferred parameters can be trained in their subsequent training, by training for \(10\) epochs on the same dataset as the source trajectory being trained on. We started each training from the transferred parameter \(\theta^{t}_{2,\pi_{t}}\) at the best trajectory step \(t\) in Figure 4. Figure 5 shows the validation accuracies for each epoch in the training of the transferred parameters. In the random initialization scenario, although the differences between the methods seem quite small (Figure 4(a), 4(b)), the mean values still follow a similar trend as in the transfer experiments in Figure 4. This is because the Naive baseline also achieves non-trivial accuracy already when transferred in this scenario. On the other hand, in the pre-trained initialization scenario, the parameters transferred by our methods and the Oracle baseline learns the datasets faster than the parameters transferred by the Naive baseline (Figure 4(c), 4(d)). Thus the benefit of the transferred parameters seems to be greater in the pre-trained initialization scenario than in the random initialization scenario.

### Mode connectivity analysis

Here we investigate the loss landscape properties of the transferred parameters on CIFAR-10, especially from the viewpoint of mode connectivity. Figure 5(a) visualizes the \(1\)-dimensional loss landscape along the linear path between the appropriately permuted source parameter \(\pi\theta^{T}_{1}\) and the transferred parameter \(\theta^{T}_{2,\pi}\), where we set \(\pi=\pi_{T}\) for GMT, \(\pi=\pi_{\texttt{naive}}\) and \(\pi=\pi_{\texttt{oracle}}\) for the Naive and Oracle baselines. In the figure, we refer to the appropriately permuted source parameters \(\theta^{T}_{2,\pi}\) by Source*. The results show that the transferred parameters lie in the same basin as the source parameter for all methods including baselines. On the other hand, Figure 5(b) visualizes the \(1\)-dimensional landscape along the linear path between the true parameter \(\theta^{T}_{2}\) (i.e., actually trained from \(\theta^{0}_{2}\)) and the transferred parameter \(\theta^{T}_{2,\pi}\). In this visualization, we notice that only the oracle baseline is closer to linearly mode connected with the true parameter. Futhermore, as we can see in the \(2\)-dimensional landscape around the true parameter (Figure 5(c)), the transferred parameter by Oracle is exactly not linearly mode connected to the true parameter, but they still live in the same basin. It remains as an open question how we can obtain the permutation \(\pi\) by which the transferred parameter \(\theta^{T}_{2,\pi}\) lives in the same basin as the true parameter \(\theta^{T}_{2}\), without using the true parameter itself unlike the Oracle baseline.

## 5 Related Work

Loss landscape, linear mode connectivity, permutation symmetry.Loss landscape of training deep neural network has been actively studied in an effort to unravel mysteries of non-convex optimization in deep learning [21; 7; 35; 29; 36]. One of the mysteries in deep learning is the stability and consistency of their training processes and solutions, despite of the multiple sources of randomness such as random initialization, data ordering and data augmentation [11; 6; 50; 26]. Previous studies of mode connectivity both theoretically [14; 47] and empirically [9; 15] demonstrate the existence of low-loss curves between any two optimal solutions trained independently with different randomness.

Linear mode connectivity (LMC) is a special case of mode connectivity where two optimal solutions are connected by a low-loss linear path [45; 13; 44; 28; 41]. Entezari et al. [10] observed that even two solutions trained from different random initialization can be linearly connected by an appropriate permutation symmetry. Ainsworth et al. [3] developed an efficient method to find such permutations, and Jordan et al. [27] extends it to NN architectures with Batch normalization [25]. Their observations strength the expectation on some sort of similarity between two training processes even from different random initializations, via permutation symmetry. In our work, based on these observations, we attempt to transfer one training process to another initial parameter by permutation symmetry.

Another line of research related to our work is the studies of monotonic linear interpolation (MLI) between an initialization and its trained result. Goodfellow et al. [16] first observed that the losses are monotonically decreasing along the linear path between an initial parameter and the trained one. Frankle [12] and Lucas et al. [42] confirmed that the losses are monotonically non-increasing even with modern network architectures such as CNNs and ResNets [19]. Vlaar and Frankle [51] empirically analyzed which factor in NN training influences the shape of the non-increasing loss curve
along the linear interpolation, and Wang et al. [53] theoretically analyzed the plateau phenomenon in the early phase of the linear interpolation. Motivated by these observations, we introduced the notion of linear trajectories in Section 3.3 to reduce storage costs in our learning transfer.

Model editing.Our approach of transferring learning trajectories can be also considered as a kind of model editing [49; 46; 24; 23] in the parameter space because we modify a given initial parameter by adding an appropriately permuted trajectory. In particular, a recent work by Ilharco et al. [23] is closely related to our work. They proposed to arithmetically edit a pre-trained NN with a task vector, which is defined by subtracting the initial pre-trained parameter from the parameter fine-tuned on a specific task. From our viewpoint, task vectors can be seen as one-step learning trajectories (i.e., learning trajectories with \(T=1\)). Model merging (or model fusion) [48; 43; 55; 37] is also related in the sense of the calculation in the parameter space.

Efficient training for multiple NNs.There are several literatures that attempt to reduce the computation costs in training multiple NNs. Fast ensemble is an approach to reduce the cost in ensemble training by cyclically scheduled learning rate [22] or by searching different optimal basins in loss landscape [15; 11; 54; 5]. A recent work by Liu et al. [39] leverages knowledge distillation [20] from one training to accelerate the subsequent trainings. Our approach differs from theirs in that we try to establish a general principle to transfer learning trajectories. Also, the warm-starting technique investigated by Ash and Adams [4] seems to be related in that they subsequently train from a once trained network. There may be some connection between their and our approaches, which remains for future work.

Gradient matching.The gradient information obtained during training has been utilized in other areas outside of ours. For example, in dataset distillation, Zhao et al. [58] optimized a distilled dataset by minimizing layer-wise cosine similarities between gradients on the distilled dataset and the real one, starting from random initial parameters, which leads to similar training results on those datasets. Similarly, Yin et al. [56] successfully recovered private training data from its gradient by minimizing the distance between gradients. In contrast to their problem where input data is optimized, our problem requires optimizing unknown transformation for NN parameters. In addition, our problem requires matching the entire learning trajectories, which are too computationally expensive to be computed naively.

## 6 Limitations

One limitation in our approach via permutation symmetry is that it requires consistency in the NN architectures of the source and transferred models. However, at least for architectures with different neuron sizes, we expect that this limitation can be addressed by using doubly stochastic matrices instead of permutations, as in previous work on model merging [48], which remains for future work.

In addition, we mainly focused on how we can successful transfer given learning trajectories. Thus its practical applications such as model ensemble and knowledge distillation, where the latter needs the extended method for transferring between different NN architectures as explained above, have not yet been explored. Also the ad-hoc techniques employed in our experiments, such as cosine scheduling for the linear trajectory (Section 3.3) and early stopping (Section 4.1), may be sub-optimal for practical use. It remains for future work to improve these techniques and explore the practical applications.

## 7 Conclusion

In this work, we formulated the problem of how we can synthesize an unknown learning trajectory from the known one, called the learning transfer problem. We derived an algorithm that approximately solves it and developed practical techniques to reduce the storage and computation costs of the algorithm. In our experiments, we confirmed that our algorithm successfully transfers a given learning trajectory at the same level as the Oracle baseline. Moreover, we observed that the transferred parameters can be efficiently trained in the subsequent training especially in the pre-trained initialization scenario. We also analyzed the mode connectivity around the transferred parameters and the source or actually trained parameters. We hope that our investigations will be helpful for future research in this new direction to reduce training costs by transferring learning trajectories.