StitchNet: Composing Neural Networks from Pre-Trained Fragments
We propose StitchNet, a novel neural network creation paradigm that stitches
together fragments (one or more consecutive network layers) from multiple
pre-trained neural networks. StitchNet allows the creation of high-performing
neural networks without the large compute and data requirements needed under
traditional model creation processes via backpropagation training. We leverage
Centered Kernel Alignment (CKA) as a compatibility measure to efficiently guide
the selection of these fragments in composing a network for a given task
tailored to specific accuracy needs and computing resource constraints. We then
show that these fragments can be stitched together to create neural networks
with accuracy comparable to that of traditionally trained networks at a
fraction of computing resource and data requirements. Finally, we explore a
novel on-the-fly personalized model creation and inference application enabled
by this new paradigm. The code is available at
https://github.com/steerapi/stitchnet.
Surat Teerapittayanon, Marcus Comiter, Brad McDanel, H. T. Kung
# StitchNet: Composing Neural Networks from Pre-Trained Fragments

###### Abstract

We propose StitchNet, a novel neural network creation paradigm that stitches together fragments (one or more consecutive network layers) from multiple pre-trained neural networks. StitchNet allows the creation of high-performing neural networks without the large compute and data requirements needed under traditional model creation processes via back-propagation training. We leverage Centered Kernel Alignment (CKA) as a compatibility measure to efficiently guide the selection of these fragments in composing a network for a given task tailored to specific accuracy needs and computing resource constraints. We then show that these fragments can be stitched together to create neural networks with comparable accuracy to traditionally trained networks at a fraction of computing resource and data requirements. Finally, we explore a novel on-the-fly personalized model creation and inference application enabled by this new paradigm.

## 1 Introduction

AI models have become increasingly more complex to support additional functionality, multiple modalities, and higher accuracy. While the increased complexity has improved model utility and performance, it has imposed significant model training costs. Therefore, training complex models is often infeasible for resource limited environments such as those at the cloud edge.

In response to these challenges, in this paper we propose a new paradigm for creating neural networks: rather than training networks from scratch or retraining them, we create neural networks through composition by _stitching together_ fragments of existing pre-trained neural networks. A fragment is one or more consecutive layers of a neural network. We call the resulting neural network composed of one or more fragments a "StitchNet" (Figure 1). By significantly reducing the amount of computation and data resources needed for creating neural networks, StitchNets enable an entire new set of applications, such as rapid generation of personalized neural networks at the edge.

StitchNet's model creation is fundamentally different from today's predominant backpropagation-based method for creating neural networks. Given a dataset and a task as input, the traditional training method uses backpropagation with stochastic gradient descent (SGD) or other optimization algorithms to adjust the weights of the neural networks. This training process iterates through the full dataset multiple times, and therefore requires compute resources that scale with the amount of data and the complexity of the network. Training large models this way also requires substantial amounts of data. While successful, this traditional paradigm for model creation is not without its limitations. Creating complex neural networks without access to large amounts of data and compute resources is a growing challenge of increasing significance, especially in resource-constrained edge environments. In the extreme case (e.g., for very large language and computer vision models), only a few companies with access to unrivaled amounts of data and compute resources are able to create such models.

StitchNets solve this problem by creating new neural networks using fragments of already existing neural networks. The new approach takes advantage of the growing amount of neural networks that already exist, having been trained

Figure 1: Overview of the StitchNet approach. Existing networks (left) are cut into fragments (middle), which are composed into StitchNets (right) created for a particular task. No retraining is needed in this process.
previously by many groups and companies. SitticNets enable the efficient reuse of the learned knowledge resident in those pre-trained networks, which has been distilled from large amounts of data, rather than having to relearn it over and over again for new tasks as we do with traditional model creation paradigms. SitticNet's ability to reuse existing pre-trained fragments, rather than recreating from scratch or re-training for every task will help accelerate the growth and application of neural networks for solving more and more complex tasks.

However, compositing these existing fragments into a coherent and high performing neural network is non-trivial. To reuse the knowledge of pre-trained neural network fragments, we need a way to 1) measure the compatibility between any two fragments, and 2) compose compatible fragments together. In the past, Centered Kernel Alignment (CKA) [1, 10, 11] has been used to measure similarity between neural network representations. We leverage CKA to assess the compatibility of any two fragments from any neural networks and compose new neural networks from fragments of existing pre-trained neural networks to create high performing networks customized for specific tasks without the costs of traditional model creation methods. The CKA score is used to reduce the search space for identifying compatible fragments and guide the fragment selection process.

We present empirical validations on benchmark datasets, comparing the performance of SitticNets to that of the original pre-trained neural networks. We demonstrate that SitticNets achieve comparable or higher accuracy on personalized tasks compared with off-the-shelf networks, and have significantly lower computational and data requirements than creating networks from scratch or through retraining.

Our contributions are:

* The SitticNet paradigm: a novel neural network creation method that enables a new set of applications.
* A novel use of Centered Kernel Alignment (CKA) in assessing the compatibility of any two fragments for their composition.
* A technique to compose compatible fragments together for both linear and convolutional layers.
* A feasibility demonstration of SitticNets for efficient on-the-fly personalized neural network creation and inference.

## 2 Composing Fragments

The core mechanism to create SitticNets is to identify reusable fragments from a pool of existing networks and compose them into a coherent neural network model capable of performing a given task. To this end, we need a way to determine how compatible any two candidate fragments are with each other. In previous work, [10] present centered kernel alignment (CKA) [11, 12] as a way to measure similarity between neural network representations. Rather than looking at the neural network as a whole, we adopt and use CKA to as a measure of compatibility between any two _fragments_ of any neural networks.

In this section, we first define CKA as a way to measure how compatible any two fragments are with one another and therefore their ability to be composed. Using CKA, we then present a technique to stitch different fragments together. Finally, we describe the algorithm to generate SitticNets.

### Centered Kernel Alignment (CKA)

Given \(\mathbf{X}\in\mathbb{R}^{p\times n}\) as outputs of a fragment \(F_{A}\) of model \(A\) and \(\mathbf{Y}\in\mathbb{R}^{q\times n}\) as inputs of a fragment \(F_{B}\) of model \(B\) of the same dataset \(\mathbf{D}\), where \(n\) is the number of samples in the dataset, \(p\) is the output dimension of \(F_{A}\) and \(q\) is the input dimension of \(F_{B}\). Let \(\mathbf{K}_{ij}=k(\mathbf{x}_{i},\mathbf{x}_{j})\) and \(\mathbf{M}_{ij}=m(\mathbf{y}_{i},\mathbf{y}_{j})\), where \(k\) and \(m\) are any two kernels. We define the compatibility score \(\text{CKA}(\mathbf{X},\mathbf{Y})\) of fragment \(F_{A}\) and fragment \(F_{B}\) as

\[\text{CKA}(\mathbf{X},\mathbf{Y})=\frac{\text{HSIC}(\mathbf{K},\mathbf{M})}{ \sqrt{\text{HSIC}(\mathbf{K},\mathbf{K})\,\text{HSIC}(\mathbf{M},\mathbf{M})}},\]

where \(\text{HSIC}\) is the Hilbert-Schmidt Independence Criterion [1] defined as

\[\text{HSIC}(\mathbf{K},\mathbf{M})=\frac{1}{(n-1)^{2}}\text{tr}(\mathbf{K}\, \mathbf{H}\,\mathbf{M}\,\mathbf{H}),\]

where \(\mathbf{H}\) is the centering matrix \(\mathbf{H}_{n}=\mathbf{I}_{n}-\frac{1}{n}\mathbf{1}\mathbf{1}^{T}\) and tr is the trace. For linear kernels, \(k(\mathbf{x},\mathbf{y})=m(\mathbf{x},\mathbf{y})=\mathbf{x}^{T}\,\mathbf{y}\), \(\text{HSIC}\) becomes \(\text{HSIC}(\mathbf{X},\mathbf{Y})=\|\text{cov}(\mathbf{X}^{T}\,\mathbf{X}, \mathbf{Y}^{T}\,\mathbf{Y})|_{F}^{2}\), where cov is the covariance function, and \(\text{CKA}(\mathbf{X},\mathbf{Y})\) becomes

\[\frac{\|\text{cov}(\mathbf{X}^{T}\,\mathbf{X},\mathbf{Y}^{T}\,\mathbf{Y})\|_{F }^{2}}{\sqrt{\|\text{cov}(\mathbf{X}^{T}\,\mathbf{X},\mathbf{X}^{T}\,\mathbf{X}) \|_{F}^{2}\|\text{cov}(\mathbf{Y}^{T}\,\mathbf{Y},\mathbf{Y}^{T}\,\mathbf{Y}) \|_{F}^{2}}}. \tag{1}\]

We use this function (Eq. 1) as a measurement of how compatible any two fragments are, given a target dataset. To reduce memory usage for a large target dataset, CKA can be approximated by averaging over minibatches as presented in [13].

### Stitching Fragments

Once we have determined compatible fragments, the next step in creating a SitticNet is to _stitch_ the two fragments together. To do so, we find a projection tensor \(\mathbf{A}\) that projects the output space of one fragment to the input space of the other fragment we are composing. We now describe this.

Without loss of generality, we assume the output and input tensors are 2D tensors, where the first dimension is the sample dimension. If the tensors are not 2D tensors, we first flatten all other dimensions with the exception of the sample dimension.

We use Einstein summation notation, where \(i\) represents the sample dimension, \(j\) the output dimension of the incoming fragment, and \(k\) the input dimension of the outgoing fragment. Given an output tensor \(\mathbf{X}_{ij}\) of the incoming fragment and an input tensor \(\mathbf{Y}_{ik}\) of the outgoing fragment, we seek
to find \(\mathbf{A}\) such that \(\mathbf{Y}_{ik}=\mathbf{A}_{kj}\,\mathbf{X}_{ij}\). We can then solve for \(\mathbf{A}\) using the Moore-Penrose pseudoinverse:

\[\mathbf{A}_{kj}=\mathbf{Y}_{ik}\,\mathbf{X}_{ij}^{T}(\mathbf{X}_{ij}\,\mathbf{X }_{ij}^{T}).^{-1} \tag{2}\]

Once \(\mathbf{A}\) is found, we fuse \(\mathbf{A}\) with the weight of the first layer of the outgoing fragment. For linear layers, we simply do the following:

\[\mathbf{W}_{lk}^{\prime}=\mathbf{W}_{lj}\,\mathbf{A}_{kj}, \tag{3}\]

where \(l\) is the dimension of the output feature of the outgoing layer.

For convolutional layers, we first upsample or downsample the spatial dimension to match each other, and then adjust the weight along the input channel dimension as follows.

\[\mathbf{W}_{okmn}^{\prime}=\mathbf{W}_{ijmn}\mathbf{A}_{kj}, \tag{4}\]

where \(o\) is the output channel dimension, \(j\) is the original input channel dimension, \(k\) is the new input channel dimension, and \(m\) and \(n\) are the spatial dimensions.

For stitching a convolutional layer with an output tensor \(\mathbf{X}\) and a linear layer with an input tensor \(\mathbf{Y}\), we first apply adaptive average pooling so that the spatial dimension is 1x1 and flatten \(\mathbf{X}\) into a 2D tensor. Then, we follow Eq. 2 and Eq. 3 to find \(\mathbf{A}\) and fuse it with the \(\mathbf{W}\) of the linear layer.

### StitchNet Generation

```
Input: fragment pool \(\mathbf{P}=\{F_{ij}\}\), network \(i\) in \(\mathbf{P}\) up to layer \(j\)\(N_{ij}\), fragment ending in layer \(j\) of network \(i\)\(F_{ij}\), target dataset \(\mathbf{D}\) with \(M\) samples, span \(K\), threshold \(T\), maximum number of fragments \(L\), result list of StitchNets and their associated scores \(\mathbf{R}\), current StitchNet \(Q\), current score \(s\) Output: resulting list of StitchNets and their associated scores \(\mathbf{R}\) if\(Q\) is empty then \(\{F_{ij}\}\) = select starting fragments in \(\mathbf{P}\) for\(F_{ij}\) in \(\{F_{ij}\}\)do  StitchNet(\(\mathbf{P}\), \(\mathbf{D}\), \(K\), \(T\), \(L\), \(\mathbf{R}\), \(F_{ij}\), 1) if the number of fragments in \(Q\geq L\)then return\(\mathbf{R}\) \(\{F_{ij}\}\) = select \(K\) middle or terminating fragments in \(\mathbf{P}\) for\(F_{ij}\) in \(\{F_{ij}\}\)do X = \(Q\)(\(\mathbf{D}\)); Y = \(N_{ij}\)(\(\mathbf{D}\)) \(s_{n}\) = \(s\times\) CKA(\(\mathbf{X},\mathbf{Y}\)) (see section 2.1) if\(s_{n}\) = \(T\)then \(Q\) = Stitch(\(Q\), \(F_{ij}\), \(\mathbf{X}\), \(\mathbf{Y}\)) (see section 2.2) if\(F_{ij}\) is a terminating fragment then R.append(\(\{Q,s_{n}\}\)) else  StitchNet(\(\mathbf{P}\), \(\mathbf{D}\), \(K\), \(T\), \(L\), \(\mathbf{R}\), \(Q\), \(s_{n}\)) return\(\mathbf{R}\)
```

**Algorithm 1**StitchNet(\(\mathbf{P}\), \(\mathbf{D}\), \(K\), \(T\), \(L\), \(\mathbf{R}\), \(Q\), \(s\))

We now describe the main algorithm for creating StitchNet networks ("StitchNets" for short), shown in Algorithm 1. A StitchNet network is created by joining a set of pre-trained network fragments drawn from a pool \(\mathbf{P}=\{F_{ij}\}\). We use the notation \(F_{ij}\) to denote a fragment of a neural network \(i\) up to its \(j\) layer, and the notation \(N_{ij}\) to denote the computation performed by the portion of the neural network from which the fragment was taken. Other than the fragment pool \(\mathbf{P}\) and creation process hyperparameters (\(K,T,L\)), the only other input to the StitchNet creation process is a dataset \(\mathbf{D}\) for which the StitchNet will be optimized.

We now describe the creation of the pool of network fragments \(\mathbf{P}\) derived from a set of pre-trained off-the-shelf networks. These pre-trained networks are divided into one of three types of fragments: _starting_ fragments for which the input is the original network input, _terminating_ fragments for which the output is the original network output, and _middle_ fragments that are neither starting nor terminating fragments.

The first step in the StitchNet creation process is to choose the set of starting fragments. This could include all starting fragments in \(\mathbf{P}\), or a subset based on certain criteria, e.g., the smallest, biggest or closest starting fragment.

Once a set of starting fragments are selected, a StitchNet is built on top of each starting fragment having a current starting score of 1. First, a set of \(K\) candidate fragments are selected from \(\mathbf{P}\). These fragments can be selected based on CKA scores (i.e., \(K\) fragments with highest CKA scores), the number of parameters of the fragments (i.e., \(K\) fragments with the least amount of number of parameters in \(\mathbf{P}\)), the closest fragments (i.e., \(K\) fragments with the least latency in \(\mathbf{P}\) in a distributed fragments setting), or other selection methods.

For each of the candidate fragments, we then compute two intermediate neural network computations. First, we pass the dataset \(\mathbf{D}\) through the candidate StitchNet in its current form, resulting in value \(\mathbf{X}\). Second, we pass the same dataset \(\mathbf{D}\) through the neural network from which the candidate fragment \(F_{ij}\) was selected, resulting in value \(\mathbf{Y}=N_{ij}(D)\).

After running these computations, we produce CKA(\(\mathbf{X},\mathbf{Y}\)) as in Section 2.1. We then multiply the CKA with the current score \(s\) to obtain the new current score \(s_{n}\). If \(s_{n}\) is still greater than a set threshold \(T\), the candidate fragment is selected and the process continues recursively. Otherwise, the candidate fragment is rejected. The threshold can be set to balance the amount of exploration allowed per available compute resources.

This process continues until a terminating fragment is selected, the maximum number of fragments \(L\) is reached or all recursive paths are exhausted. At this point, the completed StitchNets and their associated scores \(\mathbf{R}\) are returned for user selection.

## 3 Results

We now demonstrate that StitchNets can perform comparably with traditionally trained networks but with significantly reduced computational and data requirements at both inference and creation time. Through these characteristics, StitchNets enable the immediate on-the-fly creation of neural networks for personalized tasks without traditional training.
### Fragment pool

To form the fragment pool \(\mathbf{P}\), we take five off-the-shelf networks pre-trained on the ImageNet-1K dataset [4] from Torchvision [10]: _alexnet_, _densenet121_, _mobilenet_v3_small_, _resnet50_ and _vgg16_ with IMAGENET1K_V1 weights.

These pre-trained networks are cut into fragments at each convolution and linear layer that has a single input. As shown in Figure 2, there are 8 fragments for _alexnet_, 5 fragments for _densenet121_, 13 fragments for _mobilenet_v3_small_, 6 fragments for _resnet50_ and 16 fragments for _vgg16_. This results in the creation of a fragment pool \(\mathbf{P}\) of 48 fragments consisting of 5 starting fragments, 38 middle fragments, and 5 terminating fragments. We use this fragment pool in all experiments in this paper.

### Dataset

The dataset used to evaluate StitchNets in this paper is the "Dogs vs. Cats" dataset [11]. This dataset includes 25,000 training images of dogs and cats and we use an 80:20 train:test split. We map ImageNet-1K class labels into cat and dog labels (class IDs 281-285 and 151-250, respectively). To form the target dataset \(\mathbf{D}\) for use in the stitching process of Algorithm 1, we randomly select \(M\) samples from the training set as the target dataset \(\mathbf{D}\). We choose this task because it is characteristic of the type of task for which StitchNets would be used: a user needs a custom classifier for a particular task and desired set of classes.

### StitchNet Generation

We generate StitchNets with Algorithm 1 using the fragment pool and the dataset described in Section 3.1 and 3.2. We set \(K=2\), \(T=0.5\) and \(L=16\). The number of samples \(M\) in \(\mathbf{D}\) used for the stitching process is 32.

Given these hyperparameters, a total of 89 StitchNets are generated. We evaluate them on the test set of completely unseen test samples. Summary statistics for the generated StitchNets are shown in Figure 3, including accuracy (3a), number of fragments per StitchNet (3b), CKA score (3c), and number of parameters per StitchNet (3d).

### Reduction in Inference Computation

We now demonstrate how StitchNets significantly reduce inference-time computational requirements over traditional neural network training paradigms by studying StitchNet accuracy as a function of parameters.

Figure 4 shows the resulting accuracy of the generated StitchNets as a function overall CKA score for each StitchNet and number of parameters (porportional to marker size) as a proxy for inference-time computation cost. We find a number of StitchNets outperform the pre-trained network while realizing significant computational savings. For example, StitchNet27 (denoted by a green star) achieves an accuracy of 0.86 with 3.59M parameters compared with the 0.70 accuracy of the pre-trained alexnet with 61.10M parameters. Therefore, StitchNet achieves a 22.8% increase in accuracy with a 94.1% reduction in number of parameters for the given task when compared with those of the pre-trained alexnet.

These crystallizes one of the core benefits of StitchNets: without any training, the method can discover networks that are personalized for the task, outperform the original pre-trained networks, and do so while significantly reducing inference-time compute requirements. This is due to the fact that these pre-trained networks are not trained to focus on these two specific classes, while our StitchNets are stitched together specifically for the task. In the next section, we will

Figure 2: Five pre-trained networks are fragmented into a fragment pool \(\mathbf{P}\). These fragments will be stitched together to form StitchNets.
see that very little data is required for the stitching process.

Additionally, we compare the StitchNets with the various off-the-shelf models, denoted by triangles. We find that the StitchNet generation process creates many different StitchNets that outperform the off-the-shelf models, many of which do so at reduced computational cost. Figure 5 shows the composition of some of these high-performing StitchNets, demonstrating the diversity in fragment use, ordering, and architectures.

We also validate the effectiveness of using CKA to guide the stitching procedure. We find that StitchNets with a high CKA score also have high accuracy, especially those above 0.9. This shows that CKA can be used as a proxy to measure good compatibility between connecting fragments.1

Footnote 1: Note that there exist high accuracy StitchNets with low overall CKA score. This is because neural networks are robust and highly redundant, able to tolerate a certain amount of errors while still giving quality predictions (see Section 4.1).

### Reduction in Network Creation Computation

We now demonstrate that StitchNets can be created without significant data and computation requirements. Specifically, we compare StitchNet21 (generated in Figure 5 on the target dataset of \(M=32\) samples) with fine-tuning the same five off-the-shelf networks (retraining them using the training portion of dataset of Section 3.2). For fine-tuning, we replace and train only the last layer of the pre-trained network using Stochastic Gradient Descent (SGD) with batch size 32, learning rate \(0.001\) and momentum \(0.9\). The results shown are averaged over 10 runs. For ease of comparison, we normalize the computation cost in terms of the number of samples processed through a neural network. In practice, fine-tuning requires backpropagation, which incurs additional computation per sample than StitchNet generation.

Figure 6 compares the accuracy of StitchNet21 (denoted by the red star) with the traditionally fine-tuned networks as a function of the number of training samples processed. For a given accuracy target, StitchNets process a substantially smaller number of data samples than traditionally fine-tuned networks. Specifically, to reach an accuracy of 0.95, fine-tuning of _alexnet_, _densenet121_, and _mobilenet_x3_small_ require to process more than 320 samples while StitchNet requires only 32 samples used to stitch the fragments together (realizing over a 90% reduction).

Therefore, only a small amount of training samples and computation are required for StitchNet to achieve comparable accuracy. This demonstrates that StitchNets effectively reuse the information already captured in the fragments to bootstrap network creation. This allows for personalization of tasks and on-the-fly training without substantial data requirements.

### Ensembles

We now discuss the ability to ensemble generated StitchNets to improve performance. StitchNet and ensembling methods are complimentary. The StitchNet generation algorithm produces a set of candidate models. While a user can select a single StitchNet to use at inference time, because the StitchNet generation procedure finds such efficient models, we can also take advantage of the pool of StitchNets and ensemble some while still realize substantial computational savings.

We pick 10 random models from the generated StitchNets in Section 3.3 with overall CKA \(>0.8\). We sort these models based on their overall CKA scores from high to low, and then ensemble them by averaging their predicted probabilities. The results are shown in Figure 7. The ensemble often results in higher accuracy than the individual model. As a result, this ensembling method can reduce variance in performance when on-the-fly network creation and inference (as discussed in Section 4.3) is used and there is not time for full selection of a final single StitchNet. Instead, the user can select a reasonably small subset of high performing StitchNets, which even in aggregate can be significantly smaller than a single traditionally trained network.

## 4 Discussion

We now discuss the intuition behind StitchNets, examine their complexity and relation to related methods, introduce new applications they enable, and discuss their limitations.

### Why do StitchNets work?

We first discuss why we are able to reuse existing fragments of networks to create new neural networks without retraining. One core reason for this is that neural networks tend to learn fundamental and universal features. Studies (Li et al., 2015; Lu et al., 2018; Morcos, Raghu, and Bengio, 2018; Wang et al., 2018; Lenc and Vedaldi, 2015; Kornblith et al., 2019; Tang et al., 2020) have shown that neural networks learn fundamental features such as edges for different tasks. Since these learned features are fundamental, they should

Figure 3: Histogram of (a) accuracy, (b) # fragments, (c) CKA score, (d) # parameters in the generated batch of StitchNets.
be reusable rather relearned. The challenge, however, is that although these features may be universal, they may not be compatible with one another "out of the box." Therefore, we require the stitching process introduced in Section 2.2 to project the fragments into a compatible space.

Beyond this reuse of universal features and compatibility transformations, StitchNets are also enabled by the fact that neural networks are fundamentally robust. Due to the non-linear activation and built-in redundancies, neural networks tolerate certain amounts of error. As such, the fragments need not be perfectly compatible individually to produce a network that in aggregate operates at a high level of performance.

### Complexity Comparison

We now compare the complexity of the traditional training process using backpropagation with the StitchNet gen

Figure 4: Accuracy vs the overall CKA score on “Cat vs. Dogs.” \(cka\) is the overall CKA score, \(acc\) is the accuracy. The best StitchNet (\(acc\)=0.95) performs 12% better than the best pre-trained model(s) (densenet121 and resnet50 with \(acc\)=0.85).

Figure 5: Examples of generated StitchNets.

Figure 6: Accuracy vs the number of training samples processed (i.e., data and computation required). StitchNets require only a fraction of the computation of traditional training methods to achieve comparable performance.

Figure 7: Accuracy of the ensemble models. Ensembling groups of StitchNets can reduce individual model variance.
eration process. Traditional training complexity is \(O(ndp)\), where \(n\) is the number of parameters in the network, \(p\) is the number of epochs used to train, and \(d\) is the size of the dataset. StitchNet generation complexity is \(O(nqm)+O(K^{L})\). The first term \(nqm\) is the evaluation cost of the target dataset of size \(q\) on \(m\) networks in the pool, where \(q\ll d\) and \(n\) is the number of parameters in the network (assuming networks have the same # of parameters). The second term \(K^{L}\) is the search cost, where \(K\) is the span value we search at each level and \(L\) is the max depth to search. Using a high threshold cutoff \(T\) on the overall CKA score keeps search cost \(K^{L}\) small. Therefore, for a reasonable setting of hyperparameters (\(K,T,L\)) in Algorithm 1, StitchNets realize substantial computation gains over traditional training methods since \(q\ll d\) and \(m\ll p\).

### On-the-fly network creation and inference

We now discuss a new family of applications and use cases that are enabled by StitchNets: on-the-fly neural network creation and inference. In this application, we use a batch of images on which we want to perform a task (e.g., classification or detection) as our target dataset in the StitchNet generation process. With only a minor modification to the StitchNet algorithm to additionally return task results, the StitchNet generation process can return the inference outputs along with the generated StitchNets.

We now describe how this can be used in practice. Imagine a world where fragments of pre-trained neural networks for different tasks are indexed and distributed on the Internet. Any compatible fragment can be found and composed quickly to form a new neural network for a certain task. Now, imagine we want to create a neural network for classifying local cats and dogs with only a few hundred of these unlabeled images.

Without StitchNets, we either need to train a network from scratch (which may fail due to our limited amount of training data), or find an existing pre-trained neural network, label the dataset, and finetune the network. If the existing pre-trained network is too big or too slow for our use, we will then have to train a new one from scratch. But, with limited amount of unlabeled data, this task seems impossible.

With StitchNet, we can instead generate a set of candidate StitchNets with the small target dataset of unlabeled local cats and dogs images. These StitchNets are created from the pool of existing neural network fragments that have been indexed and distributed on the Internet. The proper fragments can be identified with a search criteria (e.g., the terminating fragment should contain cat and dog classes, the depth of the network should be less than 5 for computational efficiency reasons, etc.). With little computation, we will generate StitchNets capable of detecting and classifying local cats and dogs.

### Limitations

One limitation is that the target task needs to be a subset (or a composition) of the terminating fragment tasks in the fragment pool. Additionally, while a large pool of networks and fragments can lead to higher applicability and quality of StitchNets, it can also lead to high search costs. Indexing large quantities of neural networks to form the fragment pool will require novel search methods. We see this as analogous to indexing web pages on the World Wide Web, suggesting a "Google for Fragments." Much like web search needed to index written content, large amounts of neural network "content" need to be indexed in order for their value to be unlocked. Early indexing efforts can tag fragments based on dataset characteristics, computational characteristics, etc. More advanced efforts can look at inward and outward connections of each fragment to determine its rank in results. Once a narrowed set of fragments are coarsely identified, the efficient procedure introduced in this paper can generate the StitchNets. Future work will address these types of complementary methods (indexing and distribution) that will enable StitchNets to operate at scale.

## 5 Related Work

Transfer learning (or fine-tuning) Pan and Yang (2009); Weiss et al. (2016) is the current pre-dominant way to adapt existing neural networks to target tasks. Unsupervised domain adaptation is related, where the existing network is adapted using an unlabeled target dataset. StitchNets work similarly by stitching fragments using an unlabeled target dataset to create a neural network for the target task. Most work Wang and Deng (2018); Zhang et al. (2018); Tzeng et al. (2014); Kumar et al. (2018); Shu et al. (2018); Ben-David et al. (2010); Saito et al. (2017) focuses on retraining the network, while StitchNet does not require any training.

StitchNets take advantage of the assumption that the fragments have shareable representations. This assumption helps explain why fragments can be stitched together into a coherent high-performing network: dissimilar yet complimentary fragments once projected into a similar space are compatible with one another. Several existing works including Li et al. (2015); Mehrer et al. (2018); Lu et al. (2018); Morcos et al. (2018); Renblith et al. (2019); Tang et al. (2020) have studied this shareable representation assumption.

Gygli et al. (2021) reuse network components by training networks to produce compatible features by adding regularization at training time to make the networks directly compatible. StitchNet, however, focuses on creating neural networks without training. It is therefore more generally applicable. Lenc and Vedaldi (2015) combine network components by adding a stitching layer and training the recombined network with a supervised loss for several epochs. StitchNet adds a parameter-less stitching mechanism and therefore does not require any retraining. Instead, weights are adapted to be compatible with the method introduced in 2.2.

## 6 Conclusion

StitchNet is a new paradigm that can leverage a growing global library of neural networks to fundamentally change the way networks are created. By reusing fragments of these networks to efficiently compose new networks for a given
task, SitticNet addresses two of the most fundamental issues limiting neural network creation and use: large data and computation requirements.

StitchNet does this by leveraging Centered Kernel Alignment (CKA) as a compatibility measure that guides the selection of neural network fragments, tailored to specific accuracy needs and computing resource constraints. Our work has shown that neural networks can be efficiently created from compatible neural network fragments of different models at a fraction of computing resources and data requirements with a comparable accuracy. We also introduce on-the-fly efficient neural network creation and inference application that is unlocked by this method.
