The Tunnel Effect: Building Data Representations in Deep Neural Networks
Deep neural networks are widely known for their remarkable effectiveness
across various tasks, with the consensus that deeper networks implicitly learn
more complex data representations. This paper shows that sufficiently deep
networks trained for supervised image classification split into two distinct
parts that contribute to the resulting data representations differently. The
initial layers create linearly-separable representations, while the subsequent
layers, which we refer to as \textit{the tunnel}, compress these
representations and have a minimal impact on the overall performance. We
explore the tunnel's behavior through comprehensive empirical studies,
highlighting that it emerges early in the training process. Its depth depends
on the relation between the network's capacity and task complexity.
Furthermore, we show that the tunnel degrades out-of-distribution
generalization and discuss its implications for continual learning.
Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Miłoś, Tomasz Trzciński
# The Tunnel Effect: Building Data Representations

###### Abstract

Deep neural networks are widely known for their remarkable effectiveness across various tasks, with the consensus that deeper networks implicitly learn more complex data representations. This paper shows that sufficiently deep networks trained for supervised image classification split into two distinct parts that contribute to the resulting data representations differently. The initial layers create linearly-separable representations, while the subsequent layers, which we refer to as _the tunnel_, compress these representations and have a minimal impact on the overall performance. We explore the tunnel's behavior through comprehensive empirical studies, highlighting that it emerges early in the training process. Its depth depends on the relation between the network's capacity and task complexity. Furthermore, we show that the tunnel degrades out-of-distribution generalization and discuss its implications for continual learning.

## 1 Introduction

Neural networks have been the powerhouse of machine learning in the last decade. A significant effort has been put into understanding the mechanisms underlying their effectiveness. One example is the analysis of building representations in neural networks applied to image processing [19]. The consensus is that networks learn to use layers in the hierarchy by extracting more complex features than the layers before [22; 41], meaning that each layer contributes to the final network performance.

Extensive research has shown that increasing network depth exponentially enhances capacity, measured as the number of linear regions [35; 46; 50]. However, practical scenarios reveal that deep and overparameterized neural networks tend to simplify representations with increasing

Figure 1: **The tunnel effect** for VGG19 trained on the CIFAR-10. In the tunnel ( shaded area), the performance of linear probes attached to each layer saturates (blue line), and the representations rank is steeply reduced (red dashed line).
depth [13; 53]. This paradox arises because, despite their large capacity, these networks strive to reduce dimensionality and focus on discriminative patterns during supervised training [13; 15; 42; 53]. Motivated by these contradictory findings, we aim to investigate this phenomenon further and formulate the following research questions:

_How do representations depend on the depth of a layer?_

Our investigation focuses on severely overparameterized neural networks through the prism of their representations as the core components for studying neural network behavior [20; 38].

We challenge the commonly held intuition that deeper layers are responsible for capturing more complex and task-specific features [41; 57].

Specifically, we demonstrate that deep neural networks split into two parts exhibiting distinct behavior. The first part, which we call the extractor, builds representations, while the other, dubbed _the tunnel_, propagates the representations further to the model's output, compressing them significantly. To investigate the tunnel effect, we conduct multiple experiments that support our findings and shed some light on the potential source of this behavior. Our findings can be summarized as follows:

* We discover and extensively examine the tunnel effect, namely, deep networks naturally split into _the extractor_ responsible for building representations and _the compressing tunnel_, which minimally contributes to the final performance. The extractor-tunnel split emerges early in training and persists later on.
* We show that the tunnel deteriorates the generalization ability on out-of-distribution data.
* We show that the tunnel exhibits task-agnostic behavior in a continual learning scenario. Simultaneously it leads to higher catastrophic forgetting of the model.

## 2 The tunnel effect

The paper introduces and studies the dynamics of representation building in overparameterized deep neural networks called _the tunnel effect_. The following section validates the tunnel effect hypothesis in a number of settings. Through an in-depth examination in Section 3.1, we reveal that the tunnel effect is present from the initial stages and persists throughout the training process. Section 3.2 focuses on the out-of-distribution generalization and representations compression. Section 3.3 hints at important factors that impact the depth of the tunnel. Finally, in Section 4, we confront an auxiliary question: How does the tunnel's existence impact a model's adaptability to changing tasks and its vulnerability to catastrophic forgetting? To answer these questions we formulate our main claim as:

_The tunnel effect hypothesis: Sufficiently large * neural networks develop a configuration in which network layers split into two distinct groups. The first one which we call the extractor, builds linearly-separable representations. The second one, the tunnel, compresses these representations, hindering the model's out-of-distribution generalization._

Footnote *: We note that ‘sufficiently large’ covers most modern neural architectures, which tend to be heavily overparameterized.

### Experimental setup

To examine the phenomenon, we designed the setup to include the most common architectures and datasets, and use several different metrics to validate the observations.

**Architectures** We use three different families of architectures: MLP, VGGs, and ResNets. We vary the number of layers and width of networks to test the generalizability of results. See details in Appendix A.1.

**Tasks** We use three image classification tasks to study the tunnel effect: CIFAR-10, CIFAR-100, and CINIC-10. The datasets vary in the number of classes: \(10\) for CIFAR-10 and CINIC-10 and \(100\) for CIFAR-100, and the number of samples: \(50000\) for CIFAR-10 and CIFAR-100 and \(250000\) for CINIC-10). See details in Appendix A.2.

We probe the effects using: _the average accuracy of linear probing, spectral analysis of representations, and the CKA similarity between representations_. Unless stated otherwise, we report the average of \(3\) runs.
**Accuracy of linear probing:** a linear classification layer is attached to a given layer \(\ell\) of the neural network. We train this layer on the classification task and report the average accuracy. This metric measures to what extent \(\ell\)'s representations are linearly separable.

**Numerical rank of representations:** we compute singular values of the sample covariance matrix for a given layer \(\ell\) of the neural network. Using the spectrum, we estimate the numerical rank of the given representations matrix as the number of singular values above a certain threshold \(\sigma\). The numerical rank of the representations matrix can be interpreted as the measure of the degeneracy of the matrix.

**CKA similarity:** is a metric computing similarity between two representations matrices. Using this normalized index, we can identify the blocks of similar representations within the network. The definition and more details can be found in Appendix E.

**Inter and Intra class variance:** inter-class variance refers to the measure of dispersion or dissimilarity between different classes or groups in a dataset, indicating how distinct they are from each other. Intra-class variance, on the other hand, measures the variability within a single class or group, reflecting the homogeneity or similarity of data points within that class. The exact formula for computing these values can be found in Appendix F

### The main result

Table 1 presents our main result. Namely, we report the network layer at which the tunnel begins which we define as the point at which the network reaches \(95\%\) (or \(98\%\)) of its final accuracy. We found that all tested architectures exhibit the extractor-tunnel structure across all datasets used in the evaluation, but the relative length of the tunnel varies between architectures.

\begin{table}
\begin{tabular}{c c c|c c} \hline \hline Architecture & \# layers & Dataset & \(>0.95\) & \(>0.98\) \\ \hline \hline MLP & 13 & CIFAR-10 & 4 (31\%) & 5 (38\%) \\ \hline \multirow{3}{*}{VGG} & \multirow{3}{*}{19} & CIFAR-10 & 7 (36\%) & 7 (36\%) \\  & & CIFAR-100 & 8 (42\%) & 8 (42\%) \\ \cline{1-1}  & & CINIC-10 & 7 (36\%) & 7 (36\%) \\ \hline \multirow{2}{*}{ResNet} & \multirow{2}{*}{34} & CIFAR-10 & 20 (58\%) & 29 (85\%) \\ \cline{1-1}  & & CIFAR-100 & 29 (85\%) & 30 (88\%) \\ \end{tabular}
\end{table}
Table 1: The tunnel of various lengths is present in all tested configurations. For each architecture and dataset, we report the layer for which the _average linear probing accuracy is above \(0.95\) and \(0.98\) of the final performance_. The values in the brackets describe the part of the network utilized for building representations with the extractor.

Figure 2: The tunnel effect for networks trained on CIFAR-10. The blue line depicts the linear probing accuracy, and the shaded area depicts the tunnel. The red dashed line is the numerical rank of representations. The spike in the ResNet-34 representations rank coincides with the end of the penultimate residual stage.
We now discuss the tunnel effect using MLP-12, VGG-19, and ResNet-34 on CIFAR-10 as an example. The remaining experiments (for other architectures, datasets combinations) are available in Appendix B. As shown in Figure 1 and Figure 2, the early layers of the networks, around five for MLP and eight for VGG, are responsible for building linearly-separable representations. Linear probes attached to these layers achieve most of the network's final performance. These layers mark the transition between the extractor and the tunnel part (shaded area). In the case of ResNets, the transition takes place in deeper stages of the network at the \(19^{th}\) layer.

While the linear probe performance nearly saturates in the tunnel part, the representations are further refined. Figure 2 shows that the numerical rank of the representations (red dashed line) is reduced to approximately the number of CIFAR-10 classes, which is similar to the neural collapse phenomenon observed in [42]. For ResNets, the numerical rank is more dynamic, exhibiting a spike at \(29^{th}\) layer, which coincides with the end of the penultimate residual block. Additionally, the rank is higher than in the case of MLPs and VGGs.

Figure 3 reveals that for VGG-19 the inter-class representations variation decreases throughout the tunnel, meaning that representations clusters contract towards their centers. At the same time, the average distance between the centers of the clusters grows (inter-class variance). This view aligns with the observation from Figure 2, where the rank of the representations drops to values close to the number of classes. Figure 3 (right) presents an intuitive explanation of the behavior with UMAP [31] plots of the representations before and after the tunnel.

To complement this analysis, we studied the similarity of MLPs representations using the CKA index and the L1 norm of representations differences between the layers. Figure 4 shows that the representations change significantly in early layers and remain similar in the tunnel part when measured with the CKA index (left). The L1 norm of representations differences between the layers is computed on the right side of Figure 4.
Tunnel effect analysis

This section provides empirical evidence contributing to our understanding of the tunnel effect. We hope that these observations will eventually lead to explanations of this phenomenon. In particular, we show that a) the tunnel develops early during training time, b) it compresses the representations and hinders OOD generalization, and c) its size is correlated with network capacity and dataset complexity.

### Tunnel development

**Motivation** In this section, we investigate tunnel development during training. Specifically, we try to understand whether the tunnel is a phenomenon exclusively related to the representations and which part of the training is crucial for tunnel **Experiments** We train a VGG-19 on CIFAR-10 and save intermediate checkpoints every \(10\) epochs of training. We use these checkpoints to compute the layer-wise weight change during training ( Figure 5) and the evolution of numerical rank throughout the training (Figure 6).

**Results** Figure 5 shows that the split between the extractor and the tunnel is also visible in the parameters space. It could be perceived already at the early stages, and after that, its length stays roughly constant. Tunnel layers change significantly less than layers from the extractor. This result raises the question of whether the weight change affects the network's final output. Inspired by [59], we reset the weights of these layers to the state before optimization. However, the performance of the model deteriorated significantly. This suggests that although the change within the tunnel's parameters is relatively small, it plays an important role in the model's performance. Figure 6 shows that this apparent paradox can be better understood by looking at the evolution of representations' numerical rank during the very first gradient updates of the model. Throughout these steps, the rank collapses to values near-the-number of classes. It stays in this regime until the end of the training, meaning that the representations of the model evolve within a low-dimensional subspace. It remains to be understood if (and why) low-rank representations and changing weights coincide with forming linearly-separable representations.

**Takeaway** Tunnel formation is observable in the representation and parameter space. It emerges early in training and persists throughout the whole optimization. The collapse in the numerical rank of deeper layers suggest that they preserve only the necessary information required for the task.

Figure 6: The representations rank for deeper layers collapse early in training. The curves present the evolution of representations’ numerical rank over the first \(75\) training steps for all layers of the VGG-19 trained on CIFAR-10. We present a more detailed tunnel development analysis in Appendix G.
### Compression and out-of-distribution generalization

**Motivation** Practitioners observe intermediate layers to perform better than the penultimate ones for transfer learning [5; 23; 48]. However, the reason behind their effectiveness remains unclear [9]. In this section, we investigate whether the tunnel and, specifically, the collapse of numerical rank within the tunnel impacts the performance on out-of-distribution (OOD) data.

**Experiments** We train neural networks (MLPs, VGG-19, ResNet-34) on a source task (CIFAR-10) and evaluate it with linear probes on the OOD task, in this case, a subset of 10 classes from CIFAR-100. We report the accuracy of linear probing and the numerical rank of the representations.

**Results** Our results presented in Figure 7 reveal that _the tunnel is responsible for the degradation of out-of-distribution performance_. In most of our experiments, the last layer before the tunnel is the optimal choice for training a linear classifier on external data. Interestingly, we find that the OOD performance is tightly coupled with the numerical rank of the representations, which significantly decreases throughout the tunnel.

To assess the generalization of our findings we extend the proposed experimentation setup to additional dataset. To that end, we train a model on different subsets of CIFAR-100 while evaluating it with linear probes on CIFAR-10. The results presented in Figure 8 are consistent with our initial findings. We include detailed analysis with reverse experiment (CIFAR-10 \(\rightarrow\) CIFAR-100), additional architectures and datasets in the Appendix C.

In all tested scenarios, we observe a consistent relationship between the start of the tunnel and the drop in OOD performance. An increasing number of classes in the source task result in a shorter tunnel and a later drop in OOD performance. In the fixed source task experiment (Appendix C), the drop in performance occurs around the \(7^{th}\) layer of the network for all tested target tasks, which matches the start of the tunnel. This observation aligns with our earlier findings suggesting that the tunnel is a prevalent characteristic of the model rather than an artifact of a particular training or dataset setup.

Moreover, we connect the coupling of the numerical rank of the representations with OOD performance, to a potential tension between the objective of supervised learning and the generalization of OOD setup. Analogous tension was observed in [52] where adversarial robustness is at odds with model's accuracy. The results in Figure 7 align with the findings presented in Figure 3, demonstrating how the tunnel compresses clusters of class-wise representations. In work [54], the authors show that reducing the variation within each class leads to lower model transferability. Our experiments support this observation and identify the tunnel as the primary contributor to this effect.

**Takeaway** Compression of representations happening in the tunnel severely degrades the OOD performance of the model which is tightly coupled with the drop of representations rank.

Figure 8: Fewer classes in the source task create a longer tunnel, resulting in worse OOD performance. The network is trained on subsets of CIFAR-100 with different classes, and linear probes are trained on CIFAR-100. Shaded areas depict respective tunnels.

Figure 7: The tunnel degrades the out-of-distribution performance correlated with the representations’ numerical rank. The accuracy of linear probes (blue) was trained on the out-of-distribution data subset of 10 classes from CIFAR-100. The backbone was trained on CIFAR-10. The shaded area depicts the tunnel, and the red dashed line depicts the numerical rank of representations.
### Network capacity and dataset complexity

**Motivation** In this section, we explore what factors contribute to the tunnel's emergence. Based on the results from the previous section we explore the impact of dataset complexity, network's depth, and width on tunnel emergence.

**Experiments** First, we examine the impact of networks' depth and width on the tunnel using MLPs (Figure 9), VGGs, and ResNets (Table 2) trained on CIFAR-10. Next, we train VGG-19 and ResNet34 on CIFAR-{10,100} and CINIC-10 dataset investigating the role of dataset complexity on the tunnel.

**Results** Figure 9 shows that the depth of the MLP network has no impact on the length of the extractor part. Therefore increasing the network's depth contributes only to the tunnel's length. Both extractor section and numerical rank remain relatively consistent regardless of the network's depth, starting the tunnel at the same layer. This finding suggests that overparameterized neural networks allocate a fixed capacity for a given task independent of the overall capacity of the model.

Results in Table 2 indicate that the tunnel length increases as the width of the network grows, implying that representations are formed using fewer layers. However, this trend does not hold for ResNet34, as the longest tunnel is observed with the base width of the network. In the case of VGGs, the number of layers in the network does not affect the number of layers required to form representations. This aligns with the results in Figure 9.

The results presented above were obtained from a dataset with a consistent level of complexity. The data in Table 3 demonstrates that the number of classes in the dataset directly affects the length of the tunnel. Specifically, even though the CINIC-10 training dataset is three times larger than CIFAR-10, the tunnel length remains the same for both datasets. This suggests that the number of samples in the dataset does not impact the length of the tunnel. In contrast, when examining CIFAR-100 subsets, the tunnel length for both VGGs and ResNets increase. This indicates a clear relationship between the dataset's number of classes and the tunnel's length.

**Takeaway** Deeper or wider networks result in longer tunnels. Networks trained on datasets with fewer classes have longer tunnels.

\begin{table}
\begin{tabular}{c|c c c} \hline \hline  & \(1/4\) & I & 2 \\ \hline \hline VGG-16 & 8 (50\%) & 7 (44\%) & 7 (44\%) \\ \hline VGG-19 & 8 (42\%) & 7 (37\%) & 7 (37\%) \\ \hline ResNet18 & 15 (83\%) & 13 (72\%) & 13 (72\%) \\ \hline ResNet34 & 24 (68\%) & 20 (59\%) & 24 (68\%) \\ \hline \end{tabular}
\end{table}
Table 2: Widening networks layers results in a longer tunnel and shorter extractor. Column headings describe the factor in which we scale each model’s base number of channels. The models were trained on the CIFAR-10 to the full convergence. We use the \(95\%\) threshold of probing accuracy to estimate the tunnel beginning.

\begin{table}
\begin{tabular}{c|c|c c c} \hline \hline model & dataset & 30\% & 50\% & 100\% \\ \hline \hline \multirow{3}{*}{VGG-19} & CIFAR-10 & 6 (32\%) & 7 (37\%) & 7 (37\%) \\  & CIFAR-100 & 8 (42\%) & 8 (42\%) & 9 (47\%) \\  & CINIC-10 & 6 (32\%) & 7 (37\%) & 7 (37\%) \\ \hline \multirow{2}{*}{ResNet34} & CIFAR-10 & 19 (56\%) & 19 (56\%) & 21 (61\%) \\  & CIFAR-100 & 30 (88\%) & 30 (88\%) & 31 (91\%) \\ \hline \end{tabular}
\end{table}
Table 3: Networks trained on tasks with fewer classes utilize fewer resources for building representations and exhibit longer tunnels. Column headings describe the size of the class subset used in training. Within the (architecture, dataset) pair, the number of gradient steps during training in all cases was the same. We use the \(95\%\) threshold of probing accuracy to estimate the tunnel beginning.

Figure 9: Networks allocate a fixed capacity for the task, leading to longer tunnels in deeper networks. The extractor is consistent across all scenarios, with the tunnel commencing at the 4th layer.
The tunnel effect under data distribution shift

Based on the findings from the previous section and the tunnel's negative impact on transfer learning, we investigate the dynamics of the tunnel in continual learning scenarios, where large models are often used on smaller tasks typically containing only a few classes. We focus on understanding the impact of the tunnel effect on transfer learning and catastrophic forgetting [11]. Specifically, we examine how the tunnel and extractor are altered after training on a new task.

### Exploring the effects of task incremental learning on extractor and tunnel

**Motivation** In this section, we aim to understand the tunnel and extractor dynamics in continual learning. Specifically, we examine whether the extractor and the tunnel are equally prone to catastrophic forgetting.

**Experiments** We train a VGG-19 on two tasks from CIFAR-10. Each task consists of 5 classes from the dataset. We subsequently train on the first and second tasks and save the corresponding extractors \(E_{t}\) and tunnels \(T_{t}\), where \(t\in\{1,2\}\) is the task number. We also save a separate classifying head for trained on each task, that we use during evaluation.

**Results** As presented in Table 4, in any combination changing \(T_{1}\) to \(T_{2}\) or vice versa have a marginal impact on the performance. This is quite remarkable, and suggests that the tunnel is not specific to the training task. It seems that it _compresses the representations in a task-agnostic way_. The extractor part, on the other hand, is _task-specific_ and prone to forgetting as visible in the first four rows of Table 4. In the last two rows, we present two experiments that investigate how the existence of a tunnel affects the possibility of recovering from this catastrophic forgetting. In the first one, referred to as (\(E_{2}+T_{1}(FT)\)), we use original data from Task 1 to retrain a classifying head attached on top of extractor \(E_{2}\) and the tunnel \(T_{1}\). As visible, it has minimal effect on the accuracy of the first task. In the second experiment, we attach a linear probe directly to the extractor representations (\(E_{2}(FT)\)). This difference hints at a detrimental effect of the tunnel on representations' usability in continual learning.

In Appendix D.1 we study this effect further by training a tunnels on two tasks with a different number of classes, where \(n_{1}>n_{2}\). In this scenario, we observe that tunnel trained with more classes (\(T_{1}\)) maintains the performance on both tasks, contrary to the tunnel (\(T_{2}\)) that performs poorly on Task 1. This is in line with our previous observations in Section 2.2, that the tunnel compresses to the effective number of classes.

These results present a novel perspective in the ongoing debate regarding the layers responsible for causing forgetting. However, they do not align with the observations made in the previous study [47]. In Appendix D, we delve into the origin of this discrepancy and provide a comprehensive analysis of the changes in representations with a setup introduced with this experiment and the CKA similarity.

**Takeaway** The tunnel's task-agnostic compression of representations provides immunity against catastrophic forgetting when the number of classes is equal. These findings offer fresh perspectives on studying catastrophic forgetting at specific layers, broadening the current understanding in the literature.

\begin{table}
\begin{tabular}{l|c c} \hline \hline \multicolumn{1}{c}{} & First Task & Second Task \\ \hline \hline \(E_{1}+T_{1}\) & 92.04\% & 56.8\% \\ \hline \(E_{1}+T_{2}\) & 92.5\% & 58.04 \% \\ \hline \(E_{2}+T_{2}\) & 50.84 \% & 93.94 \% \\ \hline \(E_{2}+T_{1}\) & 50.66 \% & 93.72 \% \\ \hline \hline \(E_{2}+T_{1}(FT)\) & 56.1\% & – \\ \hline \(E_{2}(FT)\) & 74.4\% & – \\ \hline \end{tabular}
\end{table}
Table 4: The tunnel part is task-agnostic and can be freely mixed with different extractors retaining the original performance. We test the model’s performance on the first or second task using a combination of extractor \(E_{t}\) and tunnel \(T_{t}\) from tasks \(t\in\{1,2\}\). The last two rows \((FT)\) show how much performance can be recovered by retraining the linear probe attached to the penultimate layer \(E_{1}+T_{1}\) or the last layer of the \(E_{2}\).
### Reducing catastrophic forgetting by adjusting network depth

**Motivation** Experiments from this section verify whether it is possible to retain the performance of the original model by training a shorter version of the network. A shallower model should also exhibit less forgetting in sequential training.

**Experiments** We train VGG-19 networks with different numbers of convolutional layers. Each network is trained on two tasks from CIFAR-10. Each task consists of 5 classes from the dataset.

**Results:** The results shown in Figure 10 indicate that training shorter networks yields similar performance compared to the original model. However, performance differences become apparent when the network becomes shorter than the extractor part in the original model. This observation aligns with previous findings suggesting that the model requires a certain capacity to perform the task effectively. Additionally, the shorter models exhibit significantly less forgetting, which corroborates the conclusions drawn in previous works [32; 34] on the importance of network depth and architecture in relation to forgetting.

**Takeaway** It is possible to train shallower networks that retain the performance of the original networks and experience significantly less forgetting. However, the shorter networks need to have at least the same capacity as the extractor part of the original network.

## 5 Limitations and future work

This paper empirically investigates the tunnel effect, opening the door for future theoretical research on tunnel dynamics. Further exploration could involve mitigating the tunnel effect through techniques like adjusting learning rates for specific layers. One limitation of our work is its validation within a specific scenario (image classification), while further studies on unsupervised or self-supervised methods with other modalities would shed more light and verify the pertinence of the tunnel elsewhere.

In the experiments, we observed that ResNet-based networks exhibited shorter tunnels than plain MLPs or VGGs. This finding raises the question of whether the presence of skip connections plays a role in tunnel formation. In Appendix H, we take the first step toward a deeper understanding of this relationship by examining the emergence of tunnels in ResNets without skip connections.

## 6 Related work

The analysis of representations in neural network training is an established field [28; 56; 58]. Previous studies have explored training dynamics and the impact of model width [18; 26; 30; 45; 51; 55], but there is still a gap in understanding training dynamics [4; 37; 47; 58]. Works have investigated different architectures' impact on continual learning [33; 34] and linear models' behavior [10; 24; 25; 29]. Our work builds upon studies examining specific layers' role in model performance [4; 9; 38; 39; 45; 59] and sheds light on the origins of observed behaviors [12; 16; 42; 62].

Previous works have explored the role of specific layers in model performance [4; 9; 38; 39; 45; 59]. While some studies have observed a block structure in neural network representations, their analysis was limited to ResNet architectures and did not consider continual learning scenarios. In our work, we investigate a similar phenomenon, expanding the range of experiments and gaining deeper insights into its origins.

In [59], authors distinguish between critical and robust layers, highlighting the importance of the former for model performance, while individual layers from the latter can be reset without impacting the final performance. Our analysis builds upon this finding and further categorizes these layers into the extractor and tunnel, providing insights into their origins and their effects on model performance and generalization ability.

Figure 10: Training shorter networks from scratch gives a similar performance to the longer counterparts (top) and results in significantly lower forgetting (bottom). The horizontal lines denote original model’s performance.
Our findings are related to the Neural Collapse phenomenon [42], which has gained recent attention [12; 16; 62]. In our experiments, we also analyze the rank of the representation matrix and observe that the examined tunnel is characterized by a low representation rank.

## 7 Conclusions

This work presents new insights into the behavior of deep neural networks during training. We discover the tunnel effect, an intriguing phenomenon in modern deep networks where they split into two distinct parts - the extractor and the tunnel. The extractor part builds representations, and the tunnel part compresses these representations to a minimum rank without contributing to the model's performance. This behavior is prevalent across multiple architectures and is positively correlated with overparameterization, i.e., it can be induced by increasing the model's size or decreasing the complexity of the task.

We discuss potential sources of the tunnel and highlight the unintuitive behavior of neural networks during the initial training phase. This novel finding has significant implications for improving the performance and robustness of deep neural networks. Moreover, we demonstrate that the tunnel hinders out-of-distribution generalization and can be detrimental in continual learning settings.

Overall, our work offers new insights into the mechanisms underlying deep neural networks and can potentially improve the performance and robustness of these powerful models.