A Quantitative Approach to Predicting Representational Learning and
  Performance in Neural Networks
A key property of neural networks (both biological and artificial) is how
they learn to represent and manipulate input information in order to solve a
task. Different types of representations may be suited to different types of
tasks, making identifying and understanding learned representations a critical
part of understanding and designing useful networks. In this paper, we
introduce a new pseudo-kernel based tool for analyzing and predicting learned
representations, based only on the initial conditions of the network and the
training curriculum. We validate the method on a simple test case, before
demonstrating its use on a question about the effects of representational
learning on sequential single versus concurrent multitask performance. We show
that our method can be used to predict the effects of the scale of weight
initialization and training curriculum on representational learning and
downstream concurrent multitasking performance.
Ryan Pyle, Sebastian Musslick, Jonathan D. Cohen, Ankit B. Patel
# A Quantitative Approach to Predicting Representational Learning and Performance in Neural Networks

###### Abstract

A key property of neural networks (both biological and artificial) is how they learn to represent and manipulate input information in order to solve a task. Different types of representations may be suited to different types of tasks, making identifying and understanding learned representations a critical part of understanding and designing useful networks. In this paper, we introduce a new pseudo-kernel based tool for analyzing and predicting learned representations, based only on the initial conditions of the network and the training curriculum. We validate the method on a simple test case, before demonstrating its use on a question about the effects of representational learning on sequential single versus concurrent multitask performance. We show that our method can be used to predict the effects of the scale of weight initialization and training curriculum on representational learning and downstream concurrent multitasking performance.

## 1 Introduction

One of, if not the, most fundamental question in neural networks research is how representations are formed through learning. In machine learning, this is important for understanding how to construct systems that learn more efficiently and generalize more effectively (Bengio et al., 2013; Witty et al., 2021). In cognitive science and neuroscience, this is important for understanding how people acquire knowledge (Rumelhart et al., 1993; Rogers and McClelland, 2004; Saxe et al., 2019), and how this impacts the type of processing (e.g., serial and control-dependent vs. parallel and automatic) used in performing task(s) (Musslick and Cohen, 2021; Musslick et al., 2020). One important focus of recent work has been on the kinds of inductive biases that influence how learning impacts representations (e.g., weight initialization, regularization in learning algorithms, etc. (Narkhede et al., 2022; Garg and Liang, 2020)) as well as training curricula (Musslick et al., 2020; Saglietti et al., 2022). This is frequently studied using numerical methods, by implementing various architectural or learning biases and then simulating the systems to examine how these impact representational learning (Caruana, 1997; Musslick et al., 2020). Recently, Sahs et al. (2022) introduced a novel analytic approach to this problem, which can be used to predict important inductive bias properties from the network's initialization. Here we extend this approach by combining it with a neural tangent kernel-based analysis in order to qualitatively predict the kinds of representations that are learned, and the consequences this has for processing. We provide an example that uses a neural network model to address how people acquire simple tasks, and the extent to which this leads to serial, control-dependent versus parallel, automatic processing and multitasking capability (Musslick et al., 2016; Musslick and Cohen, 2021; Musslick et al., 2020). We expand
upon theoretical results introduced in Sahs et al. (2022), combining them with a neural tangent kernel analysis (Jacot et al., 2018), which allows for prediction of the inductive bias (and resulting representations learned by the network and downstream task performance) from the initial conditions of the network and the training regime. In the remainder of this section, we provide additional background that motivates the example we use. Then, in the sections that follow, we describe the analysis method, its validation in a benchmark setting, and the results of applying it to a richer and more complex example.

_Shared versus separated representations and flexibility versus efficiency._ One of the central findings from machine learning research using neural networks is that cross-task generalization (sometimes referred to as transfer learning) can be improved by manipulations that promote the learning of shared representations--that is, representations that capture statistical structure that is shared across tasks (Caruana, 1997; Baxter, 1995; Collobert and Weston, 2008). One way to do so is through the design of appropriate training regimens and/or learning algorithms (e.g., multi-task learning and/or meta-learning; (Caruana, 1997; Ravi et al., 2020)). Another is through the initialization of network parameters; for example, it is known that small random initial weights help promote the learning of shared structure, by forcing the network to start with what amounts to a common initial representation for all stimuli and tasks and then differentiate the representations required for specific tasks and/or stimuli under the pressure of the loss function (Flesch et al., 2021). Interestingly, while shared representations support better generalization and faster acquisition of novel but similar tasks, this comes at a cost of parallel processing capacity, a less commonly considered property of neural networks that determines how many distinct tasks the system can perform _at the same time_--that is, its capacity for concurrent multitasking (Feng et al., 2014; Musslick et al., 2016; Petri et al., 2021). Note that our use of the term "multitasking" here should not be confused with the term "multitask" learning: the former refers to the simultaneous _performance_ of multiple tasks, while the latter refers to the simultaneous acquisition of multiple tasks. These are in tension: if two tasks share representations, they risk making conflicting use of them if the tasks are performed at the same time (i.e., within a single forward-pass); thus, the representations can be used safely only when the tasks are executed serially. This potential for conflict can be averted if the system uses _separate_ representations for each task, which is less efficient but allows multiple tasks to be performed in parallel. This tension between shared vs. separated representations reflects a more general tradeoff between the _flexibility_ afforded by shared representations (more rapid learning and generalization) but at the expense of serial processing, and the _efficiency_ afforded by separated, task-dedicated representations (parallel processing; i.e., multitasking) but at the costs of slower learning and poorer generalization (i.e., greater rigidity; Musslick and Cohen (2021); Musslick et al. (2020)). While this can be thought of as analogous to the tension between interpreted and compiled procedures in traditional symbolic computing architectures, it has not (yet) been widely considered within the context of neural network architectures in machine learning.

_Shared versus separated representations and control-dependent versus automatic processing._ The tension between shared and separated representations also relates to a cornerstone of theory in cognitive science: the classic distinction between control-dependent and automatic processing (Posner and Snyder, 1975; Shiffrin and Schneider, 1977). The former refers to "intentional," "top-down," processes that are assumed to rely on control for execution (such as mental arithmetic, or searching for a novel object in a visual display), while the latter refers to processes that occur with less or no reliance on control (from reflexes, such as scratching an itch, to more sophisticated processes such as recognizing a familiar object or reading a word). A signature characteristic of control-dependent processes is the small number of such tasks that humans can perform at the same time--often only one--in contrast to automatic processes that can be performed in parallel (motor effectors permitting). The serial constraint on control-dependent processing has traditionally been assumed to reflect limitations in the mechanism(s) responsible for control _itself_, akin to the limited capacity imposed by serial processing in the core of a traditional computer (Anderson and Lebiere, 2014; Pashler, 1994; Posner and Snyder, 1975). However, recent neural network modeling work strongly suggests an alternative account: that constraints in control-dependent processing reflect the imposition of serial execution on processes that rely on shared representations (Musslick and Cohen, 2021; Musslick et al., 2020). That is, constraints associated with control-dependent processing reflect the _purpose_ rather than an intrinsic _limitation_ of control mechanisms. This helps explain the association of control with flexibility of processing Cohen (2017); Duncan (2001); Goschke (2000); Kriete et al. (2013); Shiffrin and Schneider (1977); Verguts (2017): flexibility is afforded by shared representations, which require control to insure they are not subject to
conflicting use by competing processes. It also explains why automaticity--achieved through the development of task-dedicated representations--takes longer to acquire and leads to less generalizable behavior (Logan, 1997). Together, these explain the canonical trajectory of skill acquisition from dependence on control to automaticity: When people first learn to perform a novel task (e.g., to type, play an instrument, or drive a car) they perform it in a serial, control-dependent manner, that precludes multitasking. Presumably this is because they exploit existing representations that can be "shared" to perform the novel task as soon as possible, but at the expense of dependence on control. However, with extensive practice, they can achieve efficient performance through the development of separated, task-dedicated representations that diminish reliance on control and permit performance in parallel with other tasks (i.e., concurrent multitasking; Garner & Dux (2015); Musslick & Cohen, J. D. (2019)).

These ideas have been quantified in mathematical analyses and neural network models, and fit to a wide array of findings from over half a century of cognitive science research (Musslick et al., 2020). However, the specific conditions that predispose to, and regulate the formation of shared versus separated representations are only qualitatively understood, and theoretical work has been restricted largely to numerical analyses of learning and processing in neural network models. Some methods have sought to quantify the degree of representation sharing between two tasks in terms of correlations between activity patterns for individual tasks (Musslick et al., 2016, 2020; Petri et al., 2021; 2021) in order to predict multitasking capability. Other methods, that quantify the representational manifold of task representations (Bernardi et al., 2020) have been applied to characterize multitasking capability (Henselman-Petrusek et al., 2019). However, while these methods provide a snapshot of representation sharing at a given point in training, they do not provide direct or analytic insight into the _dynamics_ of learning shared versus separated representations, nor how the inductive bias of a system may affect the representations learned.

In this article, we expand upon the ideas introduced in Sahs et al. (2022) to show how the initial condition of a network and the training regime to which it will be subjected can predict the implicit bias and thus the kinds of representations it will learn (e.g., shared vs. separated) and the corresponding patterns of performance it will exhibit in a given task setting. This offers a new method to analyze how networks can be optimized to regulate the balance between flexibility and efficiency. The latter promises to have relevance both for understanding how this is achieved in the human brain, and for the design of more adaptive artificial agents that can function more effectively in complex and changing environments.

_Task structure and network architecture._ For the purposes of illustration and analysis, we focus on feedforward neural networks with three layers of processing units that were trained to perform sets of tasks involving simple stimulus-response mappings. Each network was comprised of an input layer, subdivided into pools of units representing inputs along orthogonal stimulus dimensions (e.g., representing colors, shapes, etc.), and an additional pool used to specify which task to perform. All of the units in the input layer projected to all of the units in the hidden layer, which all projected to all units in the output layer, with an additional projection from the task specification input pool to the output layer. The output layer, like the input layer, was divided into pools of units, in this case representing outputs along orthogonal response dimensions (e.g., representing manual, verbal, etc.).

Networks were trained in an environment comprised of several feature groups (e.g., shape, size, etc.) and response groups (e.g., verbal, manual, etc.) corresponding to the stimulus and response dimensions along which the pools of input and output units of the networks were organized. Each network was trained to perform a set of tasks, in which each task was defined by a one-to-one mapping from the inputs in one pool (i.e., along one stimulus feature dimension) to the outputs in a specified pool (i.e., along one response dimension), ignoring inputs along all of the other feature dimensions and requiring null outputs along all of the other response dimensions.1 Training and testing could be performed for one task at a time ("single task" conditions), by specifying only that task in the task input pool and requiring the correct output over the task-relevant response dimension and a null response over all others; or with two or more tasks in combination ("multitasking" conditions), in which the desired tasks were specified over the task input units, and the network was required to generate correct responses over the relevant response dimensions and null responses for all others. In all cases, an input was always provided along every stimulus dimension, and the
network had to learn to ignore those that were not relevant for performing the currently specified task(s). In each case, the question of interest was how initialization and learning impacted the final connection weights to and from the hidden layer, the corresponding representations the networks used to perform each task, and the patterns of performance in single task and multitasking conditions. Specifically, we were interested in the extent to which the networks learned shared versus separated representations for sets of tasks that shared a common feature dimension; and the extent to which the analytic methods of interest were able to predict, from the initial conditions and task specifications, the types of representations learned, and the corresponding patterns of performance (e.g., speed of learning and multitasking capability). We evaluated the evolution of representations over the course of learning in two ways: using the analytic techniques of interest, and using a recently developed visualization tool to inspect these, each of which we describe in the two sections that follow.

## 2 Understanding and Visualizing Learning Dynamics

### Analyzing Learning Dynamics Using the Neural Tangent Kernel

#### 2.1.1 The Neural Tangent Kernel (NTK)

Understanding the learning dynamics of a neural network (NN) can be done using a framework known as the NTK. This is based on a kernel function \(K(x,x^{\prime})\) that represents the'similarity' of inputs \(x\) and \(x^{\prime}\) (from here on, we use \(x\) from the training set and \(x^{\prime}\) from the test set); that is, how much influence each individual sample \(x\) from the training set has on the output decision of the NN on a test sample \(x^{\prime}\). This is embodied in a kernel expansion (Jacot et al., 2018).

Using gradient descent (GD) on a NN with scalar learning rate \(\eta\) and a \(P\times 1\) vector of real parameters \(\theta\), the parameter update can be written as

\[\theta(t+1)=\theta(t)-\eta\frac{dL(\theta)}{d\theta}.\]

Taking the gradient flow approximation \(\eta\to 0\) (e.g. as the step size approaches \(0\), resulting in a continuous flow rather than discrete steps) we have

\[\dot{\theta}=-\frac{dL(\theta)}{d\theta}\]

Assuming our loss depends only on the network output \(\hat{y}\), we can rewrite this as a sum over \(N\) training samples \(\mathcal{D}:=\{(x_{m},y_{m})\}\):

\[\dot{\theta}=-\frac{dL}{d\hat{y}}\frac{d\hat{y}}{d\theta}=\sum_{m\in\mathcal{ D}}\frac{dL}{d\hat{y}_{m}}\frac{d\hat{y}_{m}}{d\theta}:=\sum_{m\in\mathcal{D}} \epsilon_{m}\phi_{m},\]

where \(\epsilon_{m}\in\mathbb{R}\) is the loss sensitivity and \(\phi_{m}\) is the \(P\times 1\) vector of NTK features (e.g. \(\frac{d\hat{y}_{m}}{d\theta_{p}}\) for each parameter \(p\)) for sample \(x_{m}\). How does the actual NN prediction function change with learning over time? We can answer this by taking total time derivatives yielding

\[\dot{\hat{y}}(\theta)=\frac{d\hat{y}(\theta)}{d\theta}^{T}\dot{\theta}=-\frac {d\hat{y}(\theta)}{d\theta}^{T}\frac{dL(\theta)}{d\hat{y}}\frac{d\hat{y}( \theta)}{d\theta},\]

where the kernel function \(K(x,x^{\prime};\theta):=\frac{d\hat{y}(x;\theta)}{d\theta}^{T}\frac{d\hat{y}( x^{\prime};\theta)}{d\theta}\). Note that this means the network's time evolution is a kernel function, made up of the NTK at time \(t\) with parameters \(\theta=\theta(t)\) and kernel weights \(\frac{dL(\theta)}{dy}\) Expanding as a sum over the training set we have

\[\dot{\epsilon}_{n}(t)=\sum_{m\in\mathcal{D}}K_{mn}(t)\epsilon_{m}(t),\quad \forall n\in\mathcal{D}\]

where we have assumed a mean squared error loss for \(L\), which results in the loss sensitivities becoming the prediction errors \(\epsilon_{m}(t):=\frac{dL(\theta)}{dy}=y_{m}-\hat{y}_{m}(t)\). The coefficients coupling the errors are \(K_{mn}(t):=\phi_{m}(t)\cdot\phi_{n}(t)\), also known as the elements of the \(N\times N\) NTK matrix \(K(t)=[K_{mn}(t)]\).
If the model is close to linear in the parameters \(\theta\) (i.e., we are in the so-called _kernel_ or _lazy_ training regime Chizat et al. (2019), where the NN's basis functions are fixed for all time), then the NTK will not change much during training, allowing the entire learning to be readily interpretable as a linear kernel machine (Ortiz-Jimenez et al., 2021). In this case, each update to the model is fully interpretable under kernel theory, with each data point influencing how the model evolves (see Fig. 1).

But what if the model is not close to the linear/lazy/kernel regime (i.e., it is in the so-called _adaptive regime_?2 In this case the NN's basis functions do change over time, rendering the NTK function time-dependent (in which case we denote it as \(K(x,x^{\prime},t)\)). Tracking the NTK along the learning trajectory/path of the model parameters \(\theta(t)\) yields the Path-integrated NTK (Domingos, 2020).

Footnote 2: All modern NNs perform best in the adaptive regime, and there is a significant performance gap between kernel and adaptive regime models Arora et al. (2019). The power of the adaptive regime is that it allows the NN to learn, based on the data, the basis functions that are most useful. In contrast, the kernel regime has fixed basis functions solely determined by architecture and initialization, with no dependence on the data or the task being learned.

#### 2.1.2 Path-Integrated NTK (Pntk)

Let the PNTK be denoted by \(P(x,x^{\prime};t)\). Then all NNs trained in a supervised setting with gradient descent result in a (pseudo-)kernel machine3 of the form

Figure 1: An explanation of a model update using the NTK. Top Left: The model’s prediction at t = 0, 1, final. Top Right: The kernel function (t=0), or influence on the network due to each data point, tied by color. Bottom Left: The changes due to each influence function (t=0), which are weighted by the loss function. Bottom Right: A comparison between the model’s final state and the prediction made using the NTK update. Note that while this is a one step NTK prediction, if the NTK assumptions (kernel regime) hold the NTK can predict arbitrarily far ahead.
Footnote 1: We use the notation of [20] to denote the number of nodes in the network.

\[\hat{y}(x)=\sum_{m\in\mathcal{D}}P(x,x_{m})+\hat{y}_{0}(x),\]

where the initial predictions \(\hat{y}_{0}(x):=\hat{y}(x,t=0)\) and \(P\) is the PNTK pseudo-kernel of the form (Domingos, 2020):

\[P(x,x^{\prime}):=\int_{0}^{t}L^{\prime}(y^{\prime}(x^{\prime}),\hat{y}^{\prime} (x^{\prime},t^{\prime}))\frac{d\hat{y}(x,t^{\prime})}{d\theta}\cdot\frac{d\hat{ y}(x^{\prime},t^{\prime})}{d\theta}dt^{\prime}=\int_{0}^{t}L^{\prime}(y^{ \prime}(x^{\prime}),\hat{y}^{\prime}(x^{\prime},t^{\prime}))K(x,x^{\prime},t )dt^{\prime}\]

At intermediate learning times we have

\[\hat{y}(x,t)=\hat{y}(x,t=0)-\int_{0}^{t}dt^{\prime}\sum_{m\in\mathcal{D}}L^{ \prime}(y_{m},\hat{y}_{m}(\theta(t^{\prime})))K(x,x_{m};\theta(t^{\prime})),\]

where our loss function is \(L\) and the loss sensitivity \(L^{\prime}\) is with respect to the predictions \(\hat{y}\) (e.g. L' = \(\frac{dL}{dy}\)). Thus, the PNTK shows that the NN predictions can be expressed in terms of the NTK, weighted by the loss sensitivity function, along the entire learning path/trajectory.

In this article, we use the NTK and PNTK to analyze how representations in the hidden layer of the network evolve during learning of the task(s). The NTK allows a decomposition of the instantaneous changes in the predictions of the NN over training inputs and training time (e.g. how \(\hat{y}\) can be decomposed over \(x_{m}\in\mathcal{D}\) at any point in time), whereas the PNTK provides an integral over those predicted effects over a specified time window (typically from 0 to \(t\)). We can analyze the NTK and PNTK, grouped in various ways (e.g., by input dimensions, tasks, and/or output dimensions) in order to fully characterize how a NN undergoing training acquires representations of the relevant information over the course of learning.

### Visualizing Learning Dynamics Using M-PHATE

We used Multislice PHATE (or M-PHATE; Gigante et al. (2019)) to visualize the evolution of representational manifolds over the course of learning. M-PHATE is a dimensionality reduction algorithm for time-series data, that extends the successful PHATE algorithm (Moon et al., 2019) to visualize internal network geometry, and that can be used to capture temporal dynamics. It does so by using longitudinal (time series) data to generate a multislice graph, and then uses PHATE to dimensionally reduce the pairwise affinity similarity kernel of the graph. By applying this to the hidden unit activations of neural networks, it can be used to visualize the representational manifolds over the course of training. Furthermore, by grouping analyses--for example, according to feature dimensions, tasks, or response dimensions--M-PHATE can be used to reveal how representations evolve that are sensitive to these factors.

It is important to note that the M-PHATE analysis works by analyzing the _hidden_ layer activities. In contrast, the NTK kernel computes similarities using the whole network's gradients. This means that the results of the two methods are are not directly comparable, but qualitative comparisons can be made.

## 3 Validation of NTK and PNTK Analyses of Learning in Simple Networks

We begin with a validation of NTK and PNTK analyses, by using them to predict the evolution of representations in a linear network which are tractable to standard analytic methods, the results of which can be used as a benchmark. Specifically, we use the results from Saxe, McClelland, and Ganguli (henceforth SMG; Saxe et al. (2013)), which show that for a linear NN with a bottleneck hidden layer, the network will learn representations over that layer that correspond to singular values of the weight matrix, in sequence, each learned with a rate proportional to the magnitude of the singular value, up until the dimensionality of the bottleneck layer. Here, we show that NTK and PNTK analyses can be used to qualitatively predict this behavior from the initial conditions of the network (i.e., its initial weights and training set).
We take a simple case for illustrative purposes, using a network with an input dimensionality of 4, a bottleneck (hidden layer) dimensionality of 3, and an output dimensionality of 5. A target linear transformation \(W_{target}\) is generated, and then used to generate random training data \(y=W_{target}x\), where \(x\) is randomly drawn from white noise. We call this the SMG task, and replicate their results, showing behavior qualitatively similar to that reported in their previous work (Fig 2).

What can a PNTK analysis tell us over and above the original analysis? We consider two approaches to answering this question. The first exploits the fact that we are considering a simple linear system, and thus inputs can be broken down by input dimension. This allows us to compute a PNTK value that predicts how a change in each individual input dimension affects each individual output dimension across examples, which can then be compared to the true \(W_{target}\) (Fig 3). We focus on two time points: \(t=190\), that falls in the middle of the period during which the first singular vector is being acquired; and \(t=770\), that falls in the middle of the period during which the second singular vector is being acquired. The PNTK analysis confirms that the first singular vector of \(W_{target}\) is being learned at \(t=190\), the second is being learned at \(t=770\), and the first singular vector is well learned (e.g. learning acquired is significant) by \(t=190\),4 while both are well learned (e.g. the first is also remembered) by \(t=770\).

Footnote 4: What we mean by a singular vector ‘is being learned’ at time t is that it is the top singular vector of the NTK(t), while ‘is well learned’ means that it singular value (of PNTK(t)) is significant (compared to the maximum from \(W\)). This means that a well learned singular vector can also continue to be learned, while a singular vector being learned may or may not be well learned at a particular time. See Fig 6 markers for a visualization relating to this

The foregoing analysis, although easy and clear, is limited to linear systems. To assess the applicability of our analyses to non-linear networks, in our second approach we numerically compare the singular vectors of the full NTK and PNTK (at specific time points \(t=190,770\)) against the projections of the data \(x\) into the coordinate system spanned by the singular vectors of \(W_{target}\). This allows us to compare how much each data point contributes to the NTK or PNTK compared to how much it would contribute if it was perfectly learning the modes of \(W_{target}\), an approach that works for linear or nonlinear systems. These results are comparable to those of the analysis of the linear system (Fig 4), providing support for the generality of the NTK and PNTK analyses to non-linear networks.

Finally, we conduct another analysis using the PNTK, examining how the learning of each mode changes progressively over time. The PNTK allows us to examine the patterns that are being learned at each time point, by examining the eigenvectors of the NTK. Consistent with the previous work (Saxe et al., 2013), the analysis confirms that eigenvectors are learned one at a time, with larger ones learned first. More interestingly, secondary learning (e.g., the mode that is being acquired at the second fastest rate, and is the second eigenvalue/vector pair of the NTK) reveals relevant patterns, with transitions occurring at times

Figure 2: Loss over time in the SMG task for training and testing data (see text for network description). Note the tiered learning, with loss dropping over some regions of time while remaining near constant in others.
Figure 3: Differences between SVD singular vectors of \(W_{target}\) compared to NTK and PNTK, at times \(t=190\) and \(770\). At time \(170\), the PNTK (top right) shows that the first singular vector has, for the most part, been learned, though it is still being fine-tuned, as shown by the NTK (top left), prior to learning of the second singular vector. At time \(770\), the NTK (bottom left) shows the second singular vector is being learned (notice we are comparing against the _second_ singular vector of \(W_{target}\)), while the PNTK (bottom right) shows that the first two singular vectors have been incorporated by the model.
Figure 4: Comparisons between projections of \(x\) into singular vectors of \(W_{target}\) and singular vectors of NTK and Path NTK at times \(t=190,770\). At time \(170\), the NTK (top left) shows us the first mode is being learned while the Path NTK (top right) shows that the 1st mode has already been well incorporated overall. At time \(770\), the NTK (bottom left) shows the second mode is being learned (notice we are comparing against the _second_ mode of \(W_{target}\)), while the PNTK (bottom right) shows that the first mode is still remembered.
dictated by the primary learning switching singular vectors (e.g. when the primary singular vector is learned and switches to learning the second singular vector, the secondary learning switches from learning the second singular vector to learning the third singular vector; Fig 5). These results align with changes in performance of the network. Fig 6 shows the primary learning of eigenvectors and eigenvalues over time along with model performance. This reveals that the two are closely linked, with changes in NTK singular values _anticipating_ model singular vectors aligning with the true task and accompanying decreases in task loss.

Figure 5: Match between NTK learning (singular vector 1 - primary, singular vector 2 - secondary) and singular vectors of \(W_{target}\). Primary learning shows a progression, where larger singular values and vectors are learned first, while the secondary learning shows a more intricate pattern with switches tied to the primary learning.
In summary, the PNTK analysis is consistent with the results reported by SMG for a linear network, using a technique that is extendable to non-linear networks. It is important to emphasize that the PNTK analysis is _predictive_, in that the results are derived only from the conditions of the network at the time point to which the analysis is applied, but predict the representational organization of the network following subsequent learning given the training regime. Of course, the SMG theory was also predictive, but only works in the linear regime--the PNTK can readily be expanded to nonlinear applications. It also reveals interesting new patterns, particularly in the secondary learning dynamics of the network, implying that some groundwork for learning higher modes is in place before their primary learning begins.

## 4 Application of NTK and PNTK Analyses to Learning and Performance in Nonlinear Networks

In the preceding section, we validated the use of NTK and PNTK analyses of learning dynamics in simple networks. Here, we explore using this technique to examine representational learning and its relationship to network performance in a more complex nonlinear network, that addresses how initial conditions influence the development of shared versus separated representations and its impact on parallel task execution (i.e., concurrent multitasking).

### Model

#### 4.1.1 Network Architecture

The network used for all further experiments is shown in Fig. 7 (cf. Musslick et al. (2020)). It had two sets of input units: a stimulus set, \(x_{1}\), used to represent stimulus features; and a task set, \(x_{2}\), used to indicate which task(s) should be performed on a given trial. The stimulus set was further divided into \(g_{1}\) pools of units, each of which was used to represent an independent stimulus dimension comprised of \(m\) features along each dimension. Accordingly, each pool was comprised of \(m\) units, with each feature represented as a one-hot input pattern over the group. The task set was comprised of single pool with a number of units equal to the number of tasks that could be specified (see below), and one unit used to represent each task. Both sets of input units projected to a single set of hidden units, with connection weights \(w_{1}\) and \(w_{2}\) for the stimulus set and task set, respectively. The H units in the hidden layer used a sigmoidal activation function \(\phi\), producing an activation vector of

\[h=\phi(w_{1}x_{1}+w_{2}x_{2}+b_{1})\]

where \(b_{1}=-2\) is a fixed bias, ensuring that the units were inhibited if no input is was provided (see Section 4.1.3). All of the hidden units, as well as the input units in the task set, projected to all of units in the

Figure 6: Comparison between NTK learning and model loss. Left: Loss vs Overlap between NTK singular vectors and \(W_{target}\) singular vectors. Right: Loss vs PNTK singular values. Dotted lines show the location of \(t=190,770\), which are the two points used for further analysis above.
output layer, with weights \(v_{1}\) and \(v_{2}\), respectively. The output layer, like the stimulus set of the input layer, was divided into \(g_{2}\) pools of units, each of which was used to represent an independent response dimension comprised of \(m\) responses along each dimension. Thus, each pool was comprised of \(m\) units with each response represented as a one-hot input pattern over the set. Like the hidden layer, the output layer used a sigmoidal activation function \(\phi\), along with output bias \(b_{2}=b_{1}\), leading to a final output of

\[y=\phi(v_{1}h+v_{2}x_{2}+b_{2})\]

or, in terms of inputs only

\[y=\phi(v_{1}\phi(w_{1}x_{1}+w_{2}x_{2}+b_{1})+v_{2}x_{2}+b_{2})\]

Note that task input \(x_{2}\) appears twice here, as there are two independent pathways involving \(x_{2}\) (one projecting to the hidden layer and the other to the output layer).

#### 4.1.2 Task Environment

As described above, a task was defined as a one-to-one mapping from the features along a single stimulus dimension to the responses of a single output dimension, reflecting response mappings in classic multitasking paradigms (e.g., Pashler 1994). This corresponded to the mapping of the \(m\) input units from the specified pool \(g_{1}\) of the stimulus group \(x_{1}\) to the associated \(m\) output unit in the specified pool \(g_{2}\) of the output units. In aggregate, this yielded \(g_{1}\cdot g_{2}\) tasks, and thus the task set \(x_{2}\) had that many units. For a given trial, a single feature unit was activated in each pool of the stimulus set \(x_{1}\) (i.e., each of the \(g_{1}\) input pools had one of its \(m\) units activated). For performance of a single task, only a single unit was activated in the task set \(x_{2}\). The network was then required to activate the output unit in the response pool corresponding to the input unit activated in the stimulus pool specified by the task unit activated in \(x_{2}\), and to suppress activity of all other output units. For example, task 1 consisted of mapping the \(m\) units of input pool \(g_{1}=1\) to the \(m\) units of the output pool \(g_{2}=1\); as only one one of the \(m\) units was active, the task consisted of mapping the active \(m\)th element of \(g_{1}\) to the m\(th\) element of \(g_{2}\), while outputting null responses elsewhere. For multitasking performance, two or more units in the task set were activated, and the network was required to activate the response corresponding to the input for each task specified, and suppress all other output units. Multitasking was restricted to only those tasks that shared neither an input set nor output set (see Lesnick et al. (2020) for a more detailed consideration of "legal multitasking").

Figure 7: Network architecture and task environment. (A) Network: The input layer is partitioned into two sets, \(x_{1}\) for stimulus inputs and \(x_{2}\) for specification of task(s) to be performed. The stimulus set is further partitioned into four pools, each representing a stimulus dimension comprised of \(m=3\) feature units. Both input groups project to the hidden layer, comprised of \(H=200\) non-linear processing units. Both the hidden and task input units project to the output layer. The output layer is comprised of three pools of non-linear processing units, each representing a response dimension comprised of \(m=3\) response units. (B) Task: black lines show the mappings from each stimulus pool to each response pool that make up the twelve tasks on which the network was trained. Each line represents a one-to-one-mapping that the network had to learn, associating each feature within a given stimulus pool to a corresponding response in the output pool for a given task.
We report results for a network that implemented four stimulus dimensions and three response dimensions (i.e., stimulus pools \(g_{1}=4\) and output pools \(g_{2}=3\)).5 This yielded a total of 12 tasks (\(x_{2}=12\)). Since each stimulus dimension had three features, and each response dimension had three possible responses (i.e., \(m=3\)), in total the network had 24 input units (\(x_{1}=12\) stimulus input, and \(x_{2}=12\) task input units) and 12 output units, as well as \(H=200\) hidden units.6

Footnote 5: We chose a different number of stimulus and response pools to be able to distinguish partitioning of the hidden unit representations according to input versus output dimensions, or both.

Footnote 6: The number of hidden units was chosen to avoid imposing a representational bottleneck on the network, thus allowing it the opportunity to learn separate representations for the mappings of each of the twelve tasks. This was done to insure that any tendencies for the network to learn lower dimensional representation were more likely to reflect factors of interest (viz., initialization and/or training protocols) and could not be attributed to limited representational resources.

#### 4.1.3 Initialization and Training

In the experiments, we manipulated the initialization of all connection weights between a _standard initialization_ condition (random uniform distribution in [-1,.1]) and a _large initialization_ condition (random uniform distribution in [-1, 1]). All biases were set to \(b_{i}=-2\) to encourage learning of an attentional scheme over the task weights in which activation of a task input unit placed processing units to which it projected in the hidden and output layers in a more sensitive range of their nonlinear processing functions 7. No layer-specific normalization (e.g., by batch) was used. For all experiments, networks tasks were always sampled uniformly from all available tasks. However, in each we manipulated whether training was restricted to performance of only one task at a time (_single task_ condition) or required performance of multiple tasks simultaneously (_multitasking_ condition), as described in the individual experiments below. In all cases, the network was trained with stochastic gradient descent (SGD) using a base learning rate of.01 for 10000 epochs, with parameters found via hyper-parameter search.8

Footnote 7: This exploits the nonlinearity of the activation functions to implement a form of multiplicative gating without the need for any additional specialized attentional mechanisms; see Cohen et al. (1990); Musslick et al. (2020) for relevant discussions).

Footnote 8: As the PNTK analyzes a specific network instantiation, all NTK-based and MPHATE-based visualizations (except where otherwise noted) are based on one trial. We re-ran each experiment at least five times to confirm that there were no major qualitative changes in the results and to generate correlation metrics over multiple experiments. Experimental results are averaged over 10 trials

### Predicting Impact of Standard vs. Large Initialization on Representational Learning

#### 4.2.1 Effect of Initialization on Representational Learning

It has previously been observed that lower initial weights promote the formation of representational sharing among tasks that share the same input and/or output dimensions (Flesch et al., 2021; Musslick et al., 2017; 2020), consistent with other findings from work in machine learning (Sahs et al., 2022; Ding et al., 2014). Here, we sought first to validate this finding in the present architecture, and visualize it using M-PHATE, and then evaluate the extent to which it could be predicted by the PNTK analysis. To do so, we compared the standard initialization with the large initialization under the single task training condition.

Fig 8 shows an M-PHATE plot of how the patterns of activity over the hidden units evolve over the course of training. The top row shows this for the standard initialization condition, with each panel showing the patterns of hidden unit activity grouped (averaged) along different dimensions. There is clear structure in the groupings, which reflects the sharing of representations for tasks that use the same inputs or outputs. For example, grouped by task (the leftmost panel), there are four clusters of three tasks each, with each cluster comprised of tasks that share the same inputs, confirmed by examining the individual elements. Similarly, grouped by input (the second panel from the left) there are three clusters of four tasks each ), with each cluster corresponding to within-group position across the four pools. Notice that the grouping here gradually decoheres over time (as the clusters are transient, we show a zoomed in inset showing this phenomenon in Fig 8 (inset). Finally, grouped by output (the rightmost panel), there are three clusters of tasks that share the same output. Both effects can be seen in the _Task by Inputs_ grouping (third panel from the left), in which three clusters appear early in training (dark dots), each of which is comprised of tasks
Figure 8: M-PHATE visualization of hidden unit activity in the network over course of training. Dots correspond to the relative positions of the patterns of hidden unit activities (each averaged over all input stimuli that share the same feature by which they are grouped, as explained below) at each epoch in training (darker colors represent earlier epochs in training and lighter colors later epochs; see legend to the right), with blue lines showing connections across time. Plots in the top row show standard initialization condition, and in the bottom row large initialization condition. Plots in each column show hidden unit activities grouped (averaged) along dimensions indicated by the label (e.g., in the leftmost column, labelled _Task_, hidden unit activations are grouped by task, with each dot corresponding to the average pattern of hidden unit activity over all inputs for a given task). Notice that, in all cases, in the standard initialization condition (top row) hidden unit representations exhibit highly organized structure, whereas in the large initialization (bottom row) they exhibit substantially less structure (see text for additional discussion). Inset shows M-PHATE visualization of hidden unit activity in the network with the standard initialization at t = 3 for the grouped by inputs configuration. This zoomed in view shows the transient clustering into 3 groups of 4, which collapses over time.
that share the same output, followed later (lighter dots) by a separation into subclusters of tasks that share the same inputs. The early organization by outputs followed later with organization by inputs is consistent with the tendency, in multilayered networks without layer-specific weight normalization, for weights closest to the output layer to experience the steepest initial gradients and therefore the earliest effects of learning.9 These observations corroborate the general principle that lower initial weights promote representational sharing in environments for which tasks share structure.

Footnote 9: To confirm this account, and rule out the possibility that early organization by outputs was because the lower dimensional structure of the outputs (three dimensions) made it easier to learn than the input structure (four dimensions), we conducted the same experiment on a network that had fewer input dimensions (three) than output dimensions (four), and observed similar results (see Appendix, Section A.3.1).

The lower panels in Fig 8 show the results for the large initialization condition. These are in stark contrast to those for the standard initialization condition, showing little if any structure: each task develops its own representations that are roughly equidistant from the others, irrespective of shared inputs and/or outputs. The one deviation from this pattern is for grouping by output (rightmost panel), in which three clusters emerge, once again presumably reflecting the early and strong influence of the gradients on weights closest to the output of a multilayered network in the absence of any layer-specific form of weight normalization.

Together, the observations above provide strong confirmation that, in this network architecture as in many others, lower initial weights promote the development of shared representations for tasks that share structure (i.e., input or output dimensions), and showcase the utility of M-PHATE for clearly and concisely visualizing such qualitative effects. In the sections that follow, we apply NTK analyses to show that downstream performance can be predicted quantitatively from the initial conditions.

#### 4.2.2 Use of NTK to Predict Representational Learning

First, we conduct an NTK analysis of the standard initialization network, following initialization but prior to training (i.e., \(T=0\)), to assess whether this can anticipate the effects of learning. The output of the NTK analysis is a 500x500 matrix, corresponding to the 500 training inputs across all tasks. The M-PHATE observations shown in Fig 8 suggest that, over the course of training, the representations learned by the network's hidden units came to be clustered according to both the four input dimensions (shared across tasks) and their mapping to each of the three output dimensions (according to task). To determine whether this organization was predicted by, and can be observed in the NTK analysis, we carried out two variants of this analysis: one in which individual training passes were sorted by the current input (aggregated over tasks), and the other sorted by task (aggregated over inputs). In both cases, the NTK analysis is calculated per output, yielding a total of nine NTK analyses (three units \(m\) per output set \(g\)). For comparison, we also carried out the NTK analysis without sorting. In contrast to the sorted analyses (shown in Fig 9), the unsorted analyses did not reveal any discernible structure (see Fig 12 in Appendix).

Fig 9 shows the results of the NTK analyses for the primary eigenvector (i.e., the one with the largest eigenvalue) in the standard initialization and high initialization condition at \(T=0\), sorted as described above. These reveal clustering effects in the standard initialization that correspond closely to those that emerged in the hidden units over training, as observed in the M-PHATE plots in Fig 8. Each plot shows the weighting for each training pattern on the eigenvector. When these are sorted by input features (left panels), the pattern of weightings for a given response (\(m\)) is the same across output pools (\(g_{2}\)) in the standard init case, consistent with the use of the same input representations across all three tasks. Complementing this, when training patterns are sorted by task (right panels), the pattern of weightings for a given response is different for each output pool in the standard init case, indicating the role of the task units in selecting which of the four input dimensions should be mapped to that output pool for each given task. Critically, this structure within the standard init is visible in the NTK analysis _prior to training_ (i.e., at \(T=0\)) in the standard initialization condition but _not_ the large initialization condition. Further details of other secondary analysis are provided in the Appendix.

In the standard-initialization condition, structure is visible in the NTK analysis before any training occurs. This raises the question: to what extent is this specifically predictive of the effects that emerge during
Figure 9: First eigenvector of the NTK analysis for the output units prior to training, showing the 1st eigenvector over each training input for each output unit, with each plot’s input indices (x-axis) ordered either by the relevant task (sorted by inputs; left panels) or by relevant input (sorted by tasks, right panels), across both the standard initialization (top panels) and large initialization (bottom panels). The 9 plots within each panel are organized by output pool \(g_{2}\) and response unit \(m\) within each pool. Notice the strong clustering in _both_ standard init cases (see text for interpretation), but neither of the large init cases.
training? To address this, we analyzed the extent to which the groupings from the initial NTK analysis predicted the structure observed in the M-PHATE hidden representations over the course of training (that is, the extent to which the NTK analysis conducted at \(T=0\), integrating only the first time step, predicted M-PHATE clustering for t\(>\)0). For the NTK analysis grouped by input, the NTK groups code for subgroup position within each task, as seen in Fig 9. This can easily be expressed by the M-PHATE grouping as well. For the NTK analysis grouped by task, the grouping clearly aligns with the output pool relevant for each task, as seen in Fig 9. However, as previously discussed, the M-PHATE analysis uses the hidden layer activations, and thus cannot include output dimension information. Instead, we predict that the M-PHATE analogously uses the relevant input pool. Based on these grouping schemes, we can test the extent to which the organization predicted from the NTK plots prior to training (at \(T=0\)) predict the patterns of clustering that emerge in the M-PHATE plot at different points during training (\(T\)\(>\)=1). We measured the correspondence between these measures using the Adjusted Rand distance metric for between group memberships, as shown in Fig 10. The results indicate that PNTK accurately predicts M-PHATE structure when examined both by task and inputs. The task-grouped predictions reach a complete match in grouping by the end of training. The input-grouped predictions also align cleanly with the trajectory of representational structure in the M-PHATE analysis, reaching a prefect match in grouping early in training when structure is clearly observed in the M-PHATE analyses, followed by a diminution of the effect that parallels the dissolution of structure observed in the M-PHATE analyses.

## 5 Predicting the Effect of Initialization and Training Curriculum on Processing

The results reported above affirm the usefulness of the NTK and PNTK analyses in predicting representational learning, both in linear and non-linear networks. They also reaffirm the premise that initialization with small random weights favors representational sharing among tasks that share common structure (e.g., input and/or output dimensions). In this section, we report a further evaluation this effect, and the ability of the NTK and PNTK analyses to predict not only the effects of initialization on representational learning, but also on performance. Specifically, building on previous work (Musslick et al., 2016; Musslick and Cohen, 2021), we tested the hypotheses that: i) insofar as the standard initialization condition favors representational sharing, it should be associated with faster learning of new single tasks (due to more effective generalization), but at the cost a compromised ability to acquire parallel processing capacity (i.e., poorer concurrent multitasking capability) relative to the large initialization condition; and ii) this can be predicted

Figure 10: Overlap between groups predicted from PNTK (at \(t=0\)) and groups observed from M-PHATE analysis (input and task) across time (mean Adjusted Rand metric \(\pm\) s.d. across trials; see text for details of analysis). The PNTK predictions for both groups are accurate, although the input-grouped M-PHATE only aligns during the earlier stages of training.
from the NTK analysis at the start of training.10 To test this, we evaluated the acquisition of multitasking performance via fine tuning of networks first trained on single task performance in each of the two initialization conditions. For each comparison, we generated two networks from the same random initialization, with the large initialization having layer 1 weights that were uniformly multiplied up by a factor of 10. We posited that although a neural network trained in the large initialization condition (and therefore biased to learn separated representations) would take longer to learn during initial single task training, it would be faster to acquire the capacity for concurrent multitasking during subsequent fine tuning on that ability, as compared to a network trained in the standard initialization condition (and hence biased to learn shared representation). Critically, we also tested the extent to which the NTK and PNTK analyses, carried out on the network _before_ initial single task training, could accurately predict end of training generalization loss and even the impact of fine tuning on concurrent multitasking performance that occurred _after_ the initial training.

Footnote 10: While it is plausible that this should be the case, given that NTK and PNTK predict patterns of representational learning and the latter determine performance, nevertheless since neither of these relationships is perfect, it is possible that NTK and/or PNTK predict a different component of the variance in representational learning than is responsible for performance.

### Effects of Initialization on Acquisition of Multitasking Capability

To test the hypotheses outlined above, we first trained networks on single task performance in the standard and large initialization conditions. We then followed this with "fine-tuning" of each resulting network on concurrent multitasking, using the same number of training examples and epochs in each case. Specifically, we trained each network to simultaneously execute, with equal probability, either 1, 2, or 3 tasks, randomly sampled from all valid combinations (e.g. non-overlapping inputs or outputs) of the selected number of tasks.

Fig 11 shows the results for networks in the standard and large initialization conditions, both during initial single task training (left panels) and during subsequent fine tuning on concurrent multitasking performance (right panels). The upper panels show a direct comparison of the mean and standard deviation of generalization losses. While the basic effects are observed here, overall performance varied across different pairs of networks, as a function of the particular _pattern_ of initial weights assigned to them (which was the same for each pair, and simply scaled differently for the standard and large initialization conditions). The lower panels of Fig 11 show the mean of direct comparisons between each pair of networks, that controls for differences in overall performance across the pairs. As predicted (and consistent with previous results (Flesch et al., 2021; Musslick et al., 2017; 2020), the standard initialization condition led to better generalization performance over the course of single task training, due to the development of shared representations (see Fig 8, upper panels). However, those presented an obstacle to the subsequent acquisition of concurrent multitasking performance, presumably because separated (task dedicated) representations had to now be learned _de novo_. Conversely, networks in the large initialization condition exhibited poorer overall generalization performance during single task training, due to a bias toward the learning of more separated representations (as predicted by the PNTK analyses; see Fig 8, lower panels), but it was better predisposed for the subsequent acquisition of concurrent multitasking performance, as is clearly observed in the right panel of Fig 11.

### Predicting performance from the PNTK analysis

Next, we tested the extent to which an PNTK analysis applied to the network prior to initial single task training could predict performance during subsequent multitask tuning. To do so, we: i) created ten pairs of networks, each with a different set of weights generated for the standard and large initialization conditions (as described above; ii) applied k-means clustering with 12 groups to the multidimensional PNTK analysis of each network prior to training; and iii) quantified the clustering quality using the silhouette method (Baarsch and Celebi, 2012), with a higher value reflecting a greater degree of sharing among representations. We then trained each network using the procedure describe above, initially on single tasks, and then on multitask fine tuning. Finally, for each pair of networks, we correlated the silhouette score for the network in each condition with the difference in generalization performance between the two conditions (standard - large initialization) at the end of each training phase. If representational structure predicted by the PNTK analyses was responsible for generalization performance after each phase of training, then the correlations should be negative following single-task and positive following multitask fine tuning. This is because the standard
Figure 11: Top: Generalization performance (mean loss \(\pm\) s.d. across trials) during single task training in the standard and large initialization conditions (left), followed by fine tuning in multitasking in each condition (right). Bottom: Difference in generalization performance (mean[loss of large initialization loss of standard initialization] \(\pm\) s.d. across trials) during single task training (left), followed by fine tuning on current multitasking (right). During single task training, while the large initialization shows faster initial improvements in performance, it is quickly overtaken by the standard initialization condition. However, during subsequent fine tuning on concurrent multitasking, the large initialization condition shows a clear and sustained advantage.
initialization should generate higher silhouette scores (more shared representations) and correspondingly lower test loss relative to the large initialization condition after single task training (hence the negative correlation), but higher relative test loss following multitask fine tuning (hence the positive correlation); and, conversely, the large initialization should generate lower silhouette scores (more separated representations) and correspondingly higher test loss relative to the standard initialization condition after single task training (and thus, again, a negative correlation), but lower relative test loss following multitask fine tuning (again a positive correlation). The results are consistent with these predictions: the correlation was \(r=-.881\) following single task training, and \(r=.973\) following multitask fine tuning. This suggests that the PNTK analysis, carried out _prior to any training_, was able to reliably predict theoretically anticipated effects of initialization on generalization performance in the network observed after two distinct phases of training. This not only confirms that the PNTK analysis, carried out _prior to any training_, is able to predict the kinds of representations learned by the network in response to different initializations, but also that it can be used to predict the patterns of generalization performance associated with those representations observed in the network after two distinct phases of training.

## 6 Discussion and Conclusions

In this article, we show how a novel analysis method that examines the gradients of the network at the outset of training (determined by its initial weights and training curriculum), can be used to predict features of the representations that are subsequently learned through training, as well as the impact these have on network performance. Specifically, we show that NTK and PNTK analyses predict the extent to which a standard initialization scheme (small random weights) biases learning toward a generalizable code that groups similar inputs together (i.e., using shared representations), whereas large weight initialization predisposes the network to learn distinct representations for each task configuration (i.e., separated representations). We also confirm that, whereas the bias toward shared representations leads to improved generalization performance in the single task setting, this leads to destructive interference that impairs multitasking performance when more than one task is performed concurrently. Conversely, the large initialization scheme favors the formation of distinct task-specific representations, that facilitates the acquisition of multitasking capability.

Importantly, we show that these effects (i.e., the eventual task or input groupings) can be predicted from the NTK eigenvectors as early as the first iteration of training, indicating that the types of representation that will be learned and their consequences on performance can be characterized before training has begun. Here, we focused on relatively simple networks, tasks, and training regimes. Evaluating the extent to which our results extend to more complex network architectures, tasks and forms of training remains an important direction for future research. We expect that this technique may be a fruitful way to advance the probing and understanding of inductive biases that influence the learning and use of representations in neural networks (Woodworth et al., 2020; Goyal & Bengio, 2022). This, in turn, may help lay the foundation for a better understanding of the respective costs and benefits of representation sharing in both biological and artificial neural network architectures.

The work presented here, and previous theoretical work on which it builds Feng et al. (2014); Musslick et al. (2020), suggests that sharing representations between tasks limits a network's capacity for multitasking. This has received empirical support in neuroscientific research. For example, neuroimaging studies have
provided evidence that the multitasking capability of human participants is inversely related to representational overlap between tasks (Nijboer et al., 2014), and that improvements in multitasking capability are accompanied by increases in representational separation Garner and Dux (2015). The present work may aid in the development of methods that allow neuroscientists to predict the learning of shared versus separate representations before they occur. Such methods would open new paths for the quantification of individual differences or the effective evaluation of training procedures for human multitasking. Along similar lines, the work presented here may help inform the design more effective artificial systems, by providing an efficient means of predicting the impact that initial conditions and training curriculum will have on downstream parallelization of performance.
