# Large Language Models Relearn Removed Concepts

Michelle Lo♡♠
&Shay B. Cohen‡
&Fazl Barez♡♣∗
♡ Apart Research
‡ School of Informatics, University of Edinburgh
♠ Department of Computing, Imperial College London
♣ Department of Engineering Sciences, University of Oxford  Equal contribution
Correspondence<EMAIL_ADDRESS>
###### Abstract

Advances in model editing through neuron pruning hold promise for removing
undesirable concepts from large language models. However, it remains unclear
whether models have the capacity to reacquire pruned concepts after editing.
To investigate this, we evaluate concept relearning in models by tracking
concept saliency and similarity in pruned neurons during retraining. Our
findings reveal that models can quickly regain performance post-pruning by
relocating advanced concepts to earlier layers and reallocating pruned
concepts to primed neurons with similar semantics. This demonstrates that
models exhibit polysemantic capacities and can blend old and new concepts in
individual neurons. While neuron pruning provides interpretability into model
concepts, our results highlight the challenges of permanent concept removal
for improved model safety. Monitoring concept reemergence and developing
techniques to mitigate relearning of unsafe concepts will be important
directions for more robust model editing. Overall, our work strongly
demonstrates the resilience and fluidity of concept representations in LLMs
post concept removal.

Large Language Models Relearn Removed Concepts

Michelle Lo♡♠††thanks:  Equal contribution Correspondence:
<EMAIL_ADDRESS>Shay B. Cohen‡ Fazl Barez♡♣∗

♡ Apart Research ‡ School of Informatics, University of Edinburgh ♠ Department
of Computing, Imperial College London ♣ Department of Engineering Sciences,
University of Oxford

##  1 Introduction

Large language models possess neurons that encode semantic concepts across
different languages Qian et al. (2016), architectures Wu et al. (2020), and
modalities Mu and Andreas (2020).

The primary objective when pruning such models is to eliminate redundant
neurons, while preserving the most crucial ones LeCun et al. (1989). The
underlying assumption is that removing important “concept neurons” will
disrupt the model’s structured internal representation of key concepts,
causing performance degradation. However, it is in fact possible for models to
regain high performance after pruning random or important neurons Liu et al.
(2022). Models demonstrate a remarkable capacity to adapt and regain
conceptual representations. This phenomenon, which we call “neuroplasticity”,
suggests a degree of adaptability in such models.

Neuroplasticity has significant implications for model editing. Model editing
promises to remove undesirable concepts, but neuroplasticity implies that
those concepts may in fact reappear if retraining takes place. Understanding
how concepts are represented, redistributed, and recaptured is thus crucial
for the development of safer, fairer and aligned models. Furthermore,
understanding how removed concepts are recovered in language models can
significantly enhance their robustness. While the inherent plasticity of
neural networks has been acknowledged Mittal et al. (2019), there is a limited
understanding of where relearned concepts are redistributed, or what
influences relearning in specific neurons. Such insights could benefit models’
ability to recover from partial damage and information loss.

This paper investigates how large language models fine-tuned for named entity
recognition relearn and redistribute concepts after the removal of important
features, to explain how a model regains performance after experiencing damage
to crucial neurons in its internal representation structure. We instigate
neuroplasticity by pruning important concept neurons and retraining a model so
that it regains its original performance. We then analyze how the distribution
of concepts changes, and examine the relationship between concepts previously
associated with a neuron and the concepts it relearns. Our findings are four-
fold:

  * •

Neuroplasticity happens quickly, allowing the model to regain performance
within a few epochs of retraining.

  * •

Pruned concepts, initially present in later layers, are remapped to neurons in
earlier layers.

  * •

Neurons which recover the pruned concept may have been primed for relearning,
as they originally captured concepts similar to that which was pruned.

  * •

Neurons exhibit polysemantic properties as they relearn a blend of new and old
concepts.

These findings contribute to a deeper understanding of how language models
learn, adapt, and retain core conceptual representations. They underscore the
potential of earlier layers to recapture fundamental representations, which
has implications for model editing. Furthermore, our exploration of how
neuroplasticity increases polysemanticity, where a neuron can represent
multiple concepts and meanings may inform strategies to enhance the transfer
of learned representations, and facilitate interpretability analysis regarding
polysemanticity.

##  2 Related work

In terms of analysing the distribution of concept representations, it is known
that more complex concepts tend to be represented by neurons in higher layers
of language models. Durrani et al. (2020) found that neurons which capture
word morphology were primarily located in the lower and middle layers, and
those learning about advanced concepts such as syntax were found at the higher
layers. However, dynamic changes in concept distribution over training are
less well studied. Prior works artificially redistributed concepts in a large
language model by modifying the activations of specific neurons which capture
desired properties, to control model outputs Bau et al. (2019). However, these
works focused on deliberately editing representations in a trained model.
There is little work in which models redistribute concepts unsupervised,
leaving a gap in understanding about how concept redistribution naturally
occurs after pruning.

Figure 1: Process of investigating neuroplasticity in a large language model.
We identify concept neurons (dark blue) in the base model, and prune them
(white). We then retrain the model until it regains its original performance
and identify new concept neurons.

In terms of performance recovery after pruning, Liu et al. (2022) demonstrated
that randomly pruned sparse networks can be trained from scratch to match the
accuracy of equivalent dense networks, particularly in larger networks.
Similarly, Mittal et al. (2019) suggested that the inherent plasticity of deep
convolutional neural networks allows them to recover from the loss of randomly
pruned filters. Their work supports the idea that large neural networks
demonstrate plasticity, and can regain performance if important neurons might
be pruned. However, their approach uses indiscriminate random pruning of
neurons in image models. In contrast, we explicitly prune the most important
neurons, which allows us to analyze causation between removing crucial
features and performance. In order to clearly analyze changes in the model
before and after pruning, our approach also trains, prunes and retrains the
model in separate phases - as opposed to other approaches, which iteratively
prune and adjust weights Castellano et al. (1997).

Our paper is most similar to Dalvi et al. (2020), in which neurons that are
redundant with regards to a specific task are pruned from LLMs, and model
performance is re-examined. However, Dalvi et al. (2020) focus on identifying
redundant neurons and do not damage key concept neurons. On the other hand, we
focus on how the internal distribution of concept representations and model
performance changes after pruning important neurons.

##  3 Problem setting

Consider a large language model $M$. Let $N$ be the set of all neurons in $M$.
Consider a concept $C$ represented within $M$.

A concept neuron is a neuron in $N$ which activates strongly on tokens
associated with $C$. Let $S\subset N$ be the set of concept neurons in $M$
prior to ablation. Let the set of remapped neurons $R\subset N$ be the set of
concept neurons associated with $C$ in $M$, after pruning neurons in $S$ and
retraining.

We analyze the redistribution of concepts in $M$ after neuroplasticity occurs,
in terms of concept saliency and concept similarity.

###  3.1 Concept saliency

Concept saliency is a measure for how strongly a neuron encodes a
representation of a specific concept, compared to other neurons in the same
model. A high saliency score (close to 1) means that the neuron captures the
concept more strongly, relative to neurons in the rest of the model.

Concept saliency helps us address the question: where are recovered concepts
distributed in $M$ when neuroplasticity occurs? Concept saliency can be
calculated for individual neurons, enabling fine-grained analysis, but the
mean concept saliency for all neurons in a layer can also be used for layer-
wise analysis. Analysis of concept saliency facilitates investigation into how
a pruned concept is distributed before and after neuroplasticity.

To compute concept saliency scores, we obtain a global ordering of neurons, in
which neurons are ranked according to the extent to which they activate on a
given concept (see §4.1). The ordered list of neurons is divided into groups
of 100 consecutive neurons. The neurons in each group is assigned the same
(normalised) saliency score, according to the group’s position in the ordered
list. For instance, a saliency score of 1.0 indicates that a neuron is one of
the top 100 most salient neurons.

###  3.2 Concept similarity

Concept similarity measures how similar the concept currently captured by a
neuron is to the concept that it originally captured in the base model.
Formally, if $C_{r}$ is the set of concepts represented by a remapped neuron
$r\in R$ in $N$ after pruning and retraining, then $C_{r}^{\prime}$ is concept
associated with $r$ in $N$ before pruning and retraining. For example, a
neuron might initially be associated with names of animals ($C_{r}^{\prime}$),
but after pruning and retraining, it might be associated with names of places
($C_{r}$).

Similarity analysis helps us address the question: how do the concepts
represented by specific neurons change during neuroplasticity? It offers a
quantitative measure to compare the concepts originally distributed in the
base model, and the redistribution of concepts in the retrained model. This
allows us to further interpret the model’s internal representations and
investigate potential relationships between neurons’ previously acquired
concepts and the recovered concept.

To compute concept similarity:

  1. 1.

Take $H$, the set of highest activating tokens for $R$ in $N$ before we induce
neuroplasticity by pruning and retraining.

  2. 2.

Take $H^{\prime}$, the set of highest activating tokens for $R$ in $N$ after
we induce neuroplasticity.

  3. 3.

Convert $H$ and $H^{\prime}$ into sets of word embedding vectors $W$ and
$W^{\prime}$ respectively.

  4. 4.

Compute the cosine similarity between the average vectors of $W$ and
$W^{\prime}$.

A similarity score close to 1 suggests that concepts are more semantically
similar, and a score close to -1 indicates opposing concepts. If the
similarity score is above a predetermined threshold (defined as 0.5 in this
paper), we infer that there is a meaningful relationship between the newly
learned concept $C_{r}$ and the concept that was originally captured
$C_{r}^{\prime}$ by the same neuron.

##  4 Method

To explore the notion of neuroplasticity within a pretrained model $M$, we
first fine-tune the model for a task that calls for the understanding of a
specific concept $C$. Subsequently, we employ a probeless neuron search method
to pinpoint the set of concept neurons $S$ within the model that correlate
with $C$ (§4.1). In order to remove the concept from the model’s internal
representations, we prune the neurons in $S$ (§4.2). Following the pruning
process, the model is retrained until it attains its original performance
level. The recovery of performance signifies that neuroplasticity has taken
effect, and that the eradicated concept has been relearned and integrated
throughout the model. We can use the same probeless search method to determine
the set of remapped neurons $R$ in the retrained model that have acquired the
pruned concept. During the retraining period, we can examine how concept
saliency and similarity evolve across neurons in both $S$ and $R$ at two-epoch
intervals. The overall process of our methodology is shown in Figure 1. For
detailed steps, refer to the pseudo-code algorithm 1 in Appendix D.

###  4.1 Identifying top concept neurons

We use a probeless corpus-based neuron search method Antverg and Belinkov
(2022) to build a list of top neurons with regards to a concept label. This
list is used to compute concept saliency (§3.1). We used the probeless search
method implementation provided by the NeuroX toolkit Dalvi et al. (2019).

Given labeled training data in which tokens are annotated with associated
concepts, we use the NeuroX toolkit to extract neuron activations for every
input token. Features include all unit activations on every token, for every
neuron in the model. The probeless method ranks neurons according to the
difference in their values across labeled tokens.

Formally, for every concept label $z\in Z$ where $Z$ is the set of all
possible concepts label, we calculate $q(z)$, the mean vector of all
representations of words that possess the concept label $z$. Next, we
calculate the element-wise difference between the mean vectors for a specific
concept $z$ ($d$ is the number of neuron activations):

$\displaystyle r(z)=\sum_{z^{\prime}\in Z,z\neq
z^{\prime}}|q(z)-q(z^{\prime})|,\,\,r(z)\in\mathbb{R}^{d}\text{.}$ (1)

Following this, we obtain a ranking by arg-sorting the elements of $r(z)$.
Then the first neuron in the ranking corresponds to the highest value in
$r(z)$. A neuron’s ranking is determined by the absolute variation between the
mean activation of that neuron for a specific concept and its mean activation
over all other concepts. A highly ranked neuron triggers more robustly on
words related to a single concept, as compared to its average activation over
all words. By following this calculation, we identify the neurons that behave
the most differently from other neurons for concepts different than $z$.

Since the probeless method ranks neurons based on average neuron activations,
this approach is capable of handling noisy activations in non-sparse
representations. In instances where many units are activated for a specific
concept, those with significantly “large” contributions will be identified as
highly salient. Furthermore, the probeless method was chosen because it is
based purely on representations, making it free of probing limitations that
might affect ranking quality. The probeless method is also significantly
faster than alternative methods such as probing classifiers, as we are only
limited by averaging and sorting speeds.

###  4.2 Pruning

To prune the concept neurons from the model, the weights of the selected
neurons are set to zero. Formally, let $W$ be the model’s weight matrix, and
let $\Phi$ be a binary mask matrix indicating the selected neurons for pruning
(with the same dimensions as $W$). Then, the pruned weight matrix
$(W^{\prime})$ is computed as: $W^{\prime}=W\odot\Phi$ where $\odot$ denotes
element-wise multiplication.

##  5 Experimental setup

We focus on pruning the specific concept of location names from a DistilBERT
model, a DistilGPT2 model Sanh et al. (2019) and a GPT2 model Radford et al.
(2019) fine-tuned for named entity recognition. Analysis of the models across
different runs are in Appendix B, and analysis of the models after pruning
other concepts are in Appendix C.

###  5.1 Model architecture

Base models:  We use the transformers library and the CoNLL-2003 dataset Tjong
Kim Sang (2002) to fine-tune open-sourced, pretrained DistilBERT, DistilGPT2
and GPT-2 models for named entity recognition (NER). DistilBERT is a lighter
version of the encoder-only BERT model with fewer layers and parameters (66
million parameters), maintaining competitive performance while requiring less
compute resources Sanh et al. (2019). GPT-2 is a large causal, decoder-only
transformer with 1.5 billion parameters Radford et al. (2019). DistilGPT2 is a
faster, lighter version of GPT-2 with 82 million parameters Sanh et al.
(2019). Since DistilBERT, DistilGPT2 and GPT-2 represent LLMs with different
architectures and training objectives, analysis on these models are more
generalizable to other LLMs.

Training: Models are fine-tuned for three epochs using AdamW optimizer with a
linear scheduler and warmup Loshchilov and Hutter (2019). The hyperparameters
used are: learning rate of 2e-5, weight decay of 0.01, batch size of 8.
Evaluations: Entity recognition performance is evaluated using precision,
recall, and F1 computed with the seqeval library Nakayama (2018). We perform
cross-validation, where we examine the highest activating tokens for each fold
to validate the robustness of the recognized entities.

###  5.2 Identifying concept neurons

To detect top concept neurons, we use concept-annotated tokens from the
dataset BERT ConceptNet Dalvi et al. (2022). BERT ConceptNet is a dataset of
latent concepts learned within the representations of BERT. The dataset
consists of sets of sentences, for which each token in a sentence is annotated
with its associated concept. Tokens are classified hierarchically based on
their semantic properties, e.g. tokens associated with location have the label
SEM:named_entity:location. As we focus on identifying salient neurons for a
particular concept, it is feasible to view this as a binary classification
problem over 1) tokens annotated with the chosen concept, and 2) tokens that
are not annotated as related to the selected concept.

###  5.3 Pruning and retraining

We use the probeless method (§4.1) to extract a list of all neurons ordered by
concept saliency. We verify our measure of concept saliency corresponds to the
extent to which a concept is captured in Appendix B.3. We prune the the most
salient half of the model’s neurons (i.e. neurons in the top half of the
ordered list of most salient neurons).

After pruning, we retrain the model using details in §5.1. After every three
epochs, we save the model’s state so that we can analyze the pattern of
concept redistribution over the retraining process incrementally. We compare
the performance of the retrained model against the base model in terms of
precision, F1 and recall.

###  5.4 Extracting highest activating tokens

To extract the highest activating tokens (HATs) for a neuron, we use the
NeuroX library Dalvi et al. (2019) and compare the activations of the neuron
across all tokens in a given dataset of sentences. The top five tokens that
account for the largest variance in activations form the set of HATs for that
neuron. Each HAT has a corresponding activation score, which is a normalised
score depicting its contribution to the overall variance. We use Word2Vec
Mikolov et al. (2013) to convert HATs to word embeddings, to compute concept
similarity (§3.2).

##  6 Results

In our results, performance metrics indicate that neuroplasticity occurs early
in the retraining process (§6.2). Specifically, neuroplasticity occurs when
pruned concepts are relocated to earlier layers in a model (§6.3). This may be
because neurons which reacquire the pruned concept are primed for relearning,
having captured similar concepts initially (§6.4). Concept similarity scores
show that the pruned concept is relearned in addition to original concepts, in
a polysemantic manner (§6.5).

###  6.1 Random pruning baseline

To verify that the neuroplasticity effects are specifically linked to pruning
critical concept neurons, we establish a random pruning baseline where the
same number of neurons are removed randomly instead of targeting concept
neurons (Table 1). After random pruning, the model experiences a more drastic
performative drop on NER compared to concept pruning (F1 decreases from 0.443
to 0.219). Recovery during retraining is also slower, reaching only 0.330 F1
after two epochs. This aligns with expectations, as random neurons likely do
not capture coherent concepts critical for NER.

However, with continued retraining, model performance matches and slightly
exceeds the original base model, implying that representation remapping does
occur but in a less directed manner. Analyzing the random pruning baseline
helps confirm that the rapid, adaptive plasticity behaviors are unique
outcomes of pruning semantic concept neurons rather than pruning arbitrary
neurons. Concept redistribution trends also differ between random pruning
versus concept pruning. As seen in Table 4, after random pruning, while later
layers gain more salient neurons, earlier and middle layers do not gain many
more salient neurons compared to concept pruning. Hence, the key difference is
that the earlier/middle layers do not capture the pruned concept as strongly
with random pruning compared to concept pruning.

###  6.2 Rapid performance recovery after retraining

Stage | Precision | Recall | F1
---|---|---|---
Mean | SD | Mean | SD | Mean | SD
Base model | 0.398 | 0.007 | 0.499 | 0.008 | 0.443 | 0.001
Pruned | 0.210 | 0.007 | 0.229 | 0.023 | 0.219 | 0.014
Retrained (2 epochs) | 0.297 | 0.010 | 0.370 | 0.035 | 0.330 | 0.020
Retrained (4 epochs) | 0.363 | 0.008 | 0.452 | 0.017 | 0.402 | 0.002
Retrained (6 epochs) | 0.393 | 0.013 | 0.490 | 0.006 | 0.436 | 0.011
Retrained (8 epochs) | 0.403 | 0.009 | 0.517 | 0.004 | 0.453 | 0.007

Table 1: Means and standard deviations (SD) to 3 d.p. of precision, recall and
F1 score on overall NER, after randomly pruning neurons from DistilGPT2 over 3
runs.

Stage | Precision | Recall | F1
---|---|---|---
Mean | SD | Mean | SD | Mean | SD
Base model | 0.932 | 0.002 | 0.940 | 0.001 | 0.937 | 0.001
Pruned | 0.019 | 0.003 | 0.147 | 0.015 | 0.034 | 0.005
Retrained (2 epochs) | 0.909 | 0.013 | 0.922 | 0.011 | 0.915 | 0.011
Retrained (4 epochs) | 0.918 | 0.008 | 0.930 | 0.008 | 0.924 | 0.008
Retrained (6 epochs) | 0.919 | 0.007 | 0.932 | 0.003 | 0.926 | 0.006
Retrained (8 epochs) | 0.922 | 0.008 | 0.934 | 0.006 | 0.928 | 0.007

Table 2: Means and standard deviations (SDs) to 3 d.p. of precision, recall
and F1 score on overall NER throughout remapping, after pruning concept
neurons from DistilBERT over 3 runs. The dramatic drop in performance after
pruning concept neurons confirms we removed important features.

Stage | Precision | Recall | F1
---|---|---|---
Mean | SD | Mean | SD | Mean | SD
Base model | 0.394 | 0.005 | 0.483 | 0.002 | 0.434 | 0.003
Pruned | 0.363 | 0.004 | 0.410 | 0.012 | 0.385 | 0.008
Retrained (2 epochs) | 0.415 | 0.003 | 0.500 | 0.009 | 0.454 | 0.005
Retrained (4 epochs) | 0.428 | 0.009 | 0.527 | 0.010 | 0.472 | 0.010
Retrained (6 epochs) | 0.443 | 0.002 | 0.569 | 0.012 | 0.495 | 0.000
Retrained (8 epochs) | 0.458 | 0.004 | 0.568 | 0.005 | 0.507 | 0.005

Table 3: Means and standard deviations (SD) to 3 d.p. of precision, recall and
F1 score on overall NER over remapping, after pruning concept neurons from
DistilGPT2 over 3 runs. The drop in performance after pruning concept neurons
confirms we have removed important features.

Upon removing the top concept neurons, there is a substantial decline in a
model’s performance on sequence evaluation tasks. For DistilBERT, there is a
considerable decrease in precision (by 98%), recall (by 84%), and F1 score (by
96%) as shown in Table 2. Notably, the performance drop for named entity
recognition (NER) regarding location names is even more pronounced - with a
95.9% decrease in precision, 93% decrease in recall, and 94.9% decrease in F1
score. This is expected. Such drastic declines verify that we have removed the
most important neurons used by the model for NER. For DistilGPT2, there is
also a considerable decrease across all scores (Table 3). Absolute performance
scores for DistilGPT2 are worse than DistilBERT - this is also expected, as
DistilGPT2 is a decoder-only model that is less suited for sequence evaluation
tasks such as named entity recognition.

When highly significant neurons are removed, the model’s capacity to grasp
critical concepts recovers after retraining. The pruned model exhibits a
marked enhancement in performance within only two epochs. For example, the
difference between the DistilBERT model’s mean F1 score before pruning and
after retraining for two epochs is merely 0.022. Over eight epochs of
retraining, the model progressively improves in precision, recall, and F1
score, until performance matches or exceeds the baseline.

The sudden jump after two epochs of retraining and the subsequent plateau in
performance indicates that neuroplasticity happens quickly in the early stages
of retraining. In the next section, we demonstrate that this can be attributed
to the redistribution of the pruned concepts to remaining neurons in earlier
layers of the model.

###  6.3 High level concept redistribution

Concepts which were originally located in later layers are redistributed over
both later and middle layers of the model during neuroplasticity. This is
evident from 1) changes in the proportion of salient neurons per layer, and 2)
changes in mean saliency per layer across stages of retraining.

First, consider the proportion of salient neurons per layer. Table 5 and Table
7 show the distribution of salient neurons across layers before and after
inducing neuroplasticity, in a DistilBERT model and a DistilGPT2 model
respectively. Specifically, these tables show the percentage of neurons in a
given layer which have a concept saliency greater than 0.5, normalized over
the number of non-pruned neurons in the given layer.

Table 5, Table 6 and Table 7 indicate that later layers initially have the
highest concentration of representations for a pruned concept, but concept
saliency becomes concentrated in either middle or later layers of the model
after neuroplasticity. Before pruning concept neurons and retraining, the last
layer of both models has the highest percentage of salient neurons. After
ablating concept neurons and retraining the models, the pruned concept is
redistributed such that layers located earlier in DistilBERT and DistilGPT2
model (layer 4 and layer 5 respectively) shows higher concentrations of
salient neurons. For GPT-2, however, concept saliency redistributes more
evenly across the model after pruning and retraining. This could be due to
differences in how layers in larger models respond to inputs and tokens.

| Percentage of salient neurons in layer
---|---
Layer | Base model | Pruned and retrained
0 | 0.111 | 3.308
1 | 15.481 | 5.923
2 | 17.778 | 14.077
3 | 19.259 | 15.615
4 | 19.333 | 17.462
5 | 21.000 | 15.308
6 | 7.037 | 28.308

Table 4: Normalized percentage of neurons which demonstrate concept saliency,
before and after baseline DistilGPT2 model is randomly pruned and retrained.

| Percentage of salient neurons in layer
---|---
Layer | Base model | Pruned and retrained
0 | 12.28 | 8.98
1 | 11.36 | 13.47
2 | 10.35 | 16.25
3 | 11.41 | 16.25
4 | 13.51 | 20.11
5 | 19.01 | 15.98
6 | 22.08 | 8.98

Table 5: Normalized percentage of neurons which demonstrate concept saliency,
before and after the DistilBERT model is pruned and retrained. Layer 6 is the
most concept salient prior to ablation, but layer 4 is the most concept
salient after ablation and retraining.

| Percentage of salient neurons in layer
---|---
Layer | Base model | Pruned and retrained
0 | 1.000 | 0.037
1 | 15.846 | 14.963
2 | 14.923 | 17.704
3 | 13.692 | 19.074
4 | 14.154 | 19.741
5 | 14.538 | 20.704
6 | 25.846 | 7.778

Table 6: Normalized percentage of neurons which demonstrate concept saliency,
before and after the DistilGPT2 model is pruned and retrained. Concept
saliency increases in layers 4 and 5 after retraining, and decreases
dramatically for layer 6.

| Percentage of salient neurons in layer
---|---
Layer | Base model | Pruned and retrained
0 | 0.000 | 0.217
1 | 5.587 | 6.000
2 | 6.696 | 8.043
3 | 7.587 | 9.739
4 | 8.304 | 8.826
5 | 8.935 | 8.391
6 | 9.152 | 8.696
7 | 9.696 | 8.304
8 | 10.326 | 7.478
9 | 10.652 | 7.522
10 | 10.891 | 7.870
11 | 11.391 | 7.696
12 | 0.783 | 11.217

Table 7: Normalized percentage of neurons which demonstrate concept saliency,
before and after a GPT2 model is pruned and retrained. Most salient neurons
initially appear in later layers, but concept saliency redistributes
approximately evenly.

Next, consider the mean saliency scores per layer after ablation and
throughout retraining. Figure 3 and Figure 4 show that mean concept saliency
score increases after pruning and retraining for middle layers, more than for
later layers. In DistilBERT, the increase in mean concept saliency for layer 4
is greater than the increase in mean concept saliency for layers 5 and 6,
which originally had the most salient neurons (Figure 3). In DistilGPT2, the
increase in mean concept saliency for layer 5 is greater than the increase in
mean concept saliency for layer 6. Other layers do not change significantly in
saliency (Figure 4). The earliest layers in DistilGPT2 even experience a
decrease in mean concept saliency, suggesting that concept saliency becomes
especially concentrated in later and middle layers. Results for DistilGPT2 are
reflected in results for GPT-2 (Figure 5).

Figure 2: Mean concept saliency for the concept of location names, for neurons
across different layers of a baseline DistilGPT2 model throughout the process
of neuroplasticity, after pruning random neurons. Figure 3: Mean concept
saliency for the concept of location names, for neurons across different
layers of DistilBERT throughout the process of neuroplasticity. The most
salient neurons are in layers 5 and 6, but earlier layers demonstrate higher
saliency scores relative to other layers than before. Figure 4: Mean concept
saliency for the concept of location names, for neurons across different
layers of DistilGPT2 throughout the process of neuroplasticity. Saliency
increases for middle and later layers (layers 5 and 6). Figure 5: Mean concept
saliency for the concept of location names, for neurons across different
layers of GPT2 throughout neuroplasticity. Results are similar to those for
DistilGPT2.

For more concrete examples based on highest activation tokens of how concept
saliency is redistributed to earlier layers, refer to Appendix B.2.

###  6.4 Pruned concept relocation

One explanation for why pruned concepts are redistributed to middle layers is
that these neurons are “primed” for relearning, since they originally captured
similar concepts to the pruned concept.

In DistilGPT2 models after pruning and retraining, most concept similarity
scores are greater than 0.5 (Figure 7). This shows similar concepts are
represented in neurons before and after pruning and retraining. In DistilBERT
models, Figure 6 suggests that neurons which relearned the pruned concept
(i.e. have high concept saliency scores) tend to have also previously captured
similar concepts. The correlation between saliency and similarity is
replicated when pruning different concepts in DistilBERT models (Appendix
C.1). In other words, the pruned concept is relearned by neurons which
previously captured representations of similar concepts. See Appendix C.3 for
further analysis.

Figure 6: In a DistilBERT model which was retrained for 8 epochs after the
concept of location names was pruned, neurons which have strongly relearned
the pruned concept (high saliency) tend to have previously represented a
similar concept (high similarity) before pruning.  Figure 7: In a DistilGPT2
model which was retrained for 8 epochs after the concept of location names was
pruned, most neurons capture concepts after pruning and retraining that are
similar to previously represented concepts - regardless of saliency.

###  6.5 Polysemantic characteristics

Pruned concepts are redistributed to neurons that previously captured similar
concepts, such that neurons relearn both aspects of the concept that they
initially captured as well as the pruned concept. In other words, neurons in
large language models become polysemantic after retraining and capture several
concepts at once Xin et al. (2019).

Throughout retraining, neurons adapt to capture concepts that are increasingly
similar to the concepts that they originally captured prior to ablation. This
appears to be especially true for encoder-only models such as DistilBERT.
Layers which adapt to relearn the pruned concept in DistilBERT (layer 4)
experience a greater increase in mean concept similarity scores, compared to
other layers in the model (Figure 8). Earlier layers learn concepts that are
much more similar to concepts that are captured in the base model, and remain
stable in concept similarity scores throughout retraining. In contrast, the
mean similarity for later layer neurons in DistilBERT fluctuates more and
rises more as retraining progresses. In DistilGPT2, concept similarity stays
stable with minor fluctuations over retraining (Figure 9). Results for
DistilGPT2 are reflected in results for GPT-2 (Figure 10). This indicates
that, in addition to learning to represent the pruned concept, later layers
relearn to represent aspects of originally captured concepts.

Figure 8: Mean concept similarity for the concept of location names, for
different layers of a DistilBERT model during neuroplasticity. Error bars
represent 95% confidence intervals. Concept similarity increases slightly in
later layers (4, 5, 6) as retraining proceeds. Neurons in these layers are
relearning old concepts during neuroplasticity. Figure 9: Mean concept
similarity for the concept of location names, for different layers of a
DistilGPT2 model during neuroplasticity. Error bars represent 95% confidence
intervals. Concept similarity stays stable with minor fluctuations. Figure 10:
Mean concept similarity for the concept of location names, for different
layers of a GPT2 model during neuroplasticity. Results are similar to those
for DistilGPT2, indicating generalization. Figure 11: Mean concept similarity
for the concept of location names, for different layers of a baseline
(randomly pruned) DistilGPT2 model during neuroplasticity.

Stage |  HATs and activations
---|---
Base model |  ("1970", 1.0), ("Bronze", 0.960), ("Administrative", 0.936), ("Thoroughbred", 0.913), ("Egyptian", 0.899)
Retrained (2 epochs) |  ("Palestinian", 1.0), ("Maltese", 0.998), ("Egyptian", 0.983), ("Venezuelan", 0.964), ("Damascus", 0.959)
Retrained (4 epochs) |  ("Maltese", 1.0), ("Palestinian", 0.956), ("Nuremberg", 0.940), ("Egyptian", 0.934), ("Nordic", 0.927)
Retrained (6 epochs) |  ("Maltese", 1.0), ("Nuremberg", 0.978), ("Malaysian", 0.955), ("Nordic", 0.948), ("Palestinian", 0.944)
Retrained (8 epochs) |  ("Maltese", 1.0), ("Malaysian", 0.944), ("Bronze", 0.934), ("Egyptian", 0.929), ("Nordic", 0.929)

Table 8: Highest activating tokens for neuron 4148 in DistilBERT throughout
stages of neuroplasticity, for the concept of location names. The neuron
learns to activate on both new tokens related to location, and old tokens (in
bold) related to its original concept captured in the base model.

Analysis of the HATs for individual neurons provides concrete examples of how
both new and old concepts are captured after neuroplasticity occurs. In one
such example (Table 8), neuron 4148 in layer 5 of a DistilBERT model learns to
activate on tokens (e.g. "Damascus") related to the pruned concept of location
names after retraining, when it previously did not do so. Over 8 epochs of
retraining, the neuron not only relearns new concepts related to location
("Maltese", "Malaysian") but also relearns concepts that it activated highly
on in the base model ("Bronze", "Egyptian"). The polysemantic neuron after
retraining captures aspects of both new and old concepts, although the newly-
learned pruned concept is represented more strongly (as seen by higher
activation scores). These results are replicated for different neurons in
Appendix C.2.

##  7 Conclusion

We explored neuroplasticity in large language models (DistilBERT, DistilGPT2
and GPT2) and implications on the distribution of concept representations. Our
analysis showed that these models redistribute concepts after the removal of
important features, allowing them to regain performance within a few epochs of
retraining. The recovery in performance can be attributed to how pruned
concepts, initially present in later layers, are remapped to remaining neurons
in earlier layers which capture similar concepts. We further identified that
neurons exhibit polysemantic properties as they relearn a blend of new and old
concepts.

These insights contribute to a deeper understanding of how language models
learn, adapt, and retain core conceptual representations. They also suggest
interesting research directions in model editing and transfer learning. For
instance, models may need to be edited throughout retraining to ensure that
pruned concepts are not recaptured by earlier layers during fine-tuning.
Studying the implications of neuroplasticity-induced polysemanticity can also
aid the development of interpretable models and the enhanced transfer of
learned representations.

## Limitations

In this paper, we only evaluate neuroplasticity in DistilBERT, DistilGPT2 and
GPT-2 models. We expect that our overall findings regarding high level concept
redistribution are generalizable, according to the similarity of results
between DistilGPT2 and GPT-2, and because different architectures are know to
have rather similar representations Wu et al. (2020). However, it is possible
that language models with different sizes and architectures may recover
performance and relearn concepts differently. We leave this exploration for
future work.

More specifically, in §6.4, we show that pruned concepts primarily relocate to
neurons that previously captured similar concepts. However, we must emphasise
that the precise relationship between the two factors is not clear. Consider
figures 6 and 14. In both cases, most data points are clustered in the region
above the threshold of 0.5 for both saliency and similarity, and there is a
large range in saliency and similarity within this region. The standard
deviations for both concept saliency and concept similarity are relatively
large (0.205 and 0.199, respectively). Hence, without further investigation,
we can only state that pruned concepts tend to relocate to neurons that
previously captured similar concepts, but we must be cautious of drawing
definite links.

Finally, we compute concept saliency using a probeless method to extract a
global ranking of neurons, ordered by the extent to which a neuron activates
on a given concept. To verify the correctness of this method, we cross-checked
our overall saliency scores with individual neurons - but our method is not
scalable if one wishes to carefully analyze many neurons at once. We are also
aware of the wider implications of this work and have added a broader impact
statement in Appendix E.

## Acknowledgements

We’d like to thank Mrinank Sharma and Clement Neo for the generous comments
and feedback on the earlier draft and Esben Kran of Apart Research for
organising the Interpretability hackathon. This project was supported by Apart
Lab.

## References

  * Antverg and Belinkov (2022) Omer Antverg and Yonatan Belinkov. 2022.  On the pitfalls of analyzing individual neurons in language models.  In _International Conference on Learning Representations_.
  * Bau et al. (2019) Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2019.  Identifying and controlling important neurons in neural machine translation.  In _International Conference on Learning Representations_.
  * Castellano et al. (1997) G. Castellano, A.M. Fanelli, and M. Pelillo. 1997.  An iterative pruning algorithm for feedforward neural networks.  _IEEE Transactions on Neural Networks_ , 8(3):519–531.
  * Dalvi et al. (2022) Fahim Dalvi, Abdul Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. 2022\.  Discovering latent concepts learned in BERT.  In _International Conference on Learning Representations_.
  * Dalvi et al. (2019) Fahim Dalvi, Avery Nortonsmith, Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, and James Glass. 2019.  Neurox: A toolkit for analyzing individual neurons in neural networks.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):9851–9852.
  * Dalvi et al. (2020) Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. 2020.  Analyzing redundancy in pretrained transformer models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4908–4926, Online. Association for Computational Linguistics.
  * Durrani et al. (2020) Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. 2020.  Analyzing individual neurons in pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4865–4880, Online. Association for Computational Linguistics.
  * LeCun et al. (1989) Yann LeCun, John Denker, and Sara Solla. 1989.  Optimal brain damage.  In _Advances in Neural Information Processing Systems_ , volume 2. Morgan-Kaufmann.
  * Liu et al. (2022) Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. 2022.  The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_.
  * Mikolov et al. (2013) Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013.  Distributed representations of words and phrases and their compositionality.  In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States_ , pages 3111–3119.
  * Mittal et al. (2019) Deepak Mittal, Shweta Bhardwaj, Mitesh M. Khapra, and Balaraman Ravindran. 2019\.  Studying the plasticity in deep convolutional neural networks using random pruning.  _Machine Vision and Applications_ , 30:203 – 216.
  * Mu and Andreas (2020) Jesse Mu and Jacob Andreas. 2020.  Compositional explanations of neurons.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 17153–17163. Curran Associates, Inc.
  * Nakayama (2018) Hiroki Nakayama. 2018.  seqeval: A python framework for sequence labeling evaluation.  Software available from https://github.com/chakki-works/seqeval.
  * Qian et al. (2016) Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016.  Investigating language universal and specific properties in word embeddings.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1478–1488, Berlin, Germany. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _ArXiv_ , abs/1910.01108.
  * Tjong Kim Sang (2002) Erik F. Tjong Kim Sang. 2002.  Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition.  In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
  * Wu et al. (2020) John Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2020.  Similarity analysis of contextual word representation models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4638–4655, Online. Association for Computational Linguistics.
  * Xin et al. (2019) Ji Xin, Jimmy Lin, and Yaoliang Yu. 2019.  What part of the neural network does this? understanding LSTMs by measuring and dissecting neurons.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5823–5830, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Experimental details

###  A.1 Datasets

We used standard splits for training, development and validation data for the
ConLL-2003 dataset Tjong Kim Sang (2002). This is a named entity recognition
dataset, which concentrates on four types of named entities: persons,
locations, organizations and names of miscellaneous entities that do not
belong to the previous three groups. The annotations are publicly available,
but the corpus used in the English dataset are under the copyright of Reuters
Ltd. The dataset covers two languages: English and German. Details of the
number of examples and label distributions are shown in tables 9 and 10. The
dataset is available from https://huggingface.co/datasets/conll2003.

English | Sentences | Tokens | LOC | MISC | ORG | PER
---|---|---|---|---|---|---
Train | 14,987 | 203,621 | 7140 | 3438 | 6321 | 6600
Validation | 3,466 | 51,362 | 1837 | 922 | 1341 | 1842
Test | 3,684 | 46,435 | 1668 | 702 | 1661 | 1617

Table 9: Number of examples and label distributions for English data in
CoNLL-2003 dataset.

German | Sentences | Tokens | LOC | MISC | ORG | PER
---|---|---|---|---|---|---
Train | 12,705 | 206,931 | 4363 | 2288 | 2427 | 2773
Validation | 3,068 | 51,444 | 1181 | 1010 | 1241 | 1401
Test | 3,160 | 51,943 | 1035 | 670 | 773 | 1195

Table 10: Number of examples and label distributions for German data in
CoNLL-2003 dataset.

We extracted concept annotations for tokens from BERT Concept Net Dalvi et al.
(2022). The publicly available dataset consists of 635,000 English sentences
and 1.7 million tokens which are annotated with 219 unique concept labels. We
used annotations for the concept SEM:named_entity:location and
SEM:named_entity:person using the given script, and take only the first 5000
lines of annotated sentences due to compute restrictions. The full dataset and
scripts are available from https://neurox.qcri.org/projects/bert-concept-
net/bert-concept-net.html

###  A.2 Infrastructure and runtime

Our experiments were run on an Nvidia Tesla A30 GPU card. On average, it took
1 hour to fine-tune the pretrained model for three epochs, identify and prune
concept neurons, and retrain the model incrementally for a total of 8 epochs.
It took 3.5 hours to rank neurons in the model by importance and identify the
highest activation tokens for each neuron in the model.

###  A.3 Alternatives to probeless method

An alternative method for identifying top concept neurons in §4.1 is to use a
linear probe. This involves training a logistic regression probe to predict
concept annotations associated with a set of sentences from neuron activations
Dalvi et al. (2019). The top concept neurons is then be computed at increasing
percentages of the weight mass and then accumulated in order to create a
global ordering of neurons by saliency.

We performed experiments with our method using both the probeless and linear
probe method. From our results, both techniques yielded less than a 1%
variation in numerical values and did not influence the overall trend.
However, the probeless method is preferred since it is less computationally
expensive than the linear probe, and is free of probing limitations that could
impact ranking quality Antverg and Belinkov (2022).

##  Appendix B Analyzing concept recovery and redistribution after
neuroplasticity

###  B.1 Performance recovery

Tables 11 and 12 display how performance on overall named entity recognition
changes throughout the remapping process for an example run on a DistilBERT
model. Table 13 displays how performance on named entity recognition,
specifically for the task of recognising location names, changes throughout
the remapping process. There is a substantial decline in precision, recall and
F1 score for location name recognition post-pruning, but performance recovers
gradually throughout retraining.

Stage | Precision | Recall | F1
---|---|---|---
Base model | 0.931 | 0.940 | 0.936
Pruned | 0.015 | 0.131 | 0.028
Retrained (2 epochs) | 0.909 | 0.919 | 0.914
Retrained (4 epochs) | 0.919 | 0.929 | 0.924
Retrained (6 epochs) | 0.922 | 0.934 | 0.928
Retrained (8 epochs) | 0.924 | 0.935 | 0.929

Table 11: Precision, recall and F1 score on overall named entity recognition
for DistilBERT throughout remapping, after pruning concept neurons related to
location names. Note the dramatic drop in performance after pruning concept
neurons, which confirms we have removed features that the model was using for
predictions.

Stage | Precision | Recall | F1
---|---|---|---
Base | 0.931 | 0.94 | 0.936
Pruned | 0.021 | 0.161 | 0.037
Retrained (2 epochs) | 0.922 | 0.934 | 0.927
Retrained (4 epochs) | 0.926 | 0.938 | 0.932
Retrained (6 epochs) | 0.925 | 0.934 | 0.930
Retrained (8 epochs) | 0.929 | 0.939 | 0.934

Table 12: Precision, recall and F1 score on overall named entity recognition
for DistilBERT throughout remapping, after pruning neurons related to location
names in a separate run.

Stage | Precision | Recall | F1
---|---|---|---
Base model | 0.956 | 0.962 | 0.959
Pruned | 0.039 | 0.067 | 0.049
Retrained (2 epochs) | 0.935 | 0.947 | 0.941
Retrained (4 epochs) | 0.948 | 0.954 | 0.951
Retrained (6 epochs) | 0.952 | 0.960 | 0.956
Retrained (8 epochs) | 0.949 | 0.960 | 0.955

Table 13: Precision, recall and F1 score for DistilBERT specifically on
recognising location names throughout remapping. Note the dramatic drop in
performance after pruning concept neurons, and the subsequent recovery in
performance throughout retraining.

Stage | Precision | Recall | F1
---|---|---|---
Base model | 0.392 | 0.940 | 0.936
Pruned | 0.486 | 0.131 | 0.028
Retrained (2 epochs) | 0.434 | 0.919 | 0.914
Retrained (4 epochs) | 0.368 | 0.929 | 0.924
Retrained (6 epochs) | 0.423 | 0.934 | 0.928
Retrained (8 epochs) | 0.394 | 0.935 | 0.929

Table 14: Precision, recall and F1 score on overall named entity recognition
for a DistilGPT2 model throughout remapping, after pruning concept neurons
related to location names. Note the drop in performance after pruning concept
neurons, which confirms we have removed features that the model was using for
predictions.

Likewise, table 14 displays how performance on overall named entity
recognition changes over one run for a DistilGPT2 model. Interestingly,
performance for the model after pruning and retraining for 8 epochs is higher
than the performance on the original fine-tuned model. This suggests
neuroplasticity could potentially improve better performance.

###  B.2 Complex concepts in later layers are redistributed to earlier ones

Table 15 shows the highest activating tokens (HATs) for the most salient
neurons in the model after retraining for two epochs. Upon comparing the HATs
in the retrained model and the base model, we confirm that the pruned concept
(location names) is remapped within the first two epochs.

Neuron |  Base model HATs |  Retrained model (2 epochs) HATs
---|---|---
4606 |  106, Homer, 39, Says, 57 |  Aires, Civic, Lumpur, Clara, Hampshire
4983 |  Wilcox, Clintons, Navarro, Peterson, Wallis |  Straits, Town, Congo, Delhi, Hague
4867 |  M20, Laws, Nafta, Flight, WTO |  Ghana, Normandy, Thousand, Troy, Holiday
4976 |  Workers, BioDefense, Regenerative, Sodra, Regeneration |  Buckinghamshire, Dove, Manitoba, Griffith, Middlesex
5134 |  Everest, Odd, Grey, Trappist, Achievement |  Cricket, Northridge, Cornwall, Madero, Scotia

Table 15: Highest activating tokens for most salient neurons in model after
pruning the concept of location names and retraining for two epochs, compared
with their highest activating tokens in the base model.

Figure 12 visualizes how concept saliency is redistributed from later layers
to earlier layers in the middle of the DistilBERT model. Neurons located in
later layers of the base model (layer 5 and 6) show higher concept saliency.
Among all the concept neurons to be pruned from the base model, neurons in
layers 5 and 6 have the highest saliency scores. This suggests that, in the
base model, neurons in later layers vary more in the extent to which they
capture the concept of location, but specific neurons represent the concept
more strongly than the rest of the model.

(a) Concept saliency for neurons in model before pruning and retraining.
Neurons in later layers are more salient.

(b) Concept saliency for neurons after retraining for 8 epochs. The most
salient neurons are still in later layers, but earlier layers demonstrate
increased saliency scores.

Figure 12: Representative sample of neurons in DistilBERT model before and
after neuroplasticity occurs, highlighted by concept saliency for location
names. Black units are neurons that were ablated.

Layer | HATs of random neurons | Saliency
---|---|---
1 | deepest, scary, Half, tests | 0.649
2 | representation, hum, Lenfant, calm | 0.654
3 | Bureau, Durham, Rookie, Christ | 0.564
4 |  Quincy, Cove, Fulham, Chorus | 0.747
5 | Wilder, Weisel, Tour, Salvador | 0.792
6 |  MA, Manchester, Premiere, Placita | 0.829

Table 16: Highest activation tokens (HATs) for random sample of neurons in
different layers of the model, which was retrained after the concept of
location was pruned. Tokens that could be associated with location are in
bold. The pruned concept reappears not only in later layers, but also in
middle layers 3 and 4.

We verify that the pruned concept is redistributed by examining HATs for
neurons across different layers after neuroplasticity. In Table 16, we
randomly sample 4 neurons from each layer of the model - which was retrained
after the concept of location names was pruned - and extract the top
activating token for each neuron. As expected from the similar saliency
scores, neurons in layers 3 and 4 now noticeably activate on location names in
addition to neurons in layers 5 and 6. Interestingly, layers 3 and 4 also
activate on tokens that could be seen as location words in one context but as
a different concept in another (e.g. “Quincy” is both the name of a city and
the name of a person). In contrast, neurons in later layers more consistently
activate solely on location names. This pattern appears throughout the model.

This indicates that not only do middle layer neurons capture the pruned
concept more strongly than before neuroplasticity occurred, but they also
capture unrelated and/or context-dependent aspects of the pruned concept.

###  B.3 Verifying our measure of concept saliency

To verify that our measure of concept saliency corresponds with the extent to
which a concept is captured, we cross-check the HATs for neurons across
different layers. We expect to see HATs in layers with higher concept saliency
scores to be more associated with the concept of location. This is validated
in Table 3, where we see that the HATs for neurons with the highest saliency
are much more related to location names, compared to neurons in previous
layers with lower saliency scores.

Neuron |  HATs | Saliency
---|---|---
1483 |  aftermath, depiction, motto, constellation, kissing | 0.393
2378 |  mash, vandalized, Dortmund, denim, Kitty | 0.526
3754 |  y, Relief, Charitable, Program, Haag | 0.678
1940 |  rail, Chef, Sometimes, aspects, understanding | 0.716
4427 |  Lithuanian, Australian, Gothic, Malaysian, Doctor | 0.810
4819 |  Outback, Burger, Fords, Alignment, Universe | 1.000

Table 17: Highest activation tokens (HATs) for neurons with different concept
saliency scores. Note that neurons with the highest concept saliency scores
are clearly related to the concept of location, whereas neurons with low
saliency are unrelated.

##  Appendix C Generalizing concept remapping to person names

###  C.1 Inducing concept remapping for the concept of person names

To verify our overall results, we induce neuroplasticity for the concept of
person names in a DistilBERT model trained on NER. We find that a similar
pattern emerges in how concepts are captured before and after remapping, and
in where concepts reappear in terms of similarity.

Stage | Precision | Recall | F1
---|---|---|---
Base model | 0.934 | 0.941 | 0.938
Pruned | 0.021 | 0.150 | 0.037
Retrained (2 epochs) | 0.897 | 0.913 | 0.905
Retrained (4 epochs) | 0.910 | 0.923 | 0.917
Retrained (6 epochs) | 0.911 | 0.928 | 0.919
Retrained (8 epochs) | 0.913 | 0.928 | 0.921

Table 18: Precision, recall and F1 score on overall named entity recognition
throughout remapping, after removing concept neurons related to person names.

Stage | Precision | Recall | F1
---|---|---|---
Base model | 0.963 | 0.967 | 0.965
Pruned | 0.011 | 0.091 | 0.020
Retrained (2 epochs) | 0.941 | 0.954 | 0.947
Retrained (4 epochs) | 0.941 | 0.957 | 0.949
Retrained (6 epochs) | 0.941 | 0.959 | 0.950
Retrained (8 epochs) | 0.937 | 0.961 | 0.949

Table 19: Precision, recall and F1 score on recognising specifically person
names throughout remapping, after removing concept neurons related to person
names.

After pruning top concept neurons related to person names, there is a
significant decline in overall performance on named entity recognition, and
performance on recognising person names, as shown in tables 18 and 19.
However, after retraining the model for 8 epochs, performance improves
gradually such that the final retrained model’s performance nearly matches its
baseline. The dramatic drop and the subsequent recovery in performance
provides further evidence for the feasibility of recovering performance
despite the removal of crucial concept neurons.

(a) Concept saliency for neurons in model before pruning and retraining.
Neurons in later layers are more salient.

(b) Concept saliency for neurons in model after retraining for 8 epochs. The
most salient neurons are in layers 5 and 6, but earlier layers demonstrate
higher saliency scores than before.

Figure 13: Representative sample of neurons in DistilBERT model before and
after neuroplasticity occurs, highlighted by concept saliency for person
names. A higher saliency means that the neuron captures the concept more
strongly than other neurons. Black units are (pruned) neurons that do not
activate on non-null tokens.

###  C.2 Redistribution of concepts after pruning concept of person names

The distribution of concept saliency for person names is also similar to the
distribution of saliency for location names. As seen from Figure 13, the
concept of person names is captured most significantly in neurons in the later
layers in the base model. The mean saliency scores in layer 5 and 6 in the
base model (0.559 and 0.610, respectively) are considerably higher than the
mean saliency scores in other layers. This reinforces the idea that higher
level concepts such as location or people names is captured in later layers of
the large language model. Interestingly, the concept of person names seems to
be more concentrated in later layers of the model compared to the concept of
location names. The range in mean concept saliency scores across layers for
person names is 0.313, compared to 0.274 for location names.

After pruning and retraining, the concept of person names is also distributed
more evenly over the middle and later layers of the model. This matches our
results from §6.3. As seen in figure 13, neurons in layers 5 and 6 remain the
most salient (mean scores 0.797 and 0.872), but neurons in layers 3 and 4 also
have noticeable saliency (mean scores 0.759 and 0.797) relative to other
layers.

We confirm that neurons relearn both new concepts and old concepts over the
retraining process, for the concept of person names. As the model is
retrained, concept similarity does not change noticeably. Most neurons post-
retraining capture new concepts which are very similar to concepts that they
captured in the base model as shown in table 20, particularly in the earlier
layers.

Layer | Mean similarity over retraining
---|---
2 epochs | 8 epochs
0 | 0.925 | 0.925
1 | 0.512 | 0.557
2 | 0.430 | 0.444
3 | 0.326 | 0.344
4 | 0.186 | 0.209
5 | 0.086 | 0.107
6 | 0.040 | 0.043

Table 20: Mean similarity scores across layers in the model at different
stages after pruning the concept of person names (two epochs retraining versus
8 epochs retraining). Similarity is highest in middle layers. Similarity
scores do not change significantly over retraining.

As a concrete example, table 21 tracks the highest activating tokens for an
example neuron throughout stages of remapping, after pruning neurons related
to person names. A subset of HATs from the base model remain throughout
remapping (“exist”), but the neuron learns to activate on new HATs related to
the pruned concept.

Stage |  HATs
---|---
Base model |  exist, investment, halt, substitution, dog
Retrained (2 epochs) |  exist, Las, Hoover, discoveries, Spencer
Retrained (4 epochs) |  exist, Las, Hoover, Spencer, socially
Retrained (6 epochs) |  exist, Las, Hoover, Spencer, socially
Retrained (8 epochs) |  exist, Las, Hoover, Spencer, es

Table 21: Highest activating tokens (HATs) for neuron 2405 throughout stages
of remapping. Note that the neuron eventually learns to activate on new tokens
related to person names, as well as old tokens that it responded to in the
base model.

###  C.3 Concept saliency versus similarity for the concept of person names

Figure 14: In a model retrained for 8 epochs after the concept of person names
was pruned, neurons with high saliency tend to also have high similarity.
Neurons with negative similarity scores all had low saliency scores.

Consider the same analysis as in §6.4 for the DistilBERT model after pruning
the concept of person names and retraining as shown in figure 14. The same
pattern appears; there exists a dense cluster of neurons in the region of
saliency above 0.5 and similarity above 0.5. Figure 14 reveals that most
neurons for which the pruned concept of person names is remapped to originally
activated strongly on a similar concept before neuroplasticity occurred. This
provides further evidence to support a tentative correlation between concept
similarity and saliency in DistilBERT.

###  C.4 Pruning sub-concepts

To verify generalizability, we prune mutually exclusive subconcepts that are
related to the same overall concept. We then check if the overall concept is
relearned. In this example, we pruned the most salient 50% of concept neurons
related to the set of (mutually exclusive) concepts
"SEM:origin:north_america:usa", "SEM:origin:north_america:mexico", and
"SEM:origin:north_america:canada" from a DistilBERT model. We then analyzed
the model after retraining to detect if the overall concept
"SEM:origin:north_america" has been relearned.

Figure 15: Mean concept saliency for the concept of "North America", for
neurons across different layers of DistilBERT throughout the process of
neuroplasticity. Layer 6 shows a dramatic decrease in saliency, whereas layer
5 shows a large increase in concept saliency.

Figure 15 shows that the concept is recovered after retraining, albeit not to
a significant extent. Most layers have a final saliency score between 0.45 to
0.65, which is lower than scores in figures in §6.3. However, layer 6 of the
model shows a significant decrease in saliency score post-ablation, whereas
layer 5 shows a signficant increase in saliency post-ablation. Representations
of the overall concept for the pruned sub-concepts are therefore redistributed
from later layers to earlier layers.

##  Appendix D Pseudocode algorithm

Algorithm 1 illustrates our method of inducing neuroplasticity and analyzing
conceptual remapping.

Algorithm 1 Neuroplasticity in LLMs

1:Pretrained Model $M$, Concept $C$

2:# Begin with a pretrained model and a concept of interest

3:

4:function FineTune($M,C$)

5: Fine-tune $M$ on task requiring $C$ $\triangleright$ Adjust the model for a
specific task

6:end function

7:

8:function IdentifyConceptNeurons($M,C$)

9: $S\leftarrow\emptyset$ $\triangleright$ Locate the neurons in the model

10: Identify neurons $S$ for $C$

11: for each neuron $s$ in $S$ do

12: Compute saliency of $s$ $\triangleright$ Define how saliency is computed

13: end for

14: return $S$

15:end function

16:

17:function Prune($M,S$)

18: Set weights in $S$ to zero $\triangleright$ Reduce the influence of the
concept

19:end function

20:

21:function RetrainModel($M$)

22: Retrain $M$ $\triangleright$ Until recovery criteria met

23: $R\leftarrow\emptyset$

24: Identify neurons $R$

25: return $R$

26:end function

27:

28:function SaliencySimilarityAnalysis($M,S,R$)

29: for each epoch do

30: if epoch is even then

31: for each neuron $r$ in $R$ do

32: Compute saliency of $r$ $\triangleright$ Saliency metric for $r$

33: if neuron $r$ is in $S$ then

34: Compute similarity for $r$ $\triangleright$ Compare $r$ with its original
state in $S$

35: end if

36: end for

37: end if

38: end for

39:end function

40:

##  Appendix E Broader impact statement

This study investigates neuroplasticity in large language models and its
implications for model pruning, editing, transfer learning, and
interpretability. The findings have the potential to improve the efficiency
and safety of language models and increase public trust in large scale AI
models.

While this work aims to understand language models further to make them safer
and aligned with user needs, better language model understanding and
optimization may also lead to potential misuse or negative consequences.
Enhanced model efficiency and capability, for instance, might be leveraged for
malicious purposes, including spreading misinformation, creating more
convincing deepfake content or even creation of dangerous pathogens. Given
these potential wider impacts, we echo the need to approach AI research
ethically and responsibly. Advancements in language model understanding,
pruning, and editing should be accompanied by studying their ethical and
safety consequences to create AI systems that are useful, safe, and aligned
with human values.


FitToWidth[1][]

# Analysis of Self-Supervised Speech Models on Children’s Speech and Infant
Vocalizations

###### Abstract

To understand why self-supervised learning (SSL) models have empirically
achieved strong performances on several speech-processing downstream tasks,
numerous studies have focused on analyzing the encoded information of the SSL
layer representations in adult speech. Limited work has investigated how pre-
training and fine-tuning affect SSL models encoding children’s speech and
vocalizations. In this study, we aim to bridge this gap by probing SSL models
on two relevant downstream tasks: (1) phoneme recognition (PR) on the speech
of adults, older children (8-10 years old), and younger children (1-4 years
old), and (2) vocalization classification (VC) distinguishing cry, fuss, and
babble for infants under 14 months old. For younger children’s PR, the
superiority of fine-tuned SSL models is largely due to their ability to learn
features that represent older children’s speech and then adapt those features
to the speech of younger children. For infant VC, SSL models pre-trained on
large-scale home recordings learn to leverage phonetic representations at
middle layers, and thereby enhance the performance of this task.

Index Terms—  Self-supervised learning, children’s speech, infant
vocalizations, canonical correlation analysis, paralinguistic features

##  1 Introduction

Automatically analyzing children’s speech and vocalizations, such as automatic
speech recognition (ASR) and vocalization classification (VC), could yield
significant advantages in the fields of education, healthcare, and
developmental psychology. For example, automatic analysis of children’s speech
can assist both clinicians and parents in identifying autism [1], speech and
language disorders [2], allowing for timely intervention. Additionally,
children’s ASR plays an important role in developing interactive educational
software that can adapt to the child’s speech and language abilities [3],
which provides opportunities to personalize feedback and increase the
efficiency of educational programming and support.

One major obstacle to developing children’s speech analysis models is the
sparsity of available labeled datasets. Recently, self-supervised learning
(SSL) models, such as wav2vec 2.0 (W2V2) [4] and HuBERT [5], have
significantly advanced children’s speech technology with limited labeled data.
SSL model first uses pre-training on a large amount of unlabeled audio data.
By fine-tuning a pre-trained SSL on a small amount of labeled data, it excels
on multiple downstream tasks. With SSL, several studies have reported strong
performances on children’s ASR [6, 7, 8], child-adult speaker recognition [9],
infant cry detection [10], and children’s VC task [11, 12].

To understand why SSL models empirically have achieved strong performances on
downstream tasks, previous studies have examined the encoded information in
SSL layer representations. Pasad et al. [13] performed layer-wise analysis on
SSL models and found acoustic and linguistic properties were encoded in
several layers, and pre-trained W2V2 model behaved similarly to an
“autoencoder” style. Li et al. [14] similarly explored W2V2 fine-tuned on
emotional corpora using CCA and discovered W2V2 discards paralinguistic
information that is irrelevant for word recognition task. Other studies
expanded layer-wise analysis for accent [15] and prosody [16] probing.

Previous studies have mainly focused on analyzing and understanding the
performances of SSL models involving adult speech. Little work has been
conducted in probing SSL models for encoding children’s speech and
vocalizations, particularly for younger children under the age of 5. In one
relevant study, Lavechin et al. [17] created a benchmark of children’s
language acquisition to probe SSL-based spoken language model on lexical and
syntactic levels. Yeung et al. [18] revealed that the ASR trained on
kindergarten speech performed much worse than the ASR trained on 1st-grade
speech when tested on kindergarten speech due to the large variability of
kindergarten phonetics, and 1st-grade children’s speech was perhaps useful for
training a kindergarten ASR. Other studies [19, 20] have analyzed and showed
variability of children’s acoustic features across ages. In this study, we
analyze how SSL models encode children’s speech and vocalizations on two
tasks: (1) phoneme recognition (PR) for the speech of adults, older children
(8-10 years old), and younger children (1-4 years old), and (2) VC
distinguishing cry, fuss, and babble for infants under 14 months. Code and
model weights are available111https://huggingface.co/lijialudew/. Our detailed
findings are listed as the following:

  * •

For PR, SSL models fine-tuned on younger children’s speech maintain older
children’s phonetics but lose most of adult phonetics. SSL models pre-trained
on children’s speech under 5 years old shows limited encoding of the phonetics
specific to this age group.

  * •

For infant VC, SSL model pre-trained on home recordings achieves superior
performance by utilizing representations of middle layers, which are encoded
with higher-level phonetic features. SSL model pre-trained on adult speech
uses lower layers that are close to input acoustic features, which results in
inferior performance. SSL representations are mostly correlated with
paralinguistic feature groups of energy, mel-frequency cepstral coefficients
(MFCC), and pitch.

##  2 Phoneme Recognition Analysis

###  2.1 Data

We use three datasets for PR analysis, including LibriSpeech [21], My Science
Tutor (MyST) [22], and Providence [23]. For LibriSpeech, we use train-
clean-100 (100-hour speech with 251 speakers), dev-clean (5.4-hour speech with
40 speakers), and test-clean (5.4-hour speech with 40 speakers) for training
and testing adult speech. MyST contains conversational speech of 1371 students
between third and fifth grades ($\sim$8-10 years old) with a virtual tutor. We
select short transcribed utterances (<15s) for training and testing to align
with shorter vocalizations produced by younger children. We have
84.2h/14.1h/15.1h for training/development/testing sets respectively.
Providence contains longitudinal audio and video recordings of six English-
speaking children (3 boys: Alex, Ethan, and William, and 3 girls: Lily, Naima,
and Violet) from ages 1 to 4 interacting with their mothers at home.
Annotators transcribed children’s speech using SAMPA phonetic symbols. We
manually filter out some of the highly noisy recordings and use transcribed
child audio for fine-tuning SSL models. In total, we train on 84.0h utterances
of four children (Ethan, William, Lily, and Naima) and test on 24.8h
utterances of the other two (Alex and Violet). To perform PR, we convert
transcripts of all three datasets into international phonetic alphabet (IPA)
format. For LibriSpeech and MyST, we use eng_to_ipa
software222https://pypi.org/project/eng-to-ipa/. For Providence, we directly
map SAMPA symbols to IPA. In total, we have 53 unique IPA phones.

###  2.2 Experimental Setting

We build PR models using W2V2 and HuBERT on SpeechBrain [24] framework. The
training objective for all fine-tuned SSL models is to minimize phone-level
CTC loss [25]. For inference, we use CTC greedy decoding without any language
model. For brevity, we use the format W2V2/HuBERT-Dataset to denote SSL models
that are either pre-trained or fine-tuned on specific Dataset using W2V2 or
HuBERT.

W2V2/HuBERT-Libri100h: We add an output linear layer of hidden node 384
followed by CTC loss on top of W2V2/HuBERT-Base, W2V2/HuBERT pre-trained on
960-hour unlabeled LibriSpeech, consisting of 12 transformer layers with a
hidden dimension of 768. We first freeze W2V2/HuBERT-Base and only fine-tune
the added linear layer for 90 epochs using LibriSpeech with phone-level CTC
loss; then we fine-tune the entire model for an additional 10 epochs until
convergence. W2V2/HuBERT-MyST: We keep the linear layer trained on
W2V2/HuBERT-Libri100h and continue fine-tuning the entire model using MyST for
another 10 epochs until convergence. To build PR model for Providence, we test
both two-level (W2V2/HuBERT-Libri100-Pro) and three-level fine-tuning
(W2V2/HuBERT-MyST-Pro). For both cases, we keep the linear layer trained on
W2V2/HuBERT-Libri100 or W2V2/HuBERT-MyST and then fine-tune the entire model
for another 10 epochs using Providence. We also explore adding a linear layer
on (1) W2V2/HuBERT-Base, and (2) W2V2-LL4300h [12], a W2V2 model pre-trained
on 4300-hour unlabeled home recordings used in our past study. We collected
these recordings from 245 families of children under 5 years old via infant-
wearable recording devices, LittleBeats (LB) [26] and LENA333LB and LENA data
collection and annotation was supported by the funding from the National
Institute on Drug Abuse (R34DA050256), National Institute of Mental Health
(R21MH112578), and USDA National Institute of Food and Agriculture
(ILLU-793-368). [27]. We directly fine-tune W2V2/HuBert-Base or W2V2-LL4300h
with the linear layer using Providence. For all experiments, learning rates
(LR) are set for Adam optimizer as 9e-3 and 1e-4 for the output linear layer
and W2V2/HuBERT respectively; scheduler with the new-bob technique adjusts LR
based on the development set performance after each epoch. Epoch with the best
performance of the development set is used for final evaluation.

###  2.3 Canonical Correlation Analysis for Phoneme Labels

Canonical correlation analysis (CCA) [28] is a statistical method for finding
the maximum correlation between linear projections of two sets of continuous-
valued vectors. In this study, we use projection-weighted CCA [29], as
described in [13], to examine the similarity between SSL layer representation
and one-hot phonemes labels for all three datasets. For extracting phoneme
representations from SSL models, we use Montreal Forced Aligner [30] to
compute phoneme timestamps. For each phoneme, we average its SSL
representation within the central third timespan. We discard the first and
last third timespans to alleviate co-articulation effects. In total, we
extract roughly 30k phonemes, with about 600 samples per phoneme, for each
dataset. The final CCA scores are calculated with a cross-validation scheme.
Total samples are divided into ten folds. One fold is used for testing, while
the remaining nine are for training. The average CCA score from three
different test folds is reported.

Dataset | LibriSpeech | MyST | Providence
---|---|---|---
Model | W2V2 | HuBERT | W2V2 | HuBERT | W2V2 | HuBERT
Libri100 | 2.9 | 3.2 | 19.7 | 20.5 | >100 | >100
MyST | 6.4 | 5.8 | 10.7 | 10.5 | >100 | >100
Libri100-Pro | 99.3 | 99.1 | 81.8 | 75.6 | 59.9 | 60.2
MyST-Pro | 99.1 | 99.4 | 70.8 | 64.1 | 59.6 (*) | 59.1 (***)
LL4300-Pro | – | – | – | – | diverged | -
Base-Pro | – | – | – | – | diverged | diverged

Table 1: Phone error rate, in percent, for SSL models evaluated on the test
set of LibriSpeech, MyST, and Providence. Results of MAPSSWE tests for
W2V2/HuBERT-Libri100-Pro and W2V2/HuBERT-MyST-Pro models are indicated with *
in parentheses.

###  2.4 Results & Analysis

Table 1 presents phone error rates (PER) for each fine-tuned model evaluated
on the testing set for LibriSpeech, MyST, and Providence datasets. Figure 1
shows the layer-wise CCA scores that can be used to explain performances in
Table 1.

Analysis of Pre-trained Models For two pre-trained W2V2 models (W2V2-Base and
W2V2-LL4300h) with the pre-training objective as reconstructing masked
acoustic input, we see an “auto-encoder style” behavior with middle layers
(6-10) encoding the richest phonetic information, while top layers exhibit a
decline in phonetic content on the speech of adults and older children (Fig 1
(a) and (b)). CCA scores on W2V2-LL4300h are lower than W2V2-Base, which
suggests the far-field adults’ speech in noisy home recordings may cause
degradation in learning phonetics. For Hubert-Base (Fig 1 (c)), top layers
embed the highest phonetic content maybe because its pre-training objective is
to predict discrete phonetic units learned from clustering. However, all three
pre-trained models fail to encode phonetics for younger children, which
explains why fine-tuning younger children’s speech on these pre-trained models
diverged (W2V2-LL4300h-Pro and W2V2/HuBERT-Base-Pro in Table 1).

Analysis of Fine-tuned Models For fine-tuned SSL models, we observe similar
trends of CCA scores of HuBERT and W2V2, so we only present the W2V2 set for
brevity444 We present complete sets of CCA scores here:
https://sites.google.com/view/ssl-analysis-children-speech. This link also
includes supplementary results of sample T-SNE [31] plots to demonstrate how
phoneme clusters of different age groups evolve across layers for fine-tuned
SSL models.. Top layers of W2V2-Libri100h and W2V2-MyST fine-tuned with
phonetic targets encode the most phonetic information (Fig 1 (d) and (e)), and
W2V2-MyST reduces the domain shift between the phonetics of adults and older
children for all layers to achieve optimal result on MyST corpus. We compare
PER for W2V2/HuBERT-Libri100h-Pro and W2V2/HuBERT-MyST-Pro on Providence using
statistical significant tests [32], MAtched Pairs Sentence-Segment Word Error
(MAPSSWE). We obtain statistically significant better PER using three-level
fine-tuning W2V2-MyST-Pro, with p-value=0.018, and HuBERT-MyST-Pro, with
p-value$<$0.001, than their corresponding two-level fine-tuning models. For
W2V2-Libri100h-Pro and W2V2-MyST-Pro (Figure 1 (f) and (g)), CCA scores show a
smaller gap between phonetics of older and younger children across all layers,
but a steady decrease with adult phonetics beyond layer 3. To recognize
younger children’s phonetics through fine-tuning, the SSL model and the output
linear layer need to first encode the phonetics of adults or older children as
a foundational basis. Note W2V2-MyST-Pro has higher CCA scores of older
children’s speech than W2V2-Libri100h-Pro for top layers, which corresponds to
lower PER on MyST on three-level fine-tuning models (W2V2/HuBERT-MyST-Pro)
than those of two-level fine-tuning (W2V2/HuBERT-Libri100h-Pro).

Fig. 1: CCA scores (y-axis) over 12 layers (x-axis) computed on LibriSpeech,
MyST, and Providence datasets for three pre-trained models, (a)-(c), and four
fine-tuned W2V2 models, (d)-(g).

##  3 Infant Vocalization Analysis

###  3.1 Data & Methodology

We follow our previous work [12] on data preprocessing, model architecture,
and experimental settings for analyzing infant vocalizations (under 14 months)
collected via LB. We separate LB home recordings into 10-min segments for
annotation. Infant vocalizations were manually labeled as cry, fuss, and
babble. In total, we have 70 labeled 10-mins recordings from 22 families. We
divide the LB audio stream into 2-second intervals of every 0.2 seconds, and
the vocalization type is determined as the majority type of centered 1 second.
We follow the leave-one-family-out data partition scheme and have 52/6/12
10-mins recordings from 15/3/4 families for training/development/testing sets,
respectively. In our past study, we used W2V2 models to perform three tasks,
including speaker diarization, infant VC, and adult VC tasks. For each task,
we employed a weighted average (WA) layer to learn the weights of mean-pooled
utterance-level representations from all 12 layers. Then we input the weighted
sum features into a feed-forward network (FFN) to obtain classification
results. The training objective is the average of cross-entropy losses over
three tasks. In this study, we focus on analyzing two models, W2V2-Base and
W2V2-LL4300h, on the infant VC task. We either freeze W2V2 and only fine-tune
the WA and FFN or fine-tune the entire model. For each experiment, we run each
model for 10 epochs and use the epoch with the best performance on the
development set for testing. We further examine the weights learned from the
WA layer and rerun the experiments with the best three layers.

Fig. 2: Weights learned (y-axis) for 12 layers (x-axis) in weighted average
layer for four W2V2 models. For each model, weights sum up to 1 across all
layers.

Model | PT/FT | Layers | Infant VC
---|---|---|---
|  | Used | Acc | F1
W2V2-Base | PT | All 12 | 68.7 | 49.5
| FT | All 12 | 65.7 | 53.5
W2V2-LL4300h | PT | All 12 | 76.0 | 64.3
| PT | 5-7 | 75.9 | 66.4
| FT | All 12 | 78.2 | 68.9
| FT | 5-7 | 75.2 | 65.1

Table 2: Accuracy and unweighted F1 scores for infant VC task. PT=Freeze pre-
trained W2V2 and fine-tune output FFN only, and FT=Fine-tune entire model.

###  3.2 Infant VC Results & Analysis

Results of infant VC are shown in Table 2. Figure 2 shows the weights learned
from the WA layer across models. W2V2-LL4300h outperformed W2V2-Base by a
large margin. For W2V2-Base pre-trained on adult speech, only lower layers 1-3
are mostly helpful, which are close to input acoustic features. This explains
why W2V2-Base produces degraded results. By fine-tuning W2V2-Base with infant
vocalizations, the learned weights become relatively uniform across layers,
possibly due to the domain shift involved in learning infant acoustics for the
entire model. For W2V2-LL4300h pre-trained on home recordings, we observe
layers 5-7 have the largest weights. Interestingly, these layers also heavily
encode phonetics of adults and older children (Fig 1 (a)). By freezing
W2V2-LL4300h and fine-tuning output FFN only with a weighted sum of layers
5-7, we obtain a 2.1% gain in terms of the F1 score compared to using a
weighted sum of all 12 layers. Fine-tuning the entire W2V2-LL4300h with a
weighted sum of all 12 layers leads to the optimal performance, where layers
6-8 contribute the most weights.

Fig. 3: CCA scores (y-axis) over 12 layers (x-axis) for paralinguistic feature
groups computed on W2V2-LL4300h. PT=Pre-train and FT=Fine-tune.

Feature Group | Dim | Low-level Descriptors
---|---|---
energy | 2 | Loudness; Harmonics-to-Noise Ratio
MFCC | 4 | Mel-Frequency Cepstral Coefficients 1-4
pitch | 1 | Pitch
formant | 6 | Formant 1, 2, and 3 frequency and bandwidth
spectral | 10 |  | Alpha Ratio 50-1000Hz and 1-5k Hz;
---
Hammarberg Index;
Spectral Slope 0–500 Hz and 500–1500 Hz;
Formant 1, 2, and 3 relative energy;
Harmonic difference H1–H2 and H1-A3
voice quality | 2 | Shimmer; Jitter

Table 3: Paralinguistic feature groups and their dimensions.

###  3.3 CCA for Paralinguistic Features

We measure the similarity between W2V2-LL4300h layer representations and
paralinguistic features using CCA. For extracting paralinguistic vectors, we
randomly select 4500 2s samples, with 1500 samples each from cry, fuss, and
babble. We compute CCA scores cross-validation, similarly to the procedures as
described in section 2.3. We use OpenSMILE [33] toolkit to extract Geneva
Minimalistic Acoustic Parameter Set [34] with 25 low-level descriptors, and we
group them into six categories as shown in Table 3. Figure 3 shows the CCA
scores. W2V2 layer representations are mostly correlated with energy, MFCC,
and pitch and less correlated with formant, spectral, and voice quality
parameters. Pitch is known as a key feature in distinguishing infant
vocalizations [35], which is also preserved in W2V2 representations. CCA
scores for paralinguistic features generally showed a steady decrease trend
from bottom layer 1 to top layer 12. We suspect that this may be due to
paralinguistic features being less useful in reconstructing masked input
regions in the W2V2 pre-training stage at top layers. CCA scores for fine-
tuned W2V2 are generally lower than pre-trained W2V2 beyond layer 6, which
suggests fine-tuned W2V2 has less emphasis on paralinguistic features.

##  4 Conclusion

In this study, we probe several pre-trained and fine-tuned SSL models on two
tasks, children’s PR and infant VC. For PR, we provide insights into how fine-
tuning affects the domain shift of SSL encoding phonetic distributions among
different age groups. Fine-tuned SSL models learn to recognize younger
children’s phoneme distinctions by adapting the phonetics learned from older
children. For infant VC, we illustrate how pre-training SSL models on home
recordings versus adult speech affects how the weighted features learned
during fine-tuning. SSL model, pre-trained on large-scale home recordings,
preserves key paralinguistic features that are essential for enhancing infant
VC. Our work sheds light on understanding how SSL models process and perceive
children’s speech and infant vocalizations.

## References

  * [1] Y. Gong, H. Yatawatte, C. Poellabauer, S. Schneider, and S. Latham, “Automatic autism spectrum disorder detection using everyday vocalizations captured by smart devices,” in _Proceedings of the 2018 ACM international conference on bioinformatics, computational biology, and health informatics_ , 2018, pp. 465–473.
  * [2] E. I. Toki, V. Zakopoulou, G. Tatsis, K. Plachouras, V. Siafaka, E. I. Kosma, S. K. Chronopoulos _et al._ , “A game-based smart system identifying developmental speech and language disorders in child communication: A protocol towards digital clinical diagnostic procedures,” in _Interactive Mobile Communication, Technologies and Learning_. Springer, 2021, pp. 559–568.
  * [3] A. Hagen, B. Pellom, and R. Cole, “Children’s speech recognition with application to interactive books and tutors,” in _2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No. 03EX721)_. IEEE, 2003, pp. 186–191.
  * [4] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: a framework for self-supervised learning of speech representations,” in _Proceedings of the 34th International Conference on NeurIPS Systems_ , 2020, pp. 12 449–12 460.
  * [5] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [6] R. Fan and A. Alwan, “DRAFT: A Novel Framework to Reduce Domain Shifting in Self-supervised Learning and Its Application to Children’s ASR,” in _Proc. Interspeech 2022_ , 2022, pp. 4900–4904.
  * [7] R. Fan, Y. Zhu, J. Wang, and A. Alwan, “Towards better domain adaptation for self-supervised models: A case study of child ASR,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1242–1252, 2022.
  * [8] R. Jain, A. Barcovschi, M. Y. Yiwere, D. Bigioi, P. Corcoran, and H. Cucu, “A wav2vec2-based experimental study on self-supervised learning methods to improve child speech recognition,” _IEEE Access_ , vol. 11, pp. 46 938–46 948, 2023.
  * [9] R. Lahiri, T. Feng, R. Hebbar, C. Lord, S. H. Kim, and S. Narayanan, “Robust Self Supervised Speech Embeddings for Child-Adult Classification in Interactions involving Children with Autism,” in _Proc. INTERSPEECH 2023_ , 2023, pp. 3557–3561.
  * [10] A. Gorin, C. Subakan, S. Abdoli, J. Wang, S. Latremouille, and C. C. Onu, “Self-supervised learning for infant cry analysis,” _2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)_ , pp. 1–5, 2023.
  * [11] A. Xu, R. Hebbar, R. Lahiri, T. Feng, L. Butler, L. Shen, H. Tager-Flusberg, and S. Narayanan, “Understanding Spoken Language Development of Children with ASD Using Pre-trained Speech Embeddings,” in _Proc. INTERSPEECH 2023_ , 2023, pp. 4633–4637.
  * [12] J. Li, M. Hasegawa-Johnson, and N. L. McElwain, “Towards Robust Family-Infant Audio Analysis Based on Unsupervised Pretraining of Wav2vec 2.0 on Large-Scale Unlabeled Family Audio,” in _Proc. INTERSPEECH 2023_ , 2023, pp. 1035–1039.
  * [13] A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self-supervised speech representation model,” in _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2021, pp. 914–921.
  * [14] Y. Li, Y. Mohamied, P. Bell, and C. Lai, “Exploration of a self-supervised speech model: A study on emotional corpora,” in _2022 IEEE Spoken Language Technology Workshop (SLT)_ , pp. 868–875.
  * [15] M. Yang, R. C. M. C. Shekar, O. Kang, and J. H. L. Hansen, “What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Model,” in _Proc. INTERSPEECH 2023_ , 2023, pp. 1923–1927.
  * [16] G.-T. Lin, C.-L. Feng, W.-P. Huang, Y. Tseng, T.-H. Lin, C.-A. Li, H.-y. Lee, and N. G. Ward, “On the utility of self-supervised models for prosody-related tasks,” in _2022 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2023, pp. 1104–1111.
  * [17] M. Lavechin, Y. Sy, H. Titeux, M. A. C. Blandón, O. Räsänen, H. Bredin, E. Dupoux, and A. Cristia, “BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models,” in _Proc. INTERSPEECH 2023_ , 2023, pp. 4588–4592.
  * [18] G. Yeung and A. Alwan, “On the difficulties of automatic speech recognition for kindergarten-aged children,” _Interspeech 2018_ , 2018.
  * [19] V. M. Shetty, S. M. Lulich, and A. Alwan, “Developmental Articulatory and Acoustic Features for Six to Ten Year Old Children,” in _Proc. INTERSPEECH 2023_ , 2023, pp. 4598–4602.
  * [20] A. Potamianos and S. Narayanan, “A review of the acoustic and linguistic properties of children’s speech,” in _2007 IEEE 9th Workshop on Multimedia Signal Processing_. IEEE, 2007, pp. 22–25.
  * [21] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 5206–5210.
  * [22] W. Ward, R. Cole, D. Bolanos, C. Buchenroth-Martin, E. Svirsky, S. V. Vuuren, T. Weston, J. Zheng, and L. Becker, “My science tutor: A conversational multimedia virtual tutor for elementary school science,” _ACM Transactions on Speech and Language Processing (TSLP)_ , vol. 7, no. 4, pp. 1–29, 2011.
  * [23] K. Demuth, J. Culbertson, and J. Alter, “Word-minimality, epenthesis and coda licensing in the early acquisition of english,” _Language and speech_ , vol. 49, no. 2, pp. 137–173, 2006.
  * [24] M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong _et al._ , “Speechbrain: A general-purpose speech toolkit,” _arXiv preprint arXiv:2106.04624_ , 2021.
  * [25] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 369–376.
  * [26] N. McElwain, B. Islam, M. Fisher, C. Nebeker, J. M. Bodway, and M. Hasegawa-Johnson, “Evaluating users’ experiences of a child multimodal wearable device: A mixed methods approach,” _JMIR Human Factors_ , 2023, in press.
  * [27] J. Gilkerson _et al._ , “The lena natural language study,” _Boulder, CO: LENA Foundation. Retrieved March_ , vol. 3, p. 2009, 2008.
  * [28] G. Andrew, R. Arora, J. Bilmes, and K. Livescu, “Deep canonical correlation analysis,” in _Proceedings of the 30th International Conference on Machine Learning_ , vol. 28, no. 3, 17–19 Jun 2013, pp. 1247–1255.
  * [29] A. S. Morcos, M. Raghu, and S. Bengio, “Insights on representational similarity in neural networks with canonical correlation,” in _Neural Information Processing Systems_ , 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:49271358
  * [30] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using kaldi.” in _Interspeech_ , vol. 2017, 2017, pp. 498–502.
  * [31] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.
  * [32] D. Pallet, W. Fisher, and J. Fiscus, “Tools for the analysis of benchmark speech recognition tests,” in _International Conference on Acoustics, Speech, and Signal Processing_ , 1990, pp. 97–100 vol.1.
  * [33] F. Eyben, M. Wöllmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” in _Proceedings of the 18th ACM international conference on Multimedia_ , 2010, pp. 1459–1462.
  * [34] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan _et al._ , “The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing,” _IEEE transactions on affective computing_ , vol. 7, no. 2, pp. 190–202, 2015.
  * [35] D. B. Fox, “An analysis of the pitch characteristics of infant vocalizations.” _Psychomusicology: A Journal of Research in Music Cognition_ , vol. 9, no. 1, p. 21, 1990.