# Test-Time Training for Speech

Sri Harsha Dumpala
Vector Institute and Dalhousie University
Chandramouli Sastry
Vector Institute and Dalhousie University
Sageev Oore
Vector Institute and Dalhousie University

###### Abstract

In this paper, we study the application of Test-Time Training (TTT) as a
solution to handling distribution shifts in speech applications. In
particular, we introduce distribution-shifts to the test datasets of standard
speech-classification tasks—for example, speaker-identification and emotion-
detection—and explore how Test-Time Training (TTT) can help adjust to the
distribution-shift. In our experiments that include distribution shifts due to
background noise and natural variations in speech such as gender and age, we
identify some key-challenges with TTT including sensitivity to optimization
hyperparameters (e.g., number of optimization steps and subset of parameters
chosen for TTT) and scalability (e.g., as each example gets its own set of
parameters, TTT is not scalable). Finally, we propose using BitFit – a
parameter-efficient fine-tuning algorithm proposed for text applications that
only considers the bias parameters for fine-tuning – as a solution to the
aforementioned challenges and demonstrate that it is consistently more stable
than fine-tuning all the parameters of the model.

##  1 Introduction

Deep learning methods achieve impressive results in a variety of speech-based
downstream tasks when the train and test data are in-distribution (Gulati et
al., 2020; Snyder et al., 2017; Zou et al., 2022). In practice, however, the
train and test distributions are usually different, i.e., there exists a
distributional shift between the train and test data. In speech, such
distributional shifts can be introduced due to inter-speaker variations such
as speaking style, gender, age, etc., or due to background induced noises such
as babble, living room, traffic, etc. These distributional shifts
significantly degrade the performance of the deep learning models
(Likhomanenko et al., 2021; Garcia-Romero et al., 2019; Parry et al., 2019).
In real-world applications, some form of distributional shifts often occurs in
the test data, making it of vital importance for deep learning models to be
robust to these shifts.

One approach to handling distributional shift is with train-time techniques
(Khurana et al., 2021; Li et al., 2020), in which we need to anticipate the
type of distributional shifts that can occur during testing, and then train
the model on data collected with this anticipated list of distributional
shifts. In practice, the anticipated list of distributional shifts is non-
exhaustive, and there is no guarantee that the trained model can generalize
well to an unseen domain at test-time.

Another interesting approach which attained significant improvements in
performance for imaging tasks is test-time training (TTT) (Sun et al., 2020;
Liu et al., 2021; Gandelsman et al., 2022). In TTT, we update the model at
inference using the test-sample. As the test sample does not have a label, a
self-supervised learning task is used for this update.

The efficacy of TTT is impacted by the choice of the self-supervised learning
task (Liu et al., 2021). Gandelsman et al. (2022) shows that masked auto-
encoding on images is a suitable task for TTT. Motivated by the success of the
transformer-based masked autoencoders (MAE) for speech (Huang et al., 2022a),
we extend a test-time training approach based on MAE (Gandelsman et al., 2022)
to speech in this work. To the best of our knowledge, this is the first work
to adapt TTT to the speech domain. We show that TTT-MAE for speech shows
significant improvements on three different downstream tasks under a variety
of distributional shifts.

Two major challenges of using TTT during inference are maintaining stable
performance robust to (reasonable) ranges of hyperparameters, and the
potential high computational cost, which is due to both (a) increased memory
requirements for updating all parameters of the model, and (b) inability to
process a batch of samples if the individual samples in the batch are
associated with different distributional shifts. We show that, for speech, it
is possible to make significant improvements with respect to all of these
issues, i.e. we improve stability, reduce memory requirements, and allow batch
processing, all by using parameter-efficient training. Specifically, we show
that using bias fine-tuning (Zaken et al., 2022), we can process a batch of
test samples, even under the condition that each test sample has a different
distributional shift.

##  2 Related work

Domain adaptation and generalization. These methods are based on the
assumption that models will have access to labelled data from the train
distribution and unlabelled (labelled) data from the test distribution. A
common strategy in domain adaptation is to learn domain invariant features
between train and test data distributions (Sun and Saenko, 2016; Tzeng et al.,
2017; Long et al., 2018). Another approach is to perform self-training on the
test distribution by generating pseudo-labels for the unlabelled data (Xie et
al., 2020). Domain generalization techniques mainly resort to adversarial
training, meta learning or adversarial data augmentation (Yang et al., 2021;
Balaji et al., 2018; Dou et al., 2019; Volpi et al., 2018). In speech, domain
adaptation and generalization techniques have been applied to tasks such as
automatic speech recognition, emotion recognition and speaker classification
(Khurana et al., 2021; Hu et al., 2021; Song et al., 2017; Li et al., 2020).
All these methods assume information about the test domain is available during
training.

Learning at inference. In the above techniques, the model is trained to
generalize to all possible distributional shifts. But anticipating every
possible distributional shift at train time is not feasible, particularly in
real world applications. Most models trained using domain generalization
techniques are fixed during inference even when the test distribution changes.
To alleviate this problem, another line of work is to adapt the model to test
samples at inference. Methods to update the model at inference can be
classified into two types: test-time adaptation and test-time training.

Test-time Adaptation (TTA): These methods allow using off-the-shelf models
without any additional training. In general terms, test-time adaptation
focuses on adapting models that were not trained with a special configuration
prior to being used at inference (Goyal et al., 2022; Boudiaf et al., 2023).
One of the first approaches of this category, called TENT (Wang et al., 2021),
requires the model and target data. It then updates the model layers by
minimizing the Shannon entropy of predictions. Mummadi et al. (2021) improves
TENT by using a log-likelihood ratio instead of entropy, and by estimating
target batch statistics. Another approach is to update batch normalization
(BN) statistics using large number of test samples (Nado et al., 2020;
Schneider et al., 2020). SITA (Khurana et al., 2022) is one such approach
which can be used on a single test data example. SITA generates a pseudo-batch
by randomly augmenting this example and then computes statistics on this
pseudo-batch. TTA in computer vision is heavily targeted on the BN layer’s
adaptation by re-estimating batch statistics on target data. In this work we
use transformer models, which have achieved state-of-the-art performance on
many speech-based downstream tasks. Since transformer models are not equipped
with BN layers, as the length of batched input sequences are different, TTA
techniques cannot be applied directly to speech. Only one work has applied TTA
to speech (Lin et al., 2022); this approach appears limited to ASR models
trained with CTC loss.

Test-time training (TTT): The basic paradigm in TTT (Sun et al., 2020) is to
use a test-time task (usually a self-supervised learning task) besides the
main task during training, and update the pre-trained model using test data
with the (self-supervised) test-time objective before the final prediction.
Sun et al. (2020) uses rotation prediction as the self-supervised task. Later,
TTT++ (Liu et al., 2021) considers contrastive loss as the self-supervised
task in addition to aligning the features by comparing the statistics of the
source data with those of the current test batch. Recently, Osowiechi et al.
(2023) minimizes the distribution shift, between the train and test
distributions, estimated using normalizing flows. Gandelsman et al. (2022)
shows that using masked autoencoding (MAE) (He et al., 2022) as the self-
supervised task for TTT achieves substantial improvements in image recognition
under various distributional shifts. TTT-based techniques are applied to other
domains such as videos (Azimi et al., 2022; Wang et al., 2023), natural
language processing (NLP) (Banerjee et al., 2021) and compressed sensing of
medical images (Darestani et al., 2022), where the self-supervised task varies
across domains. In this work, we extend TTT framework to speech domain for the
first time. Our work extends the TTT-MAE framework (Gandelsman et al., 2022)
to speech by using work on audio MAE trained with spectrograms (Huang et al.,
2022a; Chong et al., 2022).

Parameter efficient fine-tuning (PEFT). Fine-tuning entire pre-trained models
achieves state-of-the-art performance for various downstream tasks (Kenton and
Toutanova, 2019; Raffel et al., 2020; Mohamed et al., 2022). However, as the
size of these models increases rapidly, updating the models in parameter-
efficient ways becomes crucial (Yang et al., 2022; Chen et al., 2022). In the
NLP domain, PEFT techniques typically refer to either: a) insertion of new
learnable modules with fewer parameters as compared to the whole model; or b)
modifying carefully selected parameters of the model. In Houlsby et al.
(2019); Pfeiffer et al. (2020); Sung et al. (2022b); Li and Liang (2021), only
the additional parameters added to the pre-trained models are fine-tuned for
the downstream tasks while Yang et al. (2022); Zaken et al. (2022) fine-tune a
subset of parameters for downstream tasks without inserting any new modules.
Zaken et al. (2022) show that just fine-tuning the bias parameters, which
constitute about 0.1% of the overall parameters, can outperform full fine-
tuning, especially for small datasets. Motivated from NLP, PEFT techniques are
also applied to pre-trained models trained using images (Jia et al., 2022;
Chen et al., 2022; Sung et al., 2022a; Lee et al., 2022), and speech (Li et
al., 2021; Huo et al., 2022). In this work, we study PEFT techniques applied
to speech in the context of TTT, focusing on the following questions: 1) Can
PEFT techniques achieve comparable or improved performance compared to full
fine-tuning in TTT? 2) Can PEFT be more stable than full fine-tuning in TTT?

Self-Supervision using masking in speech. Denoising autoencoders (Vincent et
al., 2008) is one of the earliest forms of self-supervision in speech (Lu et
al., 2012). Subsequent advancements in self-supervision have predominantly
focused on masked language modeling (MLM) (Liu et al., 2020; Baevski et al.,
2020; Hsu et al., 2021; Gong et al., 2022; Huang et al., 2022b). Many of these
works employ transformer networks trained with MLM frameworks to achieve
state-of-the-art performance on various speech-related downstream tasks. More
recently, the concept of a masked autoencoder (MAE) based on the Vision
Transformer (ViT) has been extended to the audio and speech domain (Huang et
al., 2022a; Baade et al., 2022; Chong et al., 2022; Niizumi et al., 2022). In
MAE, only the non-masked spectrogram patches are encoded, distinguishing it
from other approaches that encode both masked and non-masked input
(wave/spectrogram) segments for self-supervised pre-training. This distinction
makes MAE-based models computationally appealing. In this work, we analyze the
effect of distributional shift on the performance of MAE for speech and use
TTT-based approach to enhance the robustness of MAE in the presence of such
distributional shifts.

##  3 Method

Pre-training MAE. Our masked autoencoder (MAE) for speech, following Huang et
al. (2022a); Chong et al. (2022), aims to reconstruct the masked patches of
the speech Mel-spectrogram with an asymmetrical encoder-decoder architecture.
Below, we provide a brief overview of the MAE.

First, we transform the input speech waveform into 128-dimensional Mel-
spectrograms using a Hanning window of size 25ms for every 10ms. Next, we
divide the spectrogram into a sequence of non-overlapping patches, each patch
sized 16 $\times$ 16\. These patches are then flattened and embedded with a
linear projection layer. To provide positional information, fixed sinusoidal
positional embeddings are added to the embedded patches. Afterward, we
randomly mask 75% of the patches while preserving positional indices of all
the patches. This enables the decoder to reconstruct the spectrogram. For the
encoder, only unmasked patches are used to generate latent representations.
The decoder then tries to reconstruct the original spectrogram, given latent
representations of the encoder and masked patches as input. The latent
representations and masked patches are organized in the initial order before
being provided as input to the decoder. During training, the objective is to
minimize mean squared error (MSE) between reconstructed and input
spectrograms, averaged over the masked patches.

Train-time training. For the downstream tasks, we only use the encoder and
discard the decoder. The latent representations generated by the encoder are
provided as input to the task-specific classifier head. In this paper, we
consider three different ways to use the pre-trained encoder for downstream
tasks: 1) Linear probing: freeze the encoder to use it as a feature extractor
and train only the classifier head; 2) Fine-tuning: train both encoder and
classifier head, end-to-end, for the downstream task; 3) Linear-probing and
fine-tuning (LP-FT): first train only the classifier head using linear
probing, and then fine-tune both encoder and classifier head end-to end as
explained in (Kumar et al., 2022).

Test-time training. Similar to (Sun et al., 2020; Gandelsman et al., 2022), we
use a Y-shaped architecture: a shared encoder network $e$ followed by two
heads, a self-supervised head $g$ and a task-specific classifier head $h$.
Here, $e$ and $g$ are the encoder and decoder networks of the pre-trained MAE,
respectively. The classifier head $h$, uses a linear projection from the
dimension of the encoder features to the number of classes, depending on the
downstream task.

When using TTT, we use linear-probing to update the weights of the classifier
head $h$ and freeze the weights of the shared encoder $e$ during train-time
training. As explained in Gandelsman et al. (2022), we found that linear-
probing is more suitable for TTT as compared to full-model fine-tuning. At
test-time, the parameters of the shared encoder are updated to minimize the
self-supervised loss. We explore the following approaches to update the
weights of the encoder during test-time:

1) Full fine-tuning: In full fine-tuning, all parameters of the shared encoder
are updated to minimize the self-supervised loss across various augmentations
of a single test sample. However, the large size of pre-trained models, such
as the MAE used in this study with 75M parameters, makes full fine-tuning
computationally expensive during test time. Moreover, extensive steps of full
fine-tuning during test time, as shown in Figure 1, can result in performance
degradation. Since there is no validation data available at test time, early
stopping is not an option.

Therefore, it is desirable to maintain stability in performance during Test-
Time Training (TTT). However, even when using the SGD optimizer as suggested
in (Gandelsman et al., 2022), full fine-tuning still exhibits performance
degradation in speech-related tasks. Furthermore, TTT techniques entail a
higher computational cost as TTT is approached as a one-sample learning
problem. This means that the model can only be adapted to one test sample at a
time, and batch processing is not feasible under the assumption that each test
sample is subject to a different distribution shift.

Table 1: Trainable parameters in different blocks of our MAE for speech. Number of Parameters in one Encoder block are same as the number of parameters in each of the first, middle or last blocks of the encoder. | #parameters(Proportion %)
---|---
MAE | 74751488 (100.00)
Encoder | 64457216 (86.23)
Decoder | 10294272 (13.77)
One Encoder block | 7087872 (9.48)
Bias - MAE | 96000 (0.13)
Bias - Encoder | 77568 (0.10)

To address these challenges, we investigate parameter-efficient fine-tuning
techniques (PEFT) that have proven to be highly effective in supervised
learning tasks within the field of NLP. However, their application in the
context of TTT has not previously been explored.

2) Parameter-efficient fine-tuning (PEFT): Here we explore different PEFT
techniques to adapt the encoder to the test sample during inference. We
conduct experiments using four different PEFT techniques for TTT: (i) First
block: updating only the final/last block of the encoder, (ii) Last block:
updating only the final/last block of the encoder, (iii) Middle block:
updating only the middle block of the encoder, and (iv) Bias: updating only
bias parameters of the encoder using the self-supervised task on the test
sample. As shown in Table 1, the number of trainable parameters for updating
any one block (first, middle, or last) accounts for 9.48% of the total
parameters, while updating the bias parameters involves just 0.1% of the total
parameters. In this study, we primarily focus on bias fine-tuning for TTT due
to the following reasons:

  * •

Bias fine-tuning is much more lightweight compared to full fine-tuning,
training 830 times fewer parameters (0.78M vs. 64M).

  * •

Zaken et al. (2022) demonstrated that, in supervised learning with limited
training data, fine-tuning only bias parameters yields superior performance to
full fine-tuning. Since TTT-MAE can be regarded as a one-sample unsupervised
domain adaptation technique, we investigate effectiveness of updating only
bias parameters during test-time training.

  * •

TTT techniques incur higher computational costs as TTT is approached as a one-
sample learning problem. This means that the model can only be adapted to one
test sample at a time, and batch processing is not feasible when each test
sample is drawn from a different distribution. We show that bias fine-tuning
allows processing an entire batch of test samples.

Figure 1: We compare the accuracy (speaker identification) across TTT steps
between three different variants of TTT (full, last-layer and bias fine-
tuning). For most distributional shifts, full fine-tuning shows degradation in
performance with longer test-time training (after 20 steps) whereas bias and
last layer fine-tuning show relatively stable performance even after 25 steps

Distributional shifts in speech. In this work, we identify and examine two
types of distributional shifts: (1) those resulting from background noise that
degrades/distorts speech quality, and (2) natural distributional shifts
resulting from inter-speaker variations, including gender, age, and speaking
style.

To generate degraded speech, we introduce background noise to clean speech
signals at a specified signal-to-noise ratio (SNR) (see Figure 2). We identify
two categories of background noise: (1) Time-invariant, where noise
characteristics remain constant over time. Examples include additive white
Gaussian noise (AWGN) and air conditioner (AC). (2) Time-varying, where noise
characteristics change over time. Examples include background babble, living
room, restaurant, reverberation, and traffic. Distributional shifts introduced
by these types of noise are generally difficult to learn, even with
adversarial training. Furthermore, some noises (e.g. babble, restaurant, and
reverberation) exhibit patterns similar to speech, which can corrupt and
contaminate information contained in the original signal, such as linguistic
content, speaker characteristics, emotions, etc.

In this study, we explore significance of TTT in handling distributional
shifts introduced by both background noise and natural variations in speech.
We show that TTT-based methods consistently outperform non-TTT techniques with
significant margins. Use of PEFT techniques, particularly BitFit, further
improve the performance and stability of TTT under different distributional
shifts. To show the effectiveness of TTT under different distributional
shifts, we conduct the following experiments: 1) Training with clean speech
and testing with speech corrupted by various background noises. 2) Training
models on one speaking style and testing with another speaking style. 3)
Training with speech from one gender and testing with the other gender. 4)
Training with either younger or older speakers and testing with the other.

Figure 2: Mel-spectrograms of speech with different distributional shifts due
to background noises added at 0 dB SNR. (c) shows the distribution shift in
Mel-spectrogram of clean speech (see (a)) when added with AWGN ((b)). (d)-(i)
shows the Mel-spectrograms of speech when added with different background
noises. Characteristics of background noises (e)-(i) vary with time thus
distorting the characteristics of speech critical for speech-based
applications. For instance, panel (e) shows how babble noise, which has
similar characteristics to speech, when added to speech, introduces patterns
similar to clean speech along both time and frequency dimensions, and thus
distorts the patterns in clean speech. Similarly, living room (see (f) and
restaurant (see (g)) noises distort speech patterns in time and frequency
dimensions.

##  4 Experiments and Results

Implementation Details In all experiments, we use 9-layer ViT by default as
the MAE encoder. For the decoder, we use a 3-layer Transformer. We use
Voxceleb2 dataset (Chung et al., 2018) for pre-training the MAE. We pre-train
the MAE for 120K steps with a batch size of 392, which takes about 7 days
using 8 Nvidia RTX6000 24GB GPUs. The AdamW optimizer with an initial learning
rate of 0.001 and a weight decay of 0.05 is applied. The learning rate has a
cosine decay schedule (Chen et al., 2020) with 20K warmup steps. We transform
raw waveform (mono-channel sampled at 16 KHz) into 128 Mel-frequency bands
extracted with 20 ms Hanning window and 10 ms stride. We use the original
speech spectrograms and apply no augmentations. We only use random masking of
the spectrograms as an augmentation, with a masking ratio of 0.75 for pre-
training.

During TTT, for each test sample, we train only the encoder (freezing the
decoder weights)for $20$ steps using SGD optimizer with a fixed learning rate
of 2.5e-3, batch size of 128, momentum of 0.9 and weight decay of 0.2. We also
show performance plots for $25$ steps of TTT (see Figure 1). During TTT, we
follow the same procedure as pre-training: mask 75% of the input patches and
provide the unmasked patches as input to the encoder whereas all the patches
are provided to the decoder. Then update the encoder weights using
reconstruction loss (MSE) on the masked patches as the objective function. We
follow the same procedure for full, bias, first-layer, middle-layer and last
layer fine-tuning.s Moreover, we do not use any augmentation on top of random
masking for TTT. We performed most of these experiments using a single Nvidia
A40 48GB GPU. Unless and otherwise specified, we report results for TTT after
20 TTT steps.

Table 2: Details of the datasets used for pre-training and downstream tasks. Total Duration is in hours and Average Length is in seconds. The tasks Speaker ID, Emotion Recog., Ltd Vocab. ASR refer to Speaker identification, emotion recognition and limited vocabulary ASR, respectively. Dataset | Task | #Classes | #Speakers | #Total | Total | Average
---|---|---|---|---|---|---
|  |  |  | Samples | Duration | Length
VoxCeleb2 | Pre-training | – | 5994 | 770000 | 2300 | 8.6
VCTK | Speaker ID | 109 | 109 | 42075 | 44 | 3.8
CREMA-D | Emotion Recog. | 4 | 91 | 4397 | 3.04 | 2.5
IEMOCAP | Emotion Recog. | 4 | 10 | 5531 | 7.00 | 4.6
RAVDESS | Emotion Recog. | 4 | 24 | 672 | 0.70 | 3.7
TESS | Emotion Recog. | 4 | 2 | 1600 | 0.92 | 2.1
Speech Commands | Ltd. Vocab. ASR | 12 | 2618 | 105829 | 29.5 | 1.0

Dataset details. Table 2 provide details of the datasets used in this work. We
perform speaker identification using VCTK (Yamagishi et al., 2019); emotion
recognition using CREMA-D (Cao et al., 2014), IEMOCAP (Busso et al., 2008),
RAVDESS (Livingstone and Russo, 2018) and TESS (Dupuis and Pichora-Fuller,
2011) datasets; low-vocabulary speech recognition using Speech commands
(Warden, 2018) dataset. We use original speech samples from each of the
datasets in the pre-training and train-time training phases. To evaluate
models’ performance under different distributional shifts, we introduce
diverse background noises sourced from Microsoft’s Scalable Noisy Speech
Dataset (MS-SNSD) (Reddy et al., 2019). These noises are exclusively added
during the testing phase and are not used in pre-training or train-time
training of the models.

Test-time Training vs No Test-time Training. In Figure 3, we compare TTT for
speech with different non-TTT techniques (linear probing, fine-tuning, LP-FT
(Kumar et al., 2022) and TENT (Wang et al., 2021)) for different background
noises unseen during training. TTT outperforms the non-TTT techniques for
every unseen background noise condition. When comparing between non-TTT
techniques, for most of the background noises, simple linear-probing performs
better than fine-tuning. Similar to images in (Kumar et al., 2022), LP-FT
performs better than both linear-probing and fine-tuning. TENT, a test-time
adaptation technique, performs better or comparable to LP-FT but performs
inferior to TTT. This can be attributed to the fact that TENT requires a large
set of test samples to learn the distribution of the test sample but performs
poorly under one-sample testing condition, as illustrated in (Khurana et al.,
2022).

(a) Speaker identification (Accuracy (%)) on VCTK

(b) Emotion classification (UAR(%)) on Crema-D

Figure 3: Compare TTT with non-TTT approaches (linear probing, fine-tuning,
LP-FT, TENT) under different different shifts due to background noises (noises
added at 0 dB). TTT significantly outperforms the non-TTT approaches across
all the distributional shifts. Results averaged over 3 runs. Emotion
classification reported in terms of unweighted average recall (UAR (%))

Even though TTT achieves significant improvements in performance under
different distributional shifts, there are a few shortcomings of TTT when
applied to speech such as high memory requirements during TTT, degradation in
performance across TTT steps and inability to process batch of test samples,
as shown in Figure 1 and explained in Section 3.

We overcome these issues by incorporating PEFT techniques into TTT. PEFT
techniques are light weight as we need to fine-tune fewer parameters compared
to full fine-tuning, thus requiring lesser memory (see Table 1). We also find
that, for speech, PEFT techniques achieve better consistency in performance
compared to full fine-tuning across the TTT steps (see Figure 1). Here we
compare performance of different PEFT techniques used during TTT under
different distributional shifts in Table 3. PEFT techniques of fine-tuning a
specific block performs comparable to full-fine-tuning. Similar to (Lee et
al., 2022), we find certain layers are more suitable for a specific set of
background noises and there is no single block which is optimal for every
background noise. Selecting a layer to fine-tune during TTT is not feasible as
we only have a single test sample. To overcome this issue, we use Bitfit
(Zaken et al., 2022) for TTT, where we fine-tune only the bias parameters of
the encoder during TTT. Bitfit, a light-weighted fine-tuning approach,
consistently performs better (comparable) to full-fine-tuning across all the
distributional shifts.

Table 3: Performance of different variants of TTT under distributional shifts due to background noises added at 0dB SNR. Different variants of TTT: Full refers to Full fine-tuning; First, Middle, Last and Bias refer to fine-tuning only the first layer, middle layer, last layer and bias parameters, respectively. AWGN $\rightarrow$ additive white Gaussian noise. Bias fine-tuning performs better than other variants of TTT across most of the distribution shifts due to background noise | Speaker Identification | Emotion Recognition
---|---|---
| Full | First | Middle | Last | Bias | Full | First | Middle | Last | Bias
Air Conditioner | 41.8 | 42.6 | 40.8 | 42.5 | 43.9 | 37.7 | 37.6 | 36.1 | 38.0 | 42.8
Gaussian (AWGN) | 8.8 | 9.1 | 8.2 | 9.3 | 10.4 | 59.2 | 59.0 | 58.6 | 59.7 | 62.8
Babble | 15.7 | 17.0 | 16.0 | 15.7 | 17.3 | 40.8 | 42.3 | 41.6 | 41.2 | 42.8
Kitchen | 17.2 | 17.8 | 16.9 | 17.1 | 21.8 | 63.5 | 64.1 | 63.7 | 64.6 | 65.9
Living Room | 15.1 | 16.3 | 16.0 | 15.4 | 16.9 | 38.3 | 39.4 | 38.5 | 40.5 | 40.3
Park | 22.1 | 22.4 | 23.3 | 21.2 | 25.6 | 58.9 | 59.6 | 58.2 | 60.1 | 62.1
Restaurant | 16.1 | 18.5 | 16.3 | 17.2 | 19.4 | 41.7 | 43.1 | 41.8 | 42.5 | 43.9
Reverberation | 34.7 | 35.4 | 35.2 | 35.8 | 37.9 | 52.3 | 53.4 | 52.2 | 52.5 | 54.7
Squeaky-Chair | 60.3 | 60.6 | 60.8 | 62.5 | 62.1 | 63.7 | 64.3 | 64.4 | 63.6 | 65.5
Traffic | 52.9 | 55.9 | 54.3 | 52.7 | 57.2 | 61.7 | 65.3 | 63.2 | 65.7 | 66.8
Typing | 55.2 | 57.4 | 57.4 | 55.9 | 57.7 | 62.1 | 63.2 | 59.7 | 63.3 | 64.6
Washing-Machine | 56.0 | 60.8 | 60.5 | 60.4 | 61.4 | 64.1 | 65.1 | 65.0 | 65.4 | 66.2
Table 4: Emotion recognition under natural distributional shifts caused by (a)
speaking style variations: Train model using CREMA-D (CRE) dataset and test
with other datasets i.,e IEMOCAP (IEM), RAVDESS (RAV) and TESS. Column CRE:
matched condition – train and test on CREMA-D dataset, (b) Gender variations:
Train model using speech data from speakers of one gender and test with
speakers from other gender. F-M: train on female and test on male; M-F: train
on male and test on female; F-F and M-M: train and test speakers from the same
gender. We use CREMA-D dataset for these experiments. TTT variants (Full and
Bias fine-tuning) outperform non-TTT methods (linear probing and fine-tuning)
across different natural distribution shifts.

(a) evaluate across datasets - variation in speaking style | CRE | IEM | RAV | TESS
---|---|---|---|---
Linear | 52.2 | 35.4 | 34.7 | 33.1
Fine-tune | 61.5 | 32.5 | 33.8 | 34.3
Full | 66.8 | 45.5 | 44.3 | 39.4
Bias | 67.3 | 45.9 | 44.2 | 40.6

(b) Across genders - variation in gender | F-F | F-M | M-M | M-F
---|---|---|---|---
Linear | 54.2 | 31.6 | 52.1 | 30.9
Fine-tune | 59.4 | 29.8 | 58.6 | 30.2
Full | 66.0 | 54.1 | 61.5 | 56.8
Bias | 68.2 | 56.5 | 64.0 | 60.5

Table 5: Emotion classification under age variation. We use TESS dataset consisting of two female speakers: one Younger (Y) and one older (O). We train on one speaker and test using other speaker’s speech. Y-O $\rightarrow$ train on Y and test with O; O-Y $\rightarrow$ train on O and test with Y | Y-Y | Y-O | O-O | O-Y
---|---|---|---|---
Linear | 49.3 | 29.8 | 50.2 | 31.7
Fine-tune | 51.8 | 28.2 | 52.3 | 30.4
Full | 52.2 | 40.8 | 52.5 | 41.6
Bias | 52.4 | 41.9 | 52.9 | 42.8

Evaluation under natural distributional shifts. We evaluate performance of TTT
techniques across different natural distributional shifts caused by inter-
speaker variations, e.g., speaking style (Table 4(a)), gender (Table 4(b)),
and age (Table 5).

In Table 4(a), we compare performance of non-TTT (Linear probing and fine-
tuning) with TTT (full fine-tuning (Full) and Bias) techniques for
distributional shift due to speaking style variation. Here we train on CREMA-D
(emotively acted utterances in American English) and test with samples from
IEMOCAP (emotive conversations enacted in American English), RAVDESS
(emotively acted utterances in North American English) and TESS (emotively
acted utterances in Canadian English). TTT-based techniques achieve
significantly better performance compared to non-TTT methods for speaking
style variations.

To evaluate the performance of TTT in addressing distributional shifts caused
by gender variations for the task of emotion classification (CREMA-D dataset),
we conducted training using speech data from one gender (either female or
male) and carried out testing using speech data from the opposite gender (male
or female). Table 4(b) provides a comparison between non-TTT and TTT-based
techniques. For gender variations, non-TTT approaches exhibit a steep decline
in performance between matched (F-F and M-M) and mismatched (F-M and M-F)
conditions. In contrast, TTT approaches (full and Bias) show very small
degradation in performance between matched and mismatched conditions. For both
matched and mismatched conditions, TTT with Bias fine-tuning performs better
than full fine-tuning.

We evaluate performance of TTT under age variations using TESS dataset for the
task of emotion recognition (refer to Table 5). The TESS dataset was collected
from two female speakers: a younger (Y) speaker aged 26 years and an older
speaker aged 64 years. In the O-Y scenario, we trained the model using younger
(Y) speaker’s speech and tested it with older (O) speaker’s speech. Similarly,
in the Y-O scenario, we trained with the older speaker (O) and tested with the
younger (Y) speaker. For unmatched train-test conditions (Y-O and O-Y), TTT
techniques outperformed non-TTT techniques, with bias fine-tuning performing
better than full fine-tuning. Interestingly, even in matched conditions (Y-Y
and O-O), TTT-based techniques performed comparably or better than non-TTT
techniques for all the natural distributional shifts.

Figure 4: Illustration of TTT-Bias Fine-tuning on Batch of Test Samples: for
each example in a batch containing $B$ samples, we create a learnable
parameter corresponding to each trainable bias as shown in green boxes and the
outputs of the modules are adjusted accordingly as shown. Only the parameters
in green boxes need to be fine-tuned and since these parameters are not shared
across examples in a batch, TTT-bias can take advantage of GPU batch
processing to improve throughput.

Improved utilization of computational resources. We discuss an approach to
process a batch of test samples using TTT for the real world scenario of each
test sample associated with a different distributional shift. Since each test
sample gets its own copy of parameters, TTT can be applied with only one test
sample at a time. However, bias fine-tuning gives us an unique opportunity to
apply TTT to a batch of examples while still ensuring that each sample gets
its own copy of parameters. For example, consider a simple linear layer
containing weight matrix $W$ and a bias-vector $b$; given a batch ${\bf x}$
containing $B$ samples, the output of this layer can be written as ${\bf
y}=W{\bf x}+b$. In TTT-bias, each example in this batch gets its own $b$ while
sharing $W$; now, suppose that the linear layer gets an additional input
$\Delta b$ (initialized to zeros) that contains $B$ learnable vectors and is
used to compute the output ${\bf y}=W{\bf x}+b+\Delta b$: since $\Delta b$
contains $B$ learnable parameters, each parameter can be optimized
independently while taking advantage of the GPU batch-processing. We
illustrate this in Figure 4: in practice, this can be implemented as forward
hooks in PyTorch without changing the model code. Refer to the supplementary
material for further analysis and results.

##  5 Conclusion

In this work, we extend test-time training to speech related tasks such as
speaker identification and emotion recognition. We extend the TTT-MAE,
proposed in Gandelsman et al. (2022) for image recognition, to improve the
performance of speech-related downstream tasks under a variety of
distributional shifts. In our application of TTT-MAE to speech, we observed
that TTT is sensitive to hyperparameters such as training steps and subset of
parameters considered for optimization. To overcome these issues, we applied
PEFT techniques to make TTT more stable and scalabale. We find that PEFT
techniques, being light-weight, achieve better or comparable performance to
full fine-tuning. Specifically, we find that bias fine-tuning, motivated from
BitFit, improves both performance and stability. Further, we propose a new
approach to process batches of test samples using bias fine-tuning for TTT.

## References

  * Azimi et al. [2022] F. Azimi, S. Palacio, F. Raue, J. Hees, L. Bertinetto, and A. Dengel.  Self-supervised test-time adaptation on video data.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 3439–3448, 2022.
  * Baade et al. [2022] A. Baade, P. Peng, and D. Harwath.  Mae-ast: Masked autoencoding audio spectrogram transformer.  _arXiv preprint arXiv:2203.16691_ , 2022.
  * Baevski et al. [2020] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in Neural Information Processing Systems_ , 34, 2020.
  * Balaji et al. [2018] Y. Balaji, S. Sankaranarayanan, and R. Chellappa.  Metareg: Towards domain generalization using meta-regularization.  _Advances in neural information processing systems_ , 31, 2018.
  * Banerjee et al. [2021] P. Banerjee, T. Gokhale, and C. Baral.  Self-supervised test-time learning for reading comprehension.  _arXiv preprint arXiv:2103.11263_ , 2021.
  * Boudiaf et al. [2023] M. Boudiaf, T. Denton, B. van Merriënboer, V. Dumoulin, and E. Triantafillou.  In search for a generalizable method for source free domain adaptation.  _arXiv preprint arXiv:2302.06658_ , 2023.
  * Busso et al. [2008] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan.  Iemocap: Interactive emotional dyadic motion capture database.  _Language resources and evaluation_ , 42(4):335–359, 2008.
  * Cao et al. [2014] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma.  Crema-d: Crowd-sourced emotional multimodal actors dataset.  _IEEE transactions on affective computing_ , 5(4):377–390, 2014.
  * Chen et al. [2022] S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition.  _Advances in Neural Information Processing Systems_ , 35, 2022.
  * Chen et al. [2020] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Chong et al. [2022] D. Chong, H. Wang, P. Zhou, and Q. Zeng.  Masked spectrogram prediction for self-supervised audio pre-training.  _arXiv preprint arXiv:2204.12768_ , 2022.
  * Chung et al. [2018] J. S. Chung, A. Nagrani, and A. Zisserman.  Voxceleb2: Deep speaker recognition.  _arXiv preprint arXiv:1806.05622_ , 2018.
  * Darestani et al. [2022] M. Z. Darestani, J. Liu, and R. Heckel.  Test-time training can close the natural distribution shift performance gap in deep learning based compressed sensing.  In _International Conference on Machine Learning_ , pages 4754–4776. PMLR, 2022.
  * Dou et al. [2019] Q. Dou, D. Coelho de Castro, K. Kamnitsas, and B. Glocker.  Domain generalization via model-agnostic learning of semantic features.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Dupuis and Pichora-Fuller [2011] K. Dupuis and M. K. Pichora-Fuller.  Recognition of emotional speech for younger and older talkers: Behavioural findings from the toronto emotional speech set.  _Canadian Acoustics_ , 39(3):182–183, 2011.
  * Fayek et al. [2017] H. M. Fayek, M. Lech, and L. Cavedon.  Evaluating deep learning architectures for speech emotion recognition.  _Neural Networks_ , 92:60–68, 2017.
  * Gandelsman et al. [2022] Y. Gandelsman, Y. Sun, X. Chen, and A. A. Efros.  Test-time training with masked autoencoders.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Garcia-Romero et al. [2019] D. Garcia-Romero, D. Snyder, S. Watanabe, G. Sell, A. McCree, D. Povey, and S. Khudanpur.  Speaker recognition benchmark using the chime-5 corpus.  In _Interspeech_ , pages 1506–1510, 2019.
  * Gong et al. [2022] Y. Gong, C.-I. Lai, Y.-A. Chung, and J. Glass.  Ssast: Self-supervised audio spectrogram transformer.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 10699–10709, 2022.
  * Goyal et al. [2022] S. Goyal, M. Sun, A. Raghunathan, and Z. Kolter.  Test-time adaptation via conjugate pseudo-labels.  _arXiv preprint arXiv:2207.09640_ , 2022.
  * Gulati et al. [2020] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang.  Conformer: Convolution-augmented transformer for speech recognition.  In _Interspeech 2020_ , pages 5036–5040, 2020.
  * He et al. [2022] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Houlsby et al. [2019] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly.  Parameter-efficient transfer learning for nlp.  In _International Conference on Machine Learning_ , pages 2790–2799. PMLR, 2019.
  * Hsu et al. [2021] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460, 2021.
  * Hu et al. [2021] H. Hu, X. Yang, Z. Raeesy, J. Guo, G. Keskin, H. Arsikere, A. Rastrow, A. Stolcke, and R. Maas.  Redat: Accent-invariant representation for end-to-end asr by domain adversarial training with relabeling.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6408–6412. IEEE, 2021.
  * Huang et al. [2022a] P.-Y. Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer.  Masked autoencoders that listen.  _Advances in Neural Information Processing Systems_ , 35:28708–28720, 2022a.
  * Huang et al. [2022b] W. Huang, Z. Zhang, Y. T. Yeung, X. Jiang, and Q. Liu.  SPIRAL: self-supervised perturbation-invariant representation learning for speech pre-training.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022b.
  * Huo et al. [2022] Z. Huo, K. C. Sim, B. Li, D. Hwang, T. N. Sainath, and T. Strohman.  Resource-efficient transfer learning from speech foundation model using hierarchical feature fusion.  _arXiv preprint arXiv:2211.02712_ , 2022.
  * Jia et al. [2022] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim.  Visual prompt tuning.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII_ , pages 709–727. Springer, 2022.
  * Kenton and Toutanova [2019] J. D. M.-W. C. Kenton and L. K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186, 2019.
  * Khurana et al. [2022] A. Khurana, S. Paul, P. Rai, S. Biswas, and G. Aggarwal.  Sita: Single image test-time adaptation.  _arXiv preprint arXiv:2112.02355_ , 2022.
  * Khurana et al. [2021] S. Khurana, N. Moritz, T. Hori, and J. Le Roux.  Unsupervised domain adaptation for speech recognition via uncertainty driven self-training.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6553–6557. IEEE, 2021.
  * Kumar et al. [2022] A. Kumar, A. Raghunathan, R. Jones, T. Ma, and P. Liang.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  In _International Conference on Learning Representations_ , 2022.
  * Lee et al. [2022] Y. Lee, A. S. Chen, F. Tajwar, A. Kumar, H. Yao, P. Liang, and C. Finn.  Surgical fine-tuning improves adaptation to distribution shifts.  _arXiv preprint arXiv:2210.11466_ , 2022.
  * Li et al. [2020] L. Li, M.-W. Mak, and J.-T. Chien.  Contrastive adversarial domain adaptation networks for speaker recognition.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2020\.
  * Li et al. [2021] X. Li, C. Wang, Y. Tang, C. Tran, Y. Tang, J. Pino, A. Baevski, A. Conneau, and M. Auli.  Multilingual speech translation from efficient finetuning of pretrained models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 827–838, 2021.
  * Li and Liang [2021] X. L. Li and P. Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, 2021.
  * Likhomanenko et al. [2021] T. Likhomanenko, Q. Xu, V. Pratap, P. Tomasello, J. Kahn, G. Avidov, R. Collobert, and G. Synnaeve.  Rethinking evaluation in ASR: are our models robust enough?  In _Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association_ , pages 311–315. ISCA, 2021\.
  * Lin et al. [2022] G.-T. Lin, S.-W. Li, and H.-y. Lee.  Listen, adapt, better wer: Source-free single-utterance test-time adaptation for automatic speech recognition.  _arXiv preprint arXiv:2203.14222_ , 2022.
  * Liu et al. [2020] A. T. Liu, S.-w. Yang, P.-H. Chi, P.-c. Hsu, and H.-y. Lee.  Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6419–6423. IEEE, 2020.
  * Liu et al. [2021] Y. Liu, P. Kothari, B. Van Delft, B. Bellot-Gurlet, T. Mordan, and A. Alahi.  Ttt++: When does self-supervised test-time training fail or thrive?  _Advances in Neural Information Processing Systems_ , 34:21808–21820, 2021.
  * Livingstone and Russo [2018] S. R. Livingstone and F. A. Russo.  The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english.  _PloS one_ , 13(5), 2018.
  * Long et al. [2018] M. Long, Z. Cao, J. Wang, and M. I. Jordan.  Conditional adversarial domain adaptation.  _Advances in neural information processing systems_ , 31, 2018.
  * Lu et al. [2012] X. Lu, S. Matsuda, C. Hori, and H. Kashioka.  Speech restoration based on deep learning autoencoder with layer-wised pretraining.  In _Thirteenth Annual Conference of the International Speech Communication Association_ , 2012.
  * Mohamed et al. [2022] A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe, et al.  Self-supervised speech representation learning: A review.  _IEEE Journal of Selected Topics in Signal Processing_ , 2022.
  * Mummadi et al. [2021] C. K. Mummadi, R. Hutmacher, K. Rambach, E. Levinkov, T. Brox, and J. H. Metzen.  Test-time adaptation to distribution shift by confidence maximization and input transformation.  _arXiv preprint arXiv:2106.14999_ , 2021.
  * Nado et al. [2020] Z. Nado, S. Padhy, D. Sculley, A. D’Amour, B. Lakshminarayanan, and J. Snoek.  Evaluating prediction-time batch normalization for robustness under covariate shift.  _arXiv preprint arXiv:2006.10963_ , 2020.
  * Niizumi et al. [2022] D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino.  Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation.  _arXiv preprint arXiv:2204.12260_ , 2022.
  * Osowiechi et al. [2023] D. Osowiechi, G. A. V. Hakim, M. Noori, M. Cheraghalikhani, I. Ben Ayed, and C. Desrosiers.  Tttflow: Unsupervised test-time training with normalizing flow.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2126–2134, 2023.
  * Parry et al. [2019] J. Parry, D. Palaz, G. Clarke, P. Lecomte, R. Mead, M. Berger, and G. Hofer.  Analysis of deep learning architectures for cross-corpus speech emotion recognition.  In _Interspeech_ , pages 1656–1660, 2019.
  * Pepino et al. [2021] L. Pepino, P. Riera, and L. Ferrer.  Emotion recognition from speech using wav2vec 2.0 embeddings.  _Proc. Interspeech 2021_ , pages 3400–3404, 2021.
  * Pfeiffer et al. [2020] J. Pfeiffer, I. Vulić, I. Gurevych, and S. Ruder.  Mad-x: An adapter-based framework for multi-task cross-lingual transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7654–7673, 2020.
  * Raffel et al. [2020] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Reddy et al. [2019] C. K. Reddy, E. Beyrami, J. Pool, R. Cutler, S. Srinivasan, and J. Gehrke.  A scalable noisy speech dataset and online subjective test framework.  _arXiv preprint arXiv:1909.08050_ , 2019.
  * Schneider et al. [2020] S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge.  Improving robustness against common corruptions by covariate shift adaptation.  _Advances in Neural Information Processing Systems_ , 33:11539–11551, 2020.
  * Snyder et al. [2017] D. Snyder, D. Garcia-Romero, D. Povey, and S. Khudanpur.  Deep neural network embeddings for text-independent speaker verification.  In _Interspeech_ , volume 2017, pages 999–1003, 2017.
  * Song et al. [2017] P. Song, S. Ou, Z. Du, Y. Guo, W. Ma, J. Liu, and W. Zheng.  Learning corpus-invariant discriminant feature representations for speech emotion recognition.  _IEICE TRANSACTIONS on Information and Systems_ , 100(5):1136–1139, 2017.
  * Sun and Saenko [2016] B. Sun and K. Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In _Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14_ , pages 443–450. Springer, 2016.
  * Sun et al. [2020] Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt.  Test-time training with self-supervision for generalization under distribution shifts.  In _International conference on machine learning_ , pages 9229–9248. PMLR, 2020.
  * Sung et al. [2022a] Y.-L. Sung, J. Cho, and M. Bansal.  Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5227–5237, 2022a.
  * Sung et al. [2022b] Y.-L. Sung, J. Cho, and M. Bansal.  Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5227–5237, 2022b.
  * Tzeng et al. [2017] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell.  Adversarial discriminative domain adaptation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7167–7176, 2017.
  * Vincent et al. [2008] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol.  Extracting and composing robust features with denoising autoencoders.  In _Proceedings of the 25th international conference on Machine learning_ , pages 1096–1103, 2008.
  * Volpi et al. [2018] R. Volpi, H. Namkoong, O. Sener, J. C. Duchi, V. Murino, and S. Savarese.  Generalizing to unseen domains via adversarial data augmentation.  _Advances in neural information processing systems_ , 31, 2018.
  * Wang et al. [2021] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell.  Tent: Fully test-time adaptation by entropy minimization.  In _International Conference on Learning Representations_ , 2021.
  * Wang et al. [2023] R. Wang, Y. Sun, Y. Gandelsman, X. Chen, A. A. Efros, and X. Wang.  Test-time training on video streams, 2023.
  * Warden [2018] P. Warden.  Speech commands: A dataset for limited-vocabulary speech recognition.  _arXiv preprint arXiv:1804.03209_ , 2018.
  * Xie et al. [2020] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le.  Self-training with noisy student improves imagenet classification.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10687–10698, 2020.
  * Yamagishi et al. [2019] J. Yamagishi, C. Veaux, K. MacDonald, et al.  Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92).  _University of Edinburgh. The Centre for Speech Technology Research (CSTR)_ , 2019.
  * Yang et al. [2021] F.-E. Yang, Y.-C. Cheng, Z.-Y. Shiau, and Y.-C. F. Wang.  Adversarial teacher-student representation learning for domain generalization.  _Advances in Neural Information Processing Systems_ , 34:19448–19460, 2021.
  * Yang et al. [2022] Z. Yang, M. Ding, Y. Guo, Q. Lv, and J. Tang.  Parameter-efficient tuning makes a good classification head.  _arXiv preprint arXiv:2210.16771_ , 2022.
  * Zaken et al. [2022] E. B. Zaken, Y. Goldberg, and S. Ravfogel.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , pages 1–9, 2022.
  * Zou et al. [2022] H. Zou, Y. Si, C. Chen, D. Rajan, and E. S. Chng.  Speech emotion recognition with co-attention based multi-level acoustic information.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7367–7371. IEEE, 2022.

## Limitations

In this work, we evaluate test-time training techniques for speech
applications. We consider parameter-efficient fine-tuning alternatives and
identify that bias fine-tuning makes the test-time training method robust to
optimization hyperparameters while also improving inference speed.
Nonetheless, test-time training of large pre-trained models requires
appropriate compute resources such as GPUs for faster inference. While we
demonstrate that bias fine-tuning allows for batch processing of test-examples
and increases throughput, inference with TTT is still not as scalable as non-
TTT inference.

## Broader Impact

In this work, we introduce TTT for speech and show that TTT improves
performance of various speech-related downstream tasks under different
distributional shifts relative to the training data. This includes
distributional shifts caused due to speaking style, gender and age based
differences – see Table 4 for speaking style and gender variations, and Table
5 for age variations. Acquiring labels such that the dataset is unbiased can
be prohibitively expensive or simply unfeasible. For example, biases can get
introduced into training datasets because the examples do not accurately
represent all possible variations in speaker demographics such as gender, age,
accent, geographical, cultural and economical background; our results show
that TTT can be used to adapt models trained on biased datasets. Like any
classification technology that infers personal information (e.g. speaker
detection), such technologies come with broad potential risks and benefits
depending on the downstream applications. For instance, with improved
performance of machine learning models, privacy of personal information might
either be at increased risk, or better protected depending on how the
technology is used. As another, particularly relevant, example, our own work
in this area is driven to support our medical collaborations with health care
practitioners, where there is a great need and value to mitigate the effects
of distributional shift between training data and the test data, to de-bias
the system for more effective diagnostic-support use.

## Dataset Details

Voxceleb2 Dataset: In this work, We pre-trained the MAE models using the
Voxceleb2 dataset [Chung et al., 2018]. Voxceleb2 dataset contains 1.1M video
clips collected from over 6000 celebrities, extracted from 150K unique videos
uploaded to Youtube. Average duration of the video clips is around 8 seconds.
For this work, we only considered video clips with at least 3 seconds in
duration, resulting in a total of 1M video clips to pre-train the models.

Voice Cloning ToolKit (VCTK) Dataset: The VCTK dataset [Yamagishi et al.,
2019] consists of approximately 42,000 clean audio clips uttered by 109 native
English speakers with various accents. Each speaker reads out about 400
sentences. The average length of each audio clip is around 4 seconds and the
total duration of the audio clips is approximately 44 hours. In this paper, we
use the VCTK dataset for the task of speaker identification. We randomly split
the utterances spoken by each speaker into 3 splits: 80% training set, 10%
validation set and remaining 10% as test set. We have a total of 33662
training samples, 4206 validation samples and 4206 test samples.

Crowd-Sourced Emotional Multimodal Actors Dataset (CREMA-D): CREMA-D [Cao et
al., 2014] is a crowd sourced audiovisual dataset consisting of emotive
utterances collected from 91 speakers (48 male actors and 43 female
actresses). Actors spoke from a group of twelve sentences. In this work, we
used data corresponding to four emotional categories: anger, happy, sad and
neutral, resulting in 4900 utterances. Total durations of the utterances is
3.4 hours with an average length of 2.5 seconds.

Interactive Emotional Dyadic Motion Capture (IEMOCAP): IEMOCAP Busso et al.
[2008] consists of scripted and improvised dialogues by 10 speakers. It is
composed of 5 recording sessions, each including speech from one male speaker
and one female speaker. In this work, we used 4 emotional classes: anger,
happy, sad and neutral. Following previous works [Fayek et al., 2017, Pepino
et al., 2021], we relabeled excitement samples as happy. This resulted in a
total of 5531 utterances with a total duration of 7 hours.

Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): RAVDESS
dataset consists of audio-visual recordings from twelve male and twelve female
actors with North American accent reciting English sentences. In this work, we
used utterances with 4 emotional classes (anger, happy, sad and neutral)
resulting in 672 utterances with a total duration of 0.7 hours.

Toronto Emotional Speech Set (TESS): TESS [Dupuis and Pichora-Fuller, 2011]
dataset consists of 200 target words spoken in the carrier phrase “Say the
word ___” by two actresses, aged 26 and 64, in different emotive states. In
this work, we used utterances spoken in 4 different emotions i.e., anger,
happy, sad and neutral. We have a total of 1600 utterances, 800 by each
actress, with a total duration of 0.92 hours and an average length of 2.1
seconds.

Speech Commands V2 (SPCV2) Dataset: The Speech Command V2 [Warden, 2018]
dataset consists of 105,829 1-second recordings of 35 common speech commands.
In [Warden, 2018], the 35 commands are segregated into 11 different classes –
10 common commands are selected as 10 different classes and the remaining 25
commands are considered as one class. The silence audio samples were created
from background noise sounds, which are not used as a class in SPCV2. The
training/validation/testing set has 84,843/9,981/11,005 1-second recordings,
respectively.

Additional Rationale for Dataset Choices Our reasons for choosing the datasets
VCTK, CREMA-D and speech commands for analysis are: 1) Due to the vast
diversity of speakers, both male and female speakers. 2) Clean speech:
Recorded in a soundproof environment. For the evaluation of TTT, we wanted the
original recordings to contain as little background noise as possible. This is
particularly important for the evaluation of the systems in presence of
different background noises which should not be present in the training data.

## Batch Processing in TTT

Figure 5 shows the timing analysis for processing a batch of test samples
using (1) TTT (process single sample at a time), (2) TTT-Batch: Process a
batch of test samples using the approach illustrated in Figure 4. As shown in
Figure 5, as the batch size increases, the throughput of TTT-Batch improves.

Figure 5: Timing analysis of TTT-Bias Fine-tuning on Batch of Test Samples.
Here we provide the timing analysis of the approach shown in Figure 4 to
process a batch of test samples. We vary the batch size of the test samples
and compare the time taken by TTT (process only one sample at a time) and TTT-
Batch (process batch of test samples simultaneously. We show that the
difference in time taken by TTT-Batch and TTT increases as the batch size
increases). For the purpose of this illustration, we use 32 augmentations for
each TTT step

## Results on Speech Commands Dataset

In Table 6, we compare the performance of non-TTT techniques (Linear, Fine-
tune) with TTT-based techniques (TTT-Full, TTT-Bias) using speech commands
dataset for the task of low-vocabulary speech recognition. Results show that
TTT techniques outperform non-TTT techniques under different distributional
shifts. Also in TTT-based techniques, the Bias fine-tuning performs better
(comparable) to full fine-tuning across all the distributional shifts.

Table 6: Results on Speech Commands dataset for the task of low-vocabulary speech recognition. performance in terms of Accuracy (in % averaged over 3 random seeds) when tested with samples corrupted with different noises. All noises added at 0dB SNR. Results show that the TTT-based techniques significantly improves the performance over non-TTT techniques under different distributional shifts. Bias fine-tuning performs better than full-fine-tuning across all distributional shifts. |  Clean |  AC |  Babble |  Kitchen |  Living |  Park |  Restaurant |  Reverb |  Traffic |  Typing
---|---|---|---|---|---|---|---|---|---|---
Linear | 92.3 | 51.4 | 30.9 | 54.2 | 35.8 | 50.2 | 37.8 | 40.7 | 56.1 | 53.8
Fine-tune | 95.2 | 50.9 | 29.3 | 53.6 | 34.2 | 49.3 | 38.3 | 39.2 | 55.4 | 54.5
TTT-Full | 95.4 | 62.2 | 37.4 | 63.3 | 45.5 | 58.3 | 45.5 | 55.1 | 63.7 | 64.1
TTT-Bias | 94.8 | 63.7 | 39.2 | 64.4 | 46.4 | 60.4 | 47.8 | 57.2 | 66.4 | 67.3

## Comparison with Supervised Methods

In this section, we will compare the performance of MAE for speech and MAE +
TTT with models such as CNN, LSTM and CNN-LSTM, trained using supervised
techniques from scratch. CNN model consists of 3 convolutional layers with
filters (256, 128, 128) followed by a feed-forward layer. LSTM network
consists of 2 LSTM layers each with 128 ReLU units followed by a feed-forward
layer. CNN-LSTM model consists of 3 convolutional layers followed by a single
LSTM layer consisting of 64 units. All the supervised models (CNN, LSTM and
CHNN-LSTM) are trained using Mel-spectrograms. Same approach as explained in
Section 4 is followed to extract the Mel-spectrograms. Tables 7 and 8 provide
the compare supervised techniques (CNN, LSTM, CNN-LSTM) with pre-trained
models (MAE (Fine-tune), TTT-Full and TTT-Bias) for CREMA-D and VCTK datasets,
respectively. Pre-trained models perform better than the supervised models
trained from scratch on clean speech and speech corrupted with different
background noises. Similar to MAE (Fine-tune), supervised techniques show huge
degradation in performance when tested with speech corrupted with different
background noises. TTT-based techniques improve performance over supervised
and pre-trained models when tested with noisy speech.

Table 7: Comparison with supervised techniques (CNN, LSTM and CNN-LSTM). Emotion recognition results in terms of UAR (in % averaged over 3 random seeds) using the models trained on Crema dataset and tested with samples corrupted with different noises. All noises added at 0dB SNR. Pre-trained models perform better than supervised models trained from scratch on both, clean and speech corrupted with background noises. When tested with speech degraded with background noise, TTT-based techniques outperform both supervised and pre-trained models | Clean | Babble | Kitchen |  | Living
---
Room
Park | Restaurant | Traffic | Typing
CNN | 59.8 | 29.5 | 35.6 | 30.8 | 44.6 | 32.5 | 40.7 | 41.0
LSTM | 58.7 | 30.2 | 34.9 | 31.1 | 44.9 | 31.8 | 39.9 | 41.2
CNN-LSTM | 60.8 | 31.6 | 35.8 | 30.9 | 45.3 | 33.2 | 40.6 | 41.4
MAE (Fine-tune) | 61.5 | 33.6 | 43.7 | 34.2 | 48.4 | 33.5 | 44.6 | 43.7
TTT-Full | 66.8 | 40.8 | 64.1 | 38.3 | 58.9 | 41.7 | 61.7 | 62.1
TTT-Bias | 67.3 | 42.8 | 65.9 | 40.3 | 62.1 | 43.9 | 66.8 | 64.6
Table 8: Comparison with supervised techniques (CNN, LSTM and CNN-LSTM). Speaker identification results in terms of Accuracy (in % averaged over 3 random seeds) using the models trained on VCTK dataset and tested with samples corrupted with different noises. All noises added at 0dB SNR. Pre-trained models perform better than supervised models trained from scratch on both, clean and speech corrupted with background noises. When tested with speech degraded with background noise, TTT-based techniques outperform both supervised and pre-trained models | Clean | Babble | Kitchen |  | Living
---
Room
Park | Restaurant | Traffic | Typing
CNN | 90.5 | 7.9 | 6.6 | 9.2 | 11.5 | 8.6 | 24.6 | 31.8
LSTM | 91.1 | 8.2 | 6.5 | 8.8 | 11.8 | 8.7 | 24.8 | 32.2
CNN-LSTM | 91.6 | 8.4 | 6.8 | 9.1 | 11.7 | 8.7 | 25.3 | 32.5
MAE (Fine-tune) | 92.1 | 8.8 | 7.0 | 9.5 | 13.6 | 10.4 | 28.4 | 35.6
TTT-Full | 90.3 | 15.7 | 17.2 | 15.1 | 22.1 | 16.1 | 52.9 | 55.2
TTT-Bias | 91.4 | 17.3 | 21.8 | 16.9 | 25.6 | 19.4 | 57.2 | 57.7

## Speaking Style Variation: IEMOCAP Dataset

Table 9 shows the effect of speaking style variation on the performance of
models trained with IEMOCAP dataset (See Table 4(a) for effect of speaking
style variation on CREMA-D dataset). Here we train on IEMOCAP (emotive
conversations enacted in American English) and test with samples from CREMA-D
(emotively acted utterances in American English), RAVDESS (emotively acted
utterances in North American English) and TESS (emotively acted utterances in
Canadian English). TTT-based techniques achieve significantly better
performance compared to non-TTT methods for speaking style variations.

Table 9: Speaking style variation based distribution shift on IEMOCAP dataset: Emotion recognition results in terms of UAR (in % averaged over 3 random seeds) using models trained on IEMOCAP dataset and tested with samples from other datasets. IEMOCAP consists of conversational speech recording whereas CREMA-D consists of read speech with enacted emotions, RAVDESS is emotive read speech collected from speakers with north American accent and Tess dataset consists of read speech recordings collected from Canadian speakers. TTT-based techniques (TTT-Full and TTT-Bias) outperform non-TTT techniques under the distributional shift caused due to speaking style variation | IEMOCAP | CREMA | RAVDESS | TESS
---|---|---|---|---
Linear | 56.3 | 37.4 | 31.3 | 35.6
Finetune | 60.8 | 40.5 | 30.6 | 32.9
TTT-Full | 63.3 | 49.3 | 42.1 | 44.6
TTT-Bias | 63.2 | 51.4 | 42.8 | 45.5

## Distributional Shifts in Speech

In Figures 6 and 7, we show the spectrograms of the clean speech distorted
with different background noises, added at 0 dB SNR.

Figure 6: Mel-spectrograms of speech with distributional shifts caused due to
different background noises added at 0 dB SNR

Figure 7: Mel-spectrograms of speech with distributional shifts caused due to
different background noises added at 0 dB SNR

Table 1: Performance comparison of different variants of TTT. Additional adapter units are trained in two ways (1) Adap-1: train adapter units during both train-time training and during TTT; (2) Adap-2: train adapter units only during TTT. Adap-2 performs better than Adap-1 for most of the distribution shifts. Bias fine-tuning performs better than adapter units (Adap-1 and Adap-2) for most of the distribution shifts | Speaker Identification | Emotion Recognition
---|---|---
| Full | Adap-1 | Adap-2 | Bias | Full | Adap-1 | Adap-2 | Bias
Clean | 90.3 | 90.2 | 90.7 | 91.4 | 66.8 | 66.0 | 67.1 | 67.3
Babble | 15.7 | 14.7 | 16.5 | 17.3 | 40.8 | 38.5 | 40.3 | 42.8
Kitchen | 17.2 | 18.2 | 20.5 | 21.8 | 63.5 | 63.1 | 64.2 | 65.9
Living Room | 15.1 | 14.5 | 16.0 | 16.9 | 38.3 | 37.1 | 38.5 | 40.3
Park | 22.1 | 17.8 | 21.9 | 25.6 | 58.9 | 56.3 | 57.7 | 62.1
Restaurant | 16.1 | 14.7 | 17.6 | 19.4 | 41.7 | 39.2 | 42.7 | 43.9
Reverberation | 34.7 | 33.1 | 35.4 | 37.9 | 52.3 | 49.1 | 51.8 | 54.7
Squeaky-Chair | 60.3 | 60.1 | 62.3 | 62.1 | 63.7 | 63.4 | 65.3 | 65.5
Traffic | 52.9 | 47.2 | 50.6 | 57.2 | 61.7 | 59.2 | 60.8 | 66.8
Typing | 55.2 | 55.1 | 57.8 | 57.7 | 62.1 | 61.9 | 64.2 | 64.6
Table 2: Performance comparison of TTT-Bias (Bias) with other SSL techniques. TTT with bias fine-tuning performs better than other SSL techniques without TTT. W2V-2.0, M. Jay and USAT refer to the models Wav2vec-2.0, Mockingjay and UniSpeech-SAT, respectively. | Speaker Identification | Emotion Recognition
---|---|---
| W2V-2.0 | M.Jay | Hubert | USAT | Bias | W2V-2.0 | M.Jay | Hubert | USAT | Bias
clean | 92.0 | 87.8 | 92.3 | 93.8 | 91.4 | 61.3 | 57.4 | 61.9 | 63.1 | 67.3
Babble | 9.2 | 8.6 | 9.5 | 10.9 | 17.3 | 34.1 | 32.7 | 34.9 | 36.1 | 42.8
Kitchen | 7.3 | 6.8 | 7.4 | 9.2 | 21.8 | 45.1 | 43.4 | 44.8 | 47.5 | 65.9
Living Room | 9.7 | 9.4 | 10.1 | 12.4 | 16.9 | 34.7 | 33.5 | 35.1 | 36.4 | 40.3
Park | 14.2 | 13.1 | 14.8 | 16.3 | 25.6 | 49.6 | 47.9 | 50.2 | 52.7 | 62.1
Restaurant | 10.6 | 9.2 | 10.9 | 12.0 | 19.4 | 34.5 | 32.2 | 35.2 | 37.4 | 43.9
Reverberation | 26.4 | 24.8 | 26.2 | 27.3 | 37.9 | 37.6 | 35.4 | 38.1 | 40.1 | 54.7
Squeaky-Chair | 36.2 | 35.8 | 36.9 | 38.5 | 62.1 | 42.8 | 40.9 | 43.0 | 44.6 | 65.5
Traffic | 29.1 | 28.0 | 28.8 | 30.4 | 57.2 | 44.7 | 43.5 | 45.1 | 45.4 | 66.8
Typing | 35.5 | 34.9 | 35.8 | 37.2 | 57.7 | 44.3 | 43.2 | 44.8 | 45.2 | 64.6
Table 3: Effect of masking ratio during TTT. Speaker identification results in terms of Accuracy using the models trained on VCTK dataset and tested with samples corrupted with different noises. During TTT, masking ratio of 75% performs better than (comparable to) other ratios. We chose 75% as it reduces the computational cost and achieves top performance Ratio | Babble | Kitchen |  | Living
---
Room
Park | Restaurant | Traffic | Typing | Avg.
25% | 15.3 | 18.9 | 14.9 | 21.8 | 18.1 | 51.4 | 52.5 | 27.55
50% | 17.1 | 21.2 | 17.0 | 25.2 | 19.6 | 56.5 | 57.4 | 30.54
75% | 17.3 | 21.8 | 16.9 | 25.6 | 19.4 | 57.2 | 57.7 | 30.85
90% | 14.2 | 16.2 | 13.8 | 18.7 | 15.1 | 48.4 | 49.3 | 25.1