# Speed Co-Augmentation for Unsupervised Audio-Visual Pre-training

Jiangliu Wang
The Chinese University of Hong Kong
Jianbo Jiao
University of Birmingham
Yibing Song
Fudan University
Stephen James
Dyson Robot Learning Lab
Zhan Tong
Tencent AI Lab
Chongjian Ge
The University of Hong Kong
Pieter Abbeel
UC Berkeley
Yun-Hui Liu
The Chinese University of Hong Kong

###### Abstract

This work aims to improve unsupervised audio-visual pre-training. Inspired by
the efficacy of data augmentation in visual contrastive learning, we propose a
novel speed co-augmentation method that randomly changes the playback speeds
of both audio and video data. Despite its simplicity, the speed co-
augmentation method possesses two compelling attributes: (1) it increases the
diversity of audio-visual pairs and doubles the size of negative pairs,
resulting in a significant enhancement in the learned representations, and (2)
it changes the strict correlation between audio-visual pairs but introduces a
partial relationship between the augmented pairs, which is modeled by our
proposed SoftInfoNCE loss111Our study [6] also validates the effectiveness of
the proposed “SoftInfoNCE loss” in single-modality contrastive learning. to
further boost the performance. Experimental results show that the proposed
method significantly improves the learned representations when compared to
vanilla audio-visual contrastive learning.

##  1 Introduction

Audio-visual contrastive learning [14, 17] for unsupervised pre-training has
received growing attention due to the observation that video content is
usually accompanied by audio signals. The alignment of signals across audio
and video forms a natural correspondence to benefit contrastive learning.
Under this framework, quite a few existing approaches [13, 17] focus on
achieving better discrimination of positive and negative pairs to improve
audio-visual representation learning. While promising results have been
achieved, most works [12, 13] apply data augmentations to each modality
individually, which may potentially limit the diversity of the generated data
views and restrict the potential of augmentation for contrastive learning.

Figure 1: An intuitive example of semantic shift after applying speed co-
augmentation on audio-visual pairs. Top: when speeding up video data, the
semantic meaning of the content doesn’t change drastically. Bottom: when
speeding up audio data, the semantic meaning of the content changed
drastically.

In this work, we propose a novel technique termed "speed co-augmentation" for
unsupervised audio-visual pre-training, which involves modifying the playback
speeds of both audio and visual data simultaneously. The speed co-augmentation
method enhances the diversity of audio-visual pairs and doubles the number of
negative pairs during training, which has been shown to be a crucial aspect of
contrastive learning [5]. Our experimental results demonstrate that this
simple co-augmentation method yields a significant performance improvement of
10.0% over the baseline audio-visual contrastive learning approach on the
HMDB51 [10] dataset.

Meanwhile, it was observed that after the speed co-augmentation, the audio and
video pairs derived from the same clip are no longer strictly positively
related, as is commonly assumed. As an intuitive special example (Fig. 1), a
sped-up cow still visually looks like a cow, but a sped-up cow may auditorily
sound like a cat. To generalize, we posit that there exists a partial
relationship between the augmented audio-visual pairs, which is influenced by
the degree of speed augmentation applied. To capture this relationship, we
introduce a cross-affinity module that automatically learns the audio-visual
correlations across different views. The resulting learned correlations
quantitatively measure the audio-visual consistency and are employed for
computations of SoftInfoNCE loss, leading to a further performance boost.
Combining the proposed speed augmentation and the cross-affinity module, we
present a Speed-augmented visual-audio Contrastive Learning framework, which
we call _SvaCLR_.

##  2 Method

Our target is to train video and audio encoders via unsupervised contrastive
learning. Given an aligned pair $(v,a)$, we apply speed-up augmentations on
both audio $a$ and video $v$ data to synthesize two additional views (_i.e_.,
$\widetilde{v}$ and $\widetilde{a}$). These audio and video samples are then
fed into the audio and video encoders $f(\cdot)$ and $g(\cdot)$ to extract
representations $y$. We then project the video and audio representations
separately via projectors $h_{v}(\cdot)$ and $h_{a}(\cdot)$. The projected
embeddings $z$ are then utilized to compute the contrastive InfoNCE loss [15].
In parallel, we introduce a cross-affinity module to model the audio-visual
embedding correlations. The modeled correlations are used to reweigh the
InfoNCE loss, _i.e_., the proposed SoftInfoNCE loss, when learning audio-
visual representations. In the following, we first introduce the speed-up
augmentation with the vanilla InfoNCE loss and then introduce the cross-
affinity module to re-weigh the InfoNCE loss (_i.e_., SoftInfoNCE loss).

###  2.1 Speed co-augmentation

For speed co-augmentation, we use a speed library to diversify training data
pairs. We use $\mathcal{T}$ to represent the speed co-augmentation set in
which the maximum speed is denoted by $S$. Each time, two speed augmentation
factors for the audio and video data are selected randomly from $\mathcal{T}$
and are applied to each data, respectively. In practice, the proposed speed
co-augmentation is implemented by applying different sampling rates of the
audio and video samples.

Before computing the contrastive InfoNCE loss [15], we project the video and
audio representations separately via projectors. We use one video projector
$h_{v}(\cdot)$ to connect the video encoder and use one audio projector
$h_{a}(\cdot)$ to connect the audio encoder. The projected representations are
then utilized to compute the contrastive InfoNCE loss as follows:

$L(i,j)=\frac{\exp({z_{i}\cdot z_{j}}~{}/~{}{\eta})}{\exp({z_{i}\cdot
z_{j}}~{}/~{}{\eta})+\sum\limits_{j=1\atop j\neq i}^{N}\exp({z_{i}\cdot
z_{j}}~{}/~{}{\eta})}$ (1)

where $z_{i}=h_{a}(y_{i})$ is the audio projection, $z_{j}=h_{v}(y_{j})$ is
the video projection, and $\eta$ is a constant temperature value. The dot
product measures the similarity between the projected audio and video
representations. For the input audio $a_{i}$, the summation term is computed
by utilizing all the video clips $v_{j}$, as long as $a_{i}$ and $v_{j}$ are
from different samples (_i.e_., unpaired).

Figure 2: The proposed cross-affinity module for SoftInfoNCE loss computation.
The cross-modality attention module takes video and audio representations as
input where there are co-augmented audio-visual data. The output is a cross-
modality affinity matrix shown on the right. Each element in this matrix
represents the correlations between audio and video for each input signal.

###  2.2 Cross-affinity module

We propose a cross-affinity module to measure the correlations between the
augmented video and audio representations. Fig. 2 illustrates the proposed
module. Given the audio embedding $y_{i}$ and the video embedding $y_{j}$, the
cross-modality attention
$\lambda(a_{i}^{\mathcal{\tau}_{1}},v_{j}^{\mathcal{\tau}_{2}})$ can be
computed as follows:

$\lambda(a_{i}^{\mathcal{\tau}_{1}},v_{j}^{\mathcal{\tau}_{2}})=\operatorname{softmax}\left[{l(y_{i})\times
l(y_{j}^{\textrm{T}})}\right],$ (2)

where $l(\cdot)$ is a mapping with learnable parameters. The projected video
and audio embeddings are correlated via the matrix multiplication operation.
In practice, three different mapping function are examined, including identity
mapping, linear mapping and nonlinear mapping, from those we find the identity
mapping achieves the best results. We speculate this is because heavier
mapping could deteriorate the ability of encoders to learn general
representations.

We compute the cross-modality attention in Eq. 2 for one co-augmented audio-
visual view. The cross-modality affinity can be formulated as a two-by-two
matrix (as shown in Fig. 2 right). Each element in this matrix represents the
correlation between the speed-augmented audio and video views. By using these
elements, we reweigh the contributions of each co-augmented audio-visual view
when computing the contrastive loss.

###  2.3 Training with SoftInfoNCE

Following [17, 12], we use a 9-layered 2D ResNet [7] as the audio encoder and
R(2+1)D-18 [19] as the video encoder. The projector is a two-layered
multilayer perceptron (MLP). The training process is end-to-end, without using
a two-stage setting as in previous works [13, 14].

Given a batch of audio-visual pairs $\mathcal{A}$ and $\mathcal{V}$, where
both $\mathcal{A}$ and $\mathcal{V}$ contain $N$ samples, we denote the speed-
up augmentation set as $\mathcal{T}$, from which we can sample augmentations
$\mathcal{\tau}\sim p(\mathcal{T})$. The encoders and projectors are trained
with the following SoftInfoNCE loss:

$\mathcal{L}(f,g,\mathcal{A},\mathcal{V})=\mathbb{E}_{(\mathcal{\tau}_{1},\mathcal{\tau}_{2})\sim
p(\mathcal{T})}\left[\frac{1}{N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{N}\lambda(a_{i}^{\mathcal{\tau}_{1}},v_{j}^{\mathcal{\tau}_{2}})\cdot
L(g(a_{i}^{\mathcal{\tau}_{1}}),f(v_{j}^{\mathcal{\tau}_{2}}))\right]$

(3)

where $L(\cdot,\cdot)$ is the contrastive InfoNCE loss function as illustrated
in Eq. 1, $a_{i}\in\mathcal{A}$, and $v_{j}\in\mathcal{V}$. The cross-modality
attention $\lambda(\cdot,\cdot)$ takes the audio and video signal as input and
measures their correlations. The output correlation value further reweighs the
contrastive loss during the training process consequently.

##  3 Experiments

###  3.1 Implementation details

We use Kinetics-Sounds (K-Sounds) [2], Kinetics-400 (K400) [8], and VGGSound
[4] as the pre-training datasets to evaluate the effectiveness of our proposed
approach.

For pre-training, we use Stochastic Gradient Descent (SGD) as the optimizer.
We use 10 epochs to warm up the learning rate from $64\times 10^{-3}$ to
$64\times 10^{-2}$ and then using a cosine learning rate decay to $10\times
10^{-2}$ in the remaining 90 epochs. Training is done on 64 V100 GPUs with a
mini-bath size of 8 on each, resulting in total batch size of 512. The total
training time is around 30 hours for 100 epochs.

For fine-tuning, we follow the fine-tuning setting of GDT [17]. SGD is used as
the optimizer and the initial learning rate is set to $2.5\times 10^{-3}$,
where it is warmed up to $2\times 10^{-2}$ in the first two epochs, and
decreased by $5\times 10^{-2}$ at 6 and 10 epochs. Training is stopped at 12
epochs.

Table 1: Effectiveness of the proposed speed co-augmentation and cross-
affinity module. Speed $s=[a,b]$ represents that the lower bound speed is $a$
while the upper bound is $b$.

Pre-training Experimental Setup Downstream Acc. speed Re-weight Speed Loss
HMDB51 K-sounds@1 ✗ ✗ - InfoNCE 54.2 2.6 ✓ ✗ $s=[1,2]$ InfoNCE 63.8 (+9.6) 3.1
(+0.5) ✓ ✗ $s=[1,4]$ InfoNCE 65.1 (+10.9) 3.5 (+0.9) ✓ ✗ $s=[1,6]$ InfoNCE
64.2 (+10.0) 3.2 (+0.6) ✓ ✓ $s=[1,4]$ SoftInfoNCE 66.1 (+11.9) 4.6 (+2.0)

###  3.2 Ablation studies

Speed co-augmentation. We investigate different augmentations for audio and
video data from the same speed distribution in Table 1. It can be seen that:
(1) compared to the no speed-up augmentation, using only two speed candidates
can already significantly improve the action recognition performance from
54.2% to 63.8%. This validates the effectiveness of the proposed speed-up
augmentation. (2) When audio-video pairs are sped up from the same
distribution, best action recognition accuracy is achieved when $s=[1,4]$.
Both $s=[1,2]$ and $s=[1,6]$ settings perform inferior than the $s=[1,4]$
setting. We suspect that this is because compared to $s=[1,2]$, using
$s=[1,4]$ provides more view synthesis while $s=[1,6]$ could be a bit
difficult for the network to learn useful semantic representations.

Cross-affinity module. Based on the best speed augmentation setting
$s_{a}=s_{v}=[1,4]$, we investigate the effectiveness of the proposed cross-
affinity module in Table 1. It can be seen that with the proposed cross-
affinity module and the SoftInfoNCE loss design, the performances can be
further improved significantly (_e.g_., +11.9% for HMDB51). To better
understand what the cross-affinity module learns, we visualize the visual-
audio pairs with the speed augmentation and their corresponding relationship
weights in Fig. 3. It can be seen that videos with manifested audio signals
are more sensitive to the proposed speed augmentation, where the weight
between the video and augmented audio is relatively small after the speed
transformation.

Figure 3: Examples of the visual-audio pairs with speed co-augmentation and
their partial relationship weights. For each sample from left to right: one
frame from the video clip, spectrogram of the corresponding audio, spectrogram
of the augmented audio with speed 2, and spectrogram of the augmented audio
with speed 4. $\lambda$ reflects the degree of correlations learned by our
proposed cross-affinity module.  Table 2:  Comparison with previous approaches
on UCF101 [18] and HMDB51 [10] datasets.

Method Pre-training Experimental Setup Downstream Acc. Architecture Dataset
(duration) Resolution HMDB51 UCF101 ACC [12] R3D-18 K-Sounds (2d) $224\times
224$ 40.6 77.2 SvaCLR (Ours) R3D-18 K-Sounds (2d) $128\times 128$ 54.9 86.4
$L^{3}$-Net [2] VGG-16 K400 (28d) - 40.2 72.3 PEMT [11] SlowFast K400 (28d)
$128\times 128$ - 85.2 GDT [17] R(2+1)D-18 K400 (28d) $128\times 128$ 62.3
90.9 SvaCLR (Ours) R(2+1)D-18 K400 (28d) $128\times 128$ 66.1 91.5 SvaCLR
(Ours) R(2+1)D-18 VGGSound (23d) $128\times 128$ 67.2 92.0 Multisensory [16]
R3D-18 K400 (28d) $224\times 224$ - 82.1 SeLaVi [3] R(2+1)D-18 K400 (28d)
$224\times 224$ 47.1 84.2 XDC [1] R(2+1)D-18 K400 (28d) $224\times 224$ 52.6
86.2 AVTS [9] MC3-18 K400 (28d) $224\times 224$ 56.9 85.8 STiCA [17]
R(2+1)D-18 K400 (28d) $224\times 224$ 60.5 - AVID [14] R(2+1)D-18 K400 (28d)
$224\times 224$ 60.8 87.5 ACC [12] R3D-18 K400 (28d) $224\times 224$ 61.8 90.2
SvaCLR (Ours) R(2+1)D-18 K400 (28d) $224\times 224$ 66.8 92.2 SvaCLR (Ours)
R(2+1)D-18 VGGSound (23d) $224\times 224$ 67.8 92.6

###  3.3 Comparison with previous works

Action recognition.

We compare our proposed SvaCLR with other approaches in Table 2. From the
results we can see that: (1) When pre-trained on a medium-scale dataset,
K-Sounds, our approach significantly outperforms previous works. We improve
performances on UCF101 and HMDB51 datasets by large margins, 9.2% and 14.3%.
This demonstrates that our proposed co-augmentation method enlarges the
diversity of the training views and benefits contrastive learning a lot. (2)
Our approach demonstrates great scalability in terms of dataset size. When
pre-trained on a large dataset K400, our approach exceeds the state-of-the-art
audio-visual representation learning approach GDT [17] by a large margin,
especially on the HMDB51 dataset, where we outperform GDT by 3.8%. Note that
GDT applies hierarchical data augmentations while we only use one-speed
augmentation. (3) Our approach also demonstrates scalability in terms of
resolution. We can further improve the performances by using a large input
size.

#### Audio-Video Retrieval.

To further evaluate the cross-modality ability of the proposed approach, we
propose to use an audio-video retrieval task on K-Sounds [2]. We compare to
audio-visual contrastive learning with vanilla InfoNCE loss and current state-
of-the-art GDT [17] in Table 3. We show that our approach achieves the best
performances on both audio-to-video retrieval task and video-to-audio
retrieval task. It is interesting to note that pre-trained on K-sounds can
achieve better performance to retrieve top-1 nearest neighbor. But its ability
to generalize to more visual-audio pairs is restricted that it performs worse
than pre-training on a larger dataset K400 to retrieve top-5, top-10, and
top-20 nearest neighbors.

Table 3: Comparison with other approaches on video-audio retrieval task.
“Baseline” represents the vanilla audio-video contrastive learning with
InfoNCE loss.

Method Pre-train Dataset Video $\rightarrow$ Audio Audio $\rightarrow$ Video
R@1 R@5 R@10 R@20 R@1 R@5 R@10 R@20 Baseline K400 2.6 13.8 26.1 44.3 3.6 13.7
25.3 41.0 GDT [17] K400 3.1 14.3 24.6 40.8 3.6 16.1 26.4 43.9 SvaCLR (Ours)
K-Sounds 5.1 14.1 25.4 42.3 4.7 16.0 27.2 43.6 SvaCLR (Ours) K400 4.6 17.2
28.3 44.6 4.2 16.4 27.6 44.3

##  4 Conclusions

We proposed a speed co-augmentation method for unsupervised audio-visual pre-
training. We observed that speed co-augmentation leads to a partial
relationship between audio-visual pairs. To combat this, we propose a cross-
affinity module, which can adaptively model the cross-modality partial
relationship and further improve performances. Extensive experimental results
show that our approach significantly improves the performances.

## References

  * [1] Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran.  Self-supervised learning by cross-modal audio-video clustering.  In NeurIPS, 2019.
  * [2] Relja Arandjelovic and Andrew Zisserman.  Look, listen and learn.  In ICCV, 2017.
  * [3] Yuki Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi.  Labelling unlabelled videos from scratch with multi-modal self-supervision.  NeurIPS, 2020.
  * [4] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman.  Vggsound: A large-scale audio-visual dataset.  In ICASSP, 2020.
  * [5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  2020\.
  * [6] Chongjian Ge, Jiangliu Wang, Zhan Tong, Shoufa Chen, Yibing Song, and Ping Luo.  Soft neighbors are positive supporters in contrastive visual representation learning.  In ICLR, 2023.
  * [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [8] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [9] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Cooperative learning of audio and video models from self-supervised synchronization.  In NeurIPS, 2018.
  * [10] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre.  Hmdb: a large video database for human motion recognition.  In ICCV, 2011.
  * [11] Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas M Breuel, Jan Kautz, and Yale Song.  Parameter efficient multimodal transformers for video representation learning.  In ICLR, 2021.
  * [12] Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song.  Active contrastive learning of audio-visual video representations.  In ICLR, 2020.
  * [13] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos.  Robust audio-visual instance discrimination.  In CVPR, 2021.
  * [14] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra.  Audio-visual instance discrimination with cross-modal agreement.  In CVPR, 2021.
  * [15] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [16] Andrew Owens and Alexei A Efros.  Audio-visual scene analysis with self-supervised multisensory features.  In ECCV, 2018.
  * [17] Mandela Patrick, Yuki M Asano, Polina Kuznetsova, Ruth Fong, João F Henriques, Geoffrey Zweig, and Andrea Vedaldi.  On compositions of transformations in contrastive self-supervised learning.  In ICCV, 2021.
  * [18] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [19] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In CVPR, 2018.


[]AvivShamsian []AvivNavon []NetaGlazer []GillHetz []JosephKeshet

# Keyword-Guided Adaptation of Automatic Speech Recognition

###### Abstract

Automatic Speech Recognition (ASR) technology has made significant progress in
recent years, providing accurate transcription across various domains.
However, some challenges remain, especially in noisy environments and
specialized jargon. In this paper, we propose a novel approach for improved
jargon word recognition by contextual biasing Whisper-based models. We employ
a keyword spotting model that leverages the Whisper encoder representation to
dynamically generate prompts for guiding the decoder during the transcription
process. We introduce two approaches to effectively steer the decoder towards
these prompts: KG-Whisper, which is aimed at fine-tuning the Whisper decoder,
and KG-Whisper-PT, which learns a prompt prefix. Our results show a
significant improvement in the recognition accuracy of specified keywords and
in reducing the overall word error rates. Specifically, in unseen language
generalization, we demonstrate an average WER improvement of $5.1\%$ over
Whisper.

###### keywords:

automatic speech recognition, keyword spotting, contextual biasing, Whisper

##  1 Introduction

Automatic speech recognition (ASR) capabilities have rapidly progressed in
recent years through advancements in deep learning models trained on massive
datasets of speech. Models like HuBERT [1] and wav2vec [2] utilize self-
supervised pre-training techniques to learn representations of acoustic and
linguistic patterns from hundreds of thousands of hours of diverse audio data.
Although such models have shown great potential in producing rich
representations useful for transcription, their lack of a high-quality decoder
limits their robustness and generalization. Recently, [3] alleviated this
limitation by presenting a new encoder-decoder transformer architecture named
Whisper. Whisper was trained on a massive multilingual corpus of transcribed
web audio data comprising 680,000 hours of speech.

However, Whisper and other state-of-the-art ASR models may suffer from
performance degradation when applied to real-world datasets presenting
numerous challenges [4]. For instance, in industrial surroundings where heavy
machinery operates continuously, background noise can reach high levels. This
acoustic environment makes it difficult for ASR systems to accurately
transcribe spoken commands or safety alerts issued by workers. Similarly, on
public transportation like a crowded bus or subway train, ambient noise from
engines or conversations can interfere with passenger attempts to a voice-
activated information system. Additionally, domain-specific utterances and
terminology, pose unique challenges for ASR systems, as they require the
identification of specialized vocabulary and language patterns. For instance,
in the medical domain, physicians detailing medical conditions or treatment
plans, and in legal contexts, where lawyers and judges employ legal jargon,
case citations, and procedural language uncommon in everyday conversation,
demonstrate such challenges. To develop speech recognition systems that are
useful in real-world situations, it is essential to overcome these practical
hurdles.

One approach for alleviating these challenges is utilizing domain-specific or
personalized context for biasing transcriptions using prior knowledge. A
common approach is injecting context into beam search or using shallow fusion
[5, 6, 7, 8], deep fusion [9, 10, 11, 12, 13], or a combination of deep and
shallow biasing [14, 15, 16, 17]. Recently, [18] proposed adapting Whisper
into specific domains using domain prompts which shows substantial reductions
in word error rates. However, this method requires finetuning with a large
amount of data and is limited in terms of the prior knowledge and terminology
one can provide. Prompting and prompt tuning Whisper was also shown beneficial
for adapting to novel tasks, such as audio-visual speech recognition and code
switching [19], and target speaker ASR [20]. However, the challenge remains to
design an efficient and effective way for contextually biasing Whisper using
domain-specific and personalized terminology.

(a) KG-Whisper - decoder finetuning.

(b) KG-Whisper-PT - prompt tuning.

Figure 1: An illustration of (a) KG-Whisper - the decoder receives keywords
during fine-tuning, while the encoder module remains frozen. (b) KG-Whisper-PT
- the entire Whisper model parameters are frozen and only a small number of
prompt tokens are tuned.

To address this issue, we propose a novel approach for contextual biasing. We
utilize recent advancements in open vocabulary keyword spotting [21, 22, 23]
to guide the ASR predictions using prior domain knowledge as a source of
contextual information. Our approach first identifies domain-specific and
personalized jargon using AdaKWS [24], a KWS system built on top of Whisper's
acoustic encoder. Next, the identified keywords are used to prompt the decoder
to encourage their incorporation into the transcribed text. We propose two
approaches for finetune Whisper for using keyword prompts. The first, termed
KG-Whisper for keyword-guided Whisper, finetunes the entire set of decoder
parameters. The second and more efficient approach based on prompt tuning
[25], termed KG-Whisper-PT, achieves on-par or superior generalization results
using only ${\sim}15$K trainable parameters. This approach preserves the
original pre-trained Whisper parameters, mitigating potential degradation
resulting from fine-tuning over specific datasets and enhancing the model's
ability to generalize to novel datasets and domains. Most similar to our
approach is the concurrent work [26]. However, the paper provides only small-
scale finetune experiments on a limited number of languages, or employ spoken
form prompts.

Throughout extensive evaluation, we empirically demonstrate that KG-Whisper
and KG-Whisper-PT constantly outperform natural baselines. Furthermore, we
evaluate the generalization of the proposed method to novel domains and
challenging acoustic settings. Finally, we evaluate the generalization
performance to novel languages not seen during training.

##  2 Method

Our goal is to reduce the overall Word Error Rate (WER) of the final
prediction and increase the recall over domain-specific (jargon) terms. To
achieve this, we integrate the predictions of a Keyword Spotting (KWS) model
into an ASR system, guiding the ASR predictions towards the set of detected
keywords.

We first present the notations and learning setup. Let $\mathcal{X}^{T}$
denote the domain of sequences of $T$ speech frames. Let $\mathcal{V}$ denote
a dictionary of tokens (sub-word units), and $\mathcal{V}^{L}$ the domain of
all token sequences containing $L$ tokens (sub-word units). Each token is
represented by a $D$-dimensional vector.

We utilize Whisper [3], a state-of-the-art transformer-based ASR. Its encoder
converts a speech sequence into a meaningful sequential representation, taking
advantage of the self-attention mechanism. Given a speech input sequence
$\mathbf{x}\in\mathcal{X}^{T}$, consisting of $T$ speech frames, the encoder
function $\mathbf{u}=f^{e}_{\phi}(\mathbf{x})$ produces an output
$\mathbf{u}\in\mathbb{R}^{F}$. Here, $F$ represents the dimension of the
acoustic representation, and $\phi$ denotes the encoder parameters.

The transformer decoder function takes the encoder representation
$\mathbf{u}$, a prompt $\mathbf{p}\in\mathcal{V}^{L}$, and the previous
tokens, $\mathbf{t}^{i-1}$. It then predicts the next token
$\hat{t}^{i}=f^{d}_{\psi}(\mathbf{u},\mathbf{p},\mathbf{t}^{i-1})$, where
$\hat{t}^{i}\in\mathcal{V}$ and $\psi$ represents the set of decoder
parameters. Given a training set of $M$ examples,
$\mathcal{S}=\\{(\mathbf{x}_{j},\mathbf{p}_{j},\mathbf{t}_{j})\\}_{j=1}^{M}$,
the ASR is trained to minimize the empirical loss:

$\min_{\\{\phi,\psi\\}}\sum_{j}\sum_{i}\mathcal{L}^{\text{CE}}\big{(}f^{d}_{\psi}(f^{e}_{\phi}(\mathbf{x}_{j}),\mathbf{p}_{j},\mathbf{t}^{i-1}_{j}),t^{i}_{j}\big{)}~{},$
(1)

where $\mathcal{L}^{\text{CE}}(\hat{t},t)$ is the cross-entropy loss between
the a predicted token, $\hat{t}$, and the correct one, $t$.

Expanding upon Whisper's architecture and optimization, our goal is to guide
its predictions toward specific keywords (which can represent out-of-
vocabulary or rare words), while preserving the overall performance of the
rest of the words in the language. Our approach is based on utilizing
decisions from an open vocabulary KWS model [24].

(a) KG-Whisper

(b) KG-Whisper-PT

Figure 2: Visualization of the cross-attention weights for KG-Whisper and KG-
Whisper-PT. We illustrate how the model's attention is directed towards the
identified keywords (Y-axis) as it predicts them within the transcribed text
(X-axis).

Let $\mathcal{K}$ represent a set of keywords, with each keyword
$k\in\mathcal{K}$ defined as a sequence of tokens within the token vocabulary
$\mathcal{V}^{L}$. The KWS [24] employs a function
$\hat{\mathbf{y}}=f^{k}_{\theta}(\mathbf{u},\mathcal{K})$ that takes as input
the encoder representation $\mathbf{u}$ and the set of keywords $\mathcal{K}$,
and predicts a binary vector $\hat{\mathbf{y}}\in\\{0,1\\}^{|\mathcal{K}|}$.
This vector indicates the presence or absence of each keyword within the input
utterance. The KWS parameters, $\theta$, are optimized to minimize the cross-
entropy loss:

$\min_{\theta}~{}\sum_{j}\sum_{(k_{j}^{+},k_{j}^{-})\in\mathcal{K}}\mathcal{L}^{\text{CE}}\big{(}f^{k}_{\theta}(\mathbf{u}_{j},\mathcal{K})\big{)}~{}.$
(2)

The KWS prediction vector $\hat{\mathbf{y}}$ is then used to generate a prompt
that is passed to the Whisper decoder. We introduce two novel approaches for
guiding Whisper's predictions toward the identified keywords. In the first
approach, _KG-Whisper_ , we fine-tune Whisper's decoder while freezing the
encoder. In the second approach, _KG-Whisper-PT_ , we tune the prompt by
adding a learned prefix to it, while freezing the encoder the decoder. In both
approaches we keep the KWS model freezed.

###  2.1 KG-Whisper: Keyword-guided tuned-decoder

Our first approach involves fine-tuning the decoder parameters $\psi$. Here,
each training example is a tuple of a speech representation $\mathbf{u}$, a
prompt $\mathbf{p}^{\text{KWS}(\mathbf{u},\mathcal{K})}$ generated from the
KWS, the previous (correct) tokens $\mathbf{t}^{i-1}$, and the next token
$t^{i}$:

$\min_{\psi}~{}\sum_{j}\sum_{i}\mathcal{L}^{\text{CE}}\big{(}f^{d}_{\psi}(\mathbf{u}_{j},\mathbf{p}^{\text{KWS}(\mathbf{u}_{j},\mathcal{K})},\mathbf{t}^{i-1}_{j}),t^{i}_{j}\big{)}~{}.$
(3)

Namely, we fine-tune the decoder parameters to minimize the CE loss over pairs
of speech representations and KWS prompts.

###  2.2 KG-Whisper-PT: Keyword-guided prompt-tuning

Table 1: WER and F1 results for Voxpopuli test dataset. | Voxpopuli |
---|---|---
| WER $\downarrow$ | F1 $\uparrow$ | # Params
KG-Whisper – Oracle | $9.41$ | $97.53$ |  $906$M
KG-Whisper-PT – Oracle | $10.86$ | $95.91$ |  $15$K
Whisper | $13.39$ | $81.77$ | $-$
Whisper + prompt | $16.70$ | $83.77$ | $-$
Whisper FT | $11.57$ | $82.54$ |  $906$M
Whisper PT | $12.83$ | $90.38$ |  $15$K
KG-Whisper | $\mathbf{9.78}$ | $91.54$ |  $906$M
KG-Whisper-PT | $11.10$ | $\mathbf{95.25}$ |  $15$K

In the second method, we learn a prefix to the prompt that the KWS generates,
$\mathbf{q}\in\mathbb{R}^{N\times D}$, composed of $N$ tokens, where each
token is a vector in $\mathbb{R}^{D}$. This is done by minimizing the CE loss,
while keeping the Whisper's encoder and decoder frozen.

$\min_{\mathbf{q}}~{}\sum_{j}\sum_{i}\mathcal{L}^{\text{CE}}\big{(}f^{d}_{\psi}(\mathbf{u}_{j},[\mathbf{q}~{}\mathbf{p}^{\text{KWS}(\mathbf{u}_{j},\mathcal{K})}],\mathbf{t}^{i-1}_{j}),t^{i}_{j}\big{)}~{},$
(4)

where $\psi$ denote the frozen parameters of the decoder.

In this paper, we highlight the application of contextual biasing with a
keyword spotter, focusing specifically on Whisper. However, we note that both
approaches outlined in this work apply to any encoder-decoder-based ASR
system.

###  2.3 Training strategy

Throughout the training phase, we emulate the KWS predictions by dynamically
generating a collection of positive and negative keywords for each audio input
within the batch. First, we randomly select the quantity of keywords in the
prompt, varying between $1$ to $5$. Next, for each keyword, we determine if
the keyword is positive or negative through a coin flip, with a $0.9$
probability assigned to positive keywords. Finally, we determine the keyword
length by randomly sampling the number of tokens for each keyword term,
ranging from $1$ to $4$. Positive keywords are sampled from the transcript of
the input audio sample, while negative keywords are generated from other
randomly selected samples within the batch. After sampling the keywords we
concatenate them with $|$ delimiter. To pass the keyword as context to the
decoder module, we place them between the
$\langle\textit{start\\_of\\_prev}\rangle\textit{(SOP)}$ and
$\langle\textit{start\\_of\\_transcript}\rangle\textit{(SOT)}$ tokens, as
depicted in Figure 1.

##  3 Experiments

In this section, we evaluate KG-Whisper and KG-Whisper-PT using several
datasets and learning setups.
Datasets. We use several well-known datasets. Voxpopuli [27]: a comprehensive
collection of recordings from the European Parliament, making it a diverse
speech dataset. The dataset includes more than $1600$ hours of annotated
speech from $16$ languages. UWB-ATCC [28]: The Ultra-Wideband Air Traffic
Control Communications dataset contains radio communications between aircraft
pilots and air traffic control systems. This dataset is extremely challenging
for ASR models due to high levels of noise exceeding $20\text{dB}$. Medical
[29]: collection of medical reports made by patients including over $6600$
utterances. Fleurs [30]: a diverse multilingual dataset with $12$ hours of
annotated audio from $102$ languages.

Data preprocessing. We follow the data preprocessing paradigm presented by [3,
24]. We resample the audio samples to $16$Khz followed by log-magnitude Mel-
spectrogram generation. Specifically, we generate 80-channel Mel-spectrograms
using 25-millisecond windows and 10-millisecond stride.

Experimental setup. We train KG-Whisper and KG-Whisper-PT on Voxpopuli dataset
for $30$K update steps using batch size of $4$ and $1e{-}7$ and $5e{-}4$
learning rate, respectively. Unless stated otherwize, we use a 12 token prefix
prompt for KG-Whisper-PT. We use a frozen version (i.e., without trainable
parameters) of AdaKWS [24] as the keyword spotter. In the evaluation step, we
query the keyword spotter with $20$ keywords for each audio sample. Among
these keywords, $3$ are positive, meaning they appear in the audio utterance,
while the remaining $17$ are negative. The keywords are sampled proportionally
to the term frequency–inverse document frequency [31] scores.

We report common metrics in ASR domain, namely Word Error Rate (WER), and the
F1 score for the positive keywords.

Baselines. We compare our methods to several natural baselines. The compared
methods include: (1) Whisper \- a pre-trained Whisper large-v2 model. (2)
Whisper + prompt \- providing keywords detected by the keyword spotter as
prompt to a pre-trained Whisper large-v2 model. (3) Whisper FT \- Whisper
large-v2 model with fine-tuned decoder. (4) KG-Whisper-Oracle \- KG-Whisper
with optimal KWS, i.e., providing the positive ground truth keywords as
prompt. (5) KG-Whisper-PT-Oracle \- KG-Whisper-PT with optimal KWS.

Table 2: Out of domain generalization: WER and F1 results for UWB-ATCC and
Medical test datasets.

UWB-ATCC Medical WER $\downarrow$ F1 $\uparrow$ WER $\downarrow$ F1 $\uparrow$
Whisper $76.55$ $24.54$ $7.33$ $80.50$ Whisper + prompt $\mathbf{55.27}$
$61.57$ $32.82$ $68.71$ KG-Whisper $56.09$ $67.75$ $6.87$ $88.74$ KG-Whisper-
PT $57.88$ $\mathbf{70.89}$ $\mathbf{6.15}$ $\mathbf{96.58}$

###  3.1 Multi-lingual Finetuning

First, we evaluate KG-Whisper and KG-Whisper-PT using the Voxpopuli dataset.
We use the training set to fine-tune the decoder of KG-Whisper, or the prefix
prompt of KG-Whisper-PT, and evaluate the performance using the test set. The
results are presented in Table 1. KG-Whisper surpasses Whisper variations by a
notable margin, demonstrating enhancements of $1.8\%$ in WER and $9\%$ in F1
metrics compared to the top-performing baseline. The increase in F1 score
suggests that our method effectively guides Whisper towards the provided
context keywords, resulting in a higher frequency of positive keyword
appearances in the predicted transcript compared to other baselines.
Furthermore, KG-Whisper-PT shows a $2.3\%$ reduction in Whisper's WER, while
utilizing a mere $15$K learnable parameters. Moreover, both KG-Whisper and KG-
Whisper-PT achieve comparable results to their Oracle baseline counterparts
which applied to the ground truth keywords as prompt. Remarkably, Whisper +
prompt exhibits inferior performance compared to the naive Whisper. We
observed that this discrepancy stems from the fact that presenting prompts
without fine-tuning encourages Whisper to generate hallucinated content.

###  3.2 Out of domain generalization

We further investigate KG-Whisper and KG-Whisper-PT ability to generalize to
out-of-domain datasets characterized by challenging acoustic environments,
domain-specific jargon words, and new languages. Here, we use the KG-Whisper
and KG-Whisper-PT models trained on the Voxpopuli dataset. Notably, our
evaluation involves datasets _not used_ during the training process of the ASR
and the KWS model, namely zero-shot learning setup. First, we evaluate our
methods on UWB-ATCC [28] dataset which presents acoustic challenges due to
poor signal-to-noise ratio. The noise, originating from radio communication,
reaches over $30$dBs at peak, making UWB-ATCC challenging even for state-of-
the-art ASR systems like Whisper. Next, using the Medical [29] dataset, we
test how well KG-Whisper and KG-Whisper-PT handle out-of-domain jargon words.
Specifically, the Medical dataset contains jargon words from the medical
domain which we assume that Whisper was less exposed to during training.
Finally, we randomly select $6$ low-resource languages from the Fleurs [30]
dataset to evaluate generalization to unseen languages that were not
represented in Whisper's training data (Estonian: ET, Hungarian: HU , Latvian:
LV, Slovenian: SI, and Uzbek: UZ). The results are presented in Tables 2 and
3. We show that KG-Whisper-PT achieves substantial improvement over Whisper
baselines, improving both WER and keywords F1 on UWB-ATCC and Medical
datasets. Additionally, our method enhances Whisper's WER performance on all
randomly selected languages from the Fleurs dataset. Overall, KG-Whisper-PT
yields an average WER improvement of $5.1\%$ over Whisper.

Table 3: Generalization to unseen languages: WER results for unseen languages
from Fleurs dataset.

ET HU LV SI UZ Whisper $24.96$ $19.14$ $25.91$ $25.82$ $86.69$ Whisper +
prompt $21.18$ $23.19$ $23.94$ $27.95$ $81.30$ KG-Whisper $18.39$ $15.90$
$22.44$ $21.24$ $\mathbf{80.39}$ KG-Whisper-PT $\mathbf{17.67}$
$\mathbf{15.68}$ $\mathbf{21.95}$ $\mathbf{20.98}$ $80.94$

Table 4: Learned prompt ablation: test results on Voxpopuli and Medical datasets with varying number of learned tokens. | Voxpopuli |  | Medical
---|---|---|---
| WER $\downarrow$ | F1 $\uparrow$ |  | WER $\downarrow$ | F1 $\uparrow$
4 tokens | $11.72$ | $94.66$ |  | $6.42$ | $96.37$
8 tokens | $11.54$ | $94.97$ |  | $6.69$ | $96.22$
12 tokens | $\mathbf{11.10}$ | $95.25$ |  | $\mathbf{6.15}$ | $96.58$
16 tokens | $11.29$ | $\mathbf{95.35}$ |  | $6.17$ | $\mathbf{96.61}$
20 tokens | $11.27$ | $95.18$ |  | $6.27$ | $96.38$
24 tokens | $11.95$ | $94.11$ |  | $6.91$ | $95.62$

###  3.3 Ablation study

We empirically investigate the effect of learned prompt length on the
performance of KG-Whisper-PT. Specifically, we evaluate KG-Whisper-PT on
Voxpopuli and Medical datasets while varying the number of learnable tokens
ranging from $4$ to $24$ tokens. The results, presented in Table 4,
demonstrates that the WER increases when the prompt length either decreases or
increases, with the best WER and F1 results are shown for 12 and 16 tokens.
For longer prompts, the decoder tends to introduce more widely insertion
errors. These errors originated in _hallucinations_ [32], whereas undesirable
text is generated. In addition, long prompts increase the computational demand
of the decoder. In contrast, shorter prompts provide limited context for the
decoder, causing it to omit words or part of the original speech, which leads
to an increase in deletion errors.

##  4 Conclusion

Whisper is almost as good as a human listener for recognizing speech under
varying environmental conditions. However, its performance declines in
specialized spoken domains characterized by rare words or jargon language.
This paper introduces two novel techniques designed to enhance Whisper's
detection of specific words by leveraging keyword spotting. Utilizing prior
knowledge on the domain and keyword spotter, our method provides context to
Whisper and boosts its transcription capabilities. Both methods outperform
Whisper baselines on various datasets, even in challenging acoustic
environments. Additionally, we demonstrate that our methods can generalize to
novel languages. We are confident that our paper can lead to further research
aimed at improving the resilience and robustness of speech foundation models.

## References

  * [1] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``Hubert: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [2] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _Advances in neural information processing systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [3] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' in _International Conference on Machine Learning (ICML)_. PMLR, 2023, pp. 28 492–28 518.
  * [4] Y. Gong, S. Khurana, L. Karlinsky, and J. Glass, ``Whisper-at: Noise-robust automatic speech recognizers are also strong general audio event taggers,'' _arXiv preprint arXiv:2307.03183_ , 2023.
  * [5] I. Williams, A. Kannan, P. S. Aleksic, D. Rybach, and T. N. Sainath, ``Contextual speech recognition in end-to-end neural network systems using beam search.'' in _Interspeech_ , 2018, pp. 2227–2231.
  * [6] D. Zhao, T. N. Sainath, D. Rybach, P. Rondon, D. Bhatia, B. Li, and R. Pang, ``Shallow-fusion end-to-end contextual biasing.'' in _Interspeech_ , 2019, pp. 1418–1422.
  * [7] A. Gourav, L. Liu, A. Gandhe, Y. Gu, G. Lan, X. Huang, S. Kalmane, G. Tiwari, D. Filimonov, A. Rastrow _et al._ , ``Personalization strategies for end-to-end speech recognition systems,'' in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 7348–7352.
  * [8] D. Eitan, M. Pirchi, N. Glazer, S. Meital, G. Ayach, G. Krendel, A. Shamsian, A. Navon, G. Hetz, and J. Keshet, ``Combining language models for specialized domains: A colorful approach,'' _arXiv preprint arXiv:2310.19708_ , 2023.
  * [9] G. Pundak, T. N. Sainath, R. Prabhavalkar, A. Kannan, and D. Zhao, ``Deep context: end-to-end contextual speech recognition,'' in _2018 IEEE spoken language technology workshop (SLT)_ , 2018, pp. 418–425.
  * [10] U. Alon, G. Pundak, and T. N. Sainath, ``Contextual speech recognition with difficult negative training examples,'' in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2019, pp. 6440–6444.
  * [11] F.-J. Chang, J. Liu, M. Radfar, A. Mouchtaris, M. Omologo, A. Rastrow, and S. Kunzmann, ``Context-aware transformer transducer for speech recognition,'' in _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2021, pp. 503–510.
  * [12] K. M. Sathyendra, T. Muniyappa, F.-J. Chang, J. Liu, J. Su, G. P. Strimel, A. Mouchtaris, and S. Kunzmann, ``Contextual adapters for personalized speech recognition in neural transducers,'' in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 8537–8541.
  * [13] T. N. Sainath, R. Prabhavalkar, D. Caseiro, P. Rondon, and C. Allauzen, ``Improving contextual biasing with text injection,'' in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5.
  * [14] D. Le, G. Keren, J. Chan, J. Mahadeokar, C. Fuegen, and M. L. Seltzer, ``Deep shallow fusion for rnn-t personalization,'' in _2021 IEEE Spoken Language Technology Workshop (SLT)_ , 2021, pp. 251–257.
  * [15] D. Le, M. Jain, G. Keren, S. Kim, Y. Shi, J. Mahadeokar, J. Chan, Y. Shangguan, C. Fuegen, O. Kalinli _et al._ , ``Contextualized streaming end-to-end speech recognition with trie-based deep biasing and shallow fusion,'' _arXiv preprint arXiv:2104.02194_ , 2021.
  * [16] Y. Xu, B. Liu, Q. Huang, X. Song, Z. Wu, S. Kang, and H. Meng, ``Cb-conformer: Contextual biasing conformer for biased word recognition,'' in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5.
  * [17] G. Sun, X. Zheng, C. Zhang, and P. C. Woodland, ``Can contextual biasing remain effective with whisper and gpt-2?'' _arXiv preprint arXiv:2306.01942_ , 2023.
  * [18] F.-T. Liao, Y.-C. Chan, Y.-C. Chen, C.-J. Hsu, and D.-s. Shiu, ``Zero-shot domain-sensitive speech recognition with prompt-conditioning fine-tuning,'' in _2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2023, pp. 1–8.
  * [19] P. Peng, B. Yan, S. Watanabe, and D. Harwath, ``Prompting the hidden talent of web-scale speech models for zero-shot task generalization,'' _arXiv preprint arXiv:2305.11095_ , 2023.
  * [20] H. Ma, Z. Peng, M. Shao, J. Li, and J. Liu, ``Extending whisper with prompt tuning to target-speaker asr,'' _arXiv preprint arXiv:2312.08079_ , 2023.
  * [21] H.-K. Shin, H. Han, D. Kim, S.-W. Chung, and H.-G. Kang, ``Learning audio-text agreement for open-vocabulary keyword spotting,'' _arXiv preprint arXiv:2206.15400_ , 2022.
  * [22] K. Nishu, M. Cho, and D. Naik, ``Matching latent encoding for audio-text based keyword spotting,'' _arXiv preprint arXiv:2306.05245_ , 2023.
  * [23] K. Nishu, M. Cho, P. Dixon, and D. Naik, ``Flexible keyword spotting based on homogeneous audio-text embedding,'' _arXiv preprint arXiv:2308.06472_ , 2023.
  * [24] A. Navon, A. Shamsian, N. Glazer, G. Hetz, and J. Keshet, ``Open-vocabulary keyword-spotting with adaptive instance normalization,'' in _Proceeding of the International Conference on Audio, Speech, and Signal Processing (ICASSP)_ , 2024.
  * [25] B. Lester, R. Al-Rfou, and N. Constant, ``The power of scale for parameter-efficient prompt tuning,'' _arXiv preprint arXiv:2104.08691_ , 2021.
  * [26] Y. Li, Y. Li, M. Zhang, C. Su, M. Ren, X. Qiao, X. Zhao, M. Piao, J. Yu, X. Lv, M. Ma, Y. Zhao, and H. Yang, ``A multitask training approach to enhance whisper with contextual biasing and open-vocabulary keyword spotting,'' 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:266999141
  * [27] C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, ``Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,'' _arXiv preprint arXiv:2101.00390_ , 2021.
  * [28] J. Zuluaga-Gomez, A. Prasad, I. Nigmatulina, S. S. Sarfjoo, P. Motlicek, M. Kleinert, H. Helmke, O. Ohneiser, and Q. Zhan, ``How does pre-trained wav2vec 2.0 perform on domain-shifted asr? an extensive benchmark on air traffic control communications,'' in _2022 IEEE Spoken Language Technology Workshop (SLT)_ , 2023, pp. 205–212.
  * [29] Y. Kumar, A. Koul, and S. Mahajan, ``A deep learning approaches and fastai text classification to predict 25 medical diseases from medical speech utterances, transcription and intent,'' _Soft computing_ , vol. 26, no. 17, pp. 8253–8272, 2022.
  * [30] A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, ``Fleurs: Few-shot learning evaluation of universal representations of speech,'' in _2022 IEEE Spoken Language Technology Workshop (SLT)_ , 2023, pp. 798–805.
  * [31] G. Salton and C. Buckley, ``Term-weighting approaches in automatic text retrieval,'' _Information processing & management_, vol. 24, no. 5, pp. 513–523, 1988.
  * [32] A. Koenecke, A. S. G. Choi, K. Mei, H. Schellmann, and M. Sloane, ``Careless whisper: Speech-to-text hallucination harms,'' _arXiv preprint arXiv:2402.08021_ , 2024.