annotations [323]. This study was motivated by unsupervised query-by-example,
applications of non-parametric Bayesian machine learning to speech processing,
and low-resource speech recognition, and was also inspired by the learning
process of infants. The goal of this type of work is to build spoken dialog
systems in a zero-resource setup for any language. To encourage zero-resource
research, zero-resource speech challenges have been organized since 2015.

In this section, we describe the research directions of zero-resource speech
technologies by following the series of zero-resource speech challenges.

  * •

Zero Resource Speech Challenge 2015 [279] mainly focused on building an
acoustic model without using any linguistic annotations based on subword unit
modeling and spoken term discovery tracks. For the subword unit modeling
track, the ABX score for the within- and across-speaker tasks was used as an
evaluation metric. The spoken term discovery track used the normalized edit
distance and coverage scores in addition to the precision, recall, and F1
scores for types, tokens, and boundaries. Both tracks were based on the
English and Xitsonga languages.

  * •

The Zero Resource Speech Challenge 2017 [324] focused on unseen language and
speaker aspects from the previous challenge. For example, to demonstrate the
robustness against unseen languages, the systems were developed with English,
French, and Mandarin and tested on two “surprise” languages: German and Wolof.
Similarly, robustness against unseen speakers was demonstrated by varying the
amount of speech available for each speaker.

  * •

The Zero Resource Speech Challenge 2019 [325] extended a goal of previous
challenges by synthesizing speech without text or phonetic labels but with
acoustic units obtained using zero-resource techniques. The evaluation metrics
were also extended to subjectively evaluate the quality of synthesized speech,
including its intelligibility, naturalness, and speaker similarity.

  * •

The Zero Resource Speech Challenge 2020 [261] was based on two tracks,
revisiting previous challenges with different evaluation metrics. The first
task revisited the 2019 challenge with low bit-rate subword representations
that optimize the quality of speech synthesis. The second task revisited the
2017 challenge by focusing on the discovery of word-like units from
unsegmented raw speech.

  * •

The Zero Resource Speech Challenge 2021 [326], the latest challenge, expanded
the scope to include language modeling tasks. In addition to phoneme-level
ABX, the challenge includes lexical, semantic, and syntactic evaluation
metrics computed via a language model of pseudo-acoustic labels.

These challenges have facilitated the tracking of technical trends in zero-
resource speech technologies. For example, research directions thereof have
expanded to various speech processing components to cover the entire spoken
dialogue systems. To keep up with this expansion, the challenge has continued
to develop appropriate evaluation metrics for zero-resource scenarios.
Following the success of representation learning, baseline and challenge
techniques have shifted from purely generative models [327, 328], deep
autoencoders [329, 83], and incorporation of neural-network-based TTS/VC
techniques [266] to self-supervised learning [330]. The latest challenge
included the visual modality, continuing the expansion to include more aspects
of human interaction.

####  VI-B2 Textless NLP

Textless NLP is a new research direction that leverages the progress mentioned
above in self-supervised speech representation learning to model language
directly from audio, bypassing the need for text or labels [78, 76, 77, 331].
Not only does this open the gate for language and dialect modeling without
orthographic rules, but it also offers the opportunity to model other non-
lexical information about how speech is delivered, e.g., speaker identity,
emotion, hesitation, interruptions. The generative spoken language model
(GSLM) [78] utilizes discrete representations from wav2vec 2.0, HuBERT, and
CPC algorithms as inputs to an autoregressive language model trained by using
the cross-entropy function to maximize the probability of predicting the next
discrete speech token. A synthesis module follows the language model to
produce speech waveforms given the generated discrete speech units. The
generated spoken continuations compete with supervised generations and
synthesis using a character language model in subjective human evaluations.
The model completes incomplete words (pow[..] $\rightarrow$ POWER) and
continues using words in the same general mood (dark $\rightarrow$
BLACKNESS)252525https://speechbot.github.io/gslm/ and has been extended to
model and generate dialogues [332].262626https://speechbot.github.io/dgslm/
Given its flexibility in modeling spoken content, the GSLM has been further
extended to jointly model content and prosody [77]. This prosodic-GSLM model
introduced a multistream causal Transformer, where the input and output layers
use multiple heads to model three channels: discrete speech units, duration,
and quantized pitch. The prosodic-GSLM model jointly generates novel content
and prosody congruently in the expressive style of the
prompt.272727https://speechbot.github.io/pgslm/ Going one step further, [331]
used a speech emotion conversion framework to modify the perceived emotion of
a speech utterance while preserving its lexical content and speaker identity.
Other studies have extended the idea of textless language processing or audio
discrete representation to applications such as spoken question answering
[333], speech separation [334], TTS [335], and speech-to-speech translation
[336].

##  VII Discussion and Conclusion

In this overview, we have presented the historical context of self-supervised
learning and provided a thorough methodological review of important self-
supervised speech representation models. Specifically, we have categorized the
approaches into three categories, generative, contrastive and predictive,
differing in terms of how the pretext task is defined. We have presented an
overview of existing benchmarks and reviewed the efforts towards efficient
zero-resource learning. Although the field is progressing rapidly, with new
approaches reaching higher levels of performance, a couple of patterns have
emerged: (1) The solid performance of Wav2vec 2.0 for speech recognition and
many downstream tasks, as well as the public availability of its pre-trained
multilingual variants, enabled wide adoption in the community making it a
“standard” go-to model. (2) The simplicity and stability of the HuBERT
approach, as well as the resemblance of its training procedure to classic
frame-level ASR systems, made it an easy choice for research extensions on
improving representation quality, speech translation, and textless NLP.

Below we highlight various shortcomings of existing work and  future research
directions:

  * •

Using the representation model. So far, there are two main ways to use
representation models: Freeze the representation models and use them as
feature extractors, or fine-tune the representation models on downstream
tasks. Some efficient methods for leveraging SSL models exist in the NLP
community. Adapters [337, 338, 339] are lightweight modules inserted into SSL
models, and in downstream tasks, the parameters of SSL models are frozen, and
only the adapters are trained. The prompt/instruction learning methods [20]
also freeze the SSL parameters and control the output of SSL by adding
additional information, which is called prompt, in the input. Both adapter-
based methods and prompt/instruction learning yield competitive performance
compared with fine-tuning in NLP applications, but there is only little
related work for speech [340, 341]. In addition, prompt for speech SSL does
not achieve comparable performance on sequence generation tasks like phoneme
recognition and slot filling, so how to use prompt is still an open question.

  * •

Increasing the efficiency of the representation model. As discussed in Section
V-C, larger representation models lead to better downstream performance.
Despite the success of these large models, they incur high costs in terms of
memory and time for pre-training, fine-tuning, and even when used only to
extract representations without gradient calculation. This makes them
unsuitable for edge devices but also limits the ability to scale these models
to very large datasets – and leads to a large energy consumption. Preliminary
studies have been conducted on compressing speech representation models
through network pruning [342] or knowledge distillation [343]. There has been
quite some effort towards more efficient general neural network models via
conditional computing [344] and neural network quantization [345] as well as
extensive work on improving the specific efficiency of Transformer models,
especially with the focus on self-attention [346], but these technology has
not been widely used in speech SSL. Because speech is intrinsically
represented as sequence, one way to reduce computation is to reduce the length
of speech representation sequence but still keep the vital information in
speech. But we have not been aware of any publication in this direction when
writing this paper. On the other hand, non-streaming architectures in models
such as the bidirectional Transformer have hindered the representation model
used in streaming scenarios, leading to studies that address these problems
[347]. We anticipate research in these directions to continue in the future.

  * •

Data-efficient approaches. SOTA representation learning methods require large
volumes of unlabeled speech during pre-training, going way beyond what babies
need to understand language. Different learning approaches have different data
needs, e.g., generative approaches could be more data efficient than
contrastive or predictive approaches since they are constrained by more bits
of information to reconstruct their inputs. Comprehensive research is needed
to study the data efficiency of different approaches.

  * •

Feature Disentanglement. Speech SSL models show strengths on a surprisingly
wide range of tasks [44], suggesting that representations contain different
information. One way to further improve downstream tasks is to disentangle
different information from the representation. For example, we can decompose
the representation into content embedding and speaker embedding and use
content embedding for ASR and speaker embedding for SID. Some work has been in
this direction [348, 349, 350].

  * •

Creating robust models. As discussed in Section V-D, studies have been
conducted on the robustness of representation models [351]. However, the
failure modes of SSL models are still poorly understood, and it remains
unclear whether they provide more or less robustness to adversarial attacks
than fully supervised models. Due to the importance of this research
direction, while writing this paper, there is already some related research
about enhancing the robustness of SSL models [281, 352, 353, 354] and
identifying their vulnerability to adversarial attack [351].

  * •

Capturing higher-level semantic information. Although many representation
learning approaches can go beyond low-level phonetic modeling to capture some
lexical information [355], they still struggle in higher-level semantic tasks
easily captured by word-level counterparts like BERT. One workaround is two-
stage training [77, 332]; however, this prevents propagating rich lexical and
semantic knowledge modeled in the second stage to benefit the phonetically
focused first stage.

  * •

Using text representation models to improve speech representation. The amount
of content information in speech corpora used to train speech representation
models is far less than that of text representation models. Noting that the
BERT training corpus exceeds 3 billion words [54], and assuming a typical
speaking rate of 120 words per minute, a speech corpus containing the same
content as the BERT training data would include 400,000 hours of audio, which
exceeds the accumulated training data of all current speech representation
models. Therefore, to enable speech representation models to better learn
human language, for instance by extracting semantic information from acoustic
signals, the use of text models such as BERT and GPT seems key: nevertheless,
how to use these to improve speech representation model pre-training remains
an open question.

We believe SSL representation models have considerable room to grow. The
relationship between representation models and downstream tasks can be
compared to the relationship between operating systems and applications.
Today, even individuals can build applications with desired functions on a
smartphone because the smartphone’s operating system handles the complex
communication with the hardware and provides a convenient developer interface.
Likewise, as SSL representation models learn general knowledge from human
speech, it is easy to develop new speech processing applications on this
basis. From this viewpoint, speech representation models will play the role of
operating systems in speech processing and further facilitate the continued
development of speech technology.

## References

  * [1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _Nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [2] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath _et al._ , “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” _IEEE Signal Processing Magazine_ , vol. 29, no. 6, pp. 82–97, 2012.
  * [3] H. A. Bourlard and N. Morgan, _Connectionist speech recognition: A hybrid approach_. Springer Science & Business Media, 2012, vol. 247.
  * [4] T. Kemp and A. Waibel, “Unsupervised training of a speech recognizer: Recent experiments,” _Proceedings of European Conference on Speech Communication and Technology_ , 1999.
  * [5] L. Lamel, J.-L. Gauvain, and G. Adda, “Lightly supervised and unsupervised acoustic model training,” _Computer Speech & Language_, 2002.
  * [6] J. Ma, S. Matsoukas, O. Kimball, and R. Schwartz, “Unsupervised training on large amounts of broadcast news data,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2006.
  * [7] G. E. Hinton, “Learning multiple layers of representation,” _Trends in Cognitive Sciences_ , vol. 11, pp. 428–434, 2007.
  * [8] Y. LeCun, S. Chopra, R. Hadsell, F. J. Huang _et al._ , “A tutorial on energy-based learning,” in _Predicting Structured Data_. MIT Press, 2006.
  * [9] Y. Bengio, A. C. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” _IEEE Transactions on Pattern Analysis Machine Intelligence_ , vol. 35, no. 8, pp. 1798–1828, 2013.
  * [10] M. I. Jordan and T. M. Mitchell, “Machine learning: Trends, perspectives, and prospects,” _Science_ , vol. 349, no. 6245, pp. 255–260, 2015.
  * [11] R. Gray, “Vector quantization,” _IEEE ASSP Magazine_ , vol. 1, no. 2, pp. 4–29, 1984.
  * [12] M. Jordan and R. Jacobs, “Hierarchical mixtures of experts and the EM algorithm,” in _Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan)_ , vol. 2, 1993.
  * [13] G. E. Hinton and R. Zemel, “Autoencoders, minimum description length and Helmholtz free energy,” in _Advances in Neural Information Processing Systems_ , vol. 6, 1994.
  * [14] D. D. Lee and H. S. Seung, “Learning the parts of objects by nonnegative matrix factorization,” _Nature_ , vol. 401, pp. 788–791, 1999.
  * [15] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” _Science_ , vol. 313, no. 5786, pp. 504–507, 2006\.
  * [16] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill _et al._ , “On the opportunities and risks of foundation models,” 2021.
  * [17] L. Ericsson, H. Gouk, C. C. Loy, and T. M. Hospedales, “Self-supervised representation learning: Introduction, advances and challenges,” 2021.
  * [18] X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, “Self-supervised learning: Generative or contrastive,” _IEEE Transactions on Knowledge & Data Engineering_, no. 01, pp. 1–1, Jun 2021.
  * [19] A. Rogers, O. Kovaleva, and A. Rumshisky, “A primer in BERTology: What we know about how BERT works,” _Transactions of the Association for Computational Linguistics_ , vol. 8, pp. 842–866, 2020. [Online]. Available: https://aclanthology.org/2020.tacl-1.54
  * [20] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” 2021.
  * [21] P. Xia, S. Wu, and B. Van Durme, “Which *BERT? A survey organizing contextualized encoders,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 7516–7533. [Online]. Available: https://aclanthology.org/2020.emnlp-main.608
  * [22] X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,” _Science China Technological Sciences_ , vol. 63, no. 10, p. 1872–1897, Sep 2020. [Online]. Available: http://dx.doi.org/10.1007/s11431-020-1647-3
  * [23] L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” _IEEE Transactions on Pattern Analysis & Machine Intelligence_, vol. 43, no. 11, pp. 4037–4058, nov 2021.
  * [24] L. Borgholt, J. D. Havtorn, J. Edin, L. Maaløe, and C. Igel, “A brief overview of unsupervised neural speech representation learning,” in _The 2nd Workshop on Self-supervised Learning for Audio and Speech Processing (AAAI-SAS-2022)_ , 2022.
  * [25] S. Latif, R. Rana, S. Khalifa, R. Jurdak, J. Qadir, and B. W. Schuller, “Deep representation learning in speech processing: Challenges, recent advances, and future trends,” 2021.
  * [26] L. Rabiner and J. Wilpon, “Considerations in applying clustering techniques to speaker independent word recognition,” in _IEEE International Conference on Acoustics, Speech, and Signal Processing_ , vol. 4, 1979, pp. 578–581.
  * [27] J. Wilpon and L. Rabiner, “A modified k-means clustering algorithm for use in isolated work recognition,” _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , vol. 33, no. 3, pp. 587–594, 1985.
  * [28] J.-L. Gauvain and C.-H. Lee, “Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains,” _IEEE Transactions on Speech and Audio Processing_ , vol. 2, no. 2, pp. 291–298, 1994.
  * [29] S. Young and P. Woodland, “State clustering in hidden Markov model-based continuous speech recognition,” _Computer Speech & Language_, vol. 8, no. 4, pp. 369–383, 1994.
  * [30] L. Bahl, P. Brown, P. de Souza, and R. Mercer, “Maximum mutual information estimation of hidden Markov model parameters for speech recognition,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , vol. 11, 1986, pp. 49–52.
  * [31] N. Smith and M. Gales, “Speech recognition using SVMs,” in _NIPS_ , 2001\.
  * [32] V. Venkataramani, S. Chakrabartty, and W. Byrne, “Support vector machines for segmental minimum Bayes risk decoding of continuous speech,” in _ASRU_ , 2003.
  * [33] V. Wan and S. Renals, “SVMSVM: support vector machine speaker verification methodology,” in _ICASSP_ , 2003, pp. 221–224.
  * [34] N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19(4), pp. 788–798, 2011.
  * [35] N. Dehak, P. .Torres-Carrasquillo, D. Reynolds, and R. Dehak, “Language recognition via i-vectors and dimensionality reduction,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2011.
  * [36] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion,” _Journal of Machine Learning Research_ , 2010.
  * [37] M. U. Gutmann and A. Hyvärinen, “Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics.” _Journal of Machine Learning Research_ , vol. 13, no. 2, 2012\.
  * [38] B. Olshausen and D. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” _Nature_ , vol. 381, pp. 607–609, June 1996.
  * [39] H. Lee, A. Battle, R. Raina, and A. Y. Ng, “Efficient sparse coding algorithms,” in _Proceedings of the 19th International Conference on Neural Information Processing Systems_. MIT Press, 2006, p. 801–808.
  * [40] G. S. Sivaram, S. K. Nemala, M. Elhilali, T. D. Tran, and H. Hermansky, “Sparse coding for speech recognition,” in _2010 IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2010, pp. 4346–4349.
  * [41] M. Ranzato, Y.-L. Boureau, S. Chopra, and Y. LeCun, “A unified energy-based framework for unsupervised learning,” in _Proc. Eleventh International Conference on Artificial Intelligence and Statistics_ , 2007, pp. 371–379.
  * [42] G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” _Neural Comput._ , vol. 14, no. 8, p. 1771–1800, aug 2002\.
  * [43] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in _Proceedings of International Conference on Learning Representations_ , 2019.
  * [44] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, “SUPERB: Speech Processing Universal PERformance Benchmark,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [45] R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” _CoRR_ , vol. abs/1603.08511, 2016.
  * [46] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” _CoRR_ , vol. abs/1807.05520, 2018\.
  * [47] C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” _CoRR_ , vol. abs/1505.05192, 2015.
  * [48] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” in _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14–16, 2014, Conference Track Proceedings_ , 2014.
  * [49] D. J. Rezende, S. Mohamed, and D. Wierstra, “Stochastic backpropagation and approximate inference in deep generative models,” in _International Conference on Machine Learning_. PMLR, 2014, pp. 1278–1286.
  * [50] L. Girin, S. Leglaive, X. Bie, J. Diard, T. Hueber, and X. Alameda-Pineda, “Dynamical variational autoencoders: A comprehensive review,” _Foundations and Trends® in Machine Learning_ , vol. 15, pp. 1–175, 12 2021\.
  * [51] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in _NAACL_ , 2018.
  * [52] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019.
  * [53] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-LM: Training multi-billion parameter language models using model parallelism,” 2020.
  * [54] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional Transformers for language understanding,” in _NAACL_ , 2019.
  * [55] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.
  * [56] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [57] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _Proceedings of the 37th International Conference on Machine Learning_ , 2020.
  * [58] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _CVPR_ , 2020.
  * [59] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” 2020.
  * [60] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” in _Proceedings of Advances in Neural Information Processing Systems_ , 2020.
  * [61] S. H. K. Parthasarathi and N. Strom, “Lessons from building acoustic models with a million hours of speech,” _CoRR_ , vol. abs/1904.01624, 2019.
  * [62] D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le, “Improved Noisy Student Training for Automatic Speech Recognition,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020.
  * [63] Q. Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert, “Iterative Pseudo-Labeling for Speech Recognition,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020.
  * [64] A. Xiao, W. Zheng, G. Keren, D. Le, F. Zhang, C. Fuegen, O. Kalinli, Y. Saraf, and A. Mohamed, “Scaling ASR improves zero and few shot learning,” _CoRR_ , vol. abs/2111.05948, 2021.
  * [65] R. Caruana, “Multitask learning,” _Machine Learning_ , vol. 28, no. 1, pp. 41–75, 1997.
  * [66] J. Cui, B. Kingsbury, B. Ramabhadran, A. Sethy, K. Audhkhasi, X. Cui, E. Kislal, L. Mangu, M. Nußbaum-Thom, M. Picheny, Z. Tüske, P. Golik, R. Schlüter, H. Ney, M. J. F. Gales, K. Knill, A. Ragni, H. Wang, and P. C. Woodland, “Multilingual representations for low resource speech recognition and keyword search,” _2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_ , 2015.
  * [67] P. Bell, J. Fainberg, O. Klejch, J. Li, S. Renals, and P. Swietojanski, “Adaptation algorithms for neural network-based speech recognition: An overview,” _IEEE Open Journal of Signal Processing_ , vol. 2, pp. 33–66, 2021.
  * [68] A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self-supervised speech representation model,” in _Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding_ , 2021.
  * [69] S. Pascual, M. Ravanelli, J. Serrà, A. Bonafonte, and Y. Bengio, “Learning problem-agnostic speech representations from multiple self-supervised tasks,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2019.
  * [70] J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio, “A Recurrent Latent Variable Model for Sequential Data,” in _Proceedings of the 29th Conference on Neural Information Processing Systems (NeurIPS)_ , Montréal, Quebec, Canada, 2015, p. 9.
  * [71] M. Fraccaro, S. K. Sønderby, U. Paquet, and O. Winther, “Sequential Neural Models with Stochastic Layers,” in _Proceedings of the 30th Conference on Neural Information Processing Systems (NeurIPS)_ , Barcelona, Spain, 2016.
  * [72] W.-N. Hsu, Y. Zhang, and J. Glass, “Learning latent representations for speech generation and transformation,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2017.
  * [73] ——, “Unsupervised learning of disentangled and interpretable representations from sequential data,” in _Proceedings of Advances in Neural Information Processing Systems_ , 2017.
  * [74] E. Aksan and O. Hilliges, “STCN: Stochastic Temporal Convolutional Networks,” in _Proceedings of the 7th International Conference on Learning Representations (ICLR)_ , New Orleans, LA, USA, Feb. 2019.
  * [75] A. van den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” 2017.
  * [76] A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [77] E. Kharitonov, A. Lee, A. Polyak, Y. Adi, J. Copet, K. Lakhotia, T. A. Nguyen, M. Rivière, A. Mohamed, E. Dupoux, and W. Hsu, “Text-free prosody-aware generative spoken language modeling,” _arXiv preprint arXiv:2109.03264_ , 2021.
  * [78] K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed, and E. Dupoux, “On generative spoken language modeling from raw audio,” _Transactions of the Association for Computational Linguistics_ , vol. 9, pp. 1336–1354, 2021.
  * [79] Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” _arXiv preprint arXiv:1308.3432_ , 2013.
  * [80] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A generative model for raw audio,” _arXiv preprint arXiv:1609.03499_ , 2016.
  * [81] D. K. Burton, J. E. Shore, and J. T. Buck, “A generalization of isolated word recognition using vector quantization,” _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 1983.
  * [82] F. Soong, A. Rosenberg, and L. R. B. Juang, “A vector quantization approach to speaker recognition,” _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 1985.
  * [83] J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord, “Unsupervised speech representation learning using WaveNet autoencoders,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 12, pp. 2041–2053, 2019.
  * [84] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with Gumbel-softmax,” in _Proceedings of International Conference on Learning Representations_ , 2017.
  * [85] R. Eloff, A. Nortje, B. van Niekerk, A. Govender, L. Nortje, A. Pretorius, E. van Biljon, E. van der Westhuizen, L. van Staden, and H. Kamper, “Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2019.
  * [86] M. D. Zeiler, M. Ranzato, R. Monga, M. Mao, K. Yang, Q. V. Le, P. Nguyen, A. Senior, V. Vanhoucke, J. Dean _et al._ , “On rectified linear units for speech processing,” in _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2013, pp. 3517–3521.
  * [87] L. Badino, C. Canevari, L. Fadiga, and G. Metta, “An auto-encoder based approach to unsupervised learning of subword units,” in _2014 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2014, pp. 7634–7638.
  * [88] L. Badino, A. Mereta, and L. Rosasco, “Discovering discrete subword units with binarized autoencoders and hidden-markov-model encoders,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015\.
  * [89] H. Kamper, M. Elsner, A. Jansen, and S. Goldwater, “Unsupervised neural network based feature extraction using weak top-down constraints,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2015.
  * [90] D. Renshaw, H. Kamper, A. Jansen, and S. Goldwater, “A comparison of neural network methods for unsupervised representation learning on the Zero Resource Speech Challenge,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2015.
  * [91] S. Settle, K. Audhkhasi, K. Livescu, and M. Picheny, “Acoustically grounded word embeddings for improved acoustics-to-word speech recognition,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2019.
  * [92] Y.-A. Chung, W.-N. Hsu, H. Tang, and J. Glass, “An unsupervised autoregressive model for speech representation learning,” _arXiv preprint arXiv:1904.03240_ , 2019.
  * [93] Y.-A. Chung and J. Glass, “Generative pre-training for speech with autoregressive predictive coding,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020.
  * [94] D. O’Shaughnessy, “Linear predictive coding,” _IEEE Potentials_ , vol. 7, no. 1, pp. 29–32, 1988.
  * [95] L. Wiskott and T. J. Sejnowski, “Slow feature analysis: Unsupervised learning of invariances,” _Neural Computation_ , vol. 14, no. 4, pp. 715–770, 2002\.
  * [96] J. S. Garofolo, “TIMIT Acoustic-Phonetic Continuous Speech Corpus LDC93S1,” _Linguistic Data Consortium_ , 1993. [Online]. Available: https://catalog.ldc.upenn.edu/LDC93S1
  * [97] Y.-A. Chung and J. Glass, “Improved speech representations with multi-target autoregressive predictive coding,” in _Proceedings of Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , 2020.
  * [98] Y.-A. Chung, H. Tang, and J. Glass, “Vector-Quantized Autoregressive Predictive Coding,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020.
  * [99] S. Ling, Y. Liu, J. Salazar, and K. Kirchhoff, “Deep contextualized acoustic representations for semi-supervised speech recognition,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020.
  * [100] A. T. Liu, S.-w. Yang, P.-H. Chi, P.-c. Hsu, and H.-y. Lee, “Mockingjay: Unsupervised speech representation learning with deep bidirectional Transformer encoders,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020.
  * [101] D. Jiang, X. Lei, W. Li, N. Luo, Y. Hu, W. Zou, and X. Li, “Improving Transformer-based speech recognition using unsupervised pre-training,” _arXiv preprint arXiv:1910.09932_ , 2019.
  * [102] L. Liu and Y. Huang, “Masked pre-trained encoder base on joint CTC-Transformer,” _arXiv preprint arXiv:2005.11978_ , 2020.
  * [103] W. Wang, Q. Tang, and K. Livescu, “Unsupervised pre-training of bidirectional speech encoders via masked reconstruction,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020.
  * [104] D. Jiang, W. Li, R. Zhang, M. Cao, N. Luo, Y. Han, W. Zou, K. Han, and X. Li, “A further study of unsupervised pretraining for Transformer based speech recognition,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2021.
  * [105] X. Yue and H. Li, “Phonetically motivated self-supervised speech representation learning,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [106] A. T. Liu, S.-W. Li, and H.-y. Lee, “TERA: Self-supervised learning of Transformer encoder representation for speech,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 2351–2366, 2021.
  * [107] A. H. Liu, Y.-A. Chung, and J. Glass, “Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [108] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “XLNet: Generalized autoregressive pretraining for language understanding,” in _Proceedings of Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019\.
  * [109] X. Song, G. Wang, Y. Huang, Z. Wu, D. Su, and H. Meng, “Speech-XLNet: Unsupervised acoustic model pretraining for self-attention networks,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020.
  * [110] S. Ling and Y. Liu, “DeCoAR 2.0: Deep contextualized acoustic representations with vector quantization,” _arXiv preprint arXiv:2012.06659_ , 2020.
  * [111] J. Luo, J. Wang, N. Cheng, and J. Xiao, “Dropout regularization for self-supervised learning of Transformer encoder speech representation,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [112] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,” _Journal of Machine Learning Research_ , vol. 15, pp. 1929–1958, 2014.
  * [113] M. Ravanelli, J. Zhong, S. Pascual, P. Swietojanski, J. Monteiro, J. Trmal, and Y. Bengio, “Multi-task self-supervised learning for robust speech recognition,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020.
  * [114] Y.-A. Chung and J. Glass, “Speech2vec: A sequence-to-sequence framework for learning word embeddings from speech,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2018.
  * [115] M. Tagliasacchi, B. Gfeller, F. d. C. Quitry, and D. Roblek, “Self-supervised audio representation learning for mobile devices,” _arXiv preprint arXiv:1905.11796_ , 2019.
  * [116] M. Tagliasacchi, B. Gfeller, F. de Chaumont Quitry, and D. Roblek, “Pre-training audio representations with self-supervision,” _IEEE Signal Processing Letters_ , vol. 27, pp. 600–604, 2020.
  * [117] F. d. C. Quitry, M. Tagliasacchi, and D. Roblek, “Learning audio representations via phase prediction,” _arXiv preprint arXiv:1910.11910_ , 2019.
  * [118] Y.-A. Chung, C.-C. Wu, C.-H. Shen, H.-Y. Lee, and L.-S. Lee, “Audio word2Vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2016.
  * [119] P.-H. Chi, P.-H. Chung, T.-H. Wu, C.-C. Hsieh, S.-W. Li, and H.-y. Lee, “Audio ALBERT: A lite BERT for self-supervised learning of audio representation,” _Proceedings of IEEE Spoken Language Technology Workshop_ , 2021.
  * [120] B. Milde and C. Biemann, “Unspeech: Unsupervised speech context embeddings,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2018.
  * [121] S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” _arXiv preprint arXiv:1904.05862_ , 2019.
  * [122] M. Riviere, A. Joulin, P.-E. Mazaré, and E. Dupoux, “Unsupervised pretraining transfers well across languages,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020.
  * [123] K. Kawakami, L. Wang, C. Dyer, P. Blunsom, and A. van den Oord, “Learning robust and multilingual speech representations,” in _EMNLP_ , 2020.
  * [124] A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” in _Proceedings of International Conference on Learning Representations_ , 2020.
  * [125] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Proceedings of Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [126] S. Sadhu, D. He, C.-W. Huang, S. H. Mallidi, M. Wu, A. Rastrow, A. Stolcke, J. Droppo, and R. Maas, “wav2vec-C: A Self-Supervised Model for Speech Representation Learning,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [127] Y. Chung, Y. Zhang, W. Han, C. Chiu, J. Qin, R. Pang, and Y. Wu, “W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” 2021.
  * [128] D. Jiang, W. Li, M. Cao, W. Zou, and X. Li, “Speech SIMCLR: Combining contrastive and reconstruction objective for self-supervised speech representation learning,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [129] A. Baevski, M. Auli, and A. Mohamed, “Effectiveness of self-supervised pre-training for speech recognition,” _arXiv preprint arXiv:1911.03912_ , 2019.
  * [130] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” _arXiv preprint arXiv:2106.07447_ , 2021\.
  * [131] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, and F. Wei, “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” 2021.
  * [132] A. Baevski, W. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “data2vec: A general framework for self-supervised learning in speech, vision and language,” 2022\.
  * [133] C.-C. Chiu, J. Qin, Y. Zhang, J. Yu, and Y. Wu, “Self-supervised learning with random-projection quantizer for speech recognition,” _arXiv preprint arXiv:2202.01855_ , 2022.
  * [134] M. Schultz and T. Joachims, “Learning a distance metric from relative comparisons,” in _Advances in Neural Information Processing Systems_ , S. Thrun, L. Saul, and B. Schölkopf, Eds., 2003.
  * [135] M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” _International Conference on Artificial Intelligence and Statistics (AISTATS)_ , 2010.
  * [136] H. Jégou, M. Douze, and C. Schmid, “Product quantization for nearest neighbor search,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 33, no. 1, pp. 117–128, 2011.
  * [137] A. Razavi, A. van den Oord, and O. Vinyals, “Generating diverse high-fidelity images with VQ-VAE-2,” in _Proceedings of Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019\.
  * [138] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * [139] J. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Á. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, “Bootstrap Your Own Latent: A new approach to self-supervised learning,” _CoRR_ , vol. abs/2006.07733, 2020.
  * [140] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision Transformers,” in _IEEE International Conference on Computer Vision_ , 2021.
  * [141] R. B. Girshick, “Fast R-CNN,” in _IEEE International Conference on Computer Vision_ , 2015.
  * [142] A. Paszke _et al._ , “PyTorch: An imperative style, high-performance deep learning library,” in _Proceedings of Advances in Neural Information Processing Systems_ , 2019.
  * [143] E. D. Petajan, “Automatic lipreading to enhance speech recognition (speech reading),” Ph.D. dissertation, University of Illinois at Urbana-Champaign, 1984\.
  * [144] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, “Recent advances in the automatic recognition of audiovisual speech,” _Proceedings of the IEEE_ , vol. 91, no. 9, pp. 1306–1326, 2003.
  * [145] P. S. Aleksic and A. K. Katsaggelos, “Audio-visual biometrics,” _Proceedings of the IEEE_ , vol. 94, no. 11, pp. 2025–2044, 2006.
  * [146] M. Legerstee, “Infants use multimodal information to imitate speech sounds,” _Infant Behavior and Development_ , vol. 13, no. 3, pp. 343–354, 1990.
  * [147] D. Roy, “Learning from sights and sounds: A computational model,” _PhD Thesis, MIT Media Laboratory_ , 1999.
  * [148] B. Lee, M. Hasegawa-Johnson, C. Goudeseune, S. Kamdar, S. Borys, M. Liu, and T. Huang, “AVICAR: Audio-visual speech corpus in a car environment,” in _Eighth International Conference on Spoken Language Processing_ , 2004.
  * [149] J. S. Chung and A. Zisserman, “Lip reading in the wild,” in _Asian Conference on Computer Vision_ , 2016.
  * [150] B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” in _International Conference on Learning Representations_ , 2022.
  * [151] J. Westbury, P. Milenkovic, G. Weismer, and R. Kent, “X-ray microbeam speech production database,” _JASA_ , vol. 88, no. S1, pp. S56–S56, 1990.
  * [152] A. Wrench, “A new resource for production modelling in speech technology,” _Proc. Institute of Acoustics_ , vol. 23, no. 3, pp. 207–218, 2001.
  * [153] S. Narayanan, E. Bresch, P. K. Ghosh, L. Goldstein, A. Katsamanis, Y. Kim, A. Lammert, M. Proctor, V. Ramanarayanan, and Y. Zhu, “A multimodal real-time MRI articulatory corpus for speech research,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2011.
  * [154] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, “Multimodal deep learning,” in _Proceedings of the 28th International Conference on Machine Learning_ , 2011.
  * [155] L. Badino, C. Canevari, L. Fadiga, and G. Metta, “Integrating articulatory data in deep neural network-based acoustic modeling,” _Comp. Sp. & Lang._, vol. 36, pp. 173–195, 2016.
  * [156] N. Srivastava and R. R. Salakhutdinov, “Multimodal learning with deep Boltzmann machines,” in _Proceedings of Advances in Neural Information Processing Systems_ , 2012, pp. 2222–2230.
  * [157] H. Hotelling, “Relations between two sets of variates,” _Biometrika_ , vol. 28, no. 3/4, pp. 321–377, 1936.
  * [158] G. Andrew, R. Arora, J. A. Bilmes, and K. Livescu, “Deep canonical correlation analysis,” in _Proceedings of the 30th International Conference on Machine Learning_ , 2013.
  * [159] W. Wang, R. Arora, K. Livescu, and J. A. Bilmes, “On deep multi-view representation learning,” in _ICML_ , 2015.
  * [160] W. Wang, X. Yan, H. Lee, and K. Livescu, “Deep variational canonical correlation analysis,” _arXiv preprint arXiv:1610.03454_ , 2016.
  * [161] T. Michaeli, W. Wang, and K. Livescu, “Nonparametric canonical correlation analysis,” in _International Conference on Machine Learning_. PMLR, 2016, pp. 1967–1976.
  * [162] T. Melzer, M. Reiter, and H. Bischof, “Nonlinear feature extraction using generalized canonical correlation analysis,” in _International Conference on Artificial Neural Networks_ , pp. 353–360.
  * [163] P. L. Lai and C. Fyfe, “Kernel and nonlinear canonical correlation analysis,” _Int. J. Neural Syst._ , vol. 10, no. 5, pp. 365–377, 2000.
  * [164] ——, “A neural implementation of canonical correlation analysis,” _Neural Networks_ , vol. 12, no. 10, pp. 1391–1397, 1999.
  * [165] F. R. Bach and M. I. Jordan, “A probabilistic interpretation of canonical correlation analysis,” Department of Statistics, University of California, Berkeley, Tech. Rep. 688, 2005.
  * [166] W. Wang, R. Arora, K. Livescu, and J. Bilmes, “Unsupervised learning of acoustic features via deep canonical correlation analysis,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2015, pp. 4590–4594.
  * [167] K. M. Hermann and P. Blunsom, “Multilingual distributed representations without word alignment,” in _Proceedings of International Conference on Learning Representations_.
  * [168] P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck, “Learning deep structured semantic models for web search using clickthrough data,” in _Int. Conf. Information and Knowledge Management_ , 2013.
  * [169] B. Shi, W.-N. Hsu, and A. Mohamed, “Robust self-supervised audio-visual speech recognition,” in _Interspeech_ , 2022.
  * [170] G. Chrupała, “Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques,” _arXiv preprint arXiv:2104.13225_ , 2021.
  * [171] G. Synnaeve, M. Versteegh, and E. Dupoux, “Learning words from images and speech,” in _NIPS Workshop Learn. Semantics_ , 2014.
  * [172] D. Harwath, A. Torralba, and J. R. Glass, “Unsupervised learning of spoken language with visual context,” in _Proceedings of Advances in Neural Information Processing Systems_ , 2016.
  * [173] D. Harwath and J. R. Glass, “Deep multimodal semantic embeddings for speech and images,” in _Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding_ , 2015.
  * [174] D. Merkx, S. Frank, and M. Ernestus, “Language learning using speech to image retrieval,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2019.
  * [175] A. Rouditchenko, A. Boggust, D. Harwath, B. Chen, D. Joshi, S. Thomas, K. Audhkhasi, H. Kuehne, R. Panda, R. Feris _et al._ , “AVLnet: Learning audio-visual language representations from instructional videos,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [176] P. Peng and D. Harwath, “Fast-slow transformer for visually grounding speech,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2022.
  * [177] G. Ilharco, Y. Zhang, and J. Baldridge, “Large-scale representation learning from visually grounded untranscribed speech,” in _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , 2019.
  * [178] R. Sanabria, A. Waters, and J. Baldridge, “Talk, don’t write: A study of direct speech-based image retrieval,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [179] D. M. Chan, S. Ghosh, D. Chakrabarty, and B. Hoffmeister, “Multi-modal pre-training for automated speech recognition,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2022\.
  * [180] P. Peng and D. Harwath, “Self-supervised representation learning for speech using visual grounding and masked language modeling,” in _AAAI SAS workshop_ , 2022.
  * [181] D. Harwath, W.-N. Hsu, and J. Glass, “Learning hierarchical discrete linguistic units from visually-grounded speech,” in _Proceedings of International Conference on Learning Representations_ , 2019.
  * [182] G. Chrupała, L. Gelderloos, and A. Alishahi, “Representations of language in a model of visually grounded speech signal,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2017, pp. 613–622.
  * [183] O. Scharenborg _et al._ , “Linguistic unit discovery from multi-modal inputs in unwritten languages: Summary of the “Speaking Rosetta” JSALT 2017 Workshop,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2018.
  * [184] D. Harwath, A. Recasens, D. Surís, G. Chuang, A. Torralba, and J. Glass, “Jointly discovering visual objects and spoken words from raw sensory input,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 649–665.
  * [185] P. Peng and D. Harwath, “Word discovery in visually grounded, self-supervised speech models,” _arXiv preprint arXiv:2203.15081_ , 2022.
  * [186] L. Wang and M. Hasegawa-Johnson, “A DNN-HMM-DNN hybrid model for discovering word-like units from spoken captions and image regions,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020.
  * [187] D. Harwath, G. Chuang, and J. Glass, “Vision as an interlingua: Learning multilingual semantic embeddings of untranscribed speech,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2018, pp. 4969–4973.
  * [188] W. N. Havard, J.-P. Chevrot, and L. Besacier, “Models of visually grounded speech signal pay attention to nouns: A bilingual experiment on English and Japanese,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2019, pp. 8618–8622.
  * [189] H. Kamper and M. Roth, “Visually grounded cross-lingual keyword spotting in speech,” in _Proc. SLTU_ , 2018.
  * [190] A. Pasad, B. Shi, H. Kamper, and K. Livescu, “On the contributions of visual and textual supervision in low-resource semantic speech retrieval,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2019.
  * [191] A. Bapna, Y. an Chung, N. Wu, A. Gulati, Y. Jia, J. H. Clark, M. Johnson, J. Riesa, A. Conneau, and Y. Zhang, “SLAM: A unified encoder for speech and language modeling via speech-text joint pre-training,” 2021.
  * [192] K. Levin, A. Jansen, and B. Van Durme, “Segmental acoustic indexing for zero resource keyword search,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2015.
  * [193] G. Chen, C. Parada, and T. N. Sainath, “Query-by-example keyword spotting using long short-term memory networks,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2015.
  * [194] S. Settle, K. Levin, H. Kamper, and K. Livescu, “Query-by-example search with discriminative neural acoustic word embeddings,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2017\.
  * [195] H. Kamper, K. Livescu, and S. Goldwater, “An embedded segmental k-means model for unsupervised segmentation and clustering of speech,” in _Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding_ , 2017.
  * [196] H. Kamper, A. Jansen, and S. Goldwater, “A segmental framework for fully-unsupervised large-vocabulary speech recognition,” _Computer Speech & Language_, vol. 46, pp. 154–174, 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0885230816301905
  * [197] A. L. Maas, S. D. Miller, T. M. O’neil, A. Y. Ng, and P. Nguyen, “Word-level acoustic modeling with convolutional vector regression,” in _Proc. ICML Workshop Representation Learn._ , 2012.
  * [198] S. Bengio and G. Heigold, “Word embeddings for speech recognition,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2014.
  * [199] K. Levin, K. Henry, A. Jansen, and K. Livescu, “Fixed-dimensional acoustic embeddings of variable-length segments in low-resource settings,” in _Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding_ , 2013.
  * [200] N. Holzenberger, M. Du, J. Karadayi, R. Riad, and E. Dupoux, “Learning word embeddings: Unsupervised methods for fixed-size representations of variable-length speech segments,” in _Proceedings of the Annual Conference of the International Speech Communication Association_. ISCA, 2018.
  * [201] H. Kamper, “Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder models,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2019, pp. 6535–3539.
  * [202] P. Peng, H. Kamper, and K. Livescu, “A correspondence variational autoencoder for unsupervised acoustic word embeddings,” in _Proc. NeurIPS Workshop on Self-Supervised Learning for Speech and Audio Processing_ , 2020.
  * [203] M. A. Carlin, S. Thomas, A. Jansen, and H. Hermansky, “Rapid evaluation of speech representations for spoken term discovery,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2011\.
  * [204] H. Kamper, W. Wang, and K. Livescu, “Deep convolutional acoustic word embeddings using word-pair side information,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2016.
  * [205] S. Toshniwal, H. Shi, B. Shi, L. Gao, K. Livescu, and K. Gimpel, “A cross-task analysis of text span representations,” in _Proceedings of the 5th Workshop on Representation Learning for NLP_ , 2020, pp. 166–176.
  * [206] S. Wang, L. Thompson, and M. Iyyer, “Phrase-BERT: Improved phrase embeddings from BERT with an application to corpus exploration,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 2021, pp. 10 837–10 851.
  * [207] L. van Staden and H. Kamper, “A comparison of self-supervised speech representations as input features for unsupervised acoustic word embeddings,” in _Proceedings of IEEE Spoken Language Technology Workshop_ , 2021, pp. 927–934.
  * [208] J. Kahn _et al._ , “Libri-light: A benchmark for ASR with limited or no supervision,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020.
  * [209] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio Set: An ontology and human-labeled dataset for audio events,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2017.
  * [210] A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein, “Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation,” _ACM Transactions on Graphics (TOG)_ , vol. 37, no. 4, pp. 1–11, 2018.
  * [211] C. Cieri, D. Miller, and K. Walker, “The Fisher corpus: A resource for the next generations of speech-to-text,” in _Proceedings of International Conference on Language Resources and Evaluation_ , vol. 4, 2004, pp. 69–71.
  * [212] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: An ASR corpus based on public domain audio books,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2015.
  * [213] D. B. Paul and J. Baker, “The design for the Wall Street Journal-based CSR corpus,” in _Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23–26_ , 1992.
  * [214] R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common Voice: A massively-multilingual speech corpus,” in _Proceedings of International Conference on Language Resources and Evaluation_ , 2020.
  * [215] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “MLS: A large-scale multilingual dataset for speech research,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020.
  * [216] C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, “VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” in _ACL_ , 2021.
  * [217] M. J. Gales, K. M. Knill, A. Ragni, and S. P. Rath, “Speech recognition and keyword spotting for low-resource languages: BABEL project research at CUED,” in _Fourth International Workshop on Spoken Language Technologies for Under-resourced Languages (SLTU-2014)_ , 2014.
  * [218] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli, “XLS-R: Self-supervised cross-lingual speech representation learning at scale,” 2021\.
  * [219] S. Chen, Y. Wu, C. Wang, Z. Chen, Z. Chen, S. Liu, J. Wu, Y. Qian, F. Wei, J. Li, and X. Yu, “UniSpeech-SAT: Universal speech representation learning with speaker aware pre-training,” 2021.
  * [220] G. Chen, S. Chai, G.-B. Wang, J. Du, W.-Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang, M. Jin, S. Khudanpur, S. Watanabe, S. Zhao, W. Zou, X. Li, X. Yao, Y. Wang, Z. You, and Z. Yan, “GigaSpeech: An Evolving, Multi-Domain ASR Corpus with 10,000 Hours of Transcribed Audio,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [221] F. Hernandez, V. Nguyen, S. Ghannay, N. Tomashenko, and Y. Esteve, “TED-LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation,” in _International Conference on Speech and Computer_. Springer, 2018, pp. 198–208.
  * [222] A. Rousseau, P. Deléglise, and Y. Esteve, “TED-LIUM: An automatic speech recognition dedicated corpus,” in _Proceedings of International Conference on Language Resources and Evaluation_ , 2012, pp. 125–129.
  * [223] J. J. Godfrey, E. C. Holliman, and J. McDaniel, “SWITCHBOARD: Telephone speech corpus for research and development,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 1992.
  * [224] J. Valk and T. Alumäe, “VoxLingua107: A dataset for spoken language recognition,” in _Proceedings of IEEE Spoken Language Technology Workshop_ , 2021.
  * [225] Y. Liu, P. Fung, Y. Yang, C. Cieri, S. Huang, and D. Graff, “HKUST/MTS: A very large scale Mandarin telephone speech corpus,” in _Proceedings of International Symposium on Chinese Spoken Language Processing_ , 2006.
  * [226] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline,” in _O-COCOSDA_. IEEE, 2017, pp. 1–5.
  * [227] Beijing DataTang Technology Co., Ltd., “aidatatang_200zh, a free Chinese Mandarin speech corpus,” http://www.datatang.com.
  * [228] Magic Data Technology Co., Ltd., “MAGICDATA Mandarin Chinese Read Speech Corpus,” 2019, http://www.imagicdatatech.com/index.php/home/dataopensource/data_info/id/101.
  * [229] Surfingtech, “ST-CMDS-20170001_1, Free ST Chinese Mandarin Corpus.”
  * [230] Primewords Information Technology Co., Ltd., “Primewords Chinese Corpus Set 1,” 2018, https://www.primewords.cn.
  * [231] M. Ravanelli, L. Cristoforetti, R. Gretter, M. Pellin, A. Sosi, and M. Omologo, “The DIRHA-English corpus and related tasks for distant-speech recognition in domestic environments,” in _Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding_ , 2015.
  * [232] J. Barker, S. Watanabe, E. Vincent, and J. Trmal, “The fifth ‘CHiME’ Speech Separation and Recognition Challenge: Dataset, task and baselines,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2018.
  * [233] V. Hozjan, Z. Kacic, A. Moreno, A. Bonafonte, and A. Nogueiras, “Interface databases: Design and collection of a multilingual emotional speech database,” in _Proceedings of International Conference on Language Resources and Evaluation_ , 2002.
  * [234] A. B. Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph,” in _Proceedings of Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , 2018.
  * [235] C. Veaux, J. Yamagishi, and K. MacDonald, “CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit,” 2016.
  * [236] A. Nagrani, J. S. Chung, and A. Zisserman, “VoxCeleb: A large-scale speaker identification dataset,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2017.
  * [237] L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio, “Speech Model Pre-Training for End-to-End Spoken Language Understanding,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2019.
  * [238] X. Anguera, L.-J. Rodriguez-Fuentes, A. Buzo, F. Metze, I. Szöke, and M. Penagarikano, “QUESST2014: Evaluating query-by-example speech search in a zero-resource setting with real-life queries,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2015\.
  * [239] A. C. Kocabiyikoglu, L. Besacier, and O. Kraif, “Augmenting LibriSpeech with French translations: A multimodal corpus for direct speech translation evaluation,” in _Proceedings of International Conference on Language Resources and Evaluation_ , 2018.
  * [240] C. Wang, A. Wu, and J. Pino, “CoVoST 2 and massively multilingual speech-to-text translation,” _arXiv preprint arXiv:2007.10310_ , 2020.
  * [241] M. Y. Tachbelie, S. T. Abate, and L. Besacier, “Using different acoustic, lexical and language modeling units for ASR of an under-resourced language–Amharic,” _Speech Communication_ , vol. 56, pp. 181–194, 2014\.
  * [242] F. A. A. Laleye, L. Besacier, E. C. Ezin, and C. Motamed, “First automatic Fongbe continuous speech recognition system: Development of acoustic models and language models,” in _FedCSIS_. IEEE, 2016, pp. 477–482.
  * [243] H. Gelas, L. Besacier, and F. Pellegrino, “Developments of Swahili resources for an automatic speech recognition system,” in _Spoken Language Technologies for Under-Resourced Languages_ , 2012.
  * [244] E. Gauthier, L. Besacier, S. Voisin, M. Melese, and U. P. Elingui, “Collecting resources in sub-Saharan African languages for automatic speech recognition: A case study of Wolof,” in _LREC 2016_ , 2016.
  * [245] D. Snyder, G. Chen, and D. Povey, “MUSAN: A music, speech, and noise corpus,” _arXiv preprint arXiv:1510.08484_ , 2015.
  * [246] D. Stowell, M. D. Wood, H. Pamuła, Y. Stylianou, and H. Glotin, “Automatic acoustic detection of birds through deep learning: The first Bird Audio Detection Challenge,” _Methods in Ecology and Evolution_ , vol. 10, no. 3, pp. 368–380, 2019.
  * [247] P. Warden, “Speech Commands: A dataset for limited-vocabulary speech recognition,” _arXiv preprint arXiv:1804.03209_ , 2018.
  * [248] T. Oponowicz, “Spoken language identification, 2018,” _https://www.kaggle.com/toponowicz/spoken-language-identification_.
  * [249] A. Mesaros, T. Heittola, and T. Virtanen, “Detection and classification of acoustic scenes and events,” _Detection and Classification of Acoustic Scenes and Events 2018 Workshop (DCASE2018)_ , pp. 9–13, 2018.
  * [250] J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan, “Neural audio synthesis of musical notes with WaveNet autoencoders,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1068–1077.
  * [251] M. Ravanelli and Y. Bengio, “Learning speaker representations with mutual information,” _arXiv preprint arXiv:1812.00271_ , 2018.
  * [252] S. Khurana, A. Laurent, W.-N. Hsu, J. Chorowski, A. Lancucki, R. Marxer, and J. Glass, “A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020.
  * [253] S. H. Weinberger and S. Kunath, “Towards a typology of English accents,” _AACL Abstract Book_ , vol. 104, 2009.
  * [254] T. Likhomanenko, Q. Xu, J. Kahn, G. Synnaeve, and R. Collobert, “slimIPL: Language-Model-Free Iterative Pseudo-Labeling,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021\.
  * [255] Y. Zhang, J. Qin, D. S. Park, W. Han, C.-C. Chiu, R. Pang, Q. V. Le, and Y. Wu, “Pushing the limits of semi-supervised learning for automatic speech recognition,” in _Workshop on Self-Supervised Learning for Speech and Audio Processing, NeurIPS_ , 2020.
  * [256] M. Hajibabaei and D. Dai, “Unified hypersphere embedding for speaker recognition,” _arXiv preprint arXiv:1807.08312_ , 2018.
  * [257] A. Hajavi and A. Etemad, “Siamese capsule network for end-to-end speaker recognition in the wild,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2021.
  * [258] Y. Wang, A. Boumadane, and A. Heba, “A fine-tuned wav2vec 2.0/HuBERT benchmark for speech emotion recognition, speaker verification and spoken language understanding,” _arXiv preprint arXiv:2111.02735_ , 2021.
  * [259] L. J. Rodríguez-Fuentes, A. Varona, M. Penagarikano, G. Bordel, and M. Diez, “GTTS-EHU systems for QUESST at MediaEval 2014,” in _MediaEval_ , 2014.
  * [260] S. Evain, H. Nguyen, H. Le, M. Z. Boito, S. Mdhaffar, S. Alisamir, Z. Tong, N. Tomashenko, M. Dinarelli, T. Parcollet, A. Allauzen, Y. Estève, B. Lecouteux, F. Portet, S. Rossato, F. Ringeval, D. Schwab, and L. Besacier, “LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [261] E. Dunbar, J. Karadayi, M. Bernard, X.-N. Cao, R. Algayres, L. Ondel, L. Besacier, S. Sakti, and E. Dupoux, “The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020.
  * [262] J. Turian, J. Shier, H. R. Khan, B. Raj, B. W. Schuller, C. J. Steinmetz, C. Malloy, G. Tzanetakis, G. Velarde, K. McNally, M. Henry, N. Pinto, C. Noufi, C. Clough, D. Herremans, E. Fonseca, J. Engel, J. Salamon, P. Esling, P. Manocha, S. Watanabe, Z. Jin, and Y. Bisk, “Hear: Holistic evaluation of audio representations,” in _Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track_ , vol. 176, 2022, pp. 125–145.
  * [263] J. Shor, A. Jansen, R. Maor, O. Lang, O. Tuval, F. de Chaumont Quitry, M. Tagliasacchi, I. Shavitt, D. Emanuel, and Y. Haviv, “Towards Learning a Universal Non-Semantic Representation of Speech,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020\.
  * [264] L. Wang, P. Luc, Y. Wu, A. Recasens, L. Smaira, A. Brock, A. Jaegle, J.-B. Alayrac, S. Dieleman, J. Carreira, and A. van den Oord, “Towards learning universal audio representations,” _arXiv preprint arXiv:2111.12124_ , 2021\.
  * [265] A. Tjandra, S. Sakti, and S. Nakamura, “Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challenge,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020.
  * [266] B. van Niekerk, L. Nortje, and H. Kamper, “Vector-Quantized Neural Networks for Acoustic Unit Discovery in the ZeroSpeech 2020 Challenge,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020.
  * [267] M. Faruqi and C. Dyer, “Community evaluation and exchange of word vectors and wordvectors.org,” in _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstration_ , 2014.
  * [268] A. Baevski, W.-N. Hsu, A. Conneau, and M. Auli, “Unsupervised speech recognition,” _arXiv preprint arXiv:2105.11084_ , 2021.
  * [269] E. Voita, R. Sennrich, and I. Titov, “The bottom-up evolution of representations in the Transformer: A study with machine translation and language modeling objectives,” in _NAACL_ , 2019.
  * [270] T. G. Grigg, D. Busbridge, J. Ramapuram, and R. Webb, “Do self-supervised and supervised methods learn similar visual representations?” in _https://arxiv.org/abs/2110.00528_ , 2021.
  * [271] B. van Niekerk, L. Nortje, M. Baas, and H. Kamper, “Analyzing speaker information in self-supervised models to improve zero-resource speech processing,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [272] S. Ling, J. Salazar, Y. Liu, and K. Kirchhoff, “BERTphone: Phonetically-aware encoder representations for utterance-level speaker and language recognition,” in _Proceedings of Odyssey: The Speaker and Language Recognition Workshop_ , 2020, pp. 9–16.
  * [273] C. Wang _et al._ , “Self-supervised learning for speech recognition with intermediate layer supervision,” _arXiv e-print 2112.08778_ , 2021.
  * [274] S. wen Yang, A. T. Liu, and H. yi Lee, “Understanding self-attention of self-supervised audio Transformers,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020.
  * [275] Y.-A. Chung, Y. Belinkov, and J. Glass, “Similarity analysis of self-supervised speech representations,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2021.
  * [276] H. Zhou, A. Baevski, and M. Auli, “A comparison of discrete latent variable models for speech representation learning,” in _https://arxiv.org/abs/2010.14230_ , 2020.
  * [277] M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux, “Unsupervised pretraining transfers well across languages,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020.
  * [278] J. Pu, Y. Yang, R. Li, O. Elibol, and J. Droppo, “Scaling effect of self-supervised models,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [279] M. Versteegh, R. Thiolliere, T. Schatz, X. N. Cao, X. Anguera, A. Jansen, and E. Dupoux, “The Zero Resource Speech Challenge 2015,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015\.
  * [280] X. Chang, T. Maekaku, P. Guo, J. Shi, Y.-J. Lu, A. S. Subramanian, T. Wang, S.-w. Yang, Y. Tsao, H.-y. Lee _et al._ , “An exploration of self-supervised pretrained representations for end-to-end speech recognition,” _arXiv preprint arXiv:2110.04590_ , 2021.
  * [281] W.-N. Hsu, A. Sriram, A. Baevski, T. Likhomanenko, Q. Xu, V. Pratap, J. Kahn, A. Lee, R. Collobert, G. Synnaeve, and M. Auli, “Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [282] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised cross-lingual representation learning for speech recognition,” _CoRR, abs/2006.13979_ , 2020.
  * [283] D.-R. Liu, K.-Y. Chen, H.-Y. Lee, and L.-s. Lee, “Completely unsupervised phoneme recognition by adversarially learning mapping relationships from audio embeddings,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , pp. 3748–3752, 2018.
  * [284] Y.-H. Wang, H. yi Lee, and L. shan Lee, “Segmental audio word2Vec: Representing utterances as sequences of vectors with applications in spoken term detection,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2018.
  * [285] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, “Improved training of Wasserstein GANs,” _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , pp. 5769–5779, 2017.
  * [286] Y.-A. Chung, W.-H. Weng, S. Tong, and J. Glass, “Unsupervised cross-modal alignment of speech and text embedding spaces,” _Proceedings of Advances in Neural Information Processing Systems_ , vol. 31, pp. 7354–7364, 2018\.
  * [287] A. Conneau, G. Lample, M. Ranzato, L. Denoyer, and H. Jégou, “Word translation without parallel data,” _Proceedings of International Conference on Learning Representations_ , 2018.
  * [288] Y.-A. Chung, W.-H. Weng, S. Tong, and J. Glass, “Towards unsupervised speech-to-text translation,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2019, pp. 7170–7174.
  * [289] M. Artetxe, G. Labaka, and E. Agirre, “A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 789–798. [Online]. Available: https://aclanthology.org/P18-1073
  * [290] C.-K. Yeh, J. Chen, C. Yu, and D. Yu, “Unsupervised speech recognition via segmental empirical output distribution matching,” _Proceedings of International Conference on Learning Representations_ , 2018.
  * [291] Y.-H. Wang, C.-T. Chung, and H.-y. Lee, “Gate activation signal analysis for gated recurrent neural networks and its correlation with phoneme boundaries,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2017.
  * [292] Y. Liu, J. Chen, and L. Deng, “Unsupervised sequence classification using sequential output statistics,” in _Proceedings of Advances in Neural Information Processing Systems_ , 2017.
  * [293] K.-Y. Chen, C.-P. Tsai, D.-R. Liu, H.-Y. Lee, and L.-s. Lee, “Completely unsupervised speech recognition by a generative adversarial network harmonized with iteratively refined hidden Markov models,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , pp. 1856–1860, 2019.
  * [294] O. Klejch, E. Wallington, and P. Bell, “Deciphering speech: a zero-resource approach to cross-lingual transfer in ASR,” _arXiv preprint arXiv:2111.06799_ , 2022.
  * [295] S. Ravi and K. Knight, “Deciphering foreign language,” in _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. Portland, Oregon, USA: Association for Computational Linguistics, Jun. 2011, pp. 12–21. [Online]. Available: https://aclanthology.org/P11-1002
  * [296] A. H. Liu, W.-N. Hsu, M. Auli, and A. Baevski, “Towards end-to-end unsupervised speech recognition,” _arXiv preprint arXiv:2204.02492_ , 2022\.
  * [297] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Proceedings of Advances in Neural Information Processing Systems_ , 2014, pp. 2672–2680.
  * [298] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein GAN,” _Proceedings of the 34th International Conference on Machine Learning_ , pp. 214–223, 2017.
  * [299] M. Artetxe, G. Labaka, E. Agirre, and K. Cho, “Unsupervised neural machine translation,” _Proceedings of International Conference on Learning Representations_ , 2018.
  * [300] G. Lample, A. Conneau, L. Denoyer, and M. Ranzato, “Unsupervised machine translation using monolingual corpora only,” _Proceedings of International Conference on Learning Representations_ , 2018.
  * [301] G.-T. Lin, C.-J. Hsu, D.-R. Liu, H.-Y. Lee, and Y. Tsao, “Analyzing the robustness of unsupervised speech recognition,” 2021.
  * [302] A. Tjandra, S. Sakti, and S. Nakamura, “Listening while speaking: Speech chain by deep learning,” in _Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding_. IEEE, 2017, pp. 301–308.
  * [303] T. Hori, R. Astudillo, T. Hayashi, Y. Zhang, S. Watanabe, and J. Le Roux, “Cycle-consistency training for end-to-end speech recognition,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2019, pp. 6271–6275.
  * [304] G. Wang, A. Rosenberg, Z. Chen, Y. Zhang, B. Ramabhadran, Y. Wu, and P. Moreno, “Improving speech recognition using consistent predictions on synthesized speech,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2020, pp. 7029–7033.
  * [305] A. Tjandra, S. Sakti, and S. Nakamura, “Machine speech chain with one-shot speaker adaptation,” _arXiv preprint arXiv:1803.10525_ , 2018.
  * [306] ——, “End-to-end feedback loss in speech chain framework via straight-through estimator,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2019, pp. 6281–6285.
  * [307] M. K. Baskar, S. Watanabe, R. Astudillo, T. Hori, L. Burget, and J. Černockỳ, “Semi-supervised sequence-to-sequence ASR using unpaired speech and text,” _arXiv preprint arXiv:1905.01152_ , 2019.
  * [308] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” _Machine Learning_ , vol. 8, no. 3, pp. 229–256, 1992.
  * [309] F. Jelinek, _Statistical Methods for Speech Recognition_. MIT press, 1997.
  * [310] M. Mohri, F. Pereira, and M. Riley, “Weighted finite-state transducers in speech recognition,” _Computer Speech & Language_, vol. 16, no. 1, pp. 69–88, 2002.
  * [311] C. Gulcehre, O. Firat, K. Xu, K. Cho, L. Barrault, H.-C. Lin, F. Bougares, H. Schwenk, and Y. Bengio, “On using monolingual corpora in neural machine translation,” _arXiv preprint arXiv:1503.03535_ , 2015.
  * [312] J. Chorowski and N. Jaitly, “Towards better decoding and language model integration in sequence to sequence models,” _arXiv preprint arXiv:1612.02695_ , 2016.
  * [313] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan _et al._ , “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2018, pp. 4779–4783.
  * [314] J. Li, R. Gadde, B. Ginsburg, and V. Lavrukhin, “Training neural speech recognition systems with synthetic speech augmentation,” _arXiv preprint arXiv:1811.00707_ , 2018.
  * [315] S. Ueno, M. Mimura, S. Sakai, and T. Kawahara, “Multi-speaker sequence-to-sequence speech synthesis for data augmentation in acoustic-to-word speech recognition,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2019, pp. 6161–6165.
  * [316] A. Rosenberg, Y. Zhang, B. Ramabhadran, Y. Jia, P. Moreno, Y. Wu, and Z. Wu, “Speech recognition with augmented synthesized speech,” in _Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding_. IEEE, 2019, pp. 996–1002.
  * [317] A. Laptev, R. Korostik, A. Svischev, A. Andrusenko, I. Medennikov, and S. Rybin, “You do not need more data: Improving end-to-end speech recognition by text-to-speech data augmentation,” in _2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI)_. IEEE, 2020, pp. 439–444.
  * [318] Y. Huang, L. He, W. Wei, W. Gale, J. Li, and Y. Gong, “Using personalized speech synthesis and neural language generator for rapid speaker adaptation,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2020, pp. 7399–7403.
  * [319] T. Hayashi, S. Watanabe, Y. Zhang, T. Toda, T. Hori, R. Astudillo, and K. Takeda, “Back-translation-style data augmentation for end-to-end ASR,” in _Proceedings of IEEE Spoken Language Technology Workshop_. IEEE, 2018, pp. 426–433.
  * [320] A. Renduchintala, S. Ding, M. Wiesner, and S. Watanabe, “Multi-Modal Data Augmentation for End-to-end ASR,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2018.
  * [321] R. Masumura, N. Makishima, M. Ihori, A. Takashima, T. Tanaka, and S. Orihashi, “Phoneme-to-Grapheme Conversion Based Large-Scale Pre-Training for End-to-End Automatic Speech Recognition,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2020\.
  * [322] R. Sennrich, B. Haddow, and A. Birch, “Improving neural machine translation models with monolingual data,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2016, pp. 86–96.
  * [323] A. Jansen, E. Dupoux, S. Goldwater, M. Johnson, S. Khudanpur, K. Church, N. Feldman, H. Hermansky, F. Metze, R. Rose _et al._ , “A summary of the 2012 JHU CLSP workshop on zero resource speech technologies and models of early language acquisition,” in _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2013, pp. 8111–8115.
  * [324] E. Dunbar, X. N. Cao, J. Benjumea, J. Karadayi, M. Bernard, L. Besacier, X. Anguera, and E. Dupoux, “The Zero Resource Speech Challenge 2017,” in _Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding_. IEEE, 2017, pp. 323–330.
  * [325] E. Dunbar, R. Algayres, J. Karadayi, M. Bernard, J. Benjumea, X.-N. Cao, L. Miskic, C. Dugrain, L. Ondel, A. W. Black _et al._ , “The Zero Resource Speech Challenge 2019: TTS without T,” _arXiv preprint arXiv:1904.11469_ , 2019.
  * [326] T. A. Nguyen, M. de Seyssel, P. Rozé, M. Rivière, E. Kharitonov, A. Baevski, E. Dunbar, and E. Dupoux, “The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling,” _arXiv preprint arXiv:2011.11588_ , 2020.
  * [327] L. Ondel, L. Burget, and J. Černockỳ, “Variational inference for acoustic unit discovery,” _Procedia Computer Science_ , vol. 81, pp. 80–86, 2016.
  * [328] M. Heck, S. Sakti, and S. Nakamura, “Feature optimized DPGMM clustering for unsupervised subword modeling: A contribution to ZeroSpeech 2017,” in _Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding_. IEEE, 2017, pp. 740–746.
  * [329] A. Tjandra, B. Sisman, M. Zhang, S. Sakti, H. Li, and S. Nakamura, “VQVAE Unsupervised Unit Discovery and Multi-Scale Code2Spec Inverter for ZeroSpeech Challenge 2019,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2019.
  * [330] T. Maekaku, X. Chang, Y. Fujita, L.-W. Chen, S. Watanabe, and A. Rudnicky, “Speech Representation Learning Combining Conformer CPC with Deep Cluster for the ZeroSpeech Challenge 2021,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [331] F. Kreuk, A. Polyak, J. Copet, E. Kharitonov, T. A. Nguyen, M. Rivière, W. Hsu, A. Mohamed, E. Dupoux, and Y. Adi, “Textless speech emotion conversion using decomposed and discrete representations,” _arXiv preprint arXiv:2111.07402_ , 2021.
  * [332] T. A. Nguyen, E. Kharitonov, J. Copet, Y. Adi, W.-N. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed, and E. Dupoux, “Generative spoken dialogue language modeling,” 2022.
  * [333] G.-T. Lin, Y.-S. Chuang, H.-L. Chung, S.-w. Yang, H.-J. Chen, S. Dong, S.-W. Li, A. Mohamed, H.-y. Lee, and L.-s. Lee, “DUAL: Discrete spoken unit adaptive learning for textless spoken question answering,” _CoRR_ , 2022\.
  * [334] J. Shi, X. Chang, T. Hayashi, Y.-J. Lu, S. Watanabe, and B. Xu, “Discretization and re-synthesis: an alternative method to solve the cocktail party problem,” _arXiv preprint arXiv:2112.09382_ , 2021.
  * [335] T. Hayashi and S. Watanabe, “Discretalk: Text-to-speech as a machine translation problem,” _arXiv preprint arXiv:2005.05525_ , 2020.
  * [336] A. Lee, H. Gong, P. Duquenne, H. Schwenk, P. Chen, C. Wang, S. Popuri, J. Pino, J. Gu, and W. Hsu, “Textless speech-to-speech translation on real data,” _CoRR_ , 2021.
  * [337] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” in _International Conference on Machine Learning_ , K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97, 09–15 Jun 2019, pp. 2790–2799.
  * [338] E. B. Zaken, S. Ravfogel, and Y. Goldberg, “BitFit: Simple parameter-efficient fine-tuning for Transformer-based masked language-models,” 2021.
  * [339] D. Guo, A. Rush, and Y. Kim, “Parameter-efficient transfer learning with diff pruning,” in _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Online: Association for Computational Linguistics, Aug. 2021, pp. 4884–4896. [Online]. Available: https://aclanthology.org/2021.acl-long.378
  * [340] B. Thomas, S. Kessler, and S. Karout, “Efficient adapter transfer of self-supervised speech models for automatic speech recognition,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2022.
  * [341] K.-W. Chang, W.-C. Tseng, S.-W. Li, and H. yi Lee, “SpeechPrompt: An exploration of prompt tuning on generative spoken language model for speech processing tasks,” _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2022.
  * [342] C.-I. J. Lai, Y. Zhang, A. H. Liu, S. Chang, Y.-L. Liao, Y.-S. Chuang, K. Qian, S. Khurana, D. Cox, and J. Glass, “PARP: Prune, adjust and re-prune for self-supervised speech recognition,” in _Proceedings of Advances in Neural Information Processing Systems_ , 2021.
  * [343] H.-J. Chang, S. wen Yang, and H. yi Lee, “DistilHuBERT: Speech representation learning by layer-wise distillation of hidden-unit BERT,” 2021\.
  * [344] E. Bengio, P.-L. Bacon, J. Pineau, and D. Precup, “Conditional Computation in Neural Networks for faster models,” _arXiv:1511.06297 [cs]_ , Jan. 2016.
  * [345] A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, “A Survey of Quantization Methods for Efficient Neural Network Inference,” Jun. 2021.
  * [346] Y. Tay, M. Dehghani, D. Bahri, and D. Metzler, “Efficient Transformers: A Survey,” Mar. 2022.
  * [347] S. Cao, Y. Kang, Y. Fu, X. Xu, S. Sun, Y. Zhang, and L. Ma, “Improving streaming Transformer based ASR under a framework of self-supervised learning,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2021.
  * [348] K. Qian, Y. Zhang, H. Gao, J. Ni, C.-I. Lai, D. Cox, M. Hasegawa-Johnson, and S. Chang, “ContentVec: An improved self-supervised speech representation by disentangling speakers,” in _Proceedings of International Conference on Machine Learning_ , 2022.
  * [349] H.-S. Choi, J. Lee, W. Kim, J. H. Lee, H. Heo, and K. Lee, “Neural analysis and synthesis: Reconstructing speech from self-supervised representations,” in _NeurIPS_ , 2021.
  * [350] D. M. Chan and S. Ghosh, “Content-context factorized representations for automated speech recognition,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2022.
  * [351] H. Wu, B. Zheng, X. Li, X. Wu, H. yi Lee, and H. Meng, “Characterizing the adversarial vulnerability of speech self-supervised learning,” 2021.
  * [352] K. P. Huang, Y.-K. Fu, Y. Zhang, and H.-y. Lee, “Improving distortion robustness of self-supervised speech processing tasks with domain adaptation,” in _Proceedings of the Annual Conference of the International Speech Communication Association_ , 2022.
  * [353] H. Wang, Y. Qian, X. Wang, Y. Wang, C. Wang, S. Liu, T. Yoshioka, J. Li, and D. Wang, “Improving noise robustness of contrastive speech representation learning with speech reconstruction,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 6062–6066.
  * [354] Q.-S. Zhu, J. Zhang, Z.-Q. Zhang, M.-H. Wu, X. Fang, and L.-R. Dai, “A noise-robust self-supervised pre-training model based speech representation learning for automatic speech recognition,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 3174–3178.
  * [355] T. A. Nguyen, B. Sagot, and E. Dupoux, “Are discrete units necessary for spoken language modeling?” 2022.

|  Abdelrahman Mohamed is a research scientist at Fundamental AI Research
(FAIR), Meta. He was a principal scientist/manager in Amazon Alexa and a
researcher in Microsoft Research. Abdelrahman was in the team that started the
Deep Learning revolution in Spoken Language Processing in 2009 and received
the IEEE Signal Processing Society Best Paper Award in 2016. He has been
focusing on improving speech representation learning, where he co-organized
multiple workshops, tutorials, and special sessions.
---|---
|  Hung-yi Lee is an associate professor in the Department of Electrical
Engineering of National Taiwan University (NTU), with a joint appointment at
the Department of Computer Science & Information Engineering. He is the co-
organizer of the special session on ”New Trends in self-supervised speech
processing” at Interspeech (2020) and the workshop on ”Self-Supervised
Learning for Speech and Audio Processing” at NeurIPS (2020) and AAAI (2022).
---|---
|  Lasse Borgholt received his Ph.D. degree from University of Copenhagen in
2022. He was supervised by Professors Christian Igel and Anders Søgaard and
Adjunct Associate Professor Lars Maaløe. He currently works as an industrial
machine learning researcher at the Danish company Corti. His primary research
interests are speech recognition and representation learning for speech.
---|---
|  Jakob D. Havtorn received his bachelor’s degree in physics and master’s
degree in mathematical modeling and computation from the Technical University
of Denmark (DTU), Copenhagen, in 2016 and 2018, respectively. He is currently
an industrial Ph.D. student at Corti and DTU, supervised by Associate
Professor Jes Frellsen, Adjunct Associate Professor Lars Maaløe and Professors
Søren Hauberg and Ole Winther. His primary research interests are deep
generative latent variable models, uncertainty quantification and
representation learning for speech.
---|---
|  Joakim Edin received his master’s degree in biomedical engineering from the
Technical University of Denmark, Copenhagen, in 2019. He worked as an
industrial machine learning researcher at the Danish company Corti prior to
starting in his current occupation as a Ph.D. student at University of
Copenhagen. He is supervised by Professor Søren Brunak and Adjunct Associate
Professor Lars Maaløe. His primary research interests are representation
learning from medical conversations and text.
---|---
|  Christian Igel received the Diploma degree in computer science from the
Technical University Dortmund, Germany, in 1997, the Dr. rer. nat. degree from
the Ruhr-University Bochum, Germany, in 2002, and the Habilitation degree from
the Ruhr-University Bochum, Germany, in 2010. From 2003 to 2010, he was a
Juniorprofessor at the Institute for Neural Computation, Ruhr-University
Bochum, Germany. He joined DIKU, the Department of Computer Science at the
University of Copenhagen (UCPH), Denmark, in 2010. He is a full professor at
DIKU and currently the Director of the SCIENCE AI Centre at UCPH. He is a
Fellow of the European Lab for Learning and Intelligent Systems (ELLIS).
---|---
|  Katrin Kirchhoff is a Director of Applied Science at Amazon Web Services,
where she heads several teams in speech and audio processing. She was a
Research Professor at the UW, Seattle, for 17 years, where she co-founded the
Signal, Speech and Language Interpretation Lab. She served on the editorial
boards of Speech Communication and Computer, Speech, and Language, and was a
member of the IEEE Speech Technical Committee.
---|---
|  Shang-Wen Li is a Research and Engineering Manager at Meta AI. He worked at
Apple Siri, Amazon Alexa and AWS. He completed his PhD in 2016 from the Spoken
Language Systems group of MIT CSAIL. He co-organized the workshop of ”Self-
Supervised Learning for Speech and Audio Processing” at NeurIPS (2020) and
AAAI (2022). His recent research focuses on self-supervised learning in speech
and its application to language understanding.
---|---
|  Karen Livescu is a Professor at TTI-Chicago. She completed her PhD at MIT
in the Spoken Language Systems group. She is an ISCA Fellow and an IEEE
Distinguished Lecturer, and has served as a program chair for ICLR,
Interspeech, and ASRU. She works on a variety of topics in speech and language
processing and machine learning. She has co-organized multiple workshops on
machine learning for speech processing, multi-view representation learning,
and self-supervised representation learning.
---|---
|  Lars Maaløe received his Ph.D. degree from the Technical University of
Denmark (DTU), Copenhagen, in 2018. He was nominated Ph.D. of the year by the
Department for Applied Mathematics and Computer Science. In the past, he
worked in the financial sector and with various tech companies such as Apple.
He is a co-founder and the current CTO of Corti, a voice-based digital
assistant powered by artificial intelligence. He also holds an appointment as
Adjunct Associate Professor at DTU.
---|---
|  Tara N. Sainath is a Principal Research scientist at Google. She received
her PhD from MIT in the Spoken Language Systems Group. She is an IEEE and ISCA
Fellow and the recipient of the 2021 IEEE SPS Industrial Innovation Award. Her
research involves applications of deep neural networks for automatic speech
recognition, and has been very active in the community organizing workshops
and special sessions on this topic.
---|---
|  Shinji Watanabe is an Associate Professor at CMU. He was a research
scientist at NTT, Japan, a visiting scholar in Georgia Tech, a senior
principal research scientist at MERL, and an associate research professor at
JHU. He has published over 300 peer-reviewed papers. He serves as a Senior
Area Editor of the IEEE TASLP. He was/has been a member of several technical
committees, including the APSIPA SLA, IEEE SPS SLTC, and MLSP.
---|---