Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASR
Xugang Lu Peng Shen Yu Tsao Hisashi Kawai



A Novel Audio-Visual Information Fusion System for Mental Disorders Detection
    Yichun Li, Shuanglin Li, Syed Mohsen Naqvi
Intelligent Sensing and Communications Research Group, Newcastle University, UK

    September 9, 2024
===============================================================================================================================


§ ABSTRACT 
Transferring linguistic knowledge from a pretrained language model (PLM) to an acoustic model has been shown to greatly improve the performance of automatic speech recognition (ASR). However, due to the heterogeneous feature distributions in cross-modalities, designing an effective model for feature alignment and knowledge transfer between linguistic and acoustic sequences remains a challenging task. Optimal transport (OT), which efficiently measures probability distribution discrepancies, holds great potential for aligning and transferring knowledge between acoustic and linguistic modalities. Nonetheless, the original OT treats acoustic and linguistic feature sequences as two unordered sets in alignment and neglects temporal order information during OT coupling estimation. Consequently, a time-consuming pretraining stage is required to learn a good alignment between the acoustic and linguistic representations. In this paper, we propose a Temporal Order Preserved OT (TOT)-based Cross-modal Alignment and Knowledge Transfer (CAKT) (TOT-CAKT) for ASR. In the TOT-CAKT, local neighboring frames of acoustic sequences are smoothly mapped to neighboring regions of linguistic sequences, preserving their temporal order relationship in feature alignment and matching. With the TOT-CAKT model framework, we conduct Mandarin ASR experiments with a pretrained Chinese PLM for linguistic knowledge transfer. Our results demonstrate that the proposed TOT-CAKT significantly improves ASR performance compared to several state-of-the-art models employing linguistic knowledge transfer, and addresses the weaknesses of the original OT-based method in sequential feature alignment for ASR.



Optimal transport, Cross-modal knowledge transfer, automatic speech recognition




§ INTRODUCTION

The combination of a pretrained language model (PLM) with an end-to-end (E2E)-based acoustical model for automatic speech recognition (ASR) has made significant progress in recent years <cit.>. The advantage of incorporating a PLM in ASR lies in the availability of unpaired large text corpora for training the PLM. Moreover, the linguistic knowledge encoded in the PLM can be utilized in ASR decoding. In most studies, the PLM is employed as an external language model (LM) for post-processing tasks such as beam search or rescoring in ASR <cit.>. However, using an external LM for post-processing compromises the speed and sometimes parallel decoding capabilities of ASR. Addressing how to transfer linguistic knowledge to acoustic encoding during model training, and subsequently conducting speech recognition without relying on any external LM post-training, is an intriguing research topic. 

In this study, our focus is on transferring linguistic knowledge from a PLM to a temporal connectionist temporal classification (CTC)-based ASR <cit.>. While there are several advanced end-to-end (E2E)-based ASR approaches that incorporate linguistic knowledge in acoustic model learning <cit.>, using a PLM, such as bidirectional encoder representation from transformers (BERT) <cit.>), facilitates linguistic knowledge transfer in ASR <cit.>, This knowledge transfer can also occur with a pretrained acoustic encoder, such as wav2vec2 <cit.>, for both linguistic and acoustic knowledge transfer <cit.>. However, due to the heterogeneous feature distributions in acoustic and linguistic spaces, it remains a challenging task to efficiently align feature representations between linguistic and acoustic modalities to facilitate knowledge transfer. In most studies, a cross-attention module is designed to integrate acoustic and text representations within a transformer decoder framework for combining acoustic and linguistic knowledge in ASR <cit.>. Yet, in the decoding stage, true text representations are unavailable, leading to the adoption of predicted text representations in decoding. This mismatch between training and testing phases weakens the benefits of linguistic information in ASR. 

For efficient alignment and matching, an effective distance metric is needed to measure the difference between acoustic and linguistic feature representations. Considering this requirement, optimal transport (OT) emerges as a suitable tool for cross-modal alignment and linguistic knowledge transfer. OT, originally proposed for optimal allocating resources  and later as a measure of discrepancies between probability distributions <cit.>, has found widespread applications in machine learning, particularly in domain adaptation <cit.>. In the field of speech, it has been employed for cross-domain spoken language recognition and speech enhancement <cit.>, as well as in speech translation and understanding <cit.>. The OT has been initially proposed for linguistic knowledge transfer learning for ASR in <cit.>. While OT is applicable for cross-domain alignment, its use in speech encounters a limitation. The original OT treats acoustic and linguistic feature sequences as two unordered sets in alignment and neglects temporal order information during OT coupling estimation. While acoustic and text speech exhibit a strong temporal order structure, requiring preservation of their temporal order relationship during alignment between an acoustic sequence and a linguistic sequence. Therefore, in <cit.>, a well pretrained acoustic model was applied in order to efficiently explore matched acoustic features to those linguistic features during cross-modal learning. And the performance was strongly depended on the goodness of the pretrained acoustic model. In this paper, we propose a Temporal Order Preserved OT (TOT)-based Cross-modal Alignment and Knowledge Transfer (CAKT) model (TOT-CAKT) for CTC-based ASR. With the TOT-CAKT, the temporal order relationship is explicitly maintained during feature alignment and matching. It is hypothesized that through this alignment and matching, linguistic knowledge can be efficiently transferred to acoustic encoding, thereby enhancing ASR performance. 

The rest of this paper is organized as follows: the proposed method is introduced in Section <ref>, where a cross-modal alignment module based on TOT and a neural adapter module for efficient linguistic feature transfer are designed. In Section <ref>, we conduct experiments to evaluate TOT-CAKT, comparing the results with several knowledge transfer learning algorithms for ASR, and provide a visualization of the learned transport coupling in OT. Finally, the conclusion is presented in Section <ref>.   



§ PROPOSED METHOD


The model framework of the proposed TOT-CAKT method is illustrated in Fig. <ref>. This model framework is modified based on a conformer-CTC-based ASR model, incorporating two key modifications. First, an `Adapter' module is added as shown in the gray blocks in Fig. <ref>. Second, an temporal order preserved OT-based cross-modal matching module is introduced in the right branch of Fig. <ref>. Both the acoustic features extracted from the conformer encoder and linguistic features derived from a PLM (with BERT utilized in this paper) are involved in the cross-modal matching process. Further details are provided in the following sections.



 §.§ Acoustic and linguistic feature representations

The acoustic feature is extracted from the acoustic encoder where a conformer-based encoder <cit.>) is adopted. The process in the `Subsampling' module involves a two-layer convolution process with a downsampling operation (a downsampling rate of 4 was used in this paper). By incorporating a positional encoding from PE_ A, the initial input to conformer blocks is obtained as H_0. The output of the conformer encoder is represented as an acoustic representation H_ ca.

    [            H_0  =  Subsampling(  X) +  PE_ A; H_ ca  =  Conformer(  H_0 ) ∈ R^l_a  × d_a ,;                                              ]

where l_a and d_a are temporal length and dimension of the acoustic feature vectors, respectively. Before engaging in cross-modal feature alignment, a linear projection termed `FC2' in the `Adapter' module is utilized to perform a feature dimension matching transform:

    H_ A  =  FC_ 2(  H_ ca) ∈ R^l_a  × d_t

In this equation, d_t corresponds to linguistic feature dimension. In the right branch of Fig. <ref>, the context-dependent linguistic feature representation is explored from a pretrained BERT model. The process is formulated as: 

    [ y_ token  =  BERTTokenizer(  y); Z_0  = [  CLS,  y_ token , SEP];     Z_i  =  BERT_i (  Z_i - 1),;                                 ]

where `BERT_i' is the i-th transformer encoder layer of BERT model, i takes values from 1 to L, with L representing the total number of BERT encoder layers. `BERTTokenizer' is a process to convert standard text to word piece based tokens <cit.>. Token symbols `CLS' and `SEP' represent the start and end of an input sequence. Z_L  ∈ℝ^l_t  × d_t is the final text representation which encodes context dependent linguistic information, l_t denotes the sequence length, and d_t represents feature dimension of text encoding representation. 


 §.§ Sinkhorn algorithm for cross-modal alignment

The original OT was formulated to transform from one probability distribution to another with minimum transport cost <cit.>. In this study, we applied OT for feature alignment on two sets. Given acoustic and linguistic feature sequences H_A and Z_L respectively, as: 

    [  H_A  = [  h_1 , h_2 ,..., h_i ,..., h_l_a ]; Z_L  = [  z_1 , z_2 ,..., z_j ,..., z_l_t ],;                                              ]

where l_a and l_t are lengths of the two sequences. Suppose the two sequences in Eq. (<ref>) are sampled from two probability distributions with weight vectors a  = [ a_1 ,a_2 ,...,a_i ,...,a_l_a ] and 
b  = [ b_1 ,b_2 ,...,b_j ,...,b_l_t ]. (a_i  = 1 /
		.
		-l_a ,b_j  = 1 /
		.
		-l_t as uniform distributions if no prior information is available). The OT distance between the two sequences is defined as:       

    L_ OT  = ^Δmin_γ∈∏(  H_A , Z_L )⟨γ , C⟩,

where γ is a transport coupling set defined as:

    ∏(  H_A , Z_L )  = ^Δ{γ∈ R_ + ^l_a  × l_t | γ 1_l_t   =  a,γ ^T  1_l_a   =  b.}

In Eq. (<ref>), 1_l_a and 1_l_t are vectors of ones with dimensions l_a and l_t, respectively. In Eq. (<ref>), C is a distance matrix (or ground metric) with element c_i,j defined as pair-wised cosine distance:

    c_i,j  =  C(  h_i , z_j )  = ^Δ  1 - cos(  h_i , z_j )

A fast estimation of OT has been introduced through the celebrated entropy-regularized OT (EOT) <cit.> where the EOT loss is defined as:

    L_ EOT(  H_A , Z_L )  = ^Δmin_γ∈∏(  H_A , Z_L )⟨γ , C⟩  - α _1   H( γ),

where α _1 is a regularization coefficient, and H( γ) is entropy of coupling matrix defined as:

    H( γ)   =  - ∑_i,jγ _i,jlogγ _i,j .

The solution of Eq. (<ref>) can be implemented with Sinkhorn algorithm as <cit.>:

    γ _α _1   = diag(  u_1 )* G*diag(  u_2 )

where G = exp(  -  C/α _1 ), u_1 and u_2 are two scaling (or re-normalization) vectors.


 §.§ Temporal order preserved OT

In the original estimation of OT in Eq. (<ref>), the two sequences in Eq. (<ref>) are treated as two sets without considering their temporal order relationship. In speech, temporal order information is crucial in OT coupling during cross-modal alignment, meaning that neighboring frames in an acoustic sequence should be progressively coupled with the neighboring tokens in a linguistic sequence. Therefore, as showed in Fig. <ref>, the temporal order information is input to the OT matching block. For the sake of clarity, the two sequences in Eq. (<ref>) can be further represented with temporal order information as:

    [ H_A  = [ (  h_1 ,1),(  h_2 ,2),...,(  h_i ,i),...,(  h_l_a  ,l_a )]; Z_L  = [ (  z_1 ,1),(  z_2 ,2),...,(  z_j ,j),...,(  z_l_t  ,l_t )];                                                                     ]
 
During the alignment of the two sequences for knowledge transfer, it is crucial to consider that elements with significant cross temporal distances might not be likely to be coupled. In other words, the coupling pairs with high probabilities between the two sequences should be distributed along the diagonal line of the temporal coherence positions. Based on this consideration, the temporal coupling prior could be defined as a two dimensional Gaussian distribution <cit.>. The fundamental concept is that the coupled pairs should not deviate significantly from the diagonal line of temporal coherence positions between the two sequences, which can be defined as:    

    p_i,j  = ^Δ1/σ√(2π)exp(  - d_i,j^2 /2σ ^2 ),

where σ is a variation variable controlling the impact of the cross-temporal distance d_i,j as defined in Eq. (<ref>).

    d_i,j  = | 0.5exi-0.1em/-0.15em
    					0.25exl_a - 0.5exj-0.1em/-0.15em
    					0.25exl_t|/√(0.5ex1-0.1em/-0.15em
    					0.25exl_a^2 + 0.5ex1-0.1em/-0.15em
    					0.25exl_t^2)

In Eq. (<ref>), the cross-temporal distance is defined on the normalized sequence lengths in acoustic and linguistic spaces. In this definition, it is evident that the farther the distance between a paired position and the temporal diagonal line, the lower possibility of their correspondence in transport coupling. By incorporating this temporal coherence prior as regularization, the new OT is defined as:

    L_TOT ( H_A , Z_L )  = ^Δmin_γ∈∏( H_A , Z_L )  < γ , C >  - α _1   H(γ ) + α _2   KL(γ ||P),

where α _1 and α _2 are two trade off parameters. In Eq. (<ref>), KL(γ||P) is the Kullback-Leibler (KL) divergence between the transport coupling matrix γ and temporal prior correspondence matrix P with elements defined in Eq. (<ref>). Building upon the definitions of KL-divergence and entropy, Eq. (<ref>) can be further expressed to:

    L_TOT ( H_A , Z_L )  = ^Δmin_γ∈∏( H_A , Z_L )  < γ ,C̃ >  - α̃H(γ ),

where α̃= α _1    + α _2, and combined ground cost matrix as

    C̃ =  C - α _2  log P

Following the procedures outlined in <cit.>, the solution of Eq. (<ref>) is obtained using the Sinkhorn algorithm as:

    γ _α̃  = diag(  u_1 )*G̃*diag(  u_2 ),

where G̃ = exp(  - C̃/α̃). Substituting variables in Eq. (<ref>) to G̃, we can obtain:

    G̃ = P^α _2/α_1    + α_2  exp ( -  C/α_1  + α_2 )

From this equation, we can see that the transport coupling between the two sequences is further constrained by their temporal order correspondence. TOT involves several hyper-parameters that can be challenging to control. For the sake of simplification, we consolidate their effects into a reduced number of hyper-parameters. For example, considering Eq. (<ref>), the impact of variation σ in Eq. (<ref>) and α_2 in Eq. (<ref>) can be combined into a single control parameter β, defined as:

    C̃ =  C + β d_i,j^2

And the Sinkhorn algorithm is applied on the cost function matrix C̃ for OT in real implementations.  


 §.§ Loss function

The proposed TOT-CAKT involves two loss functions: the cross-modal alignment and matching loss (in the right branch of Fig. <ref>) and the CTC loss (in the left branch of Fig. <ref>). In cross-modal alignment, the acoustic feature can be projected onto the linguistic space using OT as:

    [ Z̃_L      = ^Δ OT(  H_A→ Z_L) 
       = γ ^*  × H_A∈ R^l_t  × d_t , ]

where γ ^* is the optimal transport coupling based on OT. Subsequently, the alignment loss is defined as:

    L_ align  = ∑_j = 2^l_t  - 11 - cos( z̃_L^j , z_L^j ),

where z̃_L^j and z_L^j are row vectors of feature matrices Z̃_L and Z_L (matching on temporal dimensions), respectively. In Eq. (<ref>), the usage of indices from 2 to l_t -1 is for handling special symbols `CLS' and `SEP'. For efficient linguistic knowledge transfer to acoustic encoding, the following transforms are designed as indicated in Fig. <ref>:

    [ Ĥ_ ca  =  FC_ 3(  LN( H_A )) ∈ R^l_a  × d_a;          H^a,t  =  H_ ca  + s · LN(Ĥ_ ca ),;                                             ]

where s is a scaling parameter to adjust the importance of transferring linguistic projected feature. Based on this new representation H^a,t which is intended to encode both acoustic and linguistic information, the final probability prediction for ASR is formulated as:

    P̃ =  Softmax(  FC1( H^a,t)),

where `FC1' is a linear full-connected transform. The total loss in model learning is defined as:

    L  = ^Δλ .L_ CTC (P̃, y_ token ) + (1 - λ ).w.(L_ align  + L_ TOT ),

where L_ CTC (P̃, y_ token ) is CTC loss, L_ align and L_ TOT are cross-modality alignment loss and TOT loss, respectively. After the model is trained, only the left branch of Fig. <ref> is retained for ASR inference.  



§ EXPERIMENTS


ASR experiments were conducted on the open-source Mandarin speech corpus AISHELL-1 <cit.> to evaluate the proposed algorithm. The data corpus comprises three datasets: a training set with 340 speakers (150 hours), a development (or validation) set with 40 speakers (10 hours), and a test set with 20 speakers (5 hours). Data augmentation as used in <cit.> was applied. Given the tonal nature of the Mandarin language in the ASR task, in addition to using 80-dimensional log Mel-filter bank features, three extra acoustic features related to fundamental frequency, i.e.,  F0, delta F0 and delta delta F0, were utilized as raw input features. These features were extracted with a 25ms window size and a 10ms shift. 


 §.§ Model architecture

In Fig. <ref>, the `Subsampling' module consists of two CNN blocks with 256 channels, kernel size 3, stride 2, and ReLU activation function in each. The acoustic encoder is formed by stacking 16 conformer blocks <cit.>, with each having a kernel size of 15, attention dimension d_a=256, 4 attention heads, and a 2048-dimensional FFN layer. The `bert-base-chinese' from huggingface is used as the pretrained PLM for linguistic knowledge transfer <cit.>. In this Chinese BERT model, 12 transformer encoders are applied, the token (or vocabulary) size is 21128, and the dimension of linguistic feature representation is d_t=768. 


 §.§ Hyper-parameters in model learning

Several hyper-parameters are associated with the proposed model, and these parameters may have a joint (or correlated) effect in efficient linguistic knowledge transfer learning. In our preliminary experiments, for easy implementation, they were fixed as β=0.5 in Eq. (<ref>), alignment trade off parameter λ=0.3 and scale parameter w=1.0 in Eq. (<ref>). α̃ in Eq. (<ref>) and s in Eq. (<ref>) were varied in experiments. For optimization, Adam optimizer <cit.> is used with a learning rate (initially set to 0.001) schedule with 20,000 warm-up steps. The model with cross-modal transfer was trained for 130 epochs, and the final model used for evaluation was obtained by averaging models from the last 10 epochs (the original conformer-CTC model without the adapter module was pretrained or trained from scratch in different experimental settings when jointly combined with cross-modal learning). The performance was evaluated based on character error rate (CER).   



 §.§ Results


In inference stage, only the left branch (blocks in dashed red box in Fig. <ref>) is utilized, maintaining the decoding speed similar to that of the CTC-based decoding. In our experiments, only CTC greedy search-based decoding was employed, and the results are presented in table <ref>. The results of the baseline system and several state-of-the-art systems that integrate BERT for linguistic knowledge transfer are also provided for comparison. 

In this table, `Conformer+CTC' is the baseline system, trained without linguistic knowledge transfer. `Conformer+CTC/AED' denotes a hybrid CTC/AED ASR system <cit.> which used a transformer decoder with attention to text representation during model training. `NAR-BERT-ASR', `KT-RL-ATT', and `Wav2vec-BERT' are all based on integrating acoustic and linguistic features from BERT for ASR <cit.>, and even used a pretrained acoustic model (from wav2vec2.0 <cit.>) and PLM for knowledge transfer. In the OT based cross-modal learning, two experimental conditions were examined. One is that models with cross-modal learning were trained from scratch, i.e., without pretraining condition. The other is that the models were initialized with a pretrained acoustic model, then were further trained with cross-modal learning, i.e., with pretraining condition. Correspondingly, in table <ref>, `OT-BERT(w/o)' and `TOT-CAKT(w/o)' denote results of cross-modal linguistic knowledge transfer learning based on OT in <cit.> and the proposed TOT-CAKT method for without pretraining condition, respectively, `OT-BERT(w/)' and `TOT-CAKT(w/)' are results with pretraining condition. From this table, we can observe that linguistic knowledge significantly enhances the ASR performance (not in a conventional way like LM rescoring). From the results in `OT-BERT(w/o)' and `TOT-CAKT(w/o)', both the methods in <cit.> and the proposed cross-modal learning could efficiently transfer linguistic knowledge in acoustic encoding, yielding promising results. Moreover, when comparing results in `OT-BERT(w/o)' and `TOT-CAKT(w/o)', a significant performance improvement was observed when all models with cross-modal learning were trained from scratch (without pretraining condition). Furthermore, the performance of our proposed TOT-CAKT, even without pretraining could reach a level comparable to the original OT-based method, which requires a time-consuming pretraining stage (as in `OT-BERT(w/)). Finally, we further examined that when pretraining was utilized before cross-modal learning, the improvement of our proposed method (as shown in `TOT-CAKT(w/)' in table <ref>) compared to the `OT-BERT(w/)' in <cit.> was reduced. This suggests that the pretraining stage could implicitly provide temporal order information for cross-modal feature alignment and knowledge transfer. In comparison, our proposed method explicitly incorporates temporal order information in the mathematical modeling with flexible parameters for control in experiments. Based on our formulation, our future work will focus on finding optimal temporal order parameter settings.   


 §.§ Visualization of transport coupling

In the proposed TOT-CAKT, the coupled pairs between the acoustic and linguistic feature sequences are explicitly designed to correspond to their temporal coherence, i.e., acoustic segments should match well with their linguistic tokens sequentially. Two examples of the coupling matrices are shown in Fig. <ref>. In Fig. <ref>-a, the coupling matrix is learned based on OT without temporal order constraint, and Fig. <ref>-b is the coupling matrix learned with temporal order constraint. 

From this figure, it is evident that clear temporal correspondences exist between the acoustic feature sequence and linguistic token sequence in both transport couplings. Moreover, several positions with incorrect couplings in Fig. <ref>-a were corrected in Fig. <ref>-b by our proposed method, which explicitly adds a temporal order constraint.  


§ CONCLUSION


Acoustic and linguistic features belonging to different modalities require feature alignment as a crucial step in transferring linguistic knowledge from a PLM to acoustic encoding. In this paper, we proposed a novel TOT-CAKT. In the TOT-CAKT, a transfer coupling or mapping preserving temporal order information between acoustic sequence and linguistic sequence was first estimated. Subsequently, acoustic features were mapped to the linguistic space based on the transport coupling, allowing direct comparison of the mapped features to match the information encoded in the PLM. Our ASR experimental results confirmed the effectiveness of the proposed TOT-CAKT. Additionally, based on the visualization of transport coupling, we verified that TOT could eliminate unreasonable matches between the acoustic and linguistic sequences.   

In the proposed TOT-CAKT, several hyper-parameters involved in model learning are challenging to control. Additionally, some of these hyper-parameters are sensitive in implementation and may lead to stability problems in the Sinkhorn algorithm. In the current paper, the combined effects of those hyper-parameters have not been clearly explored, and a clear understanding of their combined rules has not been established, potentially hindering the identification of optimal solutions for improving ASR performance. Figuring out a set of optimal hyper-parameters in the proposed TOT-CAKT remains as our future work.   









IEEEtran
1
	
	Li2022
	J. Li, “Recent advances in end-to-end automatic speech recognition," APSIPA Transactions on Signal and Information Processing, DOI 10.1561/116.00000050, 2022.
	
	Chan2016
	W. Chan, N. Jaitly, Q. Le and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition," in Proc. of ICASSP, 2016, pp. 4960-4964.
	
	Kim2017
	S. Kim, T. Hori, and S. Watanabe, “Joint CTC-attention based end-to-end speech recognition using multi-task learning," in Proc. of ICASSP, 2017, pp. 4835–4839.
	
	Hori2017
	T. Hori, S. Watanabe, and J. R. Hershey, “Joint ctc/attention decoding for end-to-end speech recognition," in Proc. of ACL, 2017, vol. 1, pp. 518–529.
	
	Watanabe2017
	S. Watanabe, T. Hori, S. Kim, J. R. Hershey and T. Hayashi, “Hybrid CTC/Attention Architecture for End-to-End Speech Recognition," IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240-1253, 2017.
	
	RNNTASR
	A. Graves, “Sequence transduction with recurrent neural networks," arXiv preprint, arXiv:1211.3711, 2012.
	
	BERTScore
	J. Shin, Y. Lee, and K. Jung, “Effective sentence scoring method using BERT for speech recognition," in Proc. of ACML, 2019, pp. 1081-1093.
	
	MLMScore
	J. Salazar, D. Liang, T. Nguyen, K. Kirchhoff, “Masked Language Model Scoring," in Proc. of ACL, 2020, pp. 2699-2712.
	
	CTCASR
	A. Graves, and N. Jaitly, “Towards end to-end speech recognition with recurrent neural networks," in Proc. ICML, 2014, pp. 1764–1772.
	
	HierarchicalCTC
	Higuchi, K. Karube, T. Ogawa, et al., “Hierarchical conditional end-to-end asr with ctc and multi-granular subword units," in Proc. of ICASSP, 2022, pp. 7797-7801.
	
	intermediateCTC
	Y. Fujita, T. Komatsu, and Y. Kida, “Multi-sequence intermediate conditioning for ctc-based asr," arXiv preprint, arXiv:2204.00175, 2022.
	
	BERT
	J. Devlin, M. Chang, K. Lee, and K. Toutanova, “Bert: Pretraining of deep bidirectional transformers for language understanding," arXiv preprint, arXiv:1810.04805, 2018.	
	
	FNAR-BERT
	Y. Bai, J. Yi, J. Tao, Z. Tian, Z. Wen and S. Zhang, "Fast End-to-End Speech Recognition Via Non-Autoregressive Models and Cross-Modal Knowledge Transferring From BERT," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1897-1911, 2021.
	
	NARBERT
	F. Yu, K. Chen, and K. Lu, “Non-autoregressive ASR Modeling using Pre-trained Language Models for Chinese Speech Recognition," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 1474-1482, 2022
	
	
	
	
	KuboICASSP2022
	Y. Kubo, S. Karita, M. Bacchiani, “Knowledge Transfer from Large-Scale Pretrained Language Models to End-To-End Speech Recognizers," in Proc. of ICASSP, 2022, pp. 8512-8516.
	
	Choi2022
	K. Choi, H. Park, “Distilling a Pretrained Language Model to a Multilingual ASR Model," in Proc. of INTERSPEECH, 2022, pp. 2203-2207.
	
	wav2vec2.0
	A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “Wav2vec 2.0: A framework for self-supervised learning of speech representations," in Proc. of NeurIPS, 2020. 
	
	Futami2022
	H. Futami, H. Inaguma, M. Mimura, S. Sakai, T. Kawahara, “Distilling the Knowledge of BERT for CTC-based ASR," CoRR abs/2209.02030, 2022.
	
	Higuchi2023
	Y. Higuchi, T. Ogawa, T. Kobayashi, S. Watanabe, “BECTRA: Transducer-Based End-To-End ASR with Bert-Enhanced Encoder," in Proc. of ICASSP, 2023, pp. 1-5.
	
	CIFBERT1
	M. Han, F. Chen, J. Shi, S. Xu, B. Xu, “Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation," arXiv preprint, arXiv:2301.13003, 2023.
	
	wav2vecBERTSLT2022
	K. Lu and K. Chen, “A Context-aware Knowledge Transferring Strategy for CTC-based ASR," in Proc. of SLT, 2022, pp. 60-67.
	
	CTCBERT1
	K. Deng, S. Cao, Y. Zhang, L. Ma, G. Cheng, J. Xu, P. Zhang, “Improving CTC-Based Speech Recognition Via Knowledge Transferring from Pre-Trained Language Models," in Proc. of ICASSP, 2022, pp. 8517-8521.
	
	CTCBERT2
	K. Deng, Z. Yang, S. Watanabe, Y. Higuchi, G. Cheng, P. Zhang, “Improving Non-Autoregressive End-to-End Speech Recognition with Pre-Trained Acoustic and Language Models," in Proc. of ICASSP, 2022, pp. 8522-8526.
	
	DengICASSP2024
	K. Deng, Z, P. Woodland, “FastInject: Injecting Unpaired Text Data into CTC-Based ASR Training," in Proc. of ICASSP, 2024, pp. 11836-11840.
	
	CIFBERT2
	M. Han, L. Dong, Z. Liang, M. Cai, S. Zhou, Z. Ma, B. Xu, “Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selection," in Proc. of ICASSP, 2022, pp. 8532-8536.
	
	
	
	
	
	
	
	Transformer
	A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need," in Proc. of NIPS, 2017, pp. 5998-6008.		
	
	VillanoBook
	C. Villani, Optimal transport: old and new, volume 338. Springer, 2009
	
	CourtyNIPS2017
	N. Courty, R.  Flamary, A. Habrard, A. Rakotomamonjy, “Joint distribution optimal transportation for domain adaptation," in Proc. of NIPS, 2017, pp. 3733-3742.
	
	LuICASSP2021
	X. Lu, P. Shen, Y. Tsao, H. Kawai, “Unsupervised Neural Adaptation Model Based on Optimal Transport for Spoken Language Identification," in Proc. of ICASSP, 2021, pp. 7213-7217.
	
	Lin2021
	H. Lin, H. Tseng, X. Lu, Y. Tsao, “Unsupervised Noise Adaptive Speech Enhancement by Discriminator-Constrained Optimal Transport," in Proc. of NeurIPS, 2021, pp. 19935-19946.
	
	ICLR2023
	H. Tseng, H. Lin, H. Hsuan, and Y. Tsao, “Interpretations of Domain Adaptations via Layer Variational Analysis," arXiv preprint, CoRR abs/2302.01798, 2023.
	
	Cho2020
	W. Cho, D. Kwak, J. Yoon, N. Kim, “Speech to Text Adaptation: Towards an Efficient Cross-Modal Distillation," in Proc. of INTERSPEECH, 2020, pp. 896-900.
	
	Cross2021
	W. Wang, S. Ren, Y. Qian, S. Liu, Y. Shi, Y. Qian, M. Zeng, “Optimizing Alignment of Speech and Language Latent Spaces for End-To-End Speech Recognition and Understanding," in Proc. of ICASSP, 2021, pp. 7802-7806.
	
	ACL2023
	Y. Zhou, Q. Fang, Y. Feng, “CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation," arXiv preprint,  arXiv:2305.14635, 2023.
	
	ICML2023
	P. Le, H. Gong, C. Wang, J. Pino, B. Lecouteux, D. Schwab, “Pre-training for Speech Translation: CTC Meets Optimal Transport," arXiv preprint, CoRR abs/2301.11716, 2023.
	
	ASRU2023Lu
	X. Lu, P. Shen, Y. Tsao, H. Kawai, “Cross-Modal Alignment with Optimal Transport for CTC-Based ASR," IEEE-ASRU, 2023, Dec.16-20,Taipei, Taiwan.
	
	conformer2020
	A. Gulati, J. Qin, C. Chiu, et al., “Conformer: Convolution augmented transformer for speech recognition," arXiv preprint, arXiv:2005.08100, 2020
	
	Cuturi2013
	M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport," in Proc. of NIPS, 2013, vol. 26.
	
	Su2017
	B. Su, G. Hua, “Order-Preserving Wasserstein Distance for Sequence Matching," in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
	
	
	
	
	AISHELL1 
	Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng, “AIShell-1: An open-source mandarin speech corpus and a speech recognition baseline,” in Proc. of COCOSDA, 2017, pp. 1-5.
	
	Huggingface
	https://huggingface.co/
	
	Adam
	Diederik P. Kingma, Jimmy Ba, “Adam: A Method for Stochastic Optimization," in Proc. of ICLR, 2015.
	
	wenet2.0
	B. Zhang, D. Wu, Z. Peng, X. Song, Z. Yao, H. Lv, L. Xie, C. Yang, F. Pan, J. Niu, “WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit," in Proc. of INTERSPEECH, 2022, pp. 1661-1665.
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	


