Graph Neural Networks for Contextual ASR with the Tree-Constrained
  Pointer Generator
The incorporation of biasing words obtained through contextual knowledge is
of paramount importance in automatic speech recognition (ASR) applications.
This paper proposes an innovative method for achieving end-to-end contextual
ASR using graph neural network (GNN) encodings based on the tree-constrained
pointer generator method. GNN node encodings facilitate lookahead for future
word pieces in the process of ASR decoding at each tree node by incorporating
information about all word pieces on the tree branches rooted from it. This
results in a more precise prediction of the generation probability of the
biasing words. The study explores three GNN encoding techniques, namely tree
recursive neural networks, graph convolutional network (GCN), and GraphSAGE,
along with different combinations of the complementary GCN and GraphSAGE
structures. The performance of the systems was evaluated using the Librispeech
and AMI corpus, following the visual-grounded contextual ASR pipeline. The
findings indicate that using GNN encodings achieved consistent and significant
reductions in word error rate (WER), particularly for words that are rare or
have not been seen during the training process. Notably, the most effective
combination of GNN encodings obtained more than 60% WER reduction for rare and
unseen words compared to standard end-to-end systems.
Guangzhi Sun, Chao Zhang, Phil Woodland
# Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generator

###### Abstract

The incorporation of biasing words obtained through contextual knowledge is of paramount importance in automatic speech recognition (ASR) applications. This paper proposes an innovative method for achieving end-to-end contextual ASR using graph neural network (GNN) encodings based on the tree-constrained pointer generator method. GNN node encodings facilitate lookahead for future word pieces in the process of ASR decoding at each tree node by incorporating information about all word pieces on the tree branches rooted from it. This results in a more precise prediction of the generation probability of the biasing words. The study explores three GNN encoding techniques, namely tree recursive neural networks, graph convolutional network (GCN), and GraphSAGE, along with different combinations of the complementary GCN and GraphSAGE structures. The performance of the systems was evaluated using the Librispeech and AMI corpus, following the visual-grounded contextual ASR pipeline. The findings indicate that using GNN encodings achieved consistent and significant reductions in word error rate (WER), particularly for words that are rare or have not been seen during the training process. Notably, the most effective combination of GNN encodings obtained more than 60% WER reduction for rare and unseen words compared to standard end-to-end systems.

 pointer generator, contextual speech recognition, end-to-end, graph neural networks, audio-visual

## I Introduction

End-to-end ASR systems often struggle with the accurate recognition of rare "long-tail" words that were not present in the training data. To combat this issue, Contextual biasing which applies external contextual knowledge to the ASR system during inference, becomes a crucial factor in addressing the long-tail word problem in various applications [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]. Contextual knowledge is often represented as a list (referred to as a _biasing list_) of words or phrases (referred to as _biasing words_) that are likely to appear in a given context. Biasing lists can be sourced from various resources, such as a user's contact book or playlist, recently visited websites and visual information from presentation slides _etc._ Despite their infrequent occurrence and thus the limited influence on the overall word error rate (WER), biasing words significantly impacts understanding the content as biasing words are mostly content words such as nouns or proper nouns, which are crucial for downstream tasks. The inclusion of a word in a biasing list increases its likelihood of being correctly recognised, making contextual biasing a crucial component for the accurate recognition of rare content words.

End-to-end trainable ASR systems [15, 20] are designed to encapsulate all necessary knowledge within a single, static model, making it difficult to integrate dynamic context-specific knowledge at test-time. To overcome this challenge, specialised contextual biasing methods have been proposed, such as shallow fusion (SF) with a weighted finite-state transducer (WFST) or an adapted language model (LM) that incorporates contextual knowledge [1, 2, 3, 11, 12, 14], attention-based deep context approaches [4, 5, 6, 7, 8], as well as deep biasing (DB) with a prefix tree for improved efficiency when dealing with large biasing lists [6, 9, 10]. More recently, contextual biasing components with a pointer generator mechanism [46, 47, 48] that directly modifies the output distribution have been proposed [34, 38], which can be jointly optimised with ASR systems. In particular, the tree-constrained pointer generator (TCPGen) component proposed in [34] builds a neural shortcut by directly interpolating the original model distribution with the TCPGen distribution estimated from contextual knowledge structured as a prefix-tree, based on a dynamic interpolation weight predicted by the TCPGen component. TCPGen performance was further boosted by encoding the prefix-tree using a tree recursive neural network (tree-RNN) [35].

This paper substantially extends the work in [35] and proposes to use three types of graph neural networks (GNN) for prefix-tree encoding in TCPGen 1. These include tree-RNN, graph convolutional network (GCN) [43] with its variant GCNII [42], and GraphSAGE with the max-pooling aggregator [44]. While tree-RNN is a representative GNN model with a single recursive layer, GCN and GraphSAGE are two popular and effective multi-layer GNN designs. Specifically, GCN encodes the tree by utilising spectral representation, while GraphSAGE, as a spatial method, directly explores the graph topology [28]. To further enhance the performance of GNN tree encodings, this paper proposes attentive and bilinear combination approaches to exploit the complementarity between GCN and GraphSAGE. Additionally, this paper introduces an effective parameter-tying scheme for both GCN and GraphSAGE to improve their performance with deeper structures.

Footnote 1: The main code is at [https://github.com/BriansIDP/espnet/tree/GNN](https://github.com/BriansIDP/espnet/tree/GNN).

GNN encodings provide more powerful node representations in the prefix tree of TCPGen, allowing for "lookahead" functionality where each node contains not only its own word piece information but also information about its child branches. This improved node representation in TCPGen leads to more accurate generation probability predictions for biasing words, enabling better contextual biasing by incorporating information about future word pieces during each ASR decoding
step. TCPGen with GNN encodings, as a generic component for end-to-end ASR, is integrated into both attention-based encoder-decoder (AED) [15, 16, 17, 18, 19, 26] and neural transducer architecture (N-T) [20, 21, 22, 23, 24].

Experiments were conducted with two different setups: (1). a simulated contextual ASR task using LibriSpeech audiobook data, and, (2). an audio-visual speech recognition pipeline [35] with the AMI meeting data. In addition to the consistent and large reductions in error rates achieved by TCPGen with tree-RNN in [35], using the proposed GNN encodings, especially combined ones achieved further significant improvements in the word error rate (WER) of rare content words.

The remainder of this paper is organised as follows. Section II reviews related work. Section III introduces the TCPGen component. Section IV describes the details of applying proposed GNNs. Section V and VI present the experimental setup and results. Section VII gives conclusions.

## II Related work

### _End-to-end contextual speech recognition_

Recently, various contextual biasing algorithms have been developed for end-to-end ASR. One prominent research stream focuses on representing biasing lists as external weighted finite-state transducers (WFSTs), which are integrated into a class-based language model (LM) via shallow fusion (SF) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]. These methods often depend on context prefixes such as "call" or "play", limiting their ability to handle the diverse grammar in natural speech. On the other hand, deep context approaches, often using attention mechanisms, have also been proposed, which encode the biasing list into a vector to use as input for the end-to-end ASR models [4, 5, 6, 7, 8]. While deep context approaches eliminate the dependence on syntactic prefixes seen in SF methods, they require more memory and are less effective for handling large biasing lists.

The study in [9] combined the use of deep context and shallow fusion of a WFST in an N-T, leading to improved efficiency by limiting the biasing vector extraction to a subset of word pieces determined by a prefix tree representation of the biasing list, which is referred to as deep biasing (DB). The prefix-tree-based method was further expanded in [10] to include RNN LMs for shallow fusion, resulting in improved recognition of biasing words. Prior research only analysed industry datasets, however, [10] proposed and validated a simulation of contextual biasing on open-source data by incorporating a large number of distractors into the list of biasing words in the utterance. More recently, [38] and [34] simultaneously proposed a neural shortcut between the biasing list and the final model output distribution. In contrast to [38], TCPGen [34] adopted a structured prefix-tree representation for biasing lists, which also enabled the future development of tree-RNN encodings [35].

### _GNN for speech and language processing_

GNNs have been extensively employed in a multitude of speech and language tasks. In language-related applications, such as sentence-level text classification or word-level sequence labelling, GNNs are often utilized to capture the syntactic dependencies or semantic relations among words in a sentence. Furthermore, the encoding of subword-unit-based tree structures using GNNs [39, 40] has been explored for the purpose of generating more effective word representations. GNNs have also been applied to named-entity recognition [52] and neural machine translation [53, 54].

GNNs also have numerous applications in speech processing, such as text-to-speech synthesis, where GNNs model the syntactic and semantic relationships in the text. In GraphTTS [45], the authors structured the sentence into a hierarchical tree by dividing the utterance into words and then further into characters. This allowed the system to capture prosodic relationships among different parts of the input. Additionally, GNNs are used in paralinguistic tasks as the syntactic encoder, including sentiment classification and hate speech detection tasks [49, 50, 51]. In [35], a tree-RNN structure was used to encode a word piece prefix-tree in the TCPGen component for contextual biasing.

## III Tree-constrained pointer generator

TCPGen is a neural network-based module that employs a combination of symbolic prefix-tree search and a neural pointer generator for contextual biasing, allowing for end-to-end optimisation. The biasing list is structured as a prefix tree at the word piece-level. At each output stage, TCPGen computes a probability distribution over a set of valid word pieces that are constrained by the prefix tree. TCPGen also predicts a dynamic generation probability, signifying the amount of contextual biasing required at the specific output step. The final output distribution is obtained by taking a weighted summation of the TCPGen distribution and the original AED or N-T output distribution (see Figure 1).

The key symbolic representation of the external contextual knowledge in TCPGen is the prefix tree. For simplicity, examples and equations in this section are presented for a specific search path, which can be generalised easily to beam-search with multiple paths. In the example prefix tree with biasing words (turner, vignette and turn) shown in Fig. 2, if the previously decoded word piece is Tur, word pieces in_ and n form the set of valid word pieces \(\mathcal{Y}_{i}^{\text{tree}}\). Denoting \(\mathbf{x}_{1:T}\) and \(y_{i}\) as input acoustic features and output word piece, \(\mathbf{q}_{i}\) as the query vector carrying the decoding history and acoustic information, \(\mathbf{K}=[...,\mathbf{k}_{j},...]\) as the key vectors, scaled dot-product attention is performed between
and \(\mathbf{K}\) to compute the TCPGen distribution \(P^{\text{ptr}}\) and the output vector \(\mathbf{h}_{i}^{\text{ptr}}\) as shown in Eqns. (1) and (2).

\[P^{\text{ptr}}(y_{i}|y_{1:i-1},\mathbf{x}_{1:T})=\text{Softmax}( \text{Mask}(\mathbf{q}_{i}\mathbf{K}^{\text{T}}/\sqrt{d})), \tag{1}\] \[\mathbf{h}_{i}^{\text{ptr}}=\sum_{j}P^{\text{ptr}}(y_{i}=j|y_{1:i- 1},\mathbf{x}_{1:T})\,\mathbf{v}_{j}^{\text{T}}, \tag{2}\]

where \(d\) is the size of \(\mathbf{q}_{i}\) (see [33]), \(\text{Mask}(\cdot)\) sets the probabilities of word pieces that are not in \(\mathcal{Y}_{i}^{\text{ptr}}\) to zero, and \(\mathbf{v}_{j}\) is the value vector relevant to \(j\).

TCPGen can be employed in both AED and N-T. In AED, the query is the combination of the context vector and the embedding of the preceding word piece, while the keys and values are derived from the decoder word piece embedding using a shared projection matrix. The generation probability in AED is computed from the concatenation of decoder hidden states and TCPGen output vectors \(\mathbf{h}_{i}^{\text{ptr}}\), followed by Sigmoid activation function to be constrained to \((0,1)\). In N-T, the pointer generator is applied to each combination of the encoder and the predictor step, where the TCPGen distribution is calculated using the concatenation of the corresponding encoder and predictor hidden states as the query. Keys and values in N-T are computed from the predictor word piece embeddings. The generation probability for N-T is derived from the joint network output and the TCPGen output vector \(\mathbf{h}_{i}^{\text{ptr}}\). To ensure that the probability of the null symbol in N-T is unchanged, \(P_{i}^{\text{ptr}}(\varnothing|\mathbf{x}_{1:T},y_{1:i-1})\) is set to 0 and the generation probability is scaled by \(1-P_{i}^{\text{mdl}}(\varnothing|\mathbf{x}_{1:T},y_{1:i-1})\) where \(P_{i}^{\text{mdl}}\) is the original model distribution before interpolation.

For better flexibility, an _out-of-list_ (OOL) token is included in \(\mathcal{Y}_{i}^{\text{tree}}\) indicating that no suitable word piece can be found in the set of valid word pieces. To ensure that the final distribution sums to 1, the generation probability, \(P_{i}^{\text{gen}}\), is scaled as \(\hat{P}_{i}^{\text{gen}}=P_{i}^{\text{gen}}(1-P^{\text{ptr}}(\text{OOL}))\), and the final output can be calculated as shown in Eqn. (3).

\[P(y_{i})=P^{\text{mdl}}(y_{i})(1-\hat{P}_{i}^{\text{gen}})+P^{\text{ptr}}(y_{i })P_{i}^{\text{gen}}, \tag{3}\]

where dependencies, \(y_{1:i-1},\mathbf{x}_{1:T}\), are omitted for clarity. \(P^{\text{mdl}}(y_{i})\) represents the output distribution from the standard end-to-end model.

### _Biasing-driven LM discounting (BLMD) for TCPGen_

Log-linear interpolation is often used as a technique to incorporate an external LM via SF. Define the source domain data as the text of the training data for the end-to-end model, and the target domain data as the data used to train an external LM such that it generates better probability estimates for the test data. The LM discounting is defined as Eqn. (4).

\[P^{\text{sf}}(y_{i})=P^{\text{mdl}}(y_{i})\frac{P^{\text{ptr}}(y_{i})^{\alpha }}{P^{\text{src}}(y_{i})^{\beta}}, \tag{4}\]

where \(P^{\text{mdl}}(Y)\) is the probability from the end-to-end system, \(P^{\text{src}}(Y)\) is the probability of the source domain LM and \(P^{\text{ptr}}(Y)\) is the target domain LM probability. Extending this idea to contextual biasing with TCPGen, BLMD can be applied as shown in Eqn. (5).

\[P^{\text{sf}}(y_{i}) =(1-P^{\text{gen}})P^{\text{mdl}}(y_{i})\frac{P^{\text{ptr}}(y_{i} )^{\alpha_{1}}}{P^{\text{src}}(y_{i})^{\beta_{1}}}\] \[+P^{\text{gen}}P^{\text{ptr}}(y_{i})\frac{P^{\text{ptr}}(y_{i})^{ \alpha_{2}}}{P^{\text{src}}(y_{i})^{\beta_{2}}}, \tag{5}\]

where \(P^{\text{ptr}}\) is the TCPGen distribution, and the same source and target LMs are used for both distributions, but with different sets of hyper-parameters \(\alpha_{1},\beta_{1}\) and \(\alpha_{2},\beta_{2}\) tuned on the validation set.

## IV GNN Tree Encodings for TCPGen

While looking ahead into future branches of the paths being searched on the prefix tree is greatly beneficial to the correct prediction of the generation probability, node representations in standard TCPGen only contain information about the word piece on that node. To achieve lookahead functionality, GNNs are used to encapsulate future branch information into each node representation. The pipeline of applying GNN encodings in TCPGen is shown in Fig. 3.

The word piece prefix-tree is first encoded with a GNN to obtain encodings associated with each node. Then, the tree with GNN encodings is used by TCPGen, where the key and value for the TCPGen distribution are computed using the encoding of nodes in the set of valid word pieces, in place of word piece embeddings as shown in Eqn. (6).

\[\mathbf{k}_{j}=W^{\text{K}}\mathbf{h}_{n_{j}}^{\text{gm}}\qquad\mathbf{v}_{j}=W ^{\text{V}}\mathbf{h}_{n_{j}}^{\text{gm}}, \tag{6}\]

where \(W^{\text{V}}\) and \(W^{\text{K}}\) are parameter matrices, and \(\mathbf{h}^{\text{gm}}\) is the GNN node encoding obtained using different types of GNN. This paper explores three different types of GNNs, namely the tree-RNN, GCN (including its variant, GCNII) and GraphSAGE with max pooling, together with combinations of GCN and GraphSAGE as two complementary types of GNN. Details of GNN structures applied in TCPGen together with modifications are described in the following sections.

### _Tree-RNN_

Tree-RNN recursively encodes the tree from leaf nodes to the root using an RNN structure. Specifically, at node \(n_{j}\) which contains child nodes \(n_{1},...,n_{k},...,n_{K}\), the vector representation of \(n_{j}\) can be written as Eqn. (7).

\[\mathbf{h}_{n_{j}}^{\text{tmn}}=\text{ReLU}(W_{1}\mathbf{y}_{j}+\sum_{k=1:K}W_{ 2}\mathbf{h}_{n_{k}}^{\text{tmn}}), \tag{7}\]

where \(\mathbf{h}_{n_{k}}^{\text{tmn}}\) is the vector representation of node \(n_{k}\), and \(\mathbf{y}_{j}\) is the embedding vector of the word piece of node \(n_{j}\). \(W_{1}\) and \(W_{2}\) are parameter matrices jointly optimised with the ASR

Fig. 2: An example of prefix tree search and attention in TCPGen. With previous output Tur, in_ and n are two valid word pieces on which attention will be performed. A word end unit is denoted by _–_.
system by allowing gradient back-propagation through \(\mathbf{h}_{n_{k}}^{\text{tran}}\). In this way, each node recursively encodes information from its child nodes, such that the information of the entire branch rooted from it can be incorporated in the node encoding \(\mathbf{h}_{n_{k}}^{\text{tran}}\).

Before the forward pass of the main ASR model, the encoding of each node is obtained by applying Eqn. (7) recursively from leaf to root. Then, for the same example shown in Fig. 2, at the node of Tur, node encodings \(\mathbf{h}_{i_{m}}^{\text{tran}}\) and \(\mathbf{h}_{n}^{\text{tran}}\) are used to calculate the TCPGen distribution and \(\mathbf{h}_{i}^{\text{ptr}}\). Therefore, if Turner appears in the utterance, TCPGen is aware of this entire word as early as in the encoding of Tur. Such lookahead functionality achieves a more accurate prediction of the generation probability to determine when contextual biasing is needed.

Although Tree-RNN achieves the lookahead functionality, it uses a rather simple RNN structure to encode the information of all succeeding nodes on that branch into a single vector representation. Therefore, more powerful and flexible GNN encodings are explored in order to improve performance.

### _Graph Convolutional Network (GCN)_

As an alternative method to Tree-RNN, GCN is applied for tree encodings to achieve better node representations with controllable lookahead distance. GCN is a multi-layer network where each layer computes the encoding of a node as a function of its neighbours based on the graph Laplacian matrix. Each layer of GCN conducts one message passing from immediate neighbouring nodes. For a GCN with \(L\) layers, the encoding of a node covers information from a node that is an \(L\)-hop ahead on branches rooted from it.

Specifically, define \(H^{\text{gen}}(l)=[\mathbf{h}_{n_{1}}^{\text{gen}}(l),...,\mathbf{h}_{n_{N}}^{ \text{gen}}(l)]\) as the node encoding matrix of layer \(l\) whose rows \(\mathbf{h}_{n_{l}}^{\text{gen}}(l)\) are the encoding of the \(n_{j}\) nodes, the GCN layer computation is

\[H^{\text{gen}}(l+1)=f(\hat{P}H^{\text{gen}}(l)W(l)), \tag{8}\]

where \(W(l)\) is the parameter matrix of \(l\), \(f(\cdot)\) is the activation function, \(\hat{A}=A+I_{N}\) is the adjacency matrix with self-loops to enable the information of the current node to be included in the node representation, and \(\hat{D}\) is the degree matrix of \(\hat{A}\). A specific form of normalised graph Laplacian [43],

\[\hat{P}=\hat{D}^{-1/2}\hat{A}\hat{D}^{-1/2}, \tag{9}\]

is used to address the vanishing/exploding gradient problem. Note that as only future branch information is needed in TCPGen, \(\hat{A}\) only contains edges that lead to child nodes, and hence \(\hat{D}\) is computed based on this modified \(\hat{A}\). TCPGen then takes the node encodings of the final layer, \(H^{\text{gen}}(L)\), to compute key and value vectors in the same way as tree-RNN. As a practical consideration for deep networks in general, residual connections and layer normalisation are added for any GNNs with multiple layers.

Although lookahead with a configurable scope can be achieved by varying the number of layers \(L\), recent research found that the performance of GCN starts to degrade with more than three layers. Apart from the fact that deeper networks are more difficult to train, it was pointed out that the representations of the nodes in GCN are inclined to converge to a certain value and hence become indistinguishable, which is referred to as the _over-smoothing_ problem. One promising method to address this problem is to build a shortcut directly linking to the first GCN layer to ensure a certain fraction of the final representation comes from the current node itself. Thus, GCNII is also investigated in this paper with each layer computed in Eqn (10):

\[H^{\text{gi}}(l+1)=f\Big{(}[(1-\alpha)\hat{P}H^{\text{gi}}(l)+\alpha H^{\text{ gi}}(0)]W_{\beta}(l)\Big{)} \tag{10}\]

where \(H^{\text{gi}}(l)\) is the \(l\)-th layer output of GCNII, hyper-parameter \(\alpha\) scales the shortcut to the first layer, and \(W_{\beta}(l)\) is the parameter matrix defined as:

\[W_{\beta}(l)=(1-\beta_{l})I_{N}+\beta_{l}W(l) \tag{11}\]

where \(\beta_{l}\) is a layer-dependent hyper-parameter which is \(\text{ln}(1/l+1)\) in this paper.

Although GCNII has shown improved performance with deeper GCN of more than 4 layers, the maximum length in our biasing list is usually less than 10. With this depth for tree structures, the over-smoothing problem is less of a concern compared to the best structure of GCNII with 64 layers, and the network complexity is more problematic. Therefore, a simple parameter-sharing scheme is also proposed in this paper for deep GNNs to reduce network complexity. Specifically, the parameter matrices in the first \(K\) layers are shared:

\[W(1)=W(2)=...=W(K) \tag{12}\]

In contrast to other complex graph structures, message-passing operations from child nodes to the root in a tree were similar across different layers. Therefore, having the same weight matrix representing this process effectively reduces the model complexity to a degree that is adequate for tree encoding. In particular, \(K=L-1\) for GCN in this paper so that there are effectively two layer parameters to be trained while maintaining the depth of the network. The first \(K\) layers act as

Fig. 3: Pipeline of encoding prefix-tree with GNN for TCPGen. The prefix-tree is first encoded by a GNN, and the GNN-encoded tree is used by TCPGen to generate the TCPGen distribution where key and value vectors are GNN-based node encodings. The lookahead content for a 2-layer GCN as an example is denoted in {}, i.e. for node ri, \(\mathbf{h}_{n:i}\) covers information about an ne_ and o_
universal message passing and the last layer performs a final information aggregation from neighbouring nodes.

### _GraphSAGE with Max Pooling_

Node encodings for Tree-RNN and GCN are based on summation, whereas previous research [56, 57] has found that using max pooling also achieves competitive performance for word representation based on subword units. Hence, as an alternative GNN structure, GraphSAGE with a max pooling aggregator function is studied in this paper. GraphSAGE is a multi-layer GNN with each layer performing an information aggregation over a sampled set of child nodes followed by an update to the representation of the current node. Although one of the innovations in GraphSAGE is fixed-size sampling, as the training time biasing list is already a sampled subset from the full biasing list, the sampling of GraphSAGE is omitted in this paper. The computation of each layer is

\[\mathbf{h}_{\mathcal{N}_{i}}(l+1)=\max(\{\sigma(W_{1}(l)\mathbf{ h}_{n_{j}}^{\text{seg}}+\mathbf{b}(l)),\forall n_{k}\in\mathcal{N}_{j}\})\] \[\mathbf{h}_{n_{j}}^{\text{seg}}(l+1)=\sigma(W_{2}(l)\text{Concat}( \mathbf{h}_{\mathcal{N}_{j}}(l+1);\mathbf{h}_{n_{j}}^{\text{seg}}(l))) \tag{13}\]

where \(\max(\cdot)\) and \(\text{Concat}(\cdot)\) denote the element-wise max pooling and concatenation operators, \(\mathcal{N}_{j}\) is the set of child nodes of node \(n_{j}\). Although slightly better than GCN, GraphSAGE also degrades when adding more layers. Therefore, it is proposed in this paper to apply parameter-sharing for GraphSAGE, where both \(W_{1}=W_{1}(1)=W_{1}(2)=...=W_{1}(L)\) and \(W_{2}=W_{2}(1)=W_{2}(2)=...=W_{2}(L)\) are separately shared across all layers respectively.

TCPGen with GNN encodings still achieves high efficiency in handling large biasing lists. In training, with a large biasing list of 1000 words, TCPGen with a tree-RNN was 3.5 times slower than the standard AED or N-T model, with a negligible increase in space complexity. Among the three GNNs, GCN achieved the highest efficiency for training as its computation can be parallelised most, whereas the recursive computation in tree-RNN and the max-pooling in GraphSAGE hinder their training speed respectively. As a result, GCN in training is 2.5 times slower than the standard AED model, while GraphSAGE is 3 times slower. Moreover, by generating GNN encodings offline before decoding once the biasing list is available, the time and space complexity during inference is close to the standard AED or N-T for biasing lists of thousands of words.

### _Combination of GNN Encodings_

Combinations [41, 29] of GCN and GraphSAGE are explored in this paper for tree encodings, as they are conceptually complementary. GCN adopts a spectral approach where the graph Laplacian is used to aggregate information, while GraphSAGE directly exploits the graph structure and performs a max-pooling aggregation. To exploit the complementarity between the two GNNs, both additive and multiplicative combination methods are investigated here.

Additive combination performs a weighted sum of node encodings from GCN and GraphSAGE as the final node encodings before being processed by TCPGen (see Eqn. (14)).

\[\mathbf{h}_{n_{j}}^{\text{comb}}=\alpha^{\text{gen}}U_{1}\mathbf{h}_{n_{j}}^{ \text{gen}}+\alpha^{\text{seg}}U_{2}\mathbf{h}_{n_{j}}^{\text{seg}}\,. \tag{14}\]

\(U_{1}\) and \(U_{2}\) are two parameter matrices to rearrange the orders of the element in each GNN encoding as they may not encode information in the same order [41]. Note that \(\alpha^{\text{gen}}+\alpha^{\text{seg}}=1\) are weights that are either fixed or predicted via attention. The attention calculation is performed on each node \(n\) separately, as shown in Eqn. (15).

\[[\alpha_{i,n}^{\text{gen}},\alpha_{i,n}^{\text{seg}}]=\text{ Softmax}(\mathbf{q}_{i}^{T}[\mathbf{h}_{n_{j}}^{\text{gen}},\mathbf{h}_{n_{j}}^{ \text{seg}}]) \tag{15}\]

where \(\mathbf{q}_{i}\) is the same query vector used to calculate the TCPGen distribution. In this way, different sets of weights are assigned to different nodes at different decoder steps.

The multiplicative combination is performed via a low-rank approximation of the bilinear pooling method. The combination is shown in Eqn. (16)

\[\hat{\mathbf{h}}_{n_{j}}^{\text{comb}}=U_{3}(\tanh{(U_{1}\mathbf{h}_{n_{j}}^{ \text{gen}})}\odot\tanh{(U_{2}\mathbf{h}_{n_{j}}^{\text{seg}})}) \tag{16}\]

where \(U_{1}\), \(U_{2}\) and \(U_{3}\) are parameter matrices, and \(\odot\) is the element-wise product between two vectors. Following [41], a shortcut connection from each individual GNN encoding was provided to form the final combined encoding for TCPGen, as shown in Eqn. (17).

\[\mathbf{h}_{n_{j}}^{\text{comb}}=\hat{\mathbf{h}}_{n_{j}}^{\text{comb}}+U_{4 }\mathbf{h}_{n_{j}}^{\text{gen}}+U_{5}\mathbf{h}_{n_{j}}^{\text{gen}} \tag{17}\]

where \(U_{4}\) and \(U_{5}\) are another two parameter matrices.

## V Experimental setup

### _Data_

Experiments were conducted on two distinct datasets, namely the LibriSpeech audiobook corpus and the AMI meeting data, where the latter followed the audio-visual speech recognition pipeline. The LibriSpeech corpus [32], consists of 960 hours of read English from audiobooks, was used for evaluation purposes, with the dev-clean and dev-other sets used for validation, while test-clean and test-other were employed for evaluation. To investigate the impact of critical hyper-parameters, small-scale experiments were carried out using the train-clean-100 subset as the training set. Moreover, models trained on the LibriSpeech dataset were fine-tuned and evaluated on the AMI dataset in accordance with the approach proposed in [35].

The AMI meeting corpus [25] consists of 100 hours of meeting recordings involving 4-5 individuals, which were divided into the train, dev, and eval sets. To demonstrate the effectiveness of contextual biasing on data from another domain with limited training resources, a subset comprising 10% of the utterances from the AMI training set corresponding to 8 hours of audio was used to fine-tune the models previously trained on the LibriSpeech 960-hour data. There were 14 meetings from the dev set and 8 meetings from the eval set accompanied by slides that were selected to formulate the new test set for the audio-visual contextual ASR pipeline.

The 80-dim FBANK features at a 10 ms frame rate concatenated with 3-dim pitch features were used as the model input. SpecAugment [30] with the setting \((W,F,m_{F},T,p,m_{T})=(40,27,2,40,1.0,2)\) was used without any other data augmentation or speaker adaptation.
### _Biasing list selection_

To simulate real-world scenarios in LibriSpeech, the complete list of rare words comprising 200,000 distinct words as suggested in [10] was used. The full rare word list consisted of over 60% out-of-vocabulary (OOV) words that were absent from the LibriSpeech speech training set. Consistent with the method proposed in [10], the biasing lists were extracted by identifying words from the full rare word list that appeared in the reference transcription of each utterance, followed by the addition of a specific number of distractors. During inference, 10.3% of the word tokens in the test sets belonged to the full rare word list.

Fig. 4 shows the visual-grounded contextual ASR pipeline for AMI that utilises optical character recognition (OCR) output for slides. The Tesseract 4 OCR engine, equipped with LSTM models2, was first applied to the slides of each meeting series (e.g., ES2011[a-d]). Subsequently, distinct word tokens were extracted from the OCR output text files, and words in the full rare word list, which also occurred fewer than 100 times in the AMI training set, were selected to form the biasing list for that particular meeting series. These meeting-specific biasing lists were then used for the recognition of all utterances in that meeting series. The sizes of the biasing lists vary between 175 to 576, and the total number of word tokens covered by these lists was 1,751 out of 112,110 word tokens (1.5%). As shown in Fig. 4, these words mainly consisted of highly valuable content words whose accurate recognition was crucial for comprehending the utterance. Therefore, although the biasing lists had a minor impact on the overall word error rate (WER), they were essential for improving the recognition performance of critical words. Details of the meetings with slides and the extraction pipeline are available3.

Footnote 2: OCR implementation at [https://github.com/lesseract-ocr/lesseract](https://github.com/lesseract-ocr/lesseract)

Footnote 3: [https://github.com/the-anonymous-bs/AMfslides_biasing](https://github.com/the-anonymous-bs/AMfslides_biasing)

### _Model specification_

The ESPnet toolkit [31] was used for developing the systems. A unigram word piece model comprising 600 unique word pieces was created on the LibriSpeech data and was applied directly to the AMI data. Both the AED and N-T models employed a Conformer [27] encoder, which comprised 16 conformer blocks comprising 4 attention heads of size 512. The AED used a single-layer LSTM decoder of size 1024 and a location-sensitive attention mechanism featuring 4 heads of size 1024. The N-T, on the other hand, employed a 1024-dimensional predictor and a joint network consisting of a single fully-connected layer of size 1024. GNN encodings for LibriSpeech train-clean-100 experiments used 256-d GNN encodings, whereas the LibriSpeech full-scale experiments used 1024-d for GNN encoders.

LM shallow fusion and BLMD were implemented using a two-layer LSTM-LM with 2048 hidden units trained on the 800 million-word text training corpus of LibriSpeech as the target domain LM for the LibriSpeech experiments. Each source domain LM, trained on the text of the audio training data, used a single-layer LSTM with 1024 hidden units. It is worth noting that each LM had the same word pieces as the corresponding ASR system.

### _Training specifications_

During training, biasing lists with 1000 distractors were used for the experiments conducted on the LibriSpeech dataset, while 100 distractors were used for the AMI data. To create these lists, biasing words were selected from the reference transcription, and additional distractors were added. To prevent the AED model from becoming overly confident about TCPGen outputs, a dropout-inspired technique was employed during training, as described in [10]. Specifically, biasing words that were presented in the reference transcription had a 30% probability of being removed from the biasing list. The Conformer was optimised using the Noam optimiser [33]. Additionally, the hyper-parameters for the BLMD model were determined based on the respective dev sets for each dataset.

### _Evaluation metrics_

In addition to WER, the rare word error rate (R-WER) was used to evaluate the system performance on biasing words that were "rare" in the training data for that system. R-WER is the total number of _error_ word tokens that belong to the biasing list divided by the total number of word tokens in the test set that belong to the biasing list. Insertion errors were counted in R-WER if the inserted word belonged to the biasing list, in contrast to [38]. In addition, OOV WER was also computed in the same way as R-WER but for OOV words in the biasing list. There are altogether 443 such words in LibriSpeech test-clean and test-other sets. Moreover, the slides' rare word error rate (R\({}_{\text{s}}\)-WER) is reported for the AMI experiments calculated in the same way as R-WER, but for the rare words in slides. Insertions of slides biasing words were included in R\({}_{\text{s}}\)-WER.

As rare words are scarce in the dataset, significance tests were performed to ensure that the improvements found by using GNN encodings were statistically significant. Specifically,

Fig. 4: Illustration of the visual-grounded contextual ASR pipeline for the meeting series ES2011 containing meetings ES2011a to ES2011d.
independence was assumed at the book level for LibriSpeech and at the speaker level for AMI. The alternative hypothesis was defined as the GNN system performing better than the standard TCPGen (i.e. one-tailed sign test).

## VI Results

### _LibriSpeech train-clean-100 Results_

Experiments were first performed on the train-clean-100 set to find the best-performing GNN setups. The first investigation was on the number of GNN layers which determines how much lookahead is needed. The plots of WER and R-WER against the number of layers for N-T are shown in Fig. 5, and those for AED are shown in Fig. 6. Note that parameter tying only had effects when the layer number exceeded two.

For N-T, both GCN and GraphSAGE had a clear trend that the performance tended to degrade when the number of layers was increased beyond three and degraded significantly when it reached twelve layers. GraphSAGE inherently suffered less from this problem than GCN as the max pooling operator enabled the gradient for salient nodes to remain at its original value instead of being over-smoothed [42]. This degradation was mitigated by either using the GCNII structure or the parameter-tying scheme proposed in this paper. The best performance was achieved by 6-layer systems using WER as the selection criterion, where GCN and GraphSAGE with tied parameters achieved better performance than GCNII.

Similar observations were found with AED, where GCN and GraphSAGE with tied parameters achieved slightly better performance than GCNII. However, the best performance was achieved using 2-layer GCN and 3-layer GraphSAGE models. This difference is mainly caused by the label-synchronous nature of AED, where the model required knowledge for predicting only the next token, and information further into the future is less useful for this prediction in contrast to N-T.

The best-performing GCN and GraphSAGE with tied parameters were used for model combination. For additive

Fig. 5: Plot of WER (%) and R-WER (%) against the number of GNN layers for N-T on LibriSpeech test-clean data. Systems were trained on train-clean-100 for 120 epochs. Biasing lists with 1000 distractors were used. “Tied” referred to the parameter-tying scheme.

Fig. 6: Plot of WER (%) and R-WER (%) against the number of GNN layers for AED on LibriSpeech test-clean data. Systems were trained on train-clean-100 for 120 epochs. Biasing lists with 1000 distractors were used. “Tied” referred to the parameter-tying scheme.

Fig. 7: Variation of WER and R-WER against model combination weight \(\alpha_{\text{age}}\), and dynamic refers to using attention mechanism for weight calculation.
combination, different fixed combination weights gave results shown in Fig. 7, together with dynamic combination weights. As a result, dynamic combination performed better than most fixed-weight combinations, whereas the best performance was still obtained by the fixed-weight combination for both AED and N-T, with \(\alpha_{\text{age}}=0.2\). By examining the predicted dynamic weights, it was found that unless at the root node of the tree, the dynamic weight almost completely ignored GraphSAGE encodings and hence suffered from mode collapse, and hence GraphSAGE encodings were not properly trained. In fact, since GCN is usually better at handling near-future information, the model learnt to only rely on GCN.

### _LibriSpeech 960-hour Results_

The main results for LibriSpeech full-set experiments were summarised in Table I for AED and in Table II for N-T respectively. Compared to the standard TCPGen, all three types of GNNs achieved significantly better WER and R-WER (at p values smaller than 0.01) on both test sets for both AED and N-T. In particular, using multi-layer GNN, such as GCN and GraphSAGE, achieved clearly better performance to tree-RNN on AED, whereas the performance difference among those three GNN types was less obvious on N-T. The best-performing GNN structure on both AED and N-T was GCN with tied parameters. For AED, GCN achieved 16% relative WER reduction with a 20% R-WER reduction on the test-clean set and 13% relative WER reduction with 19% relative R-WER reduction on the test-other set (comparing row 4 to row 2 in Table I). For N-T, GCN achieved a 9% relative WER reduction with 21% relative R-WER reduction on the test-clean set, and a 7% WER reduction with a 17% relative R-WER reduction on the test-other set (comparing row 4 to row 2 in Table II). With similar levels of R-WER reduction, AED achieved a higher reduction in WER. As analysed in [36], TCPGen produced a much more confident prediction of \(P^{\text{gen}}\) with AED than N-T, where the main reductions in overall WER were attributed to the reduction in R-WER. The improvements using GNN indicated that the GNN encoding improved the prediction of \(P^{\text{gen}}\), which was more beneficial for the overall WER in AED.

Both additive and bilinear combinations of GNN encodings achieved superior performance to individual GNN encodings
with both AED and N-T models. For the AED model, the best performance was achieved by the additive combination with the best set of fixed weights previously found on train-clean-100 experiments, while the bilinear combination achieved very similar performance to the additive one. This led to a total of 31% relative R-WER reduction compared to the standard TCPGen (comparing row 7 to row 2 in Table I), and a total of 56% relative R-WER reduction compared to the baseline Conformer AED model. The performance improvement with the GNN combination was smaller on N-T, with the best results achieved by the bilinear pooling combination. This resulted in a 30% relative R-WER reduction compared to the standard TCPGen (comparing row 9 to row 2 in Table II), and an overall 56% relative R-WER reduction compared to the baseline Conformer N-T model.

Finally, selected systems were evaluated with BLMD, where TCPGen with GNN encodings achieved further performance improvements. The best-performing system for AED was TCPGen with the additive combination of GNN encodings, which achieved an overall 66% relative R-WER reduction on the test-clean set and 58% reduction on the test-other compared to the baseline. For N-T, an overall 57% relative R-WER reduction was achieved on test-clean, and 54% was achieved on test-other. Moreover, the OOV-WER had the same reduction pattern as R-WER for both AED and N-T. The best AED and N-T systems with combined GNN encodings for TCPGen reduced the OOV-WER by over 60%. Notably, BLMD was particularly beneficial for GNN encodings in AED systems, reducing the OOV-WER to 1/3 of the baseline value. This confirmed that even though GNN required more parameters to encode the prefix-tree, TCPGen still generalises well to unseen branches (i.e. OOV words) on the tree.

### _AMI Audio-visual Contextual ASR experiments_

The performance of various GNN encodings for TCPGen was further integrated into the audio-visual contextual ASR pipeline, and results were shown in Table III. In general, reductions in R-WER had a much smaller influence on the overall WER than with LibriSpeech, as rare words only occupied a much smaller portion. The findings were consistent with LibriSpeech, where the best-performing combination methods for AED and N-T were additive and bilinear respectively. However, the relative R-WER improvement was smaller compared to that in the LibriSpeech experiments, as the best-performing system here already had an R-WER that was very close to the overall WER whereas the R-WER in LibriSpeech was still twice as high as the overall WER. Compared to the baseline standard systems, TCPGen in AED achieved over 35% relative R-WER reduction using the best combined GNN encodings, while TCPGen with the best combined GNN encodings in N-T achieved over 30% relative R-WER reduction both with and without BLMD.

## VII Conclusion

This paper proposes three different types of GNN encodings in TCPGen for end-to-end contextual ASR, including tree-RNN, GCN and GraphSAGE. GNN encodings gave a lookahead functionality by incorporating future information on branches starting from the current node. Combination methods that take advantage of the complementarity between GCN and GraphSAGE were also explored. Experiments on LibriSpeech and AMI following an audio-visual contextual ASR pipeline showed consistent and significant WER and R-WER improvement for both AED and N-T systems using GNN encodings. The best combined GNN encodings achieved over 60% R-WER and OOV-WER reductions compared to the baseline standard systems.
