**LSTM.** Since the raw RNN only utilizes the simple tangent function, it is hard to obtain the long-term dependency of a long sentence/document. Reference [5] reinvents Long Short-Term Memory (LSTM) networks to strengthen the ability to model long-term semantic dependency in RNN. In detail, the composition function of the LSTM allows information from previous layers to flow directly to their following layers. The composition function could be defined as

$$\mathbf{f}\_{t} = \text{Sigmoid}(\mathbf{W}\_{f}^{h}\mathbf{h}\_{t-1} + \mathbf{W}\_{f}^{x}\mathbf{x}\_{t} + \mathbf{b}\_{f}),\tag{3.27}$$

$$\mathbf{i}\_{l} = \text{Sigmoid}(\mathbf{W}\_{i}^{h}\mathbf{h}\_{t-1} + \mathbf{W}\_{i}^{x}\mathbf{x}\_{t} + \mathbf{b}\_{i}),\tag{3.28}$$

$$\mathbf{b}\_{t} = \text{Sigmoid}(\mathbf{W}\_{o}^{h}\mathbf{h}\_{t-1} + \mathbf{W}\_{o}^{x}\mathbf{x}\_{t} + \mathbf{b}\_{o}),\tag{3.29}$$

$$\hat{\mathbf{c}}\_{t} = \tanh(\mathbf{W}\_{c}^{h}\mathbf{h}\_{t-1} + \mathbf{W}\_{c}^{x}\mathbf{x}\_{t} + \mathbf{b}\_{c}),\tag{3.30}$$

$$\mathbf{c}\_{t} = \mathbf{f}\_{t} \odot \mathbf{c}\_{t-1} + \mathbf{i}\_{t} \odot \hat{\mathbf{c}}\_{t},\tag{3.31}$$

$$\mathbf{h}\_t = \mathbf{o}\_t \odot \mathbf{c}\_t. \tag{3.32}$$

**Variants of LSTM.** To simplify LSTM and obtain more efficient algorithms, [3] proposes to utilize a simple but comparable RNN architecture, named Gated Recurrent Unit (GRU). Compared with LSTM, GRU has fewer parameters, which bring higher efficiency. The composition function is showed as

$$\mathbf{z}\_t = \text{Sigmoid}(\mathbf{W}\_z^h \mathbf{h}\_{t-1} + \mathbf{W}\_z^x \mathbf{x}\_t + \mathbf{b}\_z),\tag{3.33}$$

$$\mathbf{r}\_{t} = \text{Sigmoid}(\mathbf{W}\_{r}^{h}\mathbf{h}\_{t-1} + \mathbf{W}\_{r}^{x}\mathbf{x}\_{t} + \mathbf{b}\_{r}),\tag{3.34}$$

$$\hat{\mathbf{h}}\_{l} = \tanh(\mathbf{W}\_{h}(\mathbf{r}\_{l} \odot \mathbf{h}\_{t-1}) + \mathbf{W}\_{h}^{x}\mathbf{x}\_{l} + \mathbf{b}\_{h}),\tag{3.35}$$

$$\mathbf{h}\_{t} = (\mathbf{l} - \mathbf{z}\_{t}) \odot \mathbf{h}\_{t-1} + \mathbf{z}\_{t} \odot \hat{\mathbf{h}}\_{t}. \tag{3.36}$$

## *3.4.2 Recursive Neural Network*

Besides the recurrent neural network, another strategy to apply binary compositional function follows a parsing tree instead of sequential word order. Based on this philosophy, [15] proposes a recursive neural network to model different levels of semantic units. In this subsection, we will introduce some algorithms following the recursive parsing tree with different binary compositional functions.

Since all the recursive neural networks are binary trees, the basic problem we need to consider is how to derive the representation of the father component on the tree given its two children semantic components. Reference [15] proposes a recursive matrix-vector model (MV-RNN) which captures constituent parsing tree structure information by assigning a matrix-vector representation for each constituent. The vector captures the meaning of the constituent itself, and the matrix represents how it modifies the meaning of the word it combines with. Suppose we have two children components *a*, *b* and their father component *p*, the composition can be formulated as follows:

$$\mathbf{p} = f\_{\text{vec}}(a, b) = \text{g}\left(\mathbf{W}\_1 \begin{bmatrix} \mathbf{Ba} \\ \mathbf{Ab} \end{bmatrix}\right), \tag{3.37}$$

$$\mathbf{P} = f\_{matrix}(a, b) = \mathbf{W}\_2 \begin{bmatrix} \mathbf{A} \\ \mathbf{B} \end{bmatrix},\tag{3.38}$$

where **a**, **b**, **p** are the embedding vectors for each component and **A**,**B**, **P** are the matrices, **W**1 is a matrix that maps the transformed words into another semantic space, the element-wise function *g* is an activation function, and **W**2 is a matrix that maps the two matrices into one combined matrix **P** with the same dimension. The whole process is illustrated in Fig. 3.2. And then MV-RNN selects the highest node of the path in the parse tree between the two target entities to represent the input sentence.

In fact, the composition operation used in the above recursive network is similar to an RNN unit introduced in the previous subsection. And the RNN unit here can be replaced by LSTM units or GRU units. Reference [16] proposes two types of tree-structured LSTMs including the Child-Sum Tree-LSTM and the N-ary Tree-LSTM to capture constituent or dependency parsing tree structure information. For the Child-Sum Tree-LSTM, given a tree, let *C*(*t*) denote the children set of the node *t*. Its transition equations are defined as follows:

**Fig. 3.2** The architecture of the matrix-vector recursive encoder

$$\hat{\mathbf{h}}\_t = \sum\_{k \in C(t)} \mathbf{h}\_k,\tag{3.39}$$

$$\mathbf{i}\_t = \text{Sigmoid}(\mathbf{W}^{(i)}\mathbf{w}\_t + \mathbf{U}^i \hat{\mathbf{h}}\_t + \mathbf{b}^{(i)}),\tag{3.40}$$

$$\mathbf{f}\_{lk} = \text{Sigmoid}(\mathbf{W}^{(f)}\mathbf{w}\_l + \mathbf{U}^f \hat{\mathbf{h}}\_k + \mathbf{b}^{(f)}) \ (k \in \mathcal{C}(t)),\tag{3.41}$$

$$\mathbf{o}\_{t} = \text{Sigmoid}(\mathbf{W}^{(o)}\mathbf{w}\_{t} + \mathbf{U}^{o}\hat{\mathbf{h}}\_{t} + \mathbf{b}^{(o)}),\tag{3.42}$$

$$\mathbf{u}\_{l} = \tanh(\mathbf{W}^{(u)}\mathbf{w}\_{l} + \mathbf{U}^{u}\hat{\mathbf{h}}\_{l} + \mathbf{b}^{(u)}),\tag{3.43}$$

$$\mathbf{c}\_{t} = \mathbf{i}\_{t} \odot \mathbf{u}\_{t} + \sum\_{k \in C(t)} \mathbf{f}\_{tk} \odot \mathbf{c}\_{t-1},\tag{3.44}$$

$$\mathbf{h}\_t = \mathbf{o}\_t \odot \tanh(\mathbf{c}\_t). \tag{3.45}$$

The N-ary Tree-LSTM has similar transition equations as the Child-Sum Tree-LSTM. The only difference is that it limits the tree structures to have at most N branches.

## *3.4.3 Convolutional Neural Network*

Reference [6] proposes to embed an input sentence using a Convolutional Neural Network (CNN) which extracts local features by a convolution layer and combines all local features via a max-pooling operation to obtain a fixed-sized vector for the input sentence.

Formally, the convolution operation is defined as a matrix multiplication between a sequence of vectors, a convolution matrix **W**, and a bias vector **b** with a sliding window. Let us define the vector **q***i* as the concatenation of the subsequence of input representations in the *i*th window, we have

$$\mathbf{h}\_{j} = \max\_{i} \| f(\mathbf{Wq}\_{i} + \mathbf{b}) \|\_{j}, \tag{3.46}$$

where *f* indicates a nonlinear function such as sigmoid or tangent function, and **h** indicates the final representation of the sentence.

## **3.5 Summary**

In this chapter, we first introduce the semantic space for compositional semantics. Afterwards, we take phrase representation as an example to introduce various models for binary semantic composition, including additive models and multiplicative models. Finally, we introduce typical models for N-ary semantic composition including recurrent neural network, recursive neural network, and convolutional neural network. Compositional semantics allows languages to construct complex meanings from the combinations of simpler elements, and its binary semantic composition and N-ary semantic composition is the foundation of multiple NLP tasks including sentence representation, document representation, relational path representation, etc. We will give a detailed introduction to these scenarios in the following chapters.

For further understanding of compositional semantics, there are also some recommended surveys and books:

For better modeling compositional semantics, some directions require further efforts in the future:

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 4 Sentence Representation**

**Abstract** Sentence is an important linguistic unit of natural language. Sentence Representation has remained as a core task in natural language processing, because many important applications in related fields lie on understanding sentences, for example, summarization, machine translation, sentiment analysis, and dialogue system. Sentence representation aims to encode the semantic information into a real-valued representation vector, which will be utilized in further sentence classification or matching tasks. With large-scale text data available on the Internet and recent advances on deep neural networks, researchers tend to employ neural networks (e.g., convolutional neural networks and recurrent neural networks) to learn low-dimensional sentence representations and achieve great progress on relevant tasks. In this chapter, we first introduce the one-hot representation for sentences and the *n*-gram sentence representation (i.e., probabilistic language model). Then we extensively introduce neural-based models for sentence modeling, including feedforward neural network, convolutional neural network, recurrent neural network, and the latest Transformer, and pre-trained language models. Finally, we introduce several typical applications of sentence representations.

## **4.1 Introduction**

Natural language sentences consist of words or phrases, follow grammatical rules, and convey complete semantic information. Compared with words and phrases, sentences have more complex structures, including both sequential and hierarchical structures, which are essential for understanding sentences. In NLP, how to represent sentences is critical for related applications, such as sentence classification, sentiment analysis, sentence matching, and so on.

Before deep learning took off, sentences were usually represented as one-hot vectors or TF-IDF vectors, following the assumption of bag-of-words. In this case, a sentence is represented as a vocabulary-sized vector, in which each element represents the importance of a specific word (either term frequency or TF-IDF) to the sentence. However, this method confronts two issues. Firstly, the dimension of such representation vectors is usually up to thousands or millions. Thus, they usually face sparsity problem and bring in computational efficiency problem. Secondly, such a representation method follows the bag-of-words assumption and ignores the sequential and structural information, which can be crucial for understanding the semantic meanings of sentences.

Inspired by recent advances of deep learning models in computer vision and speech, researchers proposed to model sentences with deep neural networks, such as convolutional neural network, recurrent neural network, and so on. Compared with conventional word frequency-based sentence representations, deep neural networks can capture the internal structures of sentences, e.g., sequential and dependency information, through convolutional or recurrent operations. Thus, neural networkbased sentence representations have achieved great success in sentence modeling and NLP tasks.

## **4.2 One-Hot Sentence Representation**

One-hot representation is the most simple and straightforward method for word representation tasks. This method represents each word with a fixed length binary vector. Specifically, for a vocabulary *V* = {*w*1,*w*2,...,*w*|*V* |}, the one-hot representation of word *w* is **w** = [0,..., 0, 1, 0,..., 0]. Based on the one-hot word representation and the vocabulary, it can be extended to represent a sentence *s* = {*w*1,*w*2,...,*wl*} as

$$\mathbf{s} = \sum\_{k=1}^{l} \mathbf{w}\_{i},\tag{4.1}$$

where *l* indicates the length of the sentence *s*. The sentence representation **s** is the sum of the one-hot representations of *n* words within the sentence, i.e., each element in **s** represents the Term Frequency (TF) of the corresponding word.

Moreover, researchers usually take the importance of different words into consideration, rather than treat all the words equally. For example, the function words such as "a", "an", and "the" usually appear in different sentences, and reserve little meanings. Therefore, the Inverse Document Frequency (IDF) is employed to measure the importance of *wi* in *V* as follows:

$$\text{idf}\_{\text{w}\_i} = \log \frac{|D|}{\text{df}\_{\text{w}\_i}},\tag{4.2}$$

where |*D*| is the number of all documents in the corpus *D* and df*wi* represents the Document Frequency (DF) of *wi* .

With the importance of each word, the sentences are represented more precisely as follows:

$$
\hat{\mathbf{s}} = \mathbf{s} \otimes \text{idf},\tag{4.3}
$$

where ⊗ is the element-wise product.

Here, **s**ˆ is the TF-IDF representation of the sentence *s*.

## **4.3 Probabilistic Language Model**

One-hot sentence representation usually neglects the structure information in a sentence. To address this issue, researchers propose probabilistic language model, which treats *n*-grams rather than words as the basic components. An *n*-gram means a subsequence of words in a context window of length *n*, and probabilistic language model defines the probability of a sentence *s* = [*w*1,*w*2,...,*wl*] as

$$P(s) = \prod\_{i=1}^{l} P(w\_i | w\_1^{i-1}).\tag{4.4}$$

Actually, model indicated in Eq. (4.4) is not practicable due to its enormous parameter space. In practice, we simplify the model and set an *n*-sized context window, assuming that the probability of word *wi* only depends on [*wi*−*n*+1 ···*wi*−1]. More specifically, an *n*-gram language model predicts word *wi* in the sentence *s* based on its previous *n* − 1 words. Therefore, the simplified probability of a sentence is formalized as

$$P(\mathbf{s}) = \prod\_{i=1}^{l} P(\mathbf{w}\_i | \mathbf{w}\_{i-n+1}^{i-1}),\tag{4.5}$$

where the probability of selecting the word *wi* can be calculated from *n*-gram model frequency counts:

$$P(\boldsymbol{w}\_{i}|\boldsymbol{w}\_{i-n+1}^{i-1}) = \frac{P(\boldsymbol{w}\_{i-n+1}^{i})}{P(\boldsymbol{w}\_{i-n+1}^{i-1})}.\tag{4.6}$$

Typically, the conditional probabilities in *n*-gram language models are not calculated directly from the frequency counts, since it suffers severe problems when confronted with any *n*-grams that have not explicitly been seen before. Therefore, researchers proposed several types of smoothing approaches, which assign some of the total probability mass to unseen words or *n*-grams, such as "add-one" smoothing, Good-Turing discounting, or back-off models.

*n*-gram model is a typical probabilistic language model for predicting the next word in an *n*-gram sequence, which follows the Markov assumption that the probability of the target word only relies on the previous *n* − 1 words. The idea is employed by most of current sentence modeling methods. *n*-gram language model is used as an approximation of the true underlying language model. This assumption is crucial because it massively simplifies the problem of learning the parameters of language models from data. Recent works on word representation learning [3, 40, 43] are mainly based on the *n*-gram language model.

## **4.4 Neural Language Model**

Although smoothing approaches could alleviate the sparse problem in the probabilistic language model, it still performs poorly for those unseen or uncommon words and *n*-grams. Moreover, since probabilistic language models are constructed on larger and larger texts, the number of unique words (the vocabulary) increases and the number of possible sequences of words increases exponentially with the size of the vocabulary, causing a data sparsity problem. Thus statistics are needed to estimate probabilities accurately.

To address this issue, researchers propose neural language models which use continuous representations or embeddings of words and neural networks to make their predictions, in which embeddings in the continuous space help to alleviate the curse of dimensionality in language modeling, and neural networks avoid this problem by representing words in a distributed way, as nonlinear combinations of weights in a neural net [2]. An alternate description is that a neural network approximates the language function. The neural net architecture might be feedforward or recurrent, and while the former is simpler, the latter is more common.

Similar to probabilistic language models, neural language models are constructed and trained as probabilistic classifiers that learn to predict a probability distribution:

$$P(s) = \prod\_{i=1}^{l} P(w\_i | \mathbf{w}\_1^{i-1}),\tag{4.7}$$

where the conditional probability of the selecting word *wi* can be calculated by various kinds of neural networks such as feedforward neural networks, recurrent neural networks, and so on. In the following sections, we will introduce these neural language models in detail.

## *4.4.1 Feedforward Neural Network Language Model*

The goal of neural network language model is to estimate the conditional probability *P*(*wi*|*w*1,...,*wi*−1). However, the feedforward neural network (FNN) lacks an effective way to represent the long-term historical context. Therefore, it adopts the idea of *n*-gram language models to approximate the conditional probability, which assumes that each word in a word sequence more statistically depends on those words closer to it, and only *n* − 1 context words are used to calculate the conditional probability, i.e., *P*(*wi*|**w***i*−1 1 ) ≈ *P*(*wi*|**w***i*−1 *i*−*n*+1).

The overall architecture of the FNN language model is proposed by [3]. To evaluate the conditional probability of the word *wi* , it first projects its *n* − 1 context-related words to their word vector representations **x** = [**w***i*−*n*+1,..., **w***i*−1], and then feeds them into an FNN, which can be generally represented as

$$\mathbf{y} = \mathbf{M}f(\mathbf{Wx} + \mathbf{b}) + \mathbf{d},\tag{4.8}$$

where **W** is a weighted matrix to transform word vectors to hidden representations, **M** is a weighted matrix for the connections between the hidden layer and the output layer, and **b**, **d** are bias vectors. And then the conditional probability of the word *wi* can be calculated as

$$P(\mathbf{w}\_i|\mathbf{w}\_{i-n}^{i-1}) = \frac{\exp(\mathbf{y}\_{w\_i})}{\sum\_j \exp(\mathbf{y}\_j)}.\tag{4.9}$$

## *4.4.2 Convolutional Neural Network Language Model*

The Convolutional Neural Network (CNN) is the family of neural network models that features a type of layer known as the convolutional layer. This layer can extract features by a learnable filter (or kernel) at the different positions of an input. Pham et al. [47] propose the CNN language model to enhance the FNN language model. The proposed CNN network is produced by injecting a convolutional layer after the word input representation **x** = [**w***i*−*n*,..., **w***i*−1]. Formally, the convolutional layer involves a sliding window of the input vectors centered on each word vector using a parameter matrix *Wc*, which can be generally represented as

$$\mathbf{y} = \mathbf{M}(\max(\mathbf{W\_c x})),\tag{4.10}$$

where max(·) indicates a max-pooling layer. The architecture of CNN is shown in Fig. 4.1.

Moreover, [12] also introduces a convolutional neural network for language modeling with a novel gating mechanism.

## *4.4.3 Recurrent Neural Network Language Model*

To address the lack of ability for modeling long-term dependency in the FNN language model, [41] proposes a Recurrent Neural Network (RNN) language model which applies RNN in language modeling. RNNs are fundamentally different from FNNs in the sense that they operate on not only an input space but also an internal state space, and the internal state space enables the representation of sequentially extended dependencies. Therefore, the RNN language model can deal with those sentences of arbitrary length. At every time step, its input is the vector of its previous word instead of the concatenation of vectors of its *n* previous words, and the information of all other previous words can be taken into account by its internal state. Formally, the RNN language model can be defined as

$$\mathbf{h}\_{i} = f(\mathbf{W}\_{1}\mathbf{h}\_{i-1} + \mathbf{W}\_{2}\mathbf{w}\_{i} + \mathbf{b}),\tag{4.11}$$

$$\mathbf{y} = \mathbf{M}\mathbf{h}\_{i-1} + \mathbf{d},\tag{4.12}$$

where **W**1,**W**2, **M** are weighted matrices and **b**, **d** are bias vectors. Here, the RNN unit can also be implemented by LSTM or GRU. The architecture of RNN is shown in Fig. 4.2.

Recently, researchers make some comparisons among neural network language models with different architectures on both small and large corpora. The experimental results show that, generally, the RNN language model outperforms the CNN language model.

## *4.4.4 Transformer Language Model*

In 2018, Google proposed a pre-trained language model (PLM), called BERT, which achieved state-of-the-art results on a variety of NLP tasks. At that time, it was very big news. Since then, all the NLP researchers began to consider how PLMs can benefit their research tasks.

**Fig. 4.2** The architecture of RNN

In this section, we will first introduce the Transformer architecture and then talk about BERT and other PLMs in detail.

#### **<IP_ADDRESS> Transformer**

Transformer [65] is a nonrecurrent encoder-decoder architecture with a series of attention-based blocks. For the encoder, there are 6 layers and each layer is composed of a multi-head attention sublayer and a position-wise feedforward sublayer. And there is a residual connection between sublayers. The architecture of the Transformer is as shown in Fig. 4.3.

There are several attention heads in the multi-head attention sublayer. A *head* represents a scaled dot-product attention structure, which takes the query matrix **Q**, the key matrix **K**, and the value matrix **V** as the inputs, and the output is computed by

$$\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d\_k}}\right)\mathbf{V},\tag{4.13}$$

where *dk* is the dimension of query matrix.

The multi-head attention sublayer linearly projects the input hidden states *H* several times into the query matrix, the key matrix, and the value matrix for *h* heads. The dimensions of the query, key, and value vectors are *dk* , *dk* , and *d*v, respectively.

**Fig. 4.3** The architecture of Transformer

The multi-head attention sublayer could be formulated as

$$\text{Multihead}(H) = [head\_1, head\_2, \dots, head\_h] \mathbf{W}^O,\tag{4.14}$$

where *headi* = Attention(**HW***Q i* , **HW***K i* , **HW***V i* ), and **W***Q i* , **W***K i* and **W***V i* are linear projections. **W***O* is also a linear projection for the output. Here, the fully connected position-wise feedforward sublayer contains two linear transformations with ReLU activation:

$$\text{FFN}(\mathbf{x}) = \mathbf{W}\_2 \max(0, \mathbf{W}\_1 \mathbf{x} + \mathbf{b}\_1) + \mathbf{b}\_2. \tag{4.15}$$

Transformer is better than RNNs for modeling the long-term dependency, where all tokens will be equally considered during the attention operation. The Transformer was proposed to solve the problem of machine translation. Since Transformer has a very powerful ability to model sequential data, it becomes the most popular backbone of NLP applications.

#### **<IP_ADDRESS> Transformer-Based PLM**

Neural models can learn large amounts of language knowledge from language modeling. Since the language knowledge covers the demands of many downstream NLP tasks and provides powerful representations of words and sentences, some researchers found that knowledge can be transferred to other NLP tasks easily. The transferred models are called Pre-trained Language Models (PLMs).

Language modeling is the most basic and most important NLP task. It contains a variety of knowledge for language understanding, such as linguistic knowledge and factual knowledge. For example, the model needs to decide whether it should add an article before a noun. This requires linguistic knowledge about articles. Another example is the question of what is the following word after "Trump is the president of". The answer is "America", which requires factual knowledge. Since language modeling is very complex, the models can learn a lot from this task.

On the other hand, language modeling only requires plain text without any human annotation. With this feature, the models can learn complex NLP abilities from a very large-scale corpus. Since deep learning needs large amounts of data and language modeling can make full use of all texts in the world, PLMs significantly benefit the development of NLP research.

Inspired by the success of the Transformer, GPT [50] and BERT [14] begin to adopt the Transformer as the backbone of the pre-trained language models. GPT and BERT are the most representative Transformer-based pre-trained language models (PLMs). Since they achieved state-of-the-art performance on various NLP tasks, nearly all PLMs after them are based on the Transformer. In this subsection, we will talk about GPT and BERT in more detail.

GPT is the first work to pretrain a PLM based on the Transformer. The training procedure of GPT [50] contains two classic stages: generative pretraining and discriminative fine-tuning.

In the pretraining stage, the input of the model is a large-scale unlabeled corpus denoted as *U* = {*u*1, *u*2,..., *un*}. The pretraining stage aims to optimize a language model. The learning objective over the corpus is to maximize a conditional likelihood in a fixed-size window:

$$\mathcal{QC}\_{\mathbf{l}}(\partial \ell) = \sum\_{i} \log P(u\_i | u\_{i-k}, \dots, u\_{i-1}; \Theta), \tag{4.16}$$

where *k* represents the size of the window, the conditional likelihood *P* is modeled by a neural network with parameters Θ.

For a supervised dataset χ, the input is a sequence of words *s* = (*w*1,*w*2, ..,*wl*) and the output is a label *y*. The pretraining stage provides an advantageous start point of parameters that can be used to initialize subsequent supervised tasks. At this occasion, the objective is a discriminative task that maximizes the conditional possibility distribution:

$$\mathcal{LE}\_2(\chi) = \sum\_{(\mathbf{x}, \mathbf{y})} \log P(\mathbf{y}|\mathbf{w}\_1, \dots, \mathbf{w}\_l), \tag{4.17}$$

where *P*(*y*|*w*1,...,*wl*) is modeled by a K-layer Transformer. After the input tokens pass through the pretrained GPT, a hidden vector of the final layer **h***K l* will be produced. To obtain the output distribution, a linear transformation layer is added, which has the same size as the number of labels:

$$P(\mathbf{y}|\mathbf{w}\_1, \dots, \mathbf{w}\_m) = \text{Softmax}(\mathbf{W}\_{\mathbf{y}}\mathbf{h}\_l^K). \tag{4.18}$$

The final training objective is combined with a language modeling *L*1 for better generalization:

$$\mathcal{A}^{\varrho}(\chi) = \mathcal{A}\_2^{\varrho}(\chi) + \lambda \* \mathcal{A}\_1^{\varrho}(\chi),\tag{4.19}$$

where λ is a weight hyperparameter.

BERT [14] is a milestone work in the field of PLM. BERT achieved significant empirical results on 17 different NLP tasks, including SQuAD (outperform human being), GLUE (7.7% point absolute improvement), MultiNLI (4.6% point absolute improvement), etc. Compared to GPT, BERT uses a bidirectional deep Transformer as the model backbone. As illustrated in Fig. 4.4, BERT contains pretraining and fine-tuning stages.

In the pretraining stage, two objectives are designed: *Masked Language Model (MLM)* and *Next Sentence Prediction (NSP)*. (1) For MLM, tokens are randomly

**Fig. 4.4** The pretraining and fine-tuning stages for BERT

masked with a special token [MASK]. The training objective is to predict the masked tokens based on the contexts. Compared with the standard unidirectional conditional language model, which can only be trained in one direction, MLM aims to train a deep bidirectional representation model. This task is inspired by *Cloze* [64]. (2) The objective of NSP is to capture relationships between sentences for some sentence-based downstream tasks such as natural language inference (NLI) and question answering (QA). In this task, a binary classifier is trained to predict whether the sentence is the next sentence for the current. This task effectively captures the deep relationship between sentences, exploring semantic information from a different level.

After pretraining, BERT can capture various language knowledge for downstream supervised tasks. By modifying inputs and outputs, BERT can be fine-tuned for any NLP tasks, which contain the applications with the input of single text or text pairs. The input consists of sentence A and sentence B, which can represent (1) sentence pairs in paraphrase, (2) hypothesis-premise pairs in entailment, (3) question-passage pairs in QA, and (4) text-∅ for text classification task or sequence tagging. For the output, BERT can produce the token-level representation for each token, which is used to sequence tagging task or question answering. Besides, the special token [CLS] in BERT is fed into the classification layer for sequence classification.

#### **<IP_ADDRESS> PLM Family**

Pre-trained language models have rapid progress after BERT. We summarize several important directions of PLMs and show some representative models and their relationship in Fig. 4.5.

Here is a brief introduction of the PLMs after BERT. Firstly, there are some variants of BERT for better general language representation, such as RoBERTa [38] and XLNet [70]. These models mainly focus on the improvement of pretraining tasks. Secondly, some people work on pretrained generation models, such as MASS [57] and UniLM [15]. These models achieve promising results on the generation tasks instead of the Natural Language Understanding (NLU) tasks used by BERT. Thirdly, the sentence pair format of BERT inspired works on the cross-lingual and cross-modal fields. XLM [8], ViLBERT [39], and VideoBERT [59] are the important works in this direction. Lastly, there are some works [46, 81] that explore to incorporate external knowledge into PLMs since some low-frequency knowledge cannot be efficiently learned by PLMs.

## *4.4.5 Extensions*

#### **<IP_ADDRESS> Importance Sampling**

Inspired by the contrastive divergence model, [4] proposes to adopt importance sampling to accelerate the training of neural language models. They first normalize the

**Fig. 4.5** The Pre-trained language model family

outputs of neural network language model and view neural network language models as a special case of energy-based probability models as following:

$$P(\mathbf{w}\_i|\mathbf{w}\_{i-n}^{i-1}) = \frac{\exp(-\mathbf{y}\_{\mathbf{w}\_i})}{\sum\_j \exp(-\mathbf{y}\_j)}.\tag{4.20}$$

The key idea of importance sampling is to approximate the mean of log-likelihood gradient of the loss function of neural network language model by sampling several important words instead of calculating the explicit gradient. Here, the log-likelihood gradient of the loss function of neural network language model can be generally represented as

$$\frac{\partial P\left(\mathbf{w}\_{i}|\mathbf{w}\_{i-n}^{i-1}\right)}{\partial\theta} = -\frac{\partial \mathbf{y}\_{w\_{i}}}{\partial\theta} + \sum\_{j=1}^{|V|} P\left(\mathbf{w}\_{j}|\mathbf{w}\_{i-n}^{i-1}\right) \frac{\partial \mathbf{y}\_{j}}{\partial\theta}$$

$$= -\frac{\partial \mathbf{y}\_{i}}{\partial\theta} + \mathbb{E}\_{\mathbf{w}\_{k}\sim P} \left[\frac{\partial \mathbf{y}\_{k}}{\partial\theta}\right],\tag{4.21}$$

where θ indicates all parameters of the neural network language model. Here, the log-likelihood gradient of the loss function consists of two parts including positive gradient for target word *wi* and negative gradient for all words *wj* , i.e., E*wi*∼*P* [ ∂*yj* ∂θ ]. Here, the second part can be approximated by sampling important words following the probability distribution *P*:

#### 4.4 Neural Language Model 71

$$\mathbb{E}\_{\mathbf{w}\_k \sim P} \left[ \frac{\partial \mathbf{y}\_k}{\partial \theta} \right] \approx \sum\_{\mathbf{w}\_k \in V'} \frac{1}{|V'|} \frac{\partial \mathbf{y}\_k}{\partial \theta},\tag{4.22}$$

where *V* is the word set sampled under *P*.

However, since we cannot obtain probability distribution *P* in advance, it is impossible to sample important words following the probability distribution *P*. Therefore, importance sampling adopts a Monte Carlo scheme which uses an existing proposal distribution *Q* to approximate *P*, and then we have

$$\mathbb{E}\_{\mathbf{w}\_k \sim P} \left[ \frac{\partial \mathbf{y}\_k}{\partial \theta} \right] \approx \frac{1}{|V''|} \sum\_{\mathbf{w}\_l \in V''} \frac{\partial \mathbf{y}\_l}{\partial \theta} P(\mathbf{w}\_l | \mathbf{w}\_{i-n}^{i-1}) / Q(\mathbf{w}\_l), \tag{4.23}$$

where *V* is the word set sampled under *Q*. Moreover, the sample size of importance sampling approach should be increased as training processes in order to avoid divergence, which aims to ensure its effective sample size *S*:

$$S = \frac{(\sum\_{w\_l \in V''} r\_l)^2}{\sum\_{w\_l \in V''} r\_l^2},\tag{4.24}$$

where *rl* is further defined as

$$r\_l = \frac{P(\boldsymbol{w}\_l|\mathbf{w}\_{i-n}^{i-1})/\mathcal{Q}(\boldsymbol{w}\_l)}{\sum\_{\boldsymbol{w}\_j \in V^s} P(\boldsymbol{w}\_j|\mathbf{w}\_{i-n}^{i-1})/\mathcal{Q}(\boldsymbol{w}\_j)}.\tag{4.25}$$

#### **<IP_ADDRESS> Word Classification**

Besides important sampling, researchers [7, 22] also propose class-based language model, which adopts word classification to improve the performance and speed of a language model. In class-based language model, all words are assigned to a unique class, and the conditional probability of a word given its context can be decomposed into the probability of the word's class given its previous words and the probability of the word given its class and history, which is formally defined as

$$P(\boldsymbol{w}\_i|\mathbf{w}\_{i-n}^{i-1}) = \sum\_{\mathbf{c}(\boldsymbol{w}\_i)\in\mathcal{C}} P(\boldsymbol{w}\_i|\mathbf{c}(\boldsymbol{w}\_i))P(\boldsymbol{c}(\boldsymbol{w}\_i)|\mathbf{w}\_{i-n}^{i-1}),\tag{4.26}$$

where *C* indicates the set of all classes and *c*(*wi*) indicates the class of word *wi* .

Moreover, [44] proposes a hierarchical neural network language model, which extends word classification to a hierarchical binary clustering of words in a language model. Instead of simply assigning each word with a unique class, it first builds a hierarchical binary tree of words according to the word similarity obtained from WordNet. Next, it assigns a unique bit vector *c*(*wi*) = [*c*1(*wi*), *c*2(*wi*), . . . , *cl*(*wi*)]for each word, which indicates the hierarchical classes of them. And then the conditional probability of each word can be defined as

$$P(\boldsymbol{w}\_i|\mathbf{w}\_{i-n}^{i-1}) = \prod\_{j=0}^{l} P(c\_j(\boldsymbol{w}\_i)|c\_1(\boldsymbol{w}\_i), c\_2(\boldsymbol{w}\_i), \dots, c\_{j-1}(\boldsymbol{w}\_i), \mathbf{w}\_{i-n}^{i-1}).\tag{4.27}$$

The hierarchical neural network language model can achieve *O*(*k*/ log *k*) speed up as compared to a standard language model. However, the experimental results of [44] show that although the hierarchical neural network language model achieves an impressive speed up for modeling sentences, it has worse performance than the standard language model. The reason is perhaps that the introduction of hierarchical architecture or word classes imposes a negative influence on the word classification by neural network language models.

#### **<IP_ADDRESS> Caching**

Caching is also one of the important extensions in language model. A type of cachebased language model assumes that each word in recent context is more likely to appear again [58]. Hence, the conditional probability of a word can be calculated by the information from history and caching:

$$P(\boldsymbol{w}\_{i}|\mathbf{w}\_{i-n}^{i-1}) = \lambda P\_{s}(\boldsymbol{w}\_{i}|\mathbf{w}\_{i-n}^{i-1}) + (1-\lambda)P\_{c}(\boldsymbol{w}\_{i}|\mathbf{w}\_{i-n}^{i-1}),\tag{4.28}$$

where *Ps*(*wi*|**w***i*−1 *i*−*n*) indicates the conditional probability generated by standard language and *Pc*(*wi*|**w***i*−1 *i*−*n*) indicates the conditional probability generated by caching, and λ is a constant.

Another cache-based language model is also used to speed up the RNN language modeling [27]. The main idea of this approach is to store the outputs and states of language models for future predictions given the same contextual history.

## **4.5 Applications**

In this section, we will introduce two typical sentence-level NLP applications including text classification and relation extraction, as well as how to utilize sentence representation for these applications.

## *4.5.1 Text Classification*

Text classification is a typical NLP application and has lots of important real-world tasks such as parsing and semantic analysis. Therefore, it has attracted the interest of many researchers. The conventional text classification models (e.g., the LDA [6] and tree kernel [48] models) focus on capturing more contextual information and correct word order by extracting more useful and distinct features, but still expose a few issues (e.g., data sparseness) which has the significant impact on the classification accuracy. Recently, with the development of deep learning in the various fields of artificial intelligence, neural models have been introduced into the text classification field due to their abilities of text representation learning. In this section, we will introduce the two typical tasks of text classification, including sentence classification and sentiment classification.

#### **<IP_ADDRESS> Sentence Classification**

Sentence classification aims to assign a sentence an appropriate category, which is a basic task of the text classification application.

Considering the effectiveness of the CNN models in capturing sentence semantic meanings, [31] first proposes to utilize the CNN models trained on the top of pretrained word embeddings to classify sentences, which achieved promising results on several sentence classification datasets. Then, [30] introduces a dynamic CNN model to model the semantic meanings of sentences. This model handles sentences of varying lengths and uses dynamic max-pooling over linear sequences, which could help the model capture both short-range and long-range semantic relations in sentences. Furthermore, [9] proposes a novel CNN-based model named as Very Deep CNN, which operates directly at the character level. It shows that those deeper models have better results on sentence classification and can capture the hierarchical information from scattered characters to whole sentences. Yin and Schütze [74] also propose MV-CNN, which utilizes multiple types of pretrained word embeddings and extracts features from multi-granular phrases with variable-sized convolutional layers. To address the drawbacks of MV-CNN such as model complexity and the requirement for the same dimension of embeddings, [80] proposes a novel model called MG-CNN to capture multiple features from multiple sets of embeddings that are concatenated at the penultimate layer. Zhang et al. [79] present RA-CNN to jointly exploit labels on documents and their constituent sentences, which can estimate the probability that a given sentence is informative and then scales the contribution of each sentence to aggregate a document representation in proportion to the estimates.

The RNN model which aims to capture the sequential information of sentences is also widely used in sentence classification. Lai et al. [32] propose a neural network for text classification, which applies a recurrent structure to capture contextual information. Moreover, [37] introduces a multitask learning framework based on the RNN to jointly learn across multiple sentence classification tasks, which employs three different mechanisms of sharing information to model sentences with both taskspecific and shared layers. Yang et al. [71] introduce word-level and sentence-level attention mechanisms into an RNN-based model as well as a hierarchical structure to capture the hierarchical information of documents for sentence classification.

#### **<IP_ADDRESS> Sentiment Classification**

Sentiment classification is a special task of the sentence classification application, whose objective is to classify the sentimental polarities of opinions a piece of text contains, e.g., favorable or unfavorable, positive or negative. This task appeals the NLP community since it has lots of potential downstream applications such as movie review suggestions.

Similar to text classification, the sentence representation based on neural models has also been widely explored for sentiment classification. Glorot et al. [20] use a stacked denoising autoencoder in sentiment classification for the first time. Then, a series of recursive neural network models based on the recursive tree structure of sentences are conducted to learn sentence representations for sentiment classification, including the recursive autoencoder (RAE) [55], matrix-vector recursive neural network (MV-RNN) [54], and recursive neural tensor network (RNTN) [56]. Besides, [29] adopts a CNN to learn sentence representations and achieves promising performance in sentiment classification.

The RNN models also benefit sentiment classification as they are able to capture the sequential information. Li et al. [35] and Tai et al. [62] investigate a tree-structured LSTM model on text classification. There are also some hierarchical models proposed to deal with document-level sentiment classification [5, 63], which generate semantic representations at different levels (e.g., phrase, sentence, or document) within a document. Moreover, the attention mechanism is also introduced into sentiment classification, which aims to select important words from a sentence or important sentences from a document [71].

## *4.5.2 Relation Extraction*

To enrich existing KGs, researchers have devoted many efforts to automatically finding novel relational facts in text. Therefore, relation extraction (RE), which aims at extracting relational facts according to semantic information in plain text, has become a crucial NLP application. As RE is also an important downstream application of sentence representation, we will, respectively, introduce the techniques and extensions to show how to utilize sentence representation for different RE scenarios. Considering neural networks have become the backbone of the recent NLP research, we mainly focus on Neural RE (NRE) models in this section.

**Fig. 4.6** An example of sentence-level relation extraction

#### **<IP_ADDRESS> Sentence-Level NRE**

Sentence-level NRE aims at predicting the semantic relations between the given entity (or nominal) pair in a sentence. As shown in Fig. 4.6, given the input sentence *s* which consists of *n* words *s* = {*w*1,*w*2,...,*wn*} and its corresponding entity pair *e*1 and *e*2 as input, sentence-level NRE wants to obtain the conditional probability *P*(*r*|*s*, *e*1, *e*2) of relation *r* (*r* ∈ *R*) via a neural network, which can be formalized as

$$P(r|\mathbf{s}, e\_1, e\_2) = P(r|\mathbf{s}, e\_1, e\_2, \theta),\tag{4.29}$$

where θ is all parameters of the neural network and *r* is a relation in the relation set *R*.

A basic form of sentence-level NRE consists of three components: (a) an input encoder to give a representation for each input word, (b) a sentence encoder which computes either a single vector or a sequence of vectors to represent the original sentence, and (c) a relation classifier which calculates the conditional probability distribution of all relations.

**Input Encoder**. First, a sentence-level NRE system projects the discrete words of the source sentence into a continuous vector space, and obtains the input representation **w** = {**w**1, **w**2,..., **w***m*} of the source sentence.

from the word founder to the head entity Bill\_Gates is −3 and the tail entity Microsoft is 2. Besides word position embeddings, more linguistic features are also considered in addition to the word embeddings to enrich the linguistic features of the input sentence.

For each word, the NRE models often concatenate some of the above four feature embeddings as their input embeddings. Therefore, the feature embeddings of all words are concatenated and denoted as a final input sequence **w** = {**w**1, **w**2,..., **w***m*}, where **w***i* ∈ R*d* , *d* is the total dimension of all feature embeddings concatenated for each word.

**Sentence Encoder**. The sentence encoder is the core for sentence representation, which encodes input representations into either a single vector or a sequence of vectors **x** to represent sentences. We will introduce the different sentence encoders in the following.

(1) Convolutional Neural Network Encoder. Zeng et al. [76] propose to encode input sentences using a CNN model, which extracts local features by a convolutional layer and combines all local features via a max-pooling operation to obtain a fixedsized vector for the input sentence. Formally, a convolutional layer is defined as an operation on a vector sequence **w**:

$$\mathbf{p} = \text{CNN}(\mathbf{w}),\tag{4.30}$$

where CNN indicates the convolution operation inside the convolutional layer.

And the *i*th element of the sentence vector **x** can be calculated as follows:

$$\mathbf{[x]}\_i = f(\max(\mathbf{p}\_i)),\tag{4.31}$$

where *f* is a nonlinear function applied at the output, such as the hyperbolic tangent function.

#### 4.5 Applications 77

Further, PCNN [75], which is a variation of CNN, adopts a piece-wise maxpooling operation. All hidden vectors {**p**1, **p**2,...} are divided into three segments by the head and tail entities. The max-pooling operation is performed over the three segments separately, and the **x** is the concatenation of the pooling results over the three segments.

(2) Recurrent Neural Network Encoder. Zhang and Wang [78] propose to embed input sentences using an RNN model which can learn the temporal features. Formally, each input word representation is put into recurrent layers step by step. For each step *i*, the network takes the *i*th word representation vector **w***i* and the output of the previous *i* − 1 steps **h***i*−1 as input:

$$\mathbf{h}\_{i} = \text{RNN}(\mathbf{w}\_{i}, \mathbf{h}\_{i-1}), \tag{4.32}$$

where RNN indicates the transform function inside the RNN cell, which can be the LSTM units or the GRU units mentioned before.

The conventional RNN models typically deal with text sequences from start to end, and build the hidden state of each word only considering its preceding words. It has been verified that the hidden state considering its following words is more effective. Hence, the bi-directional RNN (BRNN) [52] is adopted to learn hidden states using both preceding and following words.

Similar to the previous CNN models in RE, the RNN model combines the output vectors of the recurrent layer as local features, and then uses a max-pooling operation to extract the global feature, which forms the representation of the whole input sentence. The max-pooling layer could be formulated as

$$\{\mathbf{x}\}\_j = \max\_i \{\mathbf{h}\_i\}\_j. \tag{4.33}$$

Besides max-pooling, word attention can also combine all local feature vectors together. The attention mechanism [1] learns attention weights on each step. Supposing **H** = [**h**1, **h**2,..., **h***m*] is the matrix consisting of all output vectors produced by the recurrent layer, the feature vector of the whole sentence **x** is formed by a weighted sum of these output vectors:

$$\alpha = \text{Softmax}(\mathbf{s}^\top \tanh(\mathbf{H})),\tag{4.34}$$

$$\mathbf{x} = \mathbf{H}\boldsymbol{\alpha}^{\top},\tag{4.35}$$

where **s** is a trainable query vector.

Besides, [42] proposes a model that captures information from both word sequence and tree-structured dependency by stacking bidirectional path-based LSTM-RNNs (i.e., bottom-up and top-down). More specifically, it focuses on the shortest path between the two target entities in the dependency tree, and utilizes the stacked layers to encode the shortest path for the whole sentence representation. In fact, some preliminary work [69] has shown that these paths are useful in RE, and various recursive neural models are also proposed for this. Next, we will introduce these recursive models in detail.

(3) Recursive Neural Network Encoder. The recursive encoder aims to extract features from the information of syntactic parsing trees, considering the syntactic information is beneficial for extracting relations from sentences. Generally, these encoders treat the tree structure inside syntactic parsing trees as a strategy of composition as well as a direction to combine each word feature.

Socher et al. [54] propose a recursive matrix-vector model (MV-RNN) which captures the structure information by assigning a matrix-vector representation for each constituent of the constituents in parsing trees. The vector captures the meaning of the constituent itself and the matrix represents how it modifies the meaning of the word it combines with. Tai et al. [62] further propose two types of tree-structured LSTMs including the Child-Sum Tree-LSTM and the N-ary Tree-LSTM to capture tree structure information. For the Child-Sum Tree-LSTM, given a tree, let *C*(*t*) denote the set of children of node *t*. Its transition equations are defined as follows:

$$\hat{\mathbf{h}}\_t = \sum\_{k \in C(t)} \text{TLSTM}(\mathbf{h}\_k), \tag{4.36}$$

where TLSTM(·)indicates a Tree-LSTM cell, which is simply modified from LSTM cell. The N-ary Tree-LSTM has similar transition equations as the Child-Sum Tree-LSTM. The only difference is that it limits the tree structures to have at most *N* branches.

**Relation Classifier**. When obtaining the representation **x** of the input sentence, relation classifier calculates the conditional probability *P*(*r*|*x*, *e*1, *e*2) via a softmax layer as follows:

$$P(r|\mathbf{x}, e\_1, e\_2) = \text{Softmax}(\mathbf{M}\mathbf{x} + \mathbf{b}),\tag{4.37}$$

where **M** indicates the relation matrix and **b** is a bias vector.

#### **<IP_ADDRESS> Bag-Level NRE**

Although existing neural models have achieved great success for extracting novel relational facts, it always suffers the lack of training data. To address this issue, researchers proposed a distant supervision assumption to generate training data via aligning KGs and plain text automatically. The intuition of distant supervision assumption is that all sentences that contain two entities will express their relations in KGs. For example, (New York, city\_of, United States) is a relational fact in a KG, distant supervision assumption will regard all sentences that contain these two entities as positive instances for the relation city\_of. It offers a natural way of utilizing information from multiple sentences (bag-level) rather than a single sentence (sentence-level) to decide if a relation holds between two entities.

Therefore, bag-level NRE aims to predict the semantic relations between an entity pair using all involved sentences. As shown in Fig. 4.7, given the input sentence set

**Fig. 4.7** An example of bag-level relation extraction

*S* which consists of *n* sentences *S* = {*s*1,*s*2,...,*sn*} and its corresponding entity pair *e*1 and *e*2 as inputs, bag-level NRE wants to obtain the conditional probability *P*(*r*|*S*, *e*1, *e*2) of relation *r* (*r* ∈ R) via a neural network, which can be formalized as

$$P(r|S, e\_1, e\_2) = P(r|S, e\_1, e\_2, \theta). \tag{4.38}$$

A basic form of bag-level NRE consists of four components: (a) an input encoder similar to sentence-level NRE, (b) a sentence encoder similar to sentence-level NRE, (c) a bag encoder which computes a vector representing all related sentences in a bag, and (d) a relation classifier similar to sentence-level NRE which takes bag vectors as input instead of sentence vectors. As the input encoder, sentence encoder, and relation classifier of bag-level NRE are similar to the ones of sentence-level NRE, we will thus mainly focus on introducing the bag encoder in detail.

**Bag Encoder**. The bag encoder encodes all sentence vectors into a single vector **S**. We will introduce the different bag encoders in the following:

(1) Random Encoder. It simply assumes that each sentence can express the relation between two target entities and randomly select one sentence to represent the bag. Formally, the bag representation is defined as

$$\mathbf{S} = \mathbf{s}\_{i} \ (i \in \{1, 2, \ldots, n\}),\tag{4.39}$$

where **s***i* indicates the sentence representation of *si* ∈ *S* and *i* is a random index.

(2) Max Encoder. As introduced above, not all sentences containing two target entities can express their relations. For example, the sentence New York City is the premier gateway for legal immigration to the United States does not express the relation city of. Hence, in [75], they follow the at-least-one assumption which assumes that at least one sentence that contains these two target entities can express their relations, and select the sentence with the highest probability for the relation to represent the bag. Formally, bag representation is defined as

$$\mathbf{S} = \mathbf{s}\_i \ (i = \arg\max\_i P(r|s\_i, e\_1, e\_2)). \tag{4.40}$$

(3) Average Encoder. Both random encoder or max encoder use only one sentence to represent the bag, which ignores the rich information of different sentences. To exploit the information of all sentences, [36] believes that the representation **S** of the bag depends on all sentences' representations. Each sentence representation **s***i* can give the relation information about two entities to a certain extent. The average encoder assumes that all sentences contribute equally to the representation of the bag. It means the embedding **S** of the bag is the average of all the sentence vectors:

$$\mathbf{S} = \sum\_{i} \frac{1}{n} \mathbf{s}\_i. \tag{4.41}$$

(4) Attentive Encoder. Due to the wrong label issue brought by distant supervision assumption inevitably, the performance of average encoder will be influenced by those sentences that contain no relation information. To address this issue, [36] further proposes to employ a selective attention to reduce those noisy sentences. Formally, the bag representation is defined as a weighted sum of sentence vectors:

$$\mathbf{S} = \sum\_{i} \alpha\_{i} \mathbf{s}\_{i},\tag{4.42}$$

where α*i* is defined as

$$\alpha\_{i} = \frac{\exp(\mathbf{s}\_{i}^{\top}\mathbf{A}\mathbf{r})}{\sum\_{j} \exp(\mathbf{x}\_{j}^{\top}\mathbf{A}\mathbf{r})},\tag{4.43}$$

where **A** is a diagonal matrix and **r** is the representation vector of relation *r*.

**Relation Classifier**. Similar to sentence-level NRE, when obtaining the bag representation **S**, relation classifier also calculates the conditional probability *P*(*r*|*S*, *e*1, *e*2) via a softmax layer as follows:

$$P(r|S, e\_1, e\_2) = \text{Softmax}(\mathbf{MS} + \mathbf{b}),\tag{4.44}$$

where **M** indicates the relation matrix and **b** is a bias vector.

#### **<IP_ADDRESS> Extensions**

Recently, NRE systems have achieved significant improvements in both, the supervised and distantly supervised scenarios. However, there are still many challenges in the task of RE, and many researchers have been focusing on other aspects to improve the performance of NRE as well. In this section, we will introduce these extensions in detail.

**Utilization of External Information**. Most existing NRE systems stated above only concentrate on the sentences which are extracted, regardless of the rich external information such as KGs. This heterogeneous information could provide additional knowledge from KG and is essential when extracting new relational facts.

Han et al. [24] propose a novel joint representation learning framework for knowledge acquisition. The key idea is that the joint model learns knowledge and text representations within a unified semantic space via KG-text alignments. For the text part, the sentence with two entities Mark Twain and Florida is regarded as the input for a CNN encoder, and the output of CNN is considered to be the latent relation PlaceOfBirth of this sentence. For the KG part, entity and relation representations are learned via translation-based methods. The learned representations of KG and text parts are aligned during training. Besides this preliminary attempt, many efforts have been devoted to this direction [25, 28, 51, 67, 68].

**Incorporating Relational Paths**. Although existing NRE systems have achieved promising results, they still suffer a major problem: the models can only directly learn from those sentences which contain both two-target entities. However, those sentences containing only one of the entities could also provide useful information and help build inference chains. For example, if we know that "A is the son of B" and "B is the son of C", we can infer that A is the grandson of C.

To utilize the information of both direct and indirect sentences, [77] introduces a path-based NRE model that incorporates textual relational paths. The model first employs a CNN encoder to embed the semantic meanings of sentences. Then, the model builds a relation path encoder, which measures the probability of relations given an inference chain in the text. Finally, the model combines information from both direct sentences and relational paths, and then predicts the confidence of each relationship. This work is the first effort to consider the knowledge of relation path in text for NRE, and there are also several methods later to consider the reasoning path of sentence semantic meanings for RE [11, 19].

**Document-level Relation Extraction**. In fact, not all relational facts can be extracted by sentence-level RE, i.e., a large number of relational facts are expressed in multiple sentences. Taking Fig. 4.9 as an example, multiple entities are mentioned in the document and exhibit complex interactions. In order to identify the relational fact (Riddarhuset, country, Sweden), one has to first identify the fact that Riddarhuset is located in Stockholm from Sentence 4, then identify the facts Stockholm is the capital of Sweden and Sweden is a country from Sentence 1. With the above facts, we can finally infer that the sovereign state of Riddarhuset is Sweden. This process requires reading and reasoning over multiple sentences in a document, which is intuitively beyond the reach of sentence-level RE methods. According to the statistics on a human-annotated corpus sampled from Wikipedia documents [72], at least 40.7% relational facts can only be extracted from multiple sentences, which is not negligible. Swampillai and Stevenson [61] and Verga et al. [66] also report similar observations. Therefore, it is necessary to move RE

**Fig. 4.8** An example of document-level relation extraction

forward from the sentence level to the document level. Figure 4.8 is an example for document-level RE.

**Fig. 4.9** An example from DocRED [72]

However, existing datasets for document-level RE either only have a small number of manually annotated relations and entities [34], or exhibit noisy annotations from distant supervision [45, 49], or serve specific domains or approaches [33]. To address this issue, [72] constructs a large-scale, manually annotated, and generalpurpose document-level RE dataset, named as DocRED. DocRED is constructed from Wikipedia and Wikidata, and has two key features. First, DocRED contains 132, 375 entities and 56, 354 relational facts annotated on 5, 053 Wikipedia documents, which is the largest human-annotated document-level RE dataset now. Second, over 40% of the relational facts in DocRED can only be extracted from multiple sentences. This makes DocRED require reading multiple sentences in a document to recognize entities and inferring their relations by synthesizing all information of the document.

The experimental results on DocRED show that the performance of existing sentence-level RE methods declines significantly on DocRED, indicating the task document-level RE is more challenging than sentence-level RE and remains an open problem. It also relates to the document representation which will be introduced in the next chapter.

#### **Few-shot Relation Extraction**.

As we mentioned before, the performance of the conventional RE models [23, 76] heavily depend on time-consuming and labor-intensive annotated data, which make themselves hard to generalize well. Although adopting distant supervision is a primary approach to alleviate this problem, the distantly supervised data also exhibits a long-tail distribution, where most relations have very limited instances. Furthermore, distant supervision suffers the wrong labeling problem, which makes it harder to classify long-tail relations. Hence, it is necessary to study training RE models with insufficient training instances. Figure 4.10 is an example for few-shot RE.

**Fig. 4.10** An example of few-shot relation extraction

**Table 4.1** An example for a 3 way 2 shot scenario. Different colors indicate different entities, underline for head entity, and *emphasize* for tail entity

FewRel [26] is a new large-scale supervised few-shot RE dataset, which requires models capable of handling classification task with a handful of training instances, as shown in Table 4.1. Benefiting from the FewRel dataset, there are some efforts to exploring few-shot RE [17, 53, 73] and achieve promising results. Yet, few-shot RE still remains a challenging problem for further research [18].

## **4.6 Summary**

In this chapter, we introduce sentence representation learning. Sentence representation encodes the semantic information of a sentence into a real-valued representation vector, and can be utilized in further sentence classification or matching tasks. First, we introduce the one-hot representation for sentences and probabilistic language models. Secondly, we extensively introduce several neural language models, including adopting the feedforward neural networks, the convolutional neural networks, the recurrent neural networks, and the Transformer for language models. These neural models can learn rich linguistic and semantic knowledge from language modeling. Benefiting from this, the pre-trained language models trained with large-scale corpora have achieved state-of-the-art performance on various downstream NLP tasks by transferring the learned semantic knowledge from general corpora to the target tasks. Finally, we introduce several typical applications of sentence representation including text classification and relation extraction.

For further understanding of sentence representation learning and its applications, there are also some recommended surveys and books including

In the future, for better sentence representation, some directions are requiring further efforts:

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 5 Document Representation**

**Abstract** A document is usually the highest linguistic unit of natural language. Document representation aims to encode the semantic information of the whole document into a real-valued representation vector, which could be further utilized in downstream tasks. Recently, document representation has become an essential task in natural language processing and has been widely used in many documentlevel real-world applications such as information retrieval and question answering. In this chapter, we first introduce the one-hot representation for documents. Next, we extensively introduce topic models that learn the topic distribution of words and documents. Further, we give an introduction to distributed document representation, including paragraph vector and neural document representations. Finally, we introduce several typical real-world applications of document representation, including information retrieval and question answering.

## **5.1 Introduction**

Advances in information and communication technologies offer ubiquitous access to vast amounts of information and are causing an exponential increase in the number of documents available online. While more and more textual information is available electronically, effective retrieval and mining are getting more and more difficult without the efficient organization, summarization, and indexing of document content. Therefore, document representation is playing an important role in many real-world applications, e.g., document retrieval, web search, and spam filtering. Document representation aims to represent document input into a fixed-length vector, which could describe the contents of the document, to reduce the complexity of the documents and make them easier to handle. Traditional document representation models such as one-hot document representation have achieved promising results in many document classification and clustering tasks due to their simplicity, efficiency, and often surprising accuracy.

However, the one-hot document representation model has many disadvantages. First, it loses the word order, and thus, different documents can have the same representation, as long as the same words are used. Second, it usually suffers data sparsity and high dimensionality. One-hot document representation model has very little sense about the semantics of the words or, more formally, the distances between the words. Hence, the approach for representing text documents uses multi-word terms as vector components, which are noun phrases extracted using a combination of linguistic and statistical criteria. This representation is motivated by the notion of topic models that terms should contain more semantic information than individual words. And another advantage of using terms for representing a document is its lower dimensionality compared with the traditional one-hot document representation.

Nevertheless, applying these to generation tasks remains difficult. To understand how discourse units are connected, one has to understand the communicative function of each unit, and the role it plays within the context that encapsulates it, recursively all the way up for the entire text. Identifying increasingly sophisticated humandeveloped features may be insufficient for capturing these patterns, but developing representation-based alternatives has also been difficult. Although document representation can capture aspects of coherent sentence structure, it is not clear how it could help in generating more broadly cohesive text.

Recently, neural network models have shown compelling results in generating meaningful and grammatical documents in sequence generation tasks like machine translation or parsing. It is partially attributed to the ability of these systems to capture local compositionally: the way neighboring words are combined semantically and syntactically to form meanings that they wish to express. Based on neural network models, many research works have developed a variety of ways to incorporate document-level contextual information. These models are all hybrid architectures in that they are recurrent at the sentence level, but use a different structure to summarize the context outside the sentence. Furthermore, some models explore multilevel recurrent architectures for combining local and global information in language modeling.

In this chapter, we first introduce the one-hot representation for documents. Next, we extensively introduce topic models that aim to learn latent topic distributions of words and documents. Further, we give an introduction on distributed document representation including paragraph vector and neural document representations. Finally, we introduce several typical real-world applications of document representations, including information retrieval and question answering.

## **5.2 One-Hot Document Representation**

Majority of machine learning algorithms take a fixed-length vector as the input, so documents are needed to be represented as vectors. The bag-of-words model is the most common and simple representation method for documents. Similar to one-hot sentence representation, for a document *d* = {*w*1,*w*2,...,*wl*}, a bag-ofword representation **d** can be used to represent this document. Specifically, for a vocabulary *V* = [*w*1,*w*2,...,*w*|*V*|] , the one-hot representation of word *w* is **w** = [0, 0,..., 1,..., 0]. Based on the one-hot word representation and a vocabulary *V*, it can be extended to represent a document as

$$\mathbf{d} = \sum\_{k=1}^{l} \mathbf{w}\_{i},\tag{5.1}$$

where *l* is the length of the document *d*. And similar to one-hot sentence representation, the TF-IDF method is also proposed to enhance the ability of bag-of-words representation in reflecting how important a word is to a document in a corpus.

Actually, the bag-of-words representation is mainly used as a tool of feature generation, and the most common type of features calculated from this method is word frequency appearing in the documents. This method is simple but efficient and sometimes can reach excellent performance in many real-world applications. However, the bag-of-words representation still ignores entirely the word order information, which means different documents can have the same representation as long as the same words are used. Furthermore, bag-of-words representation has little sense about the semantics of the words or, more formally, the distances between words, which means this method cannot utilize rich information hidden in the word representations.

## **5.3 Topic Model**

As our collective knowledge continues to be digitized and stored in the form of news, blogs, web pages, scientific articles, books, images, audio, videos, and social networks, it becomes more difficult to find and discover what we are looking for. We need new computational tools to help organize, search, and understand these vast amounts of information.

Right now, we work with online information using two main tools—search and links. We type keywords into a search engine and find a set of documents related to them. We look at the documents in that set, possibly navigating to other linked documents. This is a powerful way of interacting with our online archive, but something is missing.

Imagine searching and exploring documents based on the themes that run through them. We might "zoom in" and "zoom out" to find specific or broader themes; we might look at how those themes changed through time or how they are connected. Rather than finding documents through keyword search alone, we might first find the theme that we are interested in, and then examine the documents related to that theme.

For example, consider using themes to explore the complete history of the New York Times. At a broad level, some of the themes might correspond to the sections of the newspaper, such as foreign policy, national affairs, and sports. We could zoom in on a theme of interest, such as foreign policy, to reveal various aspects of it, such as Chinese foreign policy, the conflict in the Middle East, and the United States' relationship with Russia. We could then navigate through time to reveal how these specific themes have changed, tracking, for example, the changes in the conflict in the Middle East over the last 50 years. And, in all of this exploration, we would be pointed to the original articles relevant to the themes. The thematic structure would be a new kind of window through which to explore and digest the collection.

But we do not interact with electronic archives in this way. While more and more texts are available online, we do not have the human power to read and study them to provide the kind of browsing experience described above. To this end, machine learning researchers have developed **probabilistic topic modeling**, a suite of algorithms that aim to discover and annotate vast archives of documents with thematic information. Topic modeling algorithms are statistical methods that analyze the words of the original texts to explore the themes that run through them, how those themes are connected, and how they change over time. Topic modeling algorithms do not require any prior annotations or labeling of the documents. The topics emerge from the analysis of the original texts. Topic modeling enables us to organize and summarize electronic archives at a scale that would be impossible by human annotation.

## *5.3.1 Latent Dirichlet Allocation*

A variety of probabilistic topic models have been used to analyze the content of documents and the meaning of words. Hofmann first introduced the probabilistic topic approach to document modeling in his Probabilistic Latent Semantic Indexing method (pLSI). The pLSI model does not make any assumptions about how the mixture weights are generated, making it difficult to test the generalization ability of the model to new documents. Thus, Latent Dirichlet Allocation (LDA) was extended from this model by introducing a Dirichlet prior to the model. LDA is believed as a simple but efficient topic model. We first describe the basic ideas of LDA [6].

The intuition behind LDA is that documents exhibit multiple topics. LDA is a statistical model of document collections that tries to capture this intuition. It is most easily described by its generative process, the imaginary random process by which the model assumes the documents arose.

We formally define a *topic* to be a distribution over a fixed vocabulary. We assume that these topics are specified before any data has been generated. Now for each document in the collection, we generate the words in a two-stage process.

	- Randomly choose a topic from the distribution over topics in step #1.
	- Randomly choose a word from the corresponding distribution over the vocabulary.

This statistical model reflects the intuition that documents exhibit multiple topics. Each document exhibits the topics with different proportions (step #1); each word in each document is drawn from one of the topics (step #2b), where the selected topic is chosen from the per-document distribution over topics (step #2a).

We emphasize that the algorithms have no information about these subjects and the articles are not labeled with topics or keywords. The interpretable topic distributions arise by computing the hidden structure that likely generated the observed collection of documents.

#### **<IP_ADDRESS> LDA and Probabilistic Models**

LDA and other topic models are part of the broader field of *probabilistic modeling*. In generative probabilistic modeling, we treat our data as arising from a generative process that includes *hidden variables*. This generative process defines a joint probability distribution over both the observed and hidden random variables. Given the observed variables, we perform data analysis by using that joint distribution to compute the conditional distribution of the hidden variables. This conditional distribution is also called the *posterior distribution*.

LDA falls precisely into this framework. The observed variables are the words of the documents, the hidden variables are the topic structure, and the generative process is as described above. The computational problem of inferring the hidden topic structure from the documents is the problem of computing the posterior distribution, the conditional distribution of the hidden variables given the documents.

We can describe LDA more formally with the following notation. The topics are β1:*K* , where each β*k* is a distribution over the vocabulary. The topic proportions for the *d*th document are θ*d* , where θ*dk* is the topic proportion for topic *k* in document *d*. The topic assignments for the *d*th document are *zd* , where *zd*,*n* is the topic assignment for the *n*th word in document *d*. Finally, the observed words for document *d* are *wd* , where *wd*,*n* is the *n*th word in document *d*, which is an element from the fixed vocabulary.

With this notation, the generative process for LDA corresponds to the following joint distribution of the hidden and observed variables:

$$P(\beta\_{\mathbf{l}:K}, \theta\_{\mathbf{l}:D}, z\_{\mathbf{l}:D}, w\_{\mathbf{l}:D}) = \prod\_{i=1}^{K} P(\beta\_i) \prod\_{d=1}^{D} P(\theta\_d) (\prod\_{n=1}^{N} P(z\_{d,n}|\theta\_d) P(w\_{d,n}|\beta\_{\mathbf{l}:K}, z\_{d,n}) \dots) \tag{5.2}$$

Notice that this distribution specifies the number of dependencies. For example, the topic assignment *zd*,*n* depends on the per-document topic proportions θ*d* . As another example, the observed word *wd*,*n* depends on the topic assignment *zd*,*n* and all of the topics β1:*K* .

These dependencies define LDA. They are encoded in the statistical assumptions behind the generative process, in the particular mathematical form of the joint distribution, and in a third way, in the probabilistic graphical model for LDA. Probabilistic graphical models provide a graphical language for describing families of probability

**Fig. 5.1** The architecture of graphical model for Latent Dirichlet Allocation

distributions. The graphical model for LDA is in Fig. 5.1. Each node is a random variable and is labeled according to its role in the generative process. The hidden nodes, the topic proportions, assignments, and topics are unshaded. The observed nodes and the words of the documents, are shaded. We use rectangles as plate notation to denote replication. The *N* plate denotes the collection of words within documents; the *D* plate denotes the collection of documents within the collection. These three representations are equivalent ways of describing the probabilistic assumptions behind LDA.

#### **<IP_ADDRESS> Posterior Computation for LDA**

We now turn to the computational problem, computing the conditional distribution of the topic structure given the observed documents. (As we mentioned above, this is called the *posterior*.) Using our notation, the posterior is

$$P(\beta\_{\mathbf{l}\colon K}, \theta\_{\mathbf{l}\colon D}, z\_{\mathbf{l}\colon D} | \mathbf{v}\_{\mathbf{l}\colon D}) = \frac{P(\beta\_{\mathbf{l}\colon K}, \theta\_{\mathbf{l}\colon D}, z\_{\mathbf{l}\colon D}, \mathbf{v}\_{\mathbf{l}\colon D})}{P(\mathbf{v}\_{\mathbf{l}\colon D})}.\tag{5.3}$$

The numerator is the joint distribution of all the random variables, which can be easily computed for any setting of the hidden variables. The denominator is the marginal probability of the observations, which is the probability of seeing the observed corpus under any topic model. In theory, it can be computed by summing the joint distribution over every possible instantiation of the hidden topic structure.

Topic modeling algorithms form an approximation of the above equation by forming an alternative distribution over the latent topic structure that is adapted to be close to the true posterior. Topic modeling algorithms generally fall into two categories: sampling-based algorithms and variational algorithms.

Sampling-based algorithms attempt to collect samples from the posterior by approximating it with an empirical distribution. The most commonly used sampling algorithm for topic modeling is Gibbs sampling, where we construct a Markov chain, a sequence of random variables, each dependent on the previous—whose limiting distribution is posterior. The Markov chain is defined on the hidden topic variables for a particular corpus, and the algorithm is to run the chain for a long time, collect samples from the limiting distribution, and then approximate the distribution with the collected samples.

Variational methods are a deterministic alternative to sampling-based algorithms. Rather than approximating the posterior with samples, variational methods posit a parameterized family of distributions over the hidden structure and then find the member of that family that is closest to the posterior. Thus, the inference problem is transformed into an optimization problem. Variational methods open the door for innovations in optimization to have a practical impact on probabilistic modeling.

## *5.3.2 Extensions*

The simple LDA model provides a powerful tool for discovering and exploiting the hidden thematic structure in large archives of text. However, one of the main advantages of formulating LDA as a probabilistic model is that it can easily be used as a module in more complicated models for more complex goals. Since its introduction, LDA has been extended and adapted in many ways.

#### **<IP_ADDRESS> Relaxing the Assumptions of LDA**

LDA is defined by the statistical assumptions it makes about the corpus. One active area of topic modeling research is how to relax and extend these assumptions to uncover a more sophisticated structure in the texts.

One assumption that LDA makes is the bag-of-words assumption that the order of the words in the document does not matter. While this assumption is unrealistic, it is reasonable if our only goal is to uncover the coarse semantic structure of the texts. For more sophisticated goals, such as language generation, it is patently not appropriate. There have been many extensions to LDA that model words non-exchangeable. For example, [59] develops a topic model that relaxes the bag-of-words assumption by assuming that the topics generate words conditional on the previous word; [22] develops a topic model that switches between LDA and a standard HMM. These models expand the parameter space significantly but show improved language modeling performance.

Another assumption is that the order of documents does not matter. Again, this can be seen by noticing that Eq. 5.3 remains invariant to permutations of the ordering of documents in the collection. This assumption may be unrealistic when analyzing long-running collections that span years or centuries. In such collections, we may want to assume that the topics change over time. One approach to this problem is the dynamic topic model [5], a model that respects the ordering of the documents and gives a more productive posterior topical structure than LDA.

The third assumption about LDA is that the number of topics is assumed known and fixed. The Bayesian nonparametric topic model provides an elegant solution: The collection determines the number of topics during posterior inference, and new documents can exhibit previously unseen topics. Bayesian nonparametric topic models have been extended to hierarchies of topics, which find a tree of topics, moving from more general to more concrete, whose particular structure is inferred from the data [4].

#### **<IP_ADDRESS> Incorporating Meta-Data to LDA**

In many text analysis settings, the documents contain additional information such as author, title, geographic location, links, and others that we might want to account for when fitting a topic model. There has been a flurry of research on adapting topic models to include meta-data.

The author-topic model [51] is an early success story for this kind of research. The topic proportions are attached to authors; papers with multiple authors are assumed to attach each word to an author, drawn from a topic drawn from his or her topic proportions. The author-topic model allows for inferences about authors as well as documents.

Many document collections are linked. For example, scientific papers are linked by citations, or web pages are connected by hyperlinks. And several topic models have been developed to account for those links when estimating the topics. The relational topic model of [9] assumes that each document is modeled as in LDA and that the links between documents depend on the distance between their topic proportions. This is both a new topic model and a new network model. Unlike traditional statistical models of networks, the relational topic model takes into account node attributes in modeling the links.

Other work that incorporates meta-data into topic models includes models of linguistic structure [8], models that account for distances between corpora [60], and models of named entities [42]. General-purpose methods for incorporating metadata into topic models include Dirichlet-multinomial regression models [39] and supervised topic models [37].

#### **<IP_ADDRESS> Acceleration**

In the existing fast algorithms, it is difficult to decouple the access to *Cd* and *Cw* because both counts need to be updated instantly after the sampling of every token. Many algorithms have been proposed to accelerate LDA based on this equation. WarpLDA [13] is built based on a new Monte Carlo Expectation Maximization (MCEM) algorithm, which is similar to CGS, but both counts are fixed until the sampling of all tokens is finished. This scheme can be used to develop a reordering strategy to decouple the accesses to *Cd* and *Cw*, and minimize the size of randomly accessed memory.

Specifically, WarpLDA seeks a MAP solution of the latent variables Θ and Φ, with the latent topic assignments *Z* integrated out: where α and β are the Dirichlet hyperparameters. Reference [2] has shown that this MAP solution is almost identical with the solution of CGS, with proper hyperparameters.

#### 5.3 Topic Model 99

Computing log *P*(Θ, Φ|*W*, α- , β- ) directly is expensive because it needs to enumerate all the *K* possible topic assignments for each token. We, therefore, optimize its lower bound as a surrogate. Let *Q*(*Z*) be a variational distribution. Then, by Jensen's inequality, the lower bound can be *J* (Θ, Φ, *Q*(*Z*)):

$$\log P(\Theta, \Phi | W, \alpha', \beta') \ge \mathbb{E}\_{\mathcal{Q}}[\log P(W, Z | \Theta, \Phi) - \log \mathcal{Q}(Z)] + \log P(\Theta | \alpha') + \log P(\Phi | \beta')$$

$$\stackrel{\triangle}{=} \mathcal{J}^{\ell}(\Theta, \Phi, \mathcal{Q}(Z)). \tag{5.4}$$

An Expectation Maximization (EM) algorithm is implemented to find a local maximum of the posterior *P*(Θ, Φ|*W*, α- , β- ), where the E-step maximizes *J* with respect to the variational distribution *Q*(*Z*) and the M-step maximizes *J* with respect to the model parameters (Θ, Φ), while keeping *Q*(*Z*) fixed. One can prove that the optimal solution at E-step is *Q*(*Z*) = *P*(*Z*|*W*,Θ,Φ)without further assumption on *Q*. We apply Monte Carlo approximation on the expectation in Eq. 5.4,

$$\mathbb{E}\_{\mathcal{Q}}[\log P(W, Z | \Theta, \Phi) - \log \mathcal{Q}(Z)] \approx \frac{1}{S} \sum\_{s=1}^{S} \log P(W, Z^{(s)} | \Theta, \Phi) - \log \mathcal{Q}(Z^{(s)}),\tag{5.5}$$

where *Z*(1) ,..., *Z*(*S*) ∼ *Q*(*Z*) = *P*(*Z*|*W*, Θ, Φ). The sample size is set as *S* = 1 and the model uses *Z* as an abbreviation of *Z*(1) .

**Sampling** *Z*: Each dimension of *Z* can be sampled independently:

$$\mathcal{Q}(z\_{d,n} = k) \propto P(W, Z | \mathcal{Q}, \Phi) \propto \theta\_{dk} \phi\_{w\_{d,n}, k}.\tag{5.6}$$

**Optimizing** Θ,Φ: With the Monte Carlo approximation, we have

$$\begin{split} \mathcal{J} & \approx \log P(W, Z | \Theta, \Phi) + \log P(\Theta | \alpha') + \log P(\Phi | \beta') + \text{const.} \\ &= \sum\_{d, k} (C\_{dk} + \alpha'\_k - 1) \log \theta\_{dk} + \sum\_{k, \mathbf{w}} (C\_{kw} + \beta' - 1) \log \phi\_{kw} + \text{const.}, \end{split} \tag{5.7}$$

and with the optimal solutions, we have

$$
\hat{\theta}\_{dk} \propto \mathcal{C}\_{dk} + \alpha'\_k - 1, \quad \hat{\phi}\_{wk} = \frac{\mathcal{C}\_{wk} + \beta' - 1}{\mathcal{C}\_k + \bar{\beta}' - V}. \tag{5.8}
$$

Instead of computing and storing Θˆ and Φˆ, we compute and store *Cd* and *Cw* to save memory because the latter are sparse. Plug Eqs. 5.8–5.6, and let α = α- − 1, β = β- − 1, we get the full MCEM algorithm, which iteratively performs the following two steps until a given iteration number is reached:

• E-step: We can sample *zd*,*n* ∼ *Q*(*zd*,*n* = *k*) according to

$$\mathcal{Q}(z\_{d,n} = k) \propto (C\_{dk} + \alpha\_k) \frac{C\_{wk} + \beta\_w}{C\_k + \bar{\beta}}.\tag{5.9}$$

• M-step: Compute *Cd* and *Cw* by *Z*.

Note the resemblance intuitively justifies why MCEM leads to similar results with CGS. The difference between MCEM and CGS is that MCEM updates the counts *Cd* and *Cw* after sampling all *zd*,*n*s, while CGS updates the counts instantly after sampling each *zd*,*n*. The strategy that MCEM updates the counts after sampling all *zd*,*n*s is called *delayed count update*, or simply *delayed update*. MCEM can be viewed as a CGS with a delayed update, which has been widely used in other algorithms [1, 41]. While previous work uses the delayed update as a trick, we at this moment present a theoretical guarantee to converge to a MAP solution. The delayed update is essential for us to decouple the accesses of *Cd* and *Cw* to improve cache locality, without affecting the correctness.

## **5.4 Distributed Document Representation**

To address the disadvantages of bag-of-words document representation, [31] proposes paragraph vector models, including the version with Distributed Memory (PV-DM) and the version with Distributed Bag-of-Words (PV-DBOW). Moreover, researchers also proposed several hierarchical neural network models to represent documents. In this section, we will introduce these models in detail.

## *5.4.1 Paragraph Vector*

As shown in Fig. 5.2, paragraph vector maps every paragraph to a unique vector, represented by a column in the matrix **P** and maps every word to a unique vector, represented by a column in word embedding matrix **E**. The paragraph vector and word vectors are averaged or concatenated to predict the next word in a context. More formally, compared to the word vector framework, the only change in this model is in the following equation, where *h* is constructed from **E** and **P**.

$$\mathbf{y} = \text{Softmax}(h(w\_{t-k}, \dots, w\_{t+k}; \mathbf{E}, \mathbf{P})),\tag{5.10}$$

where *h* is constructed by the concatenation or average of word vectors extracted from **E** and **P**.

**Fig. 5.2** The architecture of PV-DM model

The other part of this model is that given a sequence of training words *w*1, *w*2, *w*3, …, *wl* , the objective of the paragraph vector model is to maximize the average log probability:

$$\mathcal{O} = \frac{1}{l} \sum\_{i=k}^{l-k} \log P(w\_i \mid w\_{i-k}, \dots, w\_{i+k}). \tag{5.11}$$

And the prediction task is typically done via a multi-class classifier, such as softmax. Thus, the probability equation is

$$P(w\_i \mid w\_{i-k}, \dots, w\_{i+k}) = \frac{e^{\mathbf{y}\_{w\_i}}}{\sum\_j e^{\mathbf{y}\_j}}.\tag{5.12}$$

The paragraph token can be thought of as another word. It acts as a memory that remembers what is missing from the current context, or the topic of the paragraph. For this reason, this model is often called the Distributed Memory Model of Paragraph Vectors (PV-DM).

The above method considers the concatenation of the paragraph vector with the word vectors to predict the next word in a text window. Another way is to ignore the context words in the input, but force the model to predict words randomly sampled from the paragraph in the output. In reality, what this means is that at each iteration of stochastic gradient descent, we sample a text window, then sample a random word from the text window and form a classification task given the Paragraph Vector. This technique is shown in Fig. 5.3. This version is named the Distributed Bag-of-Words version of Paragraph Vector (PV-DBOW), as opposed to the Distributed Memory version of Paragraph Vector (PV-DM) in the previous section.

**Fig. 5.3** The architecture of PV-DBOW model

In addition to being conceptually simple, this model requires to store fewer data. The data only needed to be stored is the softmax weights as opposed to both softmax weights and word vectors in the previous model. This model is also similar to the Skip-gram model in word vectors.

## *5.4.2 Neural Document Representation*

In this part, we introduce two main kinds of neural networks for document representation including document-context language model and hierarchical document autoencoder.

#### **<IP_ADDRESS> Document-Context Language Model**

Recurrent architectures can be used to combine local and global information in document language modeling. The simplest such model would be to train a single RNN, ignoring sentence boundaries as mentioned above; the last hidden state from the previous sentence *t* − 1 is used to initialize the first hidden state in sentence *t*. In such an architecture, the length of the RNN is equal to the number of tokens in the document; in typical genres such as news texts, this means training RNNs from sequences of several hundred tokens, which introduces two problems: (1) **Information decay** In a sentence with thirty tokens (not unusual in news text), the contextual information from the previous sentence must be propagated through the recurrent dynamics thirty times before it can reach the last token of the current sentence. Meaningful document-level information is unlikely to survive such a long pipeline. (2) **Learning** It is notoriously difficult to train recurrent architectures that involve many time steps. In the case of an RNN trained on an entire document, back-propagation would have to run over hundreds of steps, posing severe numerical challenges.

To address these two issues, [28] proposes to use multilevel recurrent structures to represent documents, thereby successfully efficiently leveraging document-level context in language modeling. They first proposed Context-to-Context Document-Context Language Model (ccDCLM), which assumes that contextual information from previous sentences needs to be able to "short-circuit" the standard RNN, so as to more directly impact the generation of words across longer spans of text. Formally, we have

$$\mathbf{c}\_{t-1} = \mathbf{h}\_{t-1,l},\tag{5.13}$$

where *l* is the length of sentence *t* − 1. The ccDCLM model then creates additional paths for this information to impact each hidden representation in the current sentence *t*. Writing **w***t*,*n* for the word representation of the *n*th word in the *t*th sentence, we have

$$\mathbf{h}\_{t,n} = \mathbf{g}\_{\theta}(\mathbf{h}\_{t,n-1}, f(\mathbf{w}\_{t,n}, \mathbf{c}\_{t-1}), \tag{5.14}$$

where *g*θ (·) is the activation function parameterized by θ and *f* (·) is a function that combines the context vector with the input **x***t*,*n* for the hidden state. Here we simply concatenate the representations,

$$f(\mathbf{x}\_{t,n}, \mathbf{c}\_{t-1}) = [\mathbf{x}\_{t,n}; \mathbf{c}\_{t-1}].\tag{5.15}$$

The emission probability for **y***t*,*n* is then computed from **h***t*,*n* as in the standard RNNLM. The underlying assumption of this model is that contextual information should impact the generation of each word in the current sentence. The model, therefore, introduces computational "short-circuits" for cross-sentence information, as illustrated in Fig. 5.4.

**Fig. 5.4** The architecture of ccDCLM model

**Fig. 5.5** The architecture of coDCLM model

Besides, they also proposed Context-to-Output Document-Context Language Model (coDCLM). Rather than incorporating the document context into the recurrent definition of the hidden state, the coDCLM model pushes it directly to the output, as illustrated in Fig. 5.5. Let **h***t*,*n* be the hidden state from a conventional RNNLM of sentence *t*,

$$\mathbf{h}\_{t,n} = \mathbf{g}\_{\theta}(\mathbf{h}\_{t,n-1}, \mathbf{x}\_{t,n}). \tag{5.16}$$

Then, the context vector **c***t*−1 is directly used in the output layer as

$$\mathbf{y}\_{t,n} \sim \text{Softmax}(\mathbf{W}\_h \mathbf{h}\_{t,n} + \mathbf{W}\_c \mathbf{c}\_{t-1} + \mathbf{b}).\tag{5.17}$$

#### **<IP_ADDRESS> Hierarchical Document Autoencoder**

Reference [33] also proposes hierarchical document autoencoder to represent documents. The model draws on the intuition that just as the juxtaposition of words creates a joint meaning of a sentence, the juxtaposition of sentences also creates a joint meaning of a paragraph or a document.

They first obtain representation vectors at the sentence level by putting one layer of LSTM (denoted as LSTM*wor d encode*) on top of its containing words:

$$h\_t^w(\text{enc}) = \text{LSTM}\_{encode}^{word}(\mathbf{w}\_t, h\_{t-1}^v(\text{enc})).\tag{5.18}$$

The vector output at the ending time step is used to represent the entire sentence as

$$\mathbf{s} = h\_{end\_\imath}^{\boldsymbol{w}}.\tag{5.19}$$

To build representation *eD* for the current document/paragraph, another layer of LSTM (denoted as LSTM*sentence encode* ) is placed on top of all sentences, computing representations sequentially for each time step:

**Fig. 5.6** The architecture of hierarchical document autoencoder

$$h\_t^s(\text{enc}) = \text{LSTM}\_{encode}^{sentence}(\mathbf{s}, h\_{t-1}^s(\text{enc})).\tag{5.20}$$

Representation *hs endD* computed at the final time step is used to represent the entire document: **d** = *hs endD* .

Thus one LSTM operates at the token level, leading to the acquisition of sentencelevel representations that are then used as inputs into the second LSTM that acquires document-level representations, in a hierarchical structure.

As with encoding, the decoding algorithm operates on a hierarchical structure with two layers of LSTMs. LSTM outputs at sentence level for time step *t* are obtained by

$$h\_t^s(\text{dec}) = \text{LSTM}\_{decode}^{sentence}(\mathbf{s}\_t, h\_{t-1}^s(\text{dec})).\tag{5.21}$$

The initial time step *hs* 0(*d*) = *eD*, the end-to-end output from the encoding procedure *hs t*(*d*) is used as the original input into LSTM*wor d decode* for subsequently predicting tokens within sentence *t* + 1. LSTM*wor d decode* predicts tokens at each position sequentially, the embedding of which is then combined with earlier hidden vectors for the next time-step prediction until the *ends* token is predicted. The procedure can be summarized as follows:

$$h\_t^{\text{w}}(\text{dec}) = \text{LSTM}^{\text{xentence}}\_{decode}(\mathbf{w}\_t, h\_{t-1}^{\text{w}}(\text{dec})),\tag{5.22}$$

$$P(\boldsymbol{w}|\cdot) = \text{Softmax}(\mathbf{w}, h\_{t-1}^{\boldsymbol{w}}(\text{dec})).\tag{5.23}$$

**Fig. 5.7** The architecture of hierarchical document autoencoder with attentions

During decoding, LSTM*wor d decode* generates each word token *w* sequentially and combines it with earlier LSTM-outputted hidden vectors. The LSTM hidden vector computed at the final time step is used to represent the current sentence.

This is passed to LSTM*sentence decode* , combined with *hs t* for the acquisition of *ht*+1, and outputted to the next time step in sentence decoding. For each time step *t*, LSTM*sentence decode* has to first decide whether decoding should proceed or come to a full stop: we add an additional token *end D* to the vocabulary. Decoding terminates when token *end D* is predicted. Details are shown in Fig. 5.6.

Attention models adopt a look-back strategy by linking the current decoding stage with input sentences in an attempt to consider which part of the input is most responsible for the current decoding state (Fig. 5.7).

Let *H* = {*hs* 1(*e*), *hs* 2(*e*), . . . , *hs N* (*e*)} be the collection of sentence-level hidden vectors for each sentence from the inputs, outputted from LSTM*sentence encode* . Each element in H contains information about input sequences with a strong focus on the parts surrounding each specific sentence (time step). During decoding, suppose that *es t* denotes the sentence-level embedding at current step and that *hs t*−1(dec) denotes the hidden vector outputted from LSTM*sentence decode* at previous time step *t* − 1. Attention models would first link the current-step decoding information, i.e., *hs t*−1(dec) which is outputted from LSTM*sentence dec* with each of the input sentences *i* ∈ [1, *N*], characterized by a strength indicator *vi* :

$$\nu\_i = \mathbf{U}^\top f(\mathbf{W}\_1 \cdot h\_{t-1}^s(\text{dec}) + \mathbf{W}\_2 \cdot h\_i^s(\text{enc})),\tag{5.24}$$

where **W**1,**W**2 ∈ R*K*×*K* , **U** ∈ R*K*×1. *vi* is then normalized

$$\alpha\_i = \frac{\exp(\upsilon\_i)}{\sum\_j \exp(\upsilon\_j)}.\tag{5.25}$$

The attention vector is then created by averaging weights over all input sentences:

$$\mathbf{m}\_{l} = \sum\_{i=1}^{N\_{D}} \alpha\_{i} h\_{i}^{s} \text{(enc)}\tag{5.26}$$

## **5.5 Applications**

In this section, we will introduce several applications on document level analysis based on representation learning.

## *5.5.1 Neural Information Retrieval*

Information retrieval aims to obtain relevant resources from a large-scale collection of information resources. As shown in Fig. 5.8, given the query "Steve Jobs" as input, the search engine (a typical application of information retrieval) provides relevant web pages for users. Traditional information retrieval data consists of search queries and document collections *D*. And the ground truth is available through explicit human judgments or implicit user behavior data such as click-through rate.

**Fig. 5.8** An example of information retrieval

For the given query *q* and document *d*, traditional information retrieval models estimate their relevance through lexical matches. Neural information retrieval models pay more attention to garner the query and document relevance from semantic matches. Both lexical and semantic matches are essential for neural information retrieval. Thriving from neural network black magic, it helps information retrieval models catch more sophisticated matching features and have achieved the state of the art in the information retrieval task [17].

Current neural ranking models can be categorized into two groups: representationbased and interaction-based [23]. The earlier works mainly focus on representationbased models. They learn good representations and match them in the learned representation space of queries and documents. Interaction-based methods, on the other hand, model the query-document matches from the interactions of their terms.

#### **<IP_ADDRESS> Representation-Based Neural Ranking Models**

The representation-based methods directly match the query and documents by learning two distributed representations, respectively, and then compute the matching score based on the similarity between them. In recent years, several deep neural models have been explored based on such Siamese architecture, which can be done by feedforward layers, convolutional neural networks, or recurrent neural networks.

Reference [26] proposes Deep Structured Semantic Models (DSSM) first to hash words to the letter-trigram-based representation. And then use a multilayer fully connected neural network to encode a query (or a document) as a vector. The relevance between the query and document can be simply calculated with the cosine similarity. Reference [26] trains the model by minimizing the cross-entropy loss on click-through data where each training sample consists of a query *q*, a positive document *d*+, and a uniformly sampled negative document set *D*−:

$$\mathcal{QC}\_{DSSM}(q, d^+, D^-) = -\log\left(\frac{e^{r\cdot\cos(\mathbf{q}, \mathbf{d}^+)}}{\sum\_{d \in D} e^{r\cdot\cos(\mathbf{q}, \mathbf{d})}}\right),\tag{5.27}$$

where *D* = *d*+ ∪ *D*−.

Furthermore, CDSSM [54] and ARC-I [25] utilize convolutional neural network (CNN), while LSTM-RNN [44] adopts recurrent neural network with Long Short-Term Memory (LSTM) units to represent a sentence better. Reference [53] also comes up with a more sophisticated similarity function by leveraging additional layers of the neural network.

#### **<IP_ADDRESS> Interaction-Based Neural Ranking Models**

The interaction-based neural ranking models learn word-level interaction patterns from query-document pairs, as shown in Fig. 5.9. And they provide an opportunity to compare different parts of the query with different parts of the document individually

**Fig. 5.9** The architecture of interaction-based neural ranking models

and aggregate the partial evidence of relevance. ARC-II [25] and MatchPyramind [45] utilize convolutional neural network to capture complicated patterns from word-level interactions. The Deep Relevance Matching Model (DRMM) uses pyramid pooling (histogram) to summarize the word-level similarities into ranking models [23]. There are also some works establishing position-dependent interactions for ranking models [27, 46].

Kernel-based Neural Ranking Model (K-NRM) [66] and its convolutional version Conv-KNRM [17] achieve the state of the art in neural information retrieval. K-NRM first establishes a translation matrix **M** in which each element **M***i j* is the cosine similarity of *i*th word in *q* and *j*th word in *d*. Then K-NRM utilizes kernels to convert translation matrix **M** to ranking features φ(**M**) :

$$\phi(\mathbf{M}) = \sum\_{i=1}^{n} \log \mathbf{K}(\mathbf{M}\_i),\tag{5.28}$$

$$\mathbf{K}(\mathbf{M}\_i) = \{K\_1(\mathbf{M}\_i), \dots, K\_K(\mathbf{M}\_i)\}. \tag{5.29}$$

Each RBF kernel *Kk* calculates how word pair similarities are distributed:

$$K\_k(\mathbf{M}\_i) = \sum\_j \exp\left(-\frac{(\mathbf{M}\_{ij} - \mu\_k)^2}{2\sigma\_K^2}\right). \tag{5.30}$$

Then the relevance of *q* and *d* is calculated by a ranking layer:

$$f(q,d) = \tanh(\mathbf{w}^\top \phi(\mathbf{M}) + b),\tag{5.31}$$

where **w** and *b* are trainable parameters.

Reference [66] trains the model by minimizing pair-wise loss on click-through data:

$$\mathcal{QC} = \sum\_{q} \sum\_{d^+, d^- \in D^{+,-}} \max(0, 1 - f(q, d^+) + f(q, d^-)). \tag{5.32}$$

For the given query *q*, *D*+,− are the pair-wise preferences from the ground truth. *d*+ and *d*− are two documents such that *d*+ is more relevant with *q* than *d*−. Conv-KNRM extends K-NRM to model *n*-gram semantic matches based on the convolutional neural network which can leverage snippet information.

#### **<IP_ADDRESS> Summary**

Representation-based models and interaction-based models extract match features from overall and local aspects, respectively. They can also be combined for further improvements [40].

Recently, large-scale knowledge graphs such as DBpedia, Yago, and Freebase have emerged. Knowledge graphs contain human knowledge about real-world entities and become an opportunity for search systems to understand queries and documents better. The emergence of large-scale knowledge graphs has motivated the development of entity-oriented search, which brings in entities and semantics from the knowledge graphs and has dramatically improved the effectiveness of featurebased search systems.

Entity-oriented search and neural ranking models push the boundary of matching from two different perspectives. Reference [36] incorporates semantics from knowledge graphs into the neural ranking, such as entity descriptions and entity types. This work significantly improves the effectiveness and generalization ability of interaction-based neural ranking models. However, how to fully leverage semistructured knowledge graphs and establish semantic relevance between queries and documents remains an open question.

Information retrieval has been widely used in many natural language processing tasks such as reading comprehension and question answering. Therefore, it is no doubt that neural information retrieval will lead to a new tendency for these tasks.

## *5.5.2 Question Answering*

Question Answering (QA) is one of the most important tasks and so are documentlevel applications in NLP. Many efforts have been invested in QA, especially in machine reading comprehension and open-domain QA. In this section, we will introduce the advances in these two tasks, respectively.

#### **<IP_ADDRESS> Machine Reading Comprehension**

As shown in Fig. 5.10, machine reading comprehension aims to determine the answer *a* to the question *q* given a passage *p*. The task could be viewed as a supervised learning problem: given a collection of training examples {(*pi*, *qi*, *ai*)} *n i*=1, we want to learn a mapping *f* (·) that takes the passage *pi* and corresponding question *qi* as inputs and outputs *a*ˆ*i* , where *evaluate*(*a*ˆ*i*, *ai*) is maximized. The evaluation metric is typically correlated with the answer type, which will be discussed in the following.

Generally, the current machine reading comprehension task could be divided into four categories depending on the answer types according to [10], i.e., cloze style, multiple choices, span prediction, and free-form answer.

The cloze style task such as CNN/Daily Mail [24] consists of fill-in-the-blank sentences where the question contains a placeholder to be filled in. The answer *a* is either chosen from a predefined candidate set |*A*| or from the vocabulary |*V*|. The multiple-choice task such as RACE [30] and MCTest [50] aims to select the best answer from a set of answer choices. It is typical to use accuracy to measure the performance on these two tasks: the percentage of correctly answered questions in the whole example set, since the question could be either correctly answered or not from the given hypothesized answer set.

**Fig. 5.10** An example of machine reading comprehension from SQuAD [49]

The span prediction task such as SQuAD [49] is perhaps the most widely adopted task among all, since it takes compromises between flexibility and simplicity. The task is to extract a most likely text span from the passage as the answer to the question, which is usually modeled as predicting the start position *idxstart* and end position *idxend* of the answer span. To evaluate the predicted answer span *a*ˆ, we typically use two evaluation metrics proposed by [49]. Exact match assigns full score 1.0 to the predicted answer span *a*ˆ if it exactly equals the ground truth answer *a*, otherwise 0.0. F1 score measures the degree of overlap between *a*ˆ and *a* by computing a harmonic mean of the precision and recall.

The free-form answer task such as MS MARCO [43] does not restrict the answer form or length and is also referred to as *generative question answering*. It is practical to model the task as a sequence generation problem, where the discrete token-level prediction was made. Currently, a consensus on what is the ideal evaluation metrics has not been achieved. It is common to adopt standard metrics in machine translation and summarization, including ROUGE [34] and BLEU [57].

As a critical component in the question answering system, the surging neuralbased machine reading comprehension models have greatly boosted the task of question answering in the last decades.

The first attempt [24] to apply neural networks on machine reading comprehension constructs bidirectional LSTM reader models along with attention mechanisms. The work introduces two reader models, i.e., the attentive reader and the impatient reader, as shown in Fig. 5.11. After encoding the passage and the query into hidden states using LSTMs, the attentive reader computes a scalar distribution *s*(*t*) over the passage tokens and uses it to compute the weighted sum of the passage hidden states *r*. The impatient reader extends this idea further by recurrently updating the weighted sum of passage hidden states after it has seen each query token.

The attention mechanisms used in reading comprehension could be viewed as a variant of Memory Networks [64]. Memory Networks use long-term memory units to store information for inference dynamically. Typically, given an input *x*,

**Fig. 5.11** The architecture of bidirectional LSTM reader model

the model first converts it into an internal feature representation *F*(*x*). Then, the model can update the designated memory units *mi* given the new input: *mi* = *g*(*mi*, *F*(*x*), *m*), or generate output features *o* given the new input and the memory states: *o* = *f* (*F*(*x*), *m*). Finally, the model converts the output into the response with the desired format: *r* = *R*(*o*). The key takeaway of Memory Networks is the retaining and updating of some internal memories that captivate global information. We will see how this idea is further extended in some sophisticated models.

It is no doubt that the application of attention to machine reading comprehension greatly promotes researches in this field. Following [11], the work [24] modifies the method to compute attention and simplify the prediction layer in the attentive reader. Instead of using *tanh*(·) to compute the relevance between the passage representations {**p**˜*i*} *n i*=1 and the query hidden state **q** (see Eq. 5.33), Chen et al. use the bilinear terms to directly capture the passage-query alignment (see Eq. 5.34).

$$\alpha\_i = \text{Softmax}\_i(\tanh(\mathbf{W}\_1\tilde{\mathbf{p}}\_i + \mathbf{W}\_2\mathbf{q})),\tag{5.33}$$

$$\alpha\_{i} = \text{Softmax}\_{i}(\mathbf{q}^{\top}\mathbf{W}\_{3}\tilde{\mathbf{p}}\_{i}).\tag{5.34}$$

Most machine reading comprehension models follow the same paradigm to locate the start and endpoint of the answer span. As shown in Fig. 5.12, while encoding the passage, the model retains the length of the sequence and encodes the question into a fixed-length hidden representation **q**. The question's hidden vector is then used as a pointer to scan over the passage representation {**p***i*} *n i*=1 and compute scores on every position in the passage. While maintaining this similar architecture, most machine reading comprehension models vary in the interaction methods between the passage and the question. In the following, we will introduce several classic reading comprehension architectures that follow this paradigm.

First, we introduce BiDAF, which is short for *Bi-Directional Attention Flow* [52]. The BiDAF network consists of the token embedding layer, the contextual embedding layer, the bi-directional attention flow layer, the LSTM modeling layer, and the softmax output layer, as shown in Fig. 5.13.

The token embedding layer consists of two levels. First, the character embedding layer encodes each word in character level by adopting a 1D convolutional neural network (CNN). Specifically, for each word, characters are embedded into fixedlength vectors, which are considered as 1D input for CNNs. The outputs are then max-pooled along the embedding dimension to obtain a single fixed-length vector. Second, the word embedding layer uses pretrained word vectors, i.e., GloVe [47], to map each word into a high-dimensional vector directly.

Then the concatenation of the two vectors is fed into a two-layer Highway Network [56]. Equation 5.35 shows one layer of the highway network used in the paper, where *H*1(·) and *H*2(·) represent two affine transformations:

$$\mathbf{g} = \text{Sigmoid}(H\_1(\mathbf{x})),\tag{5.35}$$

$$\mathbf{y} = \mathbf{g} \odot \text{ReLU}(H\_2(\mathbf{x})) + (1 - \mathbf{g}) \odot \mathbf{x}.\tag{5.36}$$

**Fig. 5.13** The architecture of BiDAF model

#### 5.5 Applications 115

After feeding the context and the query to the token embedding layer, we obtain **X** ∈ R*d*×*T* for the context and **Q** ∈ R*d*×*J* for the query, respectively. Afterward, the contextual embedding layer, which is a bidirectional LSTM, model the temporal interaction between words for both the context and the query.

Then, come to the attention flow layer. In this layer, the attention dependency is computed in both directions, i.e., the context-to-query (C2Q) attention and the query-to-context (Q2C) attention. For both kinds of attention, we first compute a similarity matrix **S** ∈ R*T*×*J* using the contextual embeddings of the context **H** and the query **U** obtained from the last layer (Eq. 5.37). In the equation, α(·) computes the scalar similarity of the given two vectors and **m** is a trainable weight vector.

$$\mathbf{S}\_{tj} = \alpha(\mathbf{H}\_{:,t}, \mathbf{U}\_{:,j}) \tag{5.37}$$

$$\alpha(\mathbf{h}, \mathbf{u}) = \mathbf{m}^\top [\mathbf{h}; \mathbf{u}; \mathbf{h} \odot \mathbf{u}],\tag{5.38}$$

where indicates element-wise product.

For the C2Q attention, a weighted sum of contextual query embeddings is computed given each context word. The attention distribution over the query is obtained by **a***j* = Softmax(**S***j*,:) ∈ R*J* . The final attended query vector is therefore **U**˜ :,*t* = *j at j***U**:,*j* for each context word.

For the Q2C attention, the context embeddings are merged into a single fixed length hidden vector **h**˜. The attention distribution over the context is computed by **b***t* = Softmax(max*j* **S***t j*), and **h**˜ = *t* **b***t***H**:,*t* . Lastly, the merged context embeddings are tiled *T* times along the column to produce **H**˜ .

Finally, the attended outputs are combined to yield **G**, which is defined by Eq. 5.39

$$\mathbf{G}\_{:,t} = \phi(\mathbf{H}\_{:,t}, \tilde{\mathbf{U}}\_{:,t}, \tilde{\mathbf{H}}\_{:,t}) \tag{5.39}$$

$$\beta(\mathbf{h}, \tilde{\mathbf{u}}, \mathbf{h}) = [\mathbf{h}; \tilde{\mathbf{u}}; \mathbf{h} \odot \tilde{\mathbf{u}}; \mathbf{h} \odot \mathbf{h}].\tag{5.40}$$

Afterward, the LSTM modeling layer takes **G** as input and encodes it using a two-layer bidirectional LSTM. The output **M** ∈ R2*d*×*T* is combined with **G** to yield the final start and end probability distributions over the passage.

$$P^1 = \text{Softmax}(\mathbf{u}\_1^\top \mathbf{[G; M]}),\tag{5.41}$$

$$P^2 = \text{Softmax}(\mathbf{u}\_2^\top \mathbf{[G;LSTM(M)]}),\tag{5.42}$$

where **u**1, **u**2 are two trainable weight vectors.

To train the model, the negative log likelihood loss is adopted and the goal is to maximize the probability of the golden start index *idxstart* and end index *idxend* being selected by the model,

116 5 Document Representation

$$\mathcal{QC} = -\frac{1}{N} \sum\_{i=1}^{N} \left( \log(P\_{idx\_{start}^{i}}^{1}) + \log(P\_{idx\_{start}^{i}}^{2}) \right). \tag{5.43}$$

Besides BiDAF, where attention dependencies are computed in two directions, we will also briefly introduce other interaction methods between the query and the passage. The Gated-Attention Reader proposed by [19] adopts the gated attention module, where each token representation of the passage *di* is scaled by the attended query vector **Q** after each Bi-GRU layer (Eq. 5.44).

$$
\alpha\_i = \text{Softmax}(\mathbf{Q}^\top \mathbf{d}\_i) \tag{5.44}
$$

$$
\tilde{q}\_i = \mathbf{Q}\alpha\_i \tag{5.45}
$$

$$\mathbf{x}\_i = \mathbf{d}\_i \odot \tilde{\mathbf{q}}\_i. \tag{5.46}$$

This gated attention mechanism allows the query to directly interact with the token embeddings of the passage at the semantic level. And such layer-wise interaction enables the model to learn conditional token representation given the question at different representation levels.

The Attention-over-Attention Reader [16] takes another path to model the interaction. The attention-over-attention mechanism involves calculating the attention between the passage attention α(*t*) and the averaged question attention β after obtaining the similarity matrix **M** ∈ R*n*×*m* (Eq. 5.47). This operation is considered to learn the contributions of individual question words explicitly.

$$\alpha(t) = \text{Softmax}(\mathbf{M}\_{:,t}),$$

$$\beta = \frac{1}{N} \sum\_{t=1}^{N} \text{Softmax}(\mathbf{M}\_{t,:}). \tag{5.47}$$

#### **<IP_ADDRESS> Open-Domain Question Answering**

Open-domain QA (OpenQA) has been first proposed by [21]. The task aims to answer open-domain questions using external resources such as collections of documents [58], web pages [14, 29], structured knowledge graphs [3, 7] or automatically extracted relational triples [20].

Recently, with the development of machine reading comprehension techniques [11, 16, 19, 55, 63], researchers attempt to answer open-domain questions via performing reading comprehension on plain texts. Reference [12] proposes to employ neural-based models to answer open-domain questions. As illustrated in Fig. 5.14, neural-based OpenQA system usually retrieves relevant texts of the question from a large-scale corpus and then extracts answers from these texts using reading comprehension models.

**Fig. 5.14** An example of open-domain question answering

The DrQA system consists of two components: (1) The document retriever module for finding relevant articles and (2) the document reader model for extracting answers from given contexts.

The document retriever is used as a first quick skim to narrow the searching space and focus on documents that are likely to be relevant. The retriever builds TF-IDF weighted bag-of-words vectors for the documents and the questions, and computes similarity scores for ranking. To further utilize local word order information, the retriever uses bigram counts with hash while preserving both the speed and memory efficiency.

The document reader model takes in the top 5 Wikipedia articles yielded by the document retriever and extracts the final answer to the question. For each article, the document reader predicts an answer span with a confidence score. The final prediction is made by maximizing the unnormalized exponential of prediction scores across the documents.

Given each document *d*, the document reader first builds feature representation **d**˜*i* for each word in the document. The feature representation **d**˜ is made up by the following components.

118 5 Document Representation

$$f\_{align}(d\_i) = \sum\_j \alpha\_{ij} \mathbf{E}(q\_j) \tag{5.48}$$

$$\alpha\_{ij} = \frac{\exp(\text{MLP}(\mathbf{E}(d\_i))^\top \text{MLP}(\mathbf{E}(q\_j)))}{\sum\_{j'} \exp(\text{MLP}(\mathbf{E}(d\_i))^\top \text{MLP}(\mathbf{E}(q\_{j'})))} \tag{5.49}$$

where MLP(**x**) = max(0,**Wx** + **b**) and *E*(*qj*) indicates the word embedding of the *j*th word in the question.

Finally, the feature representation is obtained by concatenating the above features:

$$\mathbf{d}\_{i} = (f\_{emb}(d\_{i}), f\_{token}(d\_{i}), f\_{exact\\_match}(d\_{i}), f\_{align}(d\_{i})).\tag{5.50}$$

Then the feature representation of the document is fed into a multilayer bidirectional LSTM (BiLSTM) to encode the contextual representation.

$$\mathbf{d}\_{\rm l}, \ldots, \mathbf{d}\_{n} = \text{BiLSTM}(\mathbf{\bar{d}}\_{\rm l}, \ldots, \mathbf{\bar{d}}\_{n}).\tag{5.51}$$

For the question, the contextual representation is simply obtained by encoding the word embeddings using a multilayer BiLSTM.

$$\mathbf{q}\_1, \dots, \mathbf{q}\_m = \text{BiLSTM}(\tilde{\mathbf{q}}\_1, \dots, \tilde{\mathbf{q}}\_m) \tag{5.52}$$

After that, the contextual representation is aggregated into a fixed-length vector using self-attention.

$$b\_j = \frac{\exp(\mathbf{u}^\top \mathbf{q}\_j)}{\sum\_{j'} \exp(\mathbf{u}^\top \mathbf{q}\_{j'})} \tag{5.53}$$

$$\mathbf{q} = \sum\_{j} b\_{j} \mathbf{q}\_{j}.\tag{5.54}$$

In the answer prediction phase, the start and end probability distributions are calculated following the paradigm mentioned in the Reading Comprehension Model section (Sect. <IP_ADDRESS>).

$$P^{start}(i) = \frac{\exp(\mathbf{d}\_i^\top \mathbf{W}^{start} \mathbf{q})}{\sum\_{i'} \exp(\mathbf{d}\_{i'}^\top \mathbf{W}^{start} \mathbf{q})} \tag{5.55}$$

$$P^{end}(i) = \frac{\exp(\mathbf{d}\_i^\top \mathbf{W}^{end} \mathbf{q})}{\sum\_{i'} \exp(\mathbf{d}\_{i'}^\top \mathbf{W}^{end} \mathbf{q})}. \tag{5.56}$$

Despite its success, the DrQA system is prone to noise in retrieved texts which may hurt the performance of the system. Hence, [15] and [61] attempt to solve the noise problem in DrQA via separating the question answering into paragraph selection and answer extraction, and they both only select the most relevant paragraph among all retrieved paragraphs to extract answers. They lose a large amount of rich information contained in those neglected paragraphs. Hence, [62] proposes strength-based and coverage-based re-ranking approaches, which can aggregate the results extracted from each paragraph by the existing DS-QA system to determine the answer better. However, the method relies on the pre-extracted answers of existing DS-QA models and still suffers the noise issue in distant supervision data because it considers all retrieved paragraphs indiscriminately. To address this issue, [35] proposes a coarseto-fine denoising OpenQA model, which employs a paragraph selector to filter out paragraphs and a paragraph reader to extract the correct answer from those denoised paragraphs.

## **5.6 Summary**

In this chapter, we have introduced document representation learning, which encodes the semantic information of the whole document into a real-valued representation vector, providing an effective way of downstream tasks utilizing the document information and has significantly improved the performances of these tasks.