instead of directly feeding the sequence of word vectors $\mathbf{x} = (x_1,...,x_n)$ to the LSTM units, we first construct the feature-rich embedding $\mathbf{x}^{'}$ for each token, which is the concatenation of the word embedding, an NER feature embedding, and a character-level representation of the word [@lample2016neural]. We use the concatenated vector as the “final” embedding $\mathbf{x}^{'}$ for the token, $$x_{i}^{'} = \textnormal{concat}(x_i, \textnormal{CharRep}_i, \textnormal{NER}_i)$$ where $\textnormal{CharRep}_i$ is the concatenation of the last hidden states of a character-based biLSTM. The intuition behind the use of NER features is that SQuAD answer spans contain a large number of named entities, numeric phrases, etc.

Then a multi-layer Bi-directional LSTM is applied to $(x_1^{'},..., x_n^{'})$ and we obtain the output state $z_{t}$ for time step $t$ by concatenation of the hidden states (forward and backward) at time step $t$ from the last layer of the BiLSTM. We apply the $\textnormal{softmax}$ to $(z_1, ..., z_n)$ to get the normalized score representation for each token, which is of size $n \times k$, where $k$ is the number of tags.

Instead of using a softmax training objective that minimizes the cross-entropy loss for each individual word, the model is trained with a CRF [@lafferty2001conditional] objective, which minimizes 