contextual representation models, where the goal is to build a generic contextual encoder.

We first review the procedure for learning a contextual encoder. Given a sequence of words $\{w_1, w_2, \dots, w_n\}$, the encoder seeks to form a rich contextual representation for every word based on their surrounding words, i.e. the context. An input layer, which is usually a word embedding or a character-CNN [@kim2016cnn], produces a context-insensitive word representation $\bm{x_i^0}$ for word $w_i$. Then, $\bm{x_i^0}$ goes through a K-layer contextualizing structure, such as LSTM [@hochreiter1997lstm], Gated CNN [@GatedCNN], or Transformer [@Transformer]. Each layer outputs a context-dependent vector $\bm{x_i^k} (k = 1, \dots ,K)$[^3]. The top layer’s output $\bm{x_i^K}$ is the final output of the encoder. Notice that when using this contextual encoder in downstream tasks, recent methods have gone beyond simply using the top layer output. They either combine different layers’ outputs $\bm{x_i^k}$ to obtain hierarchical representation (ELMo) or fine-tune the whole encoder (GPT). To avoid confusion, we refer to the contextual information captured by the encoder as the **contextual representation**. We denote the top layer’s output vectors $\bm{x_i^K}$ as $\bm{c_i}$ and refer to it as the **context vector**.

Training the contextual encoder involves embedding a context $c$ into 