the SVD of $\bm{P}$, the resulting matrix is not an approximation for $\bm{P}$. Note that in some applications, such as machine translation, the conditional distribution $p(w|c)$ is often sharp. In this case, the first term in Eq. dominates over the second term and the approximation is valid.

Experiment {#Experiment}
==========

The proposed approach is generic and can be applied to accelerating the training process of word-level contextual representation models. In this section, we take ELMo as an example and demonstrate that our approach significantly speeds up ELMo, while largely maintaining its performance.

Setup
-----

ELMo consists of a forward and a backward language model, trained on the One Billion Word Benchmark for 10 epochs. For a fair comparison, all models we introduce are trained on the same corpus for 10 epochs. All experiments are conducted on a workstation with 8 GeForce GTX 1080Ti GPUs, 40 Intel Xeon E5 CPUs, and 128G main memory. The training code is written in PyTorch such that we could evaluate on most downstream tasks with AllenNLP [@AllenNLP].

#### Models

  Model                                                                                                                                   Input                                                                                 Contextualizing   Output
  --------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- ----------------- ------------------------------------------------------------------------------------------------------------------------------------------------
  <span style="font-variant:small-caps;">[[<span style="font-variant:small-caps;">ELMo</span>]{}]{}</span>                                CNN                                                                                   LSTM              Sampled Softmax
  <span style="font-variant:small-caps;">[[<span style="font-variant:small-caps;">ELMo</span>*-S*$_{\small \textsc{}}$]{}]{}</span>       [<span style="font-variant:small-caps;">FastText$_{\small \textsc{cc}}$</span>]{}     LSTM w/ LN        [<span 