trainable parameter size. The adaptive softmax proposes to reduce $m$ for rare words. The trainable parameter size is effectively reduced but still remains sizable. For a model trained on the same corpus [@adaptive_softmax], the LSTM has around 50 million parameters while the adaptive softmax still amounts to 240 million parameters. Our approach, on the other hand, uses a pre-trained word embedding, reducing the trainable parameters of the output layer from hundreds of millions to almost zero.

#### GPU Memory Efficiency

Our approach exhibits exceptional GPU memory efficiency, due to reductions of the computational time complexity (with fewer intermediate results to keep) and trainable parameter size (with fewer parameters to store[^5]). As a result, compared to the adaptive softmax, models with our technique are 2 to 5 times more memory efficient (Section \[Exp:DetailedEfficiency\]). The memory efficiency further adds up to the speed advantage, because loading more words in each batch allows better parallelism when utilizing GPUs.

#### Scalability Across GPUs

To scale up modern deep learning systems, frameworks are designed to train models with synchronous SGD with very large batch size and consequently a large number of GPUs [@KaimingImagenetOneHour; @ImagenetInMinuetes]. However, the overhead of running on multiple GPUs is the communication 