cost spent on synchronizing the parameters and their gradients across machines, which is proportional to the size of parameters that need to be updated. For the sampled softmax, due to the use of the sparse gradient, the communication cost is proportional to the number of the sampled words. For the adaptive softmax, since full softmax is still used within the short-list and each cluster, the sparse gradient trick is not available and the communication cost is proportional to the trainable parameter size. The [<span style="font-variant:small-caps;">Semfit</span>]{} layer, on the other hand, incurs little communication cost, making it more efficient to train models on multiple GPUs.

#### Easily-accessible Word Embedding

With all these computational advantages, the only prerequisite for the [<span style="font-variant:small-caps;">Semfit</span>]{} layer is a pre-trained word embedding. Fortunately, learning word embeddings is much cheaper than learning contextual representations. For example, training a FastText embedding [@fasttext] on the One Billion Word Benchmark took merely two hours on an average CPU machine. Moreover, several existing word embeddings trained on large corpora in different domains are publicly available.

Open-vocabulary Word Embedding
------------------------------

Theoretically, we can use any pre-trained word representation as the target word embedding for the [<span style="font-variant:small-caps;">Semfit</span>]{} layer. We exploit a particular 