architecture used in this article is depicted in Figure \[fig:network-architecture\]. Every layer was followed by Dropout [@Srivastava:2014] at rate $0.2$.

;

(x0) – (projection0); (x1) – (projection1); (x2) – (projection2);

(projection0) – (hidden00); (projection1) – (hidden01); (projection2) – (hidden02);

(hidden00) – node\[midway,above\] [$C_0$]{} (hidden01); (hidden01) – node\[midway,above\] [$C_1$]{} (hidden02); (hidden02) – node\[midway,above\] [$C_2$]{} (hidden03);

(h0) -| node\[near start,above\] [$h_0$]{} (hidden01.145); (h1) -| node\[near start,above\] [$h_1$]{} (hidden02.145); (h2) – node\[midway,above\] [$h_2$]{} (h3);

(hidden00) – (hidden10); (hidden01) – (hidden11); (hidden02) – (hidden12);

(hidden10) – (hidden20); (hidden11) – (hidden21); (hidden12) – (hidden22);

(hidden20) – (hidden30); (hidden21) – (hidden31); (hidden22) – (hidden32);

(hidden30) – (hidden40); (hidden31) – (hidden41); (hidden32) – (hidden42);

(hidden40) – (output0); (hidden41) – (output1); (hidden42) – (output2);

(output0) – (y0); (output1) – (y1); (output2) – (y2);

The input of the network at time step $t$ is $w_t$, an index that identifies the vocabulary element. The output contains the predicted probabilities for every vocabulary element, but only the output corresponding to the target word is used. The vocabulary can consist of words, word classes, subwords, or subword classes. The choice of vocabulary does not make any difference with regard to the neural network training, except that large vocabularies require more memory and 