‘$\langle$UNK$\rangle$’.

   Language    $\mu$   $\sigma$        
  ----------- ------- ---------- -- -- --
   Synthetic   4.62      0.03          
    English    51.83     0.96          
    Spanish    40.80     0.89          

  : Mean and standard deviation of LM validation perplexity for the synthetic models used in Section \[synthetic\], the English models used in Section \[fernandezExps\]-\[rc\], and the Spanish models used in Section \[es\_replication\][]{data-label="tab:perps"}

We used the best English model in @gulordavaetal18 and trained 4 additional models with the same architecture[^8] but different random initializations. There was no established Spanish model architecture, so we took the best Romance model architecture[^9] reported in @gulordavaetal18 and trained 5 models. All models used in this work were trained for 40 epochs with resultant mean validation perplexities and standard deviations in Table \[tab:perps\].

Measures
--------

We evaluated the RNN LMs using information-theoretic surprisal [@shannon48; @hale2001probabilistic; @levy2008expectation]. Surprisal is defined as the inverse log probability assigned to each word ($w_i$) in a sentence given the preceding context. $$\text{surprisal}(w_i) = -\text{log}\,p(w_i|w_1 ... w_{i-1})$$

The probability is calculated by applying the softmax function to an RNN’s output layer. Surprisal has been correlated with human processing difficulty [@smith2013effect; @franketal15] allowing us to compare model behavior to human behavior. Each of the experiments done in this work looked at sentences that differed 