remains despite a HIGH attachment interpretation bias in humans. These results raise broader questions regarding the relationship between comprehension (i.e. typical language model use cases) and production (which generates the training data for language models) and point to a deeper inability of RNN LMs to learn aspects of linguistic structure from raw text alone.

Related Work {#relatedWork}
============

Much recent work has probed RNN LMs for their ability to represent syntactic phenomena. In particular, subject-verb agreement has been explored extensively [e.g., @linzenetal16; @bernardyetal2017; @Enguehard17] with results at human level performance in some cases [@gulordavaetal18]. However, additional studies have found that the models are unable to generalize sequential patterns to longer or shorter sequences that share the same abstract constructions [@trasketal2018; @vanSchijndeletal2019]. This suggests that the learned syntactic representations are very brittle.

Despite this brittleness, RNN LMs have been claimed to exhibit human-like behavior when processing garden path constructions [@vanSchijndeletal2018; @futrelletal2019; @franketal2019], reflexive pronouns and negative polarity items [@futrelletal2018], and center embedding and syntactic islands [@wilcox2019supression; @wilcox2019block]. There are some cases, like coordination islands, where RNN behavior is distinctly non-human [see @wilcox2019block], but in general this literature suggests that RNN LMs encode some type of abstract syntactic representation [e.g., @prasadetal2019]. Thus 