far though, the linguistic structures used to probe RNN LMs have often been those with unambiguously ungrammatical counterparts. This extends into the domain of semantics, where downstream evaluation platforms like GLUE and SuperGLUE evaluate LMs for correct vs. incorrect interpretations on tasks targeting language understanding [@wangetal2018; @wangetal2019].

Some recent work has relaxed this binary distinction of correct vs. incorrect or grammatical vs. ungrammatical. @lauetal17 correlate acceptability scores generated from a LM to average human acceptability ratings, suggesting that human-like gradient syntactic knowledge can be captured by such models. @futrelletal2019 also look at gradient acceptability in both RNN LMs and humans, by focusing on alternations of syntactic constituency order (e.g., heavy NP shift, dative alternation). Their results suggest that RNN LMs acquire soft constraints on word ordering, like humans. However, the alternations in @futrelletal2019, while varying in their degree of acceptability, maintain the same syntactic relations throughout the alternation (e.g., *gave a book to Tom* and *gave Tom a book* both preserve the fact that *Tom* is the indirect object). Our work expands this line of research by probing how RNN LMs behave when multiple valid interpretations, with crucially different syntactic relations, are available within a single sentence. We find that 