How poor is the stimulus? Evaluating hierarchical generalization in
  neural networks trained on child-directed speech
When acquiring syntax, children consistently choose hierarchical rules over
competing non-hierarchical possibilities. Is this preference due to a learning
bias for hierarchical structure, or due to more general biases that interact
with hierarchical cues in children's linguistic input? We explore these
possibilities by training LSTMs and Transformers - two types of neural networks
without a hierarchical bias - on data similar in quantity and content to
children's linguistic input: text from the CHILDES corpus. We then evaluate
what these models have learned about English yes/no questions, a phenomenon for
which hierarchical structure is crucial. We find that, though they perform well
at capturing the surface statistics of child-directed speech (as measured by
perplexity), both model types generalize in a way more consistent with an
incorrect linear rule than the correct hierarchical rule. These results suggest
that human-like generalization from text alone requires stronger biases than
the general sequence-processing biases of standard neural network
architectures.
Aditya Yedetore, Tal Linzen, Robert Frank, R. Thomas McCoy
How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech

###### Abstract

When acquiring syntax, children consistently choose hierarchical rules over competing non-hierarchical possibilities. Is this preference due to a learning bias for hierarchical structure, or due to more general biases that interact with hierarchical cues in children's linguistic input? We explore these possibilities by training LSTMs and Transformers--two types of neural networks without a hierarchical bias--on data similar in quantity and content to children's linguistic input: text from the CHILDES corpus. We then evaluate what these models have learned about English yes/no questions, a phenomenon for which hierarchical structure is crucial. We find that, though they perform well at capturing the surface statistics of child-directed speech (as measured by perplexity), both model types generalize in a way more consistent with an incorrect linear rule than the correct hierarchical rule. These results suggest that human-like generalization from text alone requires stronger biases than the general sequence-processing biases of standard neural network architectures.

## 1 Introduction

Syntax is driven by hierarchical structure, yet we typically encounter sentences as linear sequences of words. How do children come to recognize the hierarchical nature of the languages they acquire? Some argue that humans must have a hierarchical inductive bias--an innate predisposition for hierarchical structure (Chomsky, 1965, 1980). An alternative view (e.g., Lewis and Elman, 2001) is that no such bias is necessary: there may be clear evidence for hierarchical structure in children's input, so that children would choose hierarchical rules even without a hierarchical bias.

At first blush, recent work in natural language processing (NLP) may seem to indicate that no hierarchical bias is necessary. Neural networks trained on naturally-occurring text perform impressively on syntactic evaluations even though they have no explicit syntactic structure built into them (e.g., Gulordava et al., 2018; Wilcox et al., 2018; Warstadt et al., 2020). However, these results do not provide strong evidence about the learning biases required to learn language from the data available to humans because these models receive very different training data than humans do (Warstadt and Bowman, 2022). First, NLP models are typically trained on far more data than children receive, so models have more opportunities to encounter rare syntactic structures (Linzen, 2020). Second, most training sets in NLP are built from Internet text (e.g., Wikipedia), which differs qualitatively from the utterances that children typically hear; e.g., sentences in Wikipedia are on average 25 words long (Yasseri et al., 2012), compared to 5 words for sentences in the North American English subset of the CHILDES corpus of child-directed speech (MacWhinney, 2000).

In this work, to evaluate if neural networks without a hierarchical bias generalize like children do, we train models on text1 comparable to the sentences in children's linguistic input: English data from CHILDES. We then analyze what they have learned about the relationship between declarative sentences, such as (1a), and their corresponding yes/no questions, such as (1b):

Footnote 1: Section 6.5 discusses other input types (e.g., visual input).

1. a. Those are your checkers. b. Are those your checkers?

Crucially, nearly all naturally-occurring yes/no questions are consistent with two rules: one based
on hierarchical structure (2), and one based on linear order (3):2.3 Footnote 2: In past work these rules have been framed as transformations named Move-First and Move-Main (McCoy et al., 2020). We instead follow Berwick et al. (2011) and frame the child’s knowledge as a relationship between sentences.
2. HierarchicalQ: The auxiliary at the start of a yes/no question corresponds to the **main** auxiliary of the corresponding declarative.
3. LinearQ: The auxiliary at the start of a yes/no question corresponds to the **first** auxiliary of the corresponding declarative.

Despite the scarcity of evidence disambiguating these rules, children reliably favor HierarchicalQ (Crain and Nakayama, 1987), albeit with occasional errors consistent with LinearQ (Ambridge et al., 2008). Yes/no questions thus are a prime candidate for an aspect of English syntax for which human-like generalization requires a hierarchical bias. We evaluate yes/no question performance in LSTMs and Transformers, two neural-network architectures that have no inherent hierarchical inductive bias (McCoy et al., 2020; Petty and Frank, 2021). These architectures employ different computational mechanisms, so consistent results across both would indicate that our results are not due to idiosyncrasies of one particular architecture.

To investigate if models generalize more consistently with the hierarchical or linear rule, we evaluate them on cases where the rules make different predictions, such as (4): under HierarchicalQ, the question that corresponds to (4a) is (4b), whereas under LinearQ it is (4c).

1. [label=(4)]
2. a. The boy who **has** talked \(\underline{\mbox{\tt can}}\) read.
3. \(\underline{\mbox{\tt Can}}\) the boy who \(\underline{\mbox{\tt has talked \tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tttt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tttt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tttt\tt\tt\tt\tttt\tt\tt\tt\tt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tt\tttt\tt\tt\tt\tttt\tt\tt\tt\tttt\tt\tt\tt\tt\tttt\tt\tt\tt\tt\tt\tt\tt\tttt\tt\tt\tt\tttt\tt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tttt\tt\tt\tt\tttt\tt\tt\tttt\tt\tt\tttt\tt\tt\tttt\tt\tt\tttt\tt\tt\tt\tttt\tttt\tt\tt\tt\tt\tttt\tttt\tt\tttt\tt\tttt\tt\tt\tt\tttt\tt\tt\tttt\tttt\tt\tttt\tt\tt\tttt\tttt\tt\tt\tttt\tt\tttt\tt\tt\tttt\tttt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tttt\tt\tttt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tt\tt\tttt\tt\tt\tttt\tt\tttt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tt\tt\tt\tt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tttt\tt\tttt\tt\tt\tt\tttt\tt\tttt\tt\tttt\tt\tt\tt\tttt\tt\tt\tttt\tttt\tt\tt\tttt\tttt\tt\tttt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tt\tttt\tt\tt\tttt\tt\tttt\tt\tt\tt\tttt\tt\tttt\tt\tt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tt\tt\tttt\tt\tt\tt\tttt\tttt\tt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tt\tt\tttt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tttt\tt\tt\tt\tttt\tt\tt\tttt\tt\tt\tttt\tttt\tttt\tt\tt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tt\tt\tttt\tt\tt\tttt\tt\tt\tttt\tt\tttt\tt\tt\tttt\tt\tt\tttt\tt\tt\tt\tt\tt\tttt\tt\tt\tt\tttt\tt\tt\tt\tt\tttt\tt\tttt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tt\tttt\tt
models would generalize when faced with the type of data that children receive.

## 3 Overview of Experimental Setup

We evaluated models on yes/no questions in two ways. First, we used relative acceptability judgments (Experiment 1): We trained neural networks on the task of language modeling (predicting the next word at every point in the sentence) and evaluated whether they assigned a higher probability to sentences consistent with LinearQ or HierarchicalQ. Our second approach was based on text generation (Experiment 2): We trained networks to take in a declarative sentence and output the corresponding question, and tested whether they generalized in a way more consistent with LinearQ or HierarchicalQ. Under both framings, we trained models on data from CHILDES and evaluated them on targeted datasets constructed to differentiate LinearQ and HierarchicalQ.

## 4 Experiment 1: Relative Acceptability

### Dataset

To train models on data as similar as possible to the sentences children receive, we extracted data from CHILDES MacWhinney (2000). We used the North American English portion. We wished to replicate children's _input_, so we excluded the children's own utterances, leaving a 9.6-million-word corpus. We allocated 90% of the data to training, 5% to validation, and 5% to testing. We replaced words that appeared two or fewer times in the training set with \(<\)unk\(>\), giving a replacement rate of 0.3%. See Appendix A for more details.

### Task: Next-Word Prediction

We trained models on next-word prediction, also known as language modeling. We chose this task for two reasons. First, it is clear empirically that next-word prediction can teach neural networks a substantial amount about syntax (e.g., Hu et al., 2020). Second, it is plausible that humans perform some version of next-word prediction during sentence processing Altmann and Kamide (1999); Hale (2001); Levy (2008); Kutas et al. (2011) and that such prediction may play a role in acquisition Elman (1991). Thus, while next-word prediction is certainly not the only goal of human language learners, we view this task as a reasonable first step in emulating human language acquisition.

### Architectures

We used two neural network architectures: LSTMs Hochreiter and Schmidhuber (1997) and Transformers Vaswani et al. (2017). We chose these models for two reasons. First, they have been the most successful architectures in NLP. Thus, we have reason to believe that, of the types of low-bias models invented, these two are the ones most likely to discover linguistic regularities in our CHILDES training data. Second, the two architectures process sequences very differently (via recurrence vs. via attention). Thus, if both generalize similarly, we would have evidence that what was learned is strongly evidenced in the data, rather than due to a quirk of one particular architecture.

For our LSTMs, we used 2 layers, a hidden and embedding size of 800, a batch size of 20, a dropout rate of 0.4, and a learning rate of 10. For our Transformers, the corresponding values were 4, 800, 10, 0.2, and 5, and we used 4 attention heads. We chose these values based on a hyperparameter search described in Appendix B. All following results are averaged across 10 runs with different random seeds.

### Results: Language Model Quality

Before testing models on questions, we used perplexity to evaluate how well they captured the basic structure of their training domain. As a baseline, we used a 5-gram model with Kneser-Ney smoothing Kneser and Ney (1995) trained with KenLM Heafield (2011). The test set perplexity for the 5-gram baseline was 24.37, while the average test set perplexity for the LSTMs and Transformers was 20.05 and 19.69, respectively. For perplexity, lower is better. Thus, both neural network types outperformed the strong baseline of a smoothed 5-gram model, showing that they performed well at capturing the basic statistics of their training domain.5

Footnote 5: For an intuitive illustration of our model quality, see the sample text generated by them in Appendix H.

### General Syntactic Evaluation

As an additional way to check the validity of our setup, we evaluated our models on the Zorro dataset Huebner et al. (2021), which is based on BLiMP Warstadt et al. (2020). Zorro contains 24 evaluations, each of which targets one syntactic phenomenon (e.g., subject-verb agreement) and involves sentence pairs for which one sentence is grammatical, and the other is minimally different
but ungrammatical (e.g., by violating subject verb agreement). A model is said to get a sentence pair correct if it assigns a higher probability to the grammatical sentence than the ungrammatical one. Huebner et al. (2021) showed that Transformers trained on CHILDES data can perform well on many of the Zorro categories, so if our setup is sound, our own models should also perform well on Zorro.

See Appendix D for full results. For each syntactic phenomenon, most model re-runs scored above 0.9, though at least one scored near the chance level of 0.5. For each re-run of each architecture there is at least one phenomenon for which the model scores over 0.97, and many models score 1.00 on some phenomena. Thus, all models score well on at least some syntactic evaluations, attaining results comparable to those of Huebner et al. (2021) and providing additional support for the validity of our setup. We now test whether these models have also successfully learned the specific phenomenon that we focus on, yes/no questions--a phenomenon not included in the Zorro dataset.

### Yes/No Questions

Evaluation Dataset: Forced-Choice Acceptability JudgmentsAs a first way to test whether our models have learned HierarchicalQ, we evaluate whether they assign higher probabilities to sentences consistent with HierarchicalQ than to minimally different sentences that are ungrammatical. For this purpose, we create an evaluation dataset containing groups of 6 questions, each created by starting with a declarative sentence, such as (5), and then deleting the **first**, **main**, or neither auxiliary, and inserting the **first** or **main** auxiliary at the front of the sentence.6 For instance, in (6b), the **first** auxiliary has been proposed, and the **main** auxiliary has been deleted.

Footnote 6: It would be possible to also use a ‘prepose other’ category, where an auxiliary not in the input is inserted (McCoy et al., 2018). We excluded this category because using it would raise complications about which ‘other’ auxiliary to choose.

(5) The dog who **has** seen a boy **did** try.

(6) a. **Has** the dog who **s** been a boy **did** try? **Has** the dog who **has** seen a boy try? **Has** the dog who **has** seen a boy **did** try? **Did** the dog who seen a boy **did** try? **Did** the dog who **has** seen a boy **try? **Did** the dog who **has** seen a boy **try? **Did** the dog who **has** seen a boy **try? Within each group, we evaluate which question the model assigned the highest probability to. If a model has correctly learned HierarchicalQ, it should assign the highest probability to the question consistent with this rule, such as (6e).

Several past papers about yes/no questions have used the same general approach (Lewis and Elman, 2001; Reali and Christiansen, 2005). However, these papers considered only pairs of sentences, whereas we consider groups of 6 to allow for a wider range of possible generalizations that a model might have learned.

To generate the declaratives from which we formed groups of 6 questions, we used the context-free grammar (CFG) in Appendix F, which has a vocabulary selected from the most common words in CHILDES. Each declarative generated by the CFG (e.g., (5)) contains two auxiliary verbs: one before the sentence's main verb and one inside a relative clause modifying the subject. One potential problem is that some questions are consistent with both HierarchicalQ and LinearQ. For instance, (7a) can be formed from (7b) with the HierarchicalQ-consistent steps Prepose-Main,Delete-Main, or from (7c) with the LinearQ-consistent steps Prepose-First,Delete-Main.

(7) a. Did the boy who did see the person laugh?  b. The boy who did see the person did laugh.  c. The boy who did see the person can laugh.

To avoid this problem, we required that the auxiliary before the main verb must select for a different verb inflection than the one in the relative clause. For instance in (5), **did** selects for the verb's bare form, while **has** selects for the past participle form. Thus, the auxiliary at the start of the question could only correspond to whichever auxiliary in the declarative has the same selectional properties.7

Footnote 7: A model could succeed on this dataset with a rule that relates the auxiliary at the start of a question with the _last_ auxiliary in the declarative form. Since our models fail on this dataset, this consideration is not relevant here.

**Results: Relative Question Acceptability** For each sentence group, we used per-word perplexity to see which of the 6 candidates the models scored most highly.8 For both LSTMs and Transformers, the correct category (Prepose Main, Delete Main) was the second-rarest choice, and
the most frequent preference was for Prepose First, Delete Main, a category that is only partially correct because it references linear order in addition to hierarchical structure. (Figure 1).

Thus, neither model displays preferences consistent with the correct, fully-hierarchical generalization. The two model types showed similar scores, which may mean that these results are largely driven by the statistics of the training data that both models share, rather than the models' differing inductive biases.

One of the incorrect categories--Prepose Main, Delete None, such as (6f)--only requires reference to hierarchical structure, so it could be said to capture the hierarchical nature of yes/no questions. Nonetheless, this category was also relatively rare: combining the two fully hierarchical possibilities (Prepose Main, Delete Main and Prepose Main, Delete None) accounts for only 26% of LSTM preferences and 27% of Transformer preferences, meaning that both models over 70% of the time favored a sentence generated at least partially based on linear order.

There are two likely reasons for why our models performed so poorly on yes-no questions when they performed well on many of the phenomena in the Zorro dataset (Section 4.5). First, yes/no questions may simply be harder to learn than the other phenomena; indeed, yes/no questions are often singled out as being likely to pose difficulties for a general-purpose learner (Section 1). Alternatively, it might be that the six-way evaluation we used for yes/no questions is stricter than the binary judgments used for the Zorro dataset.

## 5 Experiment 2: Question Formation

The previous experiment was designed to operate entirely in the next-word-prediction paradigm, motivated by arguments from past literature about the strength and relative ecological validity of next-word-prediction as a training objective (see Section 4.2). However, one of this setup's shortcomings is that HierarchicalQ describes correspondences between questions and declaratives, but Experiment 1 focused on questions alone, with no consideration of declaratives.

In this second experiment, to better capture that HierarchicalQ is defined over sentence pairs, we trained models on a sentence-pair task: transforming a declarative into a question (McCoy et al., 2020). For instance, given _the child did learn_ the model must produce _did the child learn?_

We evaluated models in two ways. First, we checked if the models' predictions fully matched the correct questions. This full-sentence evaluation is demanding, and models might fail this evaluation for reasons unrelated to our core hypotheses. For instance, given _the child did learn_ the model might produce _did the baby learn_, which would be marked as incorrect, even though this lexical error is not relevant to HierarchicalQ.

As a metric that is less demanding and that also more directly targets HierarchicalQ, we measured if the first word of the output question corresponded to the first or main auxiliary of the input. Critically, LinearQ and HierarchicalQ make different predictions for the first word of a question so long as the two auxiliaries are distinct: see (4). Because this framing lets the model freely generate its output (instead of choosing one option from a pre-specified set), we allow for the possibility that the rule learned by models may not be identical to any of our manually-generated hypotheses.

Solely training models to perform this transformation involves the implicit assumption that, when children acquire English yes/no questions, the only evidence they leverage is English yes/no questions. However, other types of sentences may also provide useful evidence (Pearl and Mis, 2016): e.g., _wh_-questions also illustrate subject-auxiliary in

Figure 1: The question types that models prefer when offered a choice between 6 questions. These 6 questions are formed by modifying a declarative with a relative clause on the subject according to ‘prepose’ and ‘delete’ rules. The correct category is Prepose Main, Delete Main. Within each architecture, the proportions across all 6 question types necessarily sum to 1. Each bar shows the average across 10 model re-runs, with single-standard-deviation error bars.
version (Pullum and Scholz, 2002), while, more generally, many types of sentences could provide evidence that the syntax as a whole is hierarchical (Perfors et al., 2011). To explore this possibility, we compared a condition in which models were only trained to perform question formation (the Question Formation condition) to another in which models were first pre-trained on next-word prediction with the exact same setup as in Experiment 1 before being further trained to perform question formation (the Next-word Prediction + Question Formation condition).

### Dataset

Training SetOur question formation dataset consisted of the yes/no questions in the CHILDES Treebank (Pearl and Sprouse, 2013, 2013), a parsed subset of CHILDES containing 189,359 sentences. We used these parses to extract all yes/no questions from the CHILDES Treebank and derive their corresponding declarative forms. The resulting declarative was concatenated with the question. An example declarative/question pair is:

(8) you can spell your name. can you spell your name?

The training set consisted of 10,870 declarative/question pairs, the validation set 1,360 pairs, and the test set 1,358 pairs (we will call this test set the _randomly-partitioned test set_ to distinguish it from two other evaluation sets discussed below). We trained models to perform next-word prediction on such concatenated sentence pairs.

The first-word accuracy of the trained model was then computed based on the model's prediction for the word after the period in each test example, while the full-sentence accuracy was computed based on its predictions for all tokens after the period. All questions in the randomly-partitioned test set were withheld from both the question-formation training set and the next-word-prediction training set. Thus, models had not seen these test examples in their training, even in the Next-word Prediction + Question Formation condition in which they were trained on both tasks.

Evaluation SetsIn addition to the randomly-partitioned test set, we used CFGs to generate two targeted evaluation sets. As in Experiment 1, we selected the CFGs' vocabulary from common words in our CHILDES data. In sentences generated from the first CFG, the sentence's first auxiliary was also its main auxiliary, so LinearQ and HierarchicalQ make the same predictions. (8) exemplifies the type of declarative-question pair in this dataset. We call this dataset First-Aux = Main-Aux. For sentences generated by the second CFG, the main auxiliary was the _second_ auxiliary in the sentence; thus, these examples disambiguate LinearQ and HierarchicalQ. Example (9) is a declarative-question pair from this evaluation set.

(9) a boy who is playing cantry. can a boy who is playing try?

We call this dataset First-Aux \(\neq\) Main-Aux. See Appendix F for the CFGs used. We sampled 10,000 declarative sentences from these grammars and transformed them into questions according to HierarchicalQ to create our evaluation sets.

### Results

Randomly-Partitioned Test SetThe LSTMs and Transformers in the Question Formation condition performed well on the randomly-partitioned test set, with a full-question accuracy of 0.68 \(\pm\) 0.014 and 0.87 \(\pm\) 0.005 (averaged across 10 reruns with margins indicating one standard deviation). The models in the Next-word Prediction + Question Formation condition performed similarly well, with a full-question accuracy of 0.66 \(\pm\) 0.008 for the LSTMs and 0.93 \(\pm\) 0.004 for the Transformers. For both model types, the first-word accuracy for the question was nearly 1.00 across re-runs. We suspect that Transformers have a stronger full-question accuracy because producing the question requires copying all words from the declarative (but in a different order). Copying is likely easy for Transformers because they can attend to specific words in the prior context, while our LSTMs must compress the entire context into a fixed-size vector, which may degrade the individual word representations. Because both model types achieved near-perfect performance on the crucial first-word accuracy metric, we conclude that our models have successfully learned how to handle the types of declarative/question pairs that we extracted from the CHILDES Treebank.

Targeted Evaluation SetsOn our two targeted evaluation sets, models almost never produced the complete question correctly. Turning to the more lenient measure of first-word accuracy, for examples on which LinearQ and HierarchicalQ predict the same first output word (First-Aux = Main-Aux), the Transformer trained only on question formation performed strongly, while the Trans
former trained on both tasks, and both LSTMs, performed reasonably well (Figure 2; note models could choose any word in their vocabulary to begin the output, so chance performance is near 0.00). For the crucial cases that disambiguate the two rules (First-Aux\(\neq\)Main-Aux), both models in both conditions performed more consistently with LinearQ than HierarchicalQ. Training on next-word prediction before question formation had inconsistent effects: it modestly increased the likelihood of hierarchical generalization in LSTMs, yet it decreased that likelihood in Transformers.

Lexical SpecificityIn Appendix G, we further break down the First-Aux\(\neq\)Main-Aux results based the auxiliaries' identity. The generalization pattern varied considerably across auxiliary pairs. For some auxiliary pairs, the auxiliary chosen to begin the question was usually neither auxiliary in the input (Figure 3, left facet). For other pairs, models usually chose the first auxiliary, regardless of lexical identity (Figure 3, middle facet). Finally, for some pairs, the auxiliary chosen was usually the same one, regardless of whether it was the first or main auxiliary (Figure 3, right facet).

Generalization based on lexical identity is rarely considered in past discussions of English yes/no question acquisition. Of the papers on this phenomenon (see Clark and Lappin (2010), Lasnik and Lidz (2017), and Pearl (2021) for overviews), the only one to our knowledge that discusses lexical specificity is Frank and Mathis (2007), which studied models trained on synthetic data. Our results highlight the importance of testing for a broad range of generalizations: Lexically-specific hypotheses appear attractive for our low-bias learners, so an account of what biases can yield human-like learning should rule out these lexically-specific hypotheses along with linear ones.

## 6 Discussion

We have found that, when trained on child-directed speech, two types of standard neural networks performed reasonably well at capturing the statistical properties of the dataset, yet their handling of English yes/no questions was more consistent with a linear rule LinearQ than the correct hierarchical rule HierarchicalQ. These results support the hypothesis that a learner requires a hierarchical bias to consistently learn hierarchical rules when learning from the linguistic data children receive.

### Takeaways for LSTMs and Transformers

When trained on massive corpora, LSTMs and Transformers perform impressively on some syntactic evaluations. Based on such results, it is tempting to conclude that the general-purpose biases of these architectures suffice to yield human-like syn

Figure 3: Lexical specificity in model behavior. Each facet considers only the evaluation examples containing the two auxiliaries in the facet heading; e.g., the _can and do_ facet includes, for example, the inputs _the children who **can** play **do** learn_ and _the children who **do** play **can** learn_. The bars show the proportion of model predictions for the first word of the output that are consistent with four potential movement rules, averaged across 10 model re-runs and with error bars showing one standard deviation above and below the mean. This plot only shows an illustrative subset of auxiliary pairs for one model type (Transformers in the Next-Word Prediction + Question Formation condition); see Appendix G for the full results.

Figure 2: Proportion of model-produced questions that were consistent with the linear rule LinearQ and/or the hierarchical rule HierarchicalQ. In the First-Aux = Main-Aux dataset, the first auxiliary is the main auxiliary, so both LinearQ and HierarchicalQ produce the correct question string. The First-Aux \(\neq\)Main-Aux dataset disambiguates the two rules. Each bar shows the average across 10 model re-runs, with error bars showing one standard deviation.
tax acquisition. Our results caution against this interpretation: When we trained the same architectures on data more similar to children's input, they failed to learn the structure of English yes/no questions. Thus, at least when learning from text alone, LSTMs and Transformers do not display human-like language learning--they do not generalize as humans do _from the data that humans receive_.

### Takeaways for the Poverty of the Stimulus Debate

Below we specify four possible positions in the poverty-of-the-stimulus debate about the adequacy of children's input for inducing hierarchical rules in low-bias learners, arranged from assuming the most limited to the most expansive innate component:

1. **Any inductive biases:** Any learner trained on CHILDES will generalize like humans do.
2. **Any inductive biases that enable in-distribution learning:** Any learner that captures the statistical patterns of the training distribution will generalize to HierarchicalQ.
3. **Some non-hierarchical inductive biases:** Some general-purpose learners will generalize as humans do, but others will not.
4. **Only a hierarchical inductive bias:** No general-purpose learners will generalize as humans do: hierarchical biases are necessary.

Position (10) is clearly false: many learners cannot learn certain aspects of syntax, no matter their training data (e.g., bigram models cannot capture long-distance dependencies). Our work shows that position (11) is also false: Though our models performed well on the in-distribution test sets of Experiments 1 and 2, they did not generalize in human-like ways. This leaves positions (12) and (13), which our existing results cannot differentiate. It is possible that only learners with hierarchical inductive biases can demonstrate human-like language learning (position (13)), but also that some learners without this bias can succeed (position (12))--just not the learners we tested. For further discussion of how computational modeling can bear on learnability arguments, see Wilcox et al. (2021).

One potential solution supporting position (12) would be that learners leverage the hierarchical structure of some syntactic phenomenon to help conclude that other, impoverished phenomena are hierarchical Perfors et al. (2011); Mulligan et al. (2021). However, our results from Experiment 2 show that giving learners access to a wider range of phenomena does not automatically improve hierarchical generalization: Models' performance on question formation was not substantially improved (and in some cases was even harmed) when they were trained not just on question formation but also on next-word prediction on the entire CHILDES corpus. Thus, although training on text that contains many linguistic phenomena can give models a hierarchical inductive bias when the training is done over large Internet corpora Warstadt and Bowman (2020); Mueller et al. (2022), our results provide evidence that this conclusion does not extend to models trained on child-directed speech.

Though both (12) and (13) remain as possibilities, we believe that our results more strongly support (13). Of all currently available general-purpose learners, LSTMs and Transformers are the best at modeling the probabilistic structure of linguistic data. Therefore, if child-directed speech contains clear evidence for the hierarchical nature of yes/no questions--evidence so clear that at least some general-purpose learners could recognize it--it is likely that LSTMs and Transformers would be among the set of general-purpose learners that could use this evidence to make hierarchical generalizations in our experiments. The fact that these architectures instead predominantly favored linear generalizations therefore supports position (13).

### How to test for HierarchicalQ

We have argued that an ideal simulation of the acquisition of English yes/no questions would have the following properties:

1. The training data should be similar to children's linguistic input.
2. The training task should be ecologically valid.
3. The evaluation method should focus on correspondences between pairs of sentences rather than the acceptability of individual sentences.

Property (14) motivated our use of text from CHILDES as the training data. We are not aware of a single experimental setup that fully satisfies both Property (15) and Property (16), so we instead used two experiments, each one focusing on one property at the cost of satisfying the other one less well. Experiment 1 works entirely in the context of the relatively ecologically valid task of next-word prediction, motivated by Property (15), but its
evaluation is only based on the acceptability of individual sentences, failing to satisfy Property (16). Experiment 2 fully satisfies Property (16) by using an evaluation based on sentence pairs, at the cost of including a less ecologically-valid training component based on sentence transformations. Both experiments yielded qualitatively similar conclusions (failure of models to learn HierarchicalQ).

### Quantity of Training Data

The size of our training set was plausibly within the range from which children can acquire HierarchicalQ. Crain and Nakayama (1987) found that children between ages 3 and 5 behaved much more consistently with HierarchicalQ than LinearQ. Though these children made many errors, their errors were usually compatible with a hierarchical rule (e.g., Prepose Main, Delete None errors: see Section 4.6). By age 3, American children receive approximately 10 to 33 million words of input Hart and Risley (1995), and the 8.5 million words of our training set is close to the lower end of that range. Thus, it is reasonable to suppose that a learner that generalizes as children do would favor HierarchicalQ after being trained on our training set. Our models, in contrast, regularly preferred sentences generated in ways based on linear order (Figures 1 and 2), a category of error that is very rare in children Crain and Nakayama (1987); Ambridge et al. (2008).

In order to give our models the strongest chance of generalizing correctly, it would have been ideal to provide a quantity of data closer to 33 million words, the high end of Hart and Risley's range. Our data source did not contain enough text to make this possible, but future work could investigate ways to augment the data using other sources.

### Type of Training Data

Our training set was both qualitatively and quantitatively closer to children's input than the massive Internet corpora standardly used to train models in NLP Linzen (2020). This difference is important: Lin et al. (2019), Warstadt and Bowman (2020), and Mueller et al. (2022) all found evidence that models trained on large Internet corpora performed well on yes/no questions evaluations, whereas our models trained on CHILDES performed poorly--though we cannot be certain the differences in results are solely due to differences in the training data, since these prior papers used different model architectures, training tasks, and evaluation setups.

Though our training data are more similar to children's input than massive Internet corpora are, differences remain. Our experiments omit several aspects of a child's experience that might help them acquire syntax, such as prosody Morgan and Demuth (1996), visual information Shi et al. (2019), and meaning Fitz and Chang (2017); Abend et al. (2017), all of which might correlate with syntactic structure and thus provide cues to the correct hierarchical generalization. On the other hand, our dataset might present an easier learning scenario than children are faced with, because children must learn to segment the speech stream into words Lakhotia et al. (2021), while our models do not need to. Further, though real-world grounding could provide helpful information, learners might struggle to leverage this information due to difficulty determining what is being discussed in the physical world Gleitman et al. (2005).

## 7 Conclusion

In this work, we trained two types of neural networks (LSTMs and Transformers) on sentences of the types available to children and then analyzed what they had learned about English yes/no questions. Across several evaluation paradigms, these models failed to generalize in human-like ways: Humans display hierarchical generalization, while the models' generalization was instead based on linear order and individual words' identities. Our results support the hypothesis that human-like linguistic generalization requires biases stronger than those of LSTMs and Transformers. Future work should investigate what inductive biases enable successful generalization. One approach would be to test architectures with built-in hierarchical structure; past work has shown that such architectures have a hierarchical bias McCoy et al. (2020) and generalize better on the hierarchical phenomenon of subject-verb agreement Kuncoro et al. (2018); Lepori et al. (2020), so they may also generalize better on English yes/no questions. A final direction would be to expand the input beyond words alone so that learners can leverage hierarchical structure that is present in other modalities, such as hierarchical structure in visual scenes.

### Ethics Statement

Use of human data:While we did not collect any new human data ourselves, many of our analyses involved the use of prior datasets within the
CHILDES database. All of these datasets were collected in accordance with IRB policies at the institutions of the data collectors, and all followed standard practices in obtaining informed consent and deidentifying data.9

Footnote 9: [https://talkbank.org/share/irb/](https://talkbank.org/share/irb/)

Risks and limitations:The main risk of our proposed analyses is that future work using the same analyses might draw overly strong conclusions based on increased model performance, leading to overestimates of model strength. Such overestimates are an issue because they can lead users to place more trust in a model than is warranted.

To clarify, we view strong performance on our evaluation datasets as necessary but not sufficient to demonstrate human-like learning. Thus, if models perform poorly on our datasets (as the models we evaluated did), then we have strong reason to conclude that models are not learning in human-like ways. If future models perform better, such results would be consistent with human-like learning but would not conclusively establish that models learn as humans do, as they might instead be using some shallow heuristic that is not controlled for in our datasets. In other words, a criterion that is necessary but not sufficient facilitates strong conclusions about failure but does not facilitate strong conclusions about success. If future papers are faced with models that are more successful, such papers would ideally supplement results based on our datasets with analyses of models' internal strategies in order to more conclusively establish that what they have learned is not a spurious heuristic.
