Though state-of-the-art sentence representation models can perform tasks requiring significant knowledge of grammar, it is an open question how best to evaluate their grammatical knowledge. We explore five experimental methods inspired by prior work evaluating pretrained sentence representation models. We use a single linguistic phenomenon, negative polarity item (NPI) licensing, as a case study for our experiments. NPIs like 'any' are grammatical only if they appear in a licensing environment like negation ('Sue doesn't have any cats' vs. '*Sue has any cats'). This phenomenon is challenging because of the variety of NPI licensing environments that exist. We introduce an artificially generated dataset that manipulates key features of NPI licensing for the experiments. We find that BERT has significant knowledge of these features, but its success varies widely across different experimental methods. We conclude that a variety of methods is necessary to reveal all relevant aspects of a model's grammatical knowledge in a given domain.
 Recent sentence representation models have attained state-of-the-art results on language understanding tasks, but standard methodology for evaluating their knowledge of grammar has been slower to emerge. Recent work evaluating grammatical knowledge of sentence encoders like BERT BID6 has employed a variety of methods. For example, BID28 , BID7 , and BID30 use probing tasks to target a model's knowledge of particular grammatical features. BID22 and BID34 compare language models' probabilities for pairs of minimally different sentences differing in grammatical acceptability. BID19 , BID33 , and BID16 use Boolean acceptability judgments inspired by methodologies in generative linguistics. However, we have not yet seen any substantial direct comparison between these methods, and it is not yet clear whether they tend to yield similar conclusions about what a given model knows.We aim to better understand the trade-offs in task choice by comparing different methods inspired by previous work to evaluate sentence understanding models in a single empirical domain. We choose negative polarity item (NPI) licensing, an empirically rich phenomenon widely discussed in the theoretical linguistics literature, as our case study. NPIs are words or expressions that can only appear in environments that are, in some sense, negative. For example, any is an NPI because it is acceptable in negative sentences (1) but not positive sentences (2); negation thus serves as an NPI licensor. NPIs furthermore cannot be outside the syntactic scope of a licensor (3). Intuitively, a licensor's scope is the syntactic domain in which an NPI is licensed, and it varies from licensor to licensor. A sentence with an NPI present is only acceptable in cases where (i) there is a licensoras in (1) but not (2)-and (ii) the NPI is within the scope of that licensor-as in (1) but not (3).(1)Mary hasn't eaten any cookies.(2) *Mary has eaten any cookies.(3) *Any cookies haven't been eaten.We compare five experimental methods to test BERT's knowledge of NPI licensing. We consider: (i) a Boolean acceptability classification task to test BERT's knowledge of sentences in isolation, (ii) an absolute minimal pair task evaluating whether the absolute Boolean outputs of acceptability classifiers distinguish between minimally different pairs of sentences, (iii) a gradient minimal pair task evaluating whether the gradient outputs of acceptability classifiers distinguish between minimal pairs, (iv) a cloze test evaluating the grammatical preferences of BERT's masked language modeling head, and (v) a probing task evaluating BERT's representations for knowledge of specific grammatical features relevant to NPI licensing. We find that BERT knows about NPI licensing environments. However, our five methods give meaningfully different results. In particular, the gradient minimal pair experiment leads us to believe that BERT has systematic knowledge about all NPI licensing environments and relevant grammatical features, while the absolute minimal pair and probing experiments show that BERT's knowledge is in fact not equal across these domains. We conclude that no single method is able to accurately depict all relevant aspects of a model's grammatical knowledge; comparing both gradient and absolute measures of performance of trained models gives a more complete picture. We recommend that future studies would benefit from using multiple converging methods to evaluate model performance. We find that BERT systematically represents all features relevant to NPI licensing across most environments according to certain evaluation methods. However, these results vary widely across the different methods we compare. In particular, BERT performs nearly perfectly on the gradient minimal pairs task (at ceiling) across all of minimal pair configurations and nearly all licensing environments. Based on this method alone, we might conclude that BERT's knowledge of this domain is near perfect. However, the other methods show a more nuanced picture. BERT's knowledge of which expressions are NPIs and NPI licensors is generally stronger than its knowledge of the licensors' scope. This is especially apparent from the probing results FIG5 . BERT without acceptability fine-tuning performs close to ceiling on the licensor-detection probing task, but is inconsistent at scope-detection. Tellingly, the BoW baseline is also able to perform at ceiling on the and licensor-detection probing task. For BoW to succeed at this task, the GloVe embeddings for NPI-licensors must share some common property, most likely the fact that licensors co-occur with NPIs. It is possible that BERT is able to succeed using a similar strategy. By contrast, identifying whether an NPI is in the scope of a licensor requires at the very least word order information and not just co-occurrences.The contrast in BERT's performance on the gradient and absolute tasks tells us that these evaluations reveal different aspects of BERT's knowledge. The gradient task is strictly easier than the absolute task. On the one hand, BERT's high performance on the gradient task reveals the presence of systematic knowledge in the NPI domain. On the other hand, due to ceiling effects, the gradient task fails to reveal actual differences between environments that we clearly observe based on absolute, cloze, and probing tasks.While BERT has systematic knowledge of acceptability contrasts, this knowledge varies across environments and is not categorical. Current linguistic theory models human knowledge of natural language as categorical: In that sense BERT fails at attaining human performance. However, it is unclear whether humans themselves achieve categorical performance. Results from an MTurk study on human acceptability of our generated dataset show non-categorical agreement with the judgments in our dataset.Supplementing BERT with additional pretraining on CCG and MNLI does not improve performance, and even lowers performance in some cases. While results from BID26 lead us to hypothesize that intermediate pretraining might help, this is not what we observe on our data. This result is in direct contrast with the results from BID34 , who find that syntactic pretraining does improve performance in the NPI domain. This difference in findings is likely due to differences in models and training procedure, as their model is an RNN jointly trained on language modeling and parsing over the much smaller Penn Treebank BID21 .Future studies would benefit from employing a variety of different methodologies for assessing model performance withing a specified domain. In particular , a result showing generally good performance for a model should be regarded as possibly hiding actual differences in performance that a different task would reveal. Similarly, generally poor performance for a model does not necessarily mean that the model does not have systematic knowledge in a given domain; it may be that an easier task would reveal systematicity. We have shown that within a well-defined domain of English grammar, evaluation of sentence encoders using different tasks will reveal different aspects of the encoder's knowledge in that domain. Table 3 : Reduced paradigm for Simple questions. "Lic." is abbreviated from "Licensor". The licensor and licensor replacement are shown in bold (has in both cases). The NPI (any) and NPI replacement (the) are shown in italics. There is no scope manipulation because it is not possible to place an NPI or NPI replacement outside of the scope of an interrogative or declarative phrase. The 2 minimal pairs are shown by arrows, pointing from unacceptable to acceptable sentence. Table 4 : Results from MTurk validation. 'Environment' is the name of the licensing environment and 'label' is whether the sentence was intended as acceptable ( ) or unacceptable (*). The results of the validation ratings is in '% accept' and represents the majority vote for each sentence as acceptable/unacceptable and then averaged to give the percentage of times a sentence in a given condition was rated as acceptable by the MTurk raters. 'Diff' is calculated from the % of acceptable sentences rated acceptable minus the % of unacceptable sentences rated acceptable (100 is a perfect score, 0 means there is no difference). 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00GloVe BoW (Gradient Preference) CoLA All NPI All-but-1 NPI Avg Other NPI 1 NPI Trained on 0.78 0.69 0.67 0.89 0.78 0.71 0.65 0.95 0.84 0.84 1.00 1.00 1.00 1.00 1.00 1.00 0.99 1.00 1.00 0.99 0.98 0.86 1.00 1.00 1.00 1.00 0.99 1.00 1.00 1.00 0.97 0.95 0.98 1.00 0.97 0.99 0.94 1.00 1.00 0.95 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
Different methods for analyzing BERT suggest different (but compatible) conclusions in a case study on NPIs.