# Can Language Models Be Tricked by Language Illusions? Easier with Syntax,
Harder with Semantics

Yuhan Zhang
Linguistics
Harvard University
<EMAIL_ADDRESS>
Edward Gibson
Brain & Cognitive Sciences
Massachusetts Institute of Technology
<EMAIL_ADDRESS>
Forrest Davis
Computer Science
Colgate University
<EMAIL_ADDRESS>

###### Abstract

Language models (LMs) have been argued to overlap substantially with human
beings in grammaticality judgment tasks. But when humans systematically make
errors in language processing, should we expect LMs to behave like cognitive
models of language and mimic human behavior? We answer this question by
investigating LMs’ more subtle judgments associated with “language illusions”
– sentences that are vague in meaning, implausible, or ungrammatical but
receive unexpectedly high acceptability judgments by humans. We looked at
three illusions: the comparative illusion (e.g. “More people have been to
Russia than I have”), the depth-charge illusion (e.g. “No head injury is too
trivial to be ignored”), and the negative polarity item (NPI) illusion (e.g.
“The hunter who no villager believed to be trustworthy will ever shoot a
bear”). We found that probabilities represented by LMs were more likely to
align with human judgments of being “tricked” by the NPI illusion which
examines a structural dependency, compared to the comparative and the depth-
charge illusions which require sophisticated semantic understanding. No single
LM or metric yielded results that are entirely consistent with human behavior.
Ultimately, we show that LMs are limited both in their construal as cognitive
models of human language processing and in their capacity to recognize nuanced
but critical information in complicated language materials.

##  1 Introduction

Linguistic evaluations of language models use human language processing data
(e.g. human norming data Nair et al. (2020); Zhang et al. (2022),
acceptability judgments Linzen et al. (2016); Marvin and Linzen (2018),
behavioral or neural measures of language processing Schrimpf et al. (2021);
Kauf et al. (2022)) as benchmarks to investigate whether LMs possess knowledge
of language. This assumes that human-produced data correctly instantiates
abstract rules of a language and that humans fully utilize their linguistic
knowledge in laboratories and everyday life. However, this assumption is an
oversimplification. Humans make consistent errors during language processing
Gross (1983). Under these circumstances, should we expect language models to
behave the same as humans? Or should they circumvent human limitations and
achieve error-free performance?

Consider, for example, the well-studied case of subject-verb agreement. While
we expect an LM of Standard American English to prefer “the key to the
cabinets is on the shelf” to “the key to the cabinets are on the shelf” (as
discussed in Linzen et al., 2016), a wealth of psycholinguistic research has
systematically documented that humans can ignore errors and accept globally
ungrammatical strings (stemming from Bock and Miller, 1991). Should LMs follow
the ideal grammar or mimic human’s (sometimes) errorful behavior?111For
additional critiques of the role of ideal grammatical knowledge in evaluations
of LMs, see Pannitto and Herbelot (2020); Weissweiler et al. (2023).

We add to this discussion by investigating three language illusions. Basic
examples of each are given in 1: the comparative illusion 1, the depth-charge
illusion 1, and the negative-polarity item (NPI) illusion 1. All three in 1
are literally unnatural English sentences, despite the fact that humans often
find them surprisingly acceptable.

. . More people have been to Russia than I have. .̱ No head injury is too
trivial to be ignored. .̧ The hunter who no villager believed to be
trustworthy will ever shoot a bear.

In this paper, we relied on minimally different strings springing out from the
basic illusion sentences that are either (a) considered fully acceptable by
human participants, (b) considered fully unacceptable by human participants,
or (c) rated surprisingly acceptable by humans (i.e. instances of the relevant
illusion). We explored whether language models capture the basic contrast
between acceptable and unacceptable strings, whether they rate illusion
sentences as better than their unacceptable counterparts, and finally, whether
models capture nuanced linguistic manipulations that influence human judgments
of the illusion material. Further, we compared two ways of measuring models’
preferences, one over the whole sentence (perplexity) and another of a
privileged position in the sentence (surprisal).

If LMs pattern like human comprehension behavior that involves errors, we
expect to derive measures that similarly rate illusion sentences as more
acceptable than typical unacceptable sentences. If, on the other hand, LMs
align with ideal grammatical judgments, illusion sentences should be rated as
unacceptable. Our findings indicate that none of the language models we
investigated consistently exhibited illusion effects or demonstrated overall
human-like judgment behaviors. Nor do they possess the necessary linguistic
knowledge for error-free, literal sentence processing. These findings add more
insights into the discussion of LMs’ emulation of human behavior and their
construal as cognitive models of human language processing.

##  2 Related work

###  2.1 LMs’ linguistic abilities

We draw insights from evaluation work relying on acceptability tasks. The
construction of minimal pairs has been used to evaluate models for a variety
of linguistic processes, including subject-verb agreement (e.g. Linzen et al.,
2016), filler-gap dependency (e.g. Wilcox et al., 2018), control (e.g.
Stengel-Eskin and Van Durme, 2022), and binding (e.g. Davis, 2022). This basic
template has been expanded into a variety of benchmarks, both for
investigations of English (e.g. Warstadt et al., 2020), but also, other
languages (e.g. Chinese (Song et al., 2022); Russian (Mikhailov et al., 2022);
Japanese (Someya and Oseki, 2023)). While aggregated results suggest that
models overlap with human acceptability judgments in a variety of cases (e.g.
Hu et al., 2020), LMs can behave in distinctly non-human-like ways in
capturing the intricacies of grammatical phenomenon (e.g. Lee and Schuster,
2022), the interaction between linguistic processes (e.g. Davis and van
Schijndel, 2020), and in generalizing knowledge to infrequent items (e.g. Wei
et al., 2021).

In our experiments, we are interested in cases where human interpretations and
behaviors differ from what is expected given the literal content of the entire
string. Garden path sentences are a classic example of this basic phenomenon.
Strings like “The horse raced past the barn fell” are often difficult for
humans on first reading because the word raced is misparsed as a main verb
(e.g. the horse raced past) rather than a reduced relative clause (e.g. the
horse that was raced past the barn fell). LMs have been shown to similarly
misprocess these sentences (van Schijndel and Linzen, 2021), though they fall
short of capturing the magnitude of the processing cost (Arehalli et al.,
2022). Here we expand these investigations to language illusions that
similarly trigger errorful acceptable judgments in humans while being
unnatural and unacceptable. We find that LMs do not pattern like humans in all
cases.

###  2.2 Language illusions

Language illusions refer to ungrammatical, semantically vague, or
pragmatically implausible sentences that receive higher than expected
acceptability by humans Phillips et al. (2011). We study three language
illusions in particular: comparative illusion Montalbetti (1984) (Section 4),
depth-charge illusion Wason and Reich (1979) (Section 5), and NPI illusion
Xiang et al. (2009) (Section 6). Existing human research has found that the
illusion effects for both the comparative and the depth-charge illusion are
robust and overwhelming but the NPI illusion effect only appears during
speeded judgment tasks or word-by-word online paradigms Parker and Phillips
(2016); Wellwood et al. (2018); Paape et al. (2020); Orth et al. (2021).

For human sentence processing, it has been suggested that language illusions
provide evidence for rational inference of error-prone strings which
integrates heuristics and available context information during processing
Ferreira et al. (2002); Levy (2008); Gibson et al. (2013); Futrell et al.
(2020); Hahn et al. (2022); Zhang et al. (2023b). These phenomena raise
fundamental questions like what is the role of our grammatical knowledge in
comparison to other cognitive resources when it comes to assigning a specific
interpretation to a linguistic string, and how we can model their interactions
to make better predictions about human sentence processing.

Studying LMs’ processing of language illusions provides a way to explore
whether they can be viewed as cognitive models of human sentence processing.
As large language models like ChatGPT improve at generating grammatically
appropriate strings, it becomes ever more important to investigate whether
they are comparable to human language processing behavior at all (see Mahowald
et al., 2023, for a review). From there, we can reason about what
characteristics in the training of LMs, the architecture of LMs, and the
“abilities” of LMs enable them to carry out either literal interpretations and
detect the anomaly, or to fall into the illusion rabbit hole.

Illusion type | item | BERT | RoBERTa | GPT-2 | GPT-3
---|---|---|---|---|---
|  | PPL | Surp | PPL | Surp | PPL | Surp | PPL | Surp
Comparative | 32 | -0.36 | -0.001 | -0.56 | -0.09 | -0.22 | -0.05 | -0.30 | -0.25
Depth-charge | 32 | -0.37 | -0.15 | -0.61 | -0.45 | -0.12 | -0.41 | -0.37 | -0.98
NPI | 32 | -0.26 | -2.46 | -0.71 | -2.60 | -0.21 | -1.73 | -0.29 | -2.55
Table 1:  Estimated coefficients of the main effect (acceptable sentence
condition vs. unacceptable condition (reference)) for each statistical model.
If LMs rate acceptable sentences as more acceptable, the coefficients for
perplexity or surprisal should be significantly negative. Cells color-coded in
blue represent statistical significance level ($p<.05$) in the expected
direction. White cells represent an insignificant main effect. In other words,
blue cells indicate the statistical model output supports LMs’ ability to
distinguish sentences based on linguistic acceptability.

##  3 Methods

###  3.1 Models and Measures

We analyzed four models, two masked language models, and two autoregressive
models: BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), GPT-2 (Radford
et al., 2019), GPT-3 (Brown et al., 2020). BERT, RoBERTa, and GPT-2 were
accessed via HuggingFace (Wolf et al., 2020), and GPT-3 via OpenAI’s API.222We
used ‘bert-base-cased’, ‘roberta-base’, ‘gpt2’, and ‘text-davinci-003’. Code
for replicating the results, statistical tests, and figures can be found at
https://github.com/forrestdavis/LanguageIllusions.git . We used two measures,
sentence level perplexity and surprisal of specific target words. For
autoregressive models, the surprisal of a specific word333For words that are
subworded, the joint probability was calculated. is given by the following
equation:

$\textrm{Surp}(\textrm{w}_{i})=-\textrm{log
Prob(w}_{i}|\textrm{w}_{1}...\textrm{w}_{i-1})$ (1)

Perplexity for a sentence of N words is:

$2^{\frac{1}{N}\sum_{i=1}^{N}{\textrm{Surp}(\textrm{w}_{i})}}$ (2)

For bidirectional models, we calculated the surprisal of a word in a context
by using the masking technique in Kauf and Ivanova (2023), which corrects for
words that are subworded.444For example, consider the word ‘souvenir’. This is
subworded by BERT into ‘so’, ‘##uven’, and ‘ir’. Rather than MASK each
subpart, one at a time, (e.g. ‘so’ [MASK] ‘ir’), the right context of the
target subword is always masked (e.g. ‘so’ [MASK] [MASK]). Further, we used
this masking technique to calculate the pseudo-perplexity of a sentence
(Salazar et al., 2020).

###  3.2 Evaluation procedure

We treated LMs as psycholinguistic research subjects to generate both whole-
sentence perplexity and surprisals at critical words for carefully controlled
minimal pairs for each illusion (following, Futrell et al., 2019). Assuming
these two scores are correlated to human acceptability judgments Lau et al.
(2017), we constructed mix-effects linear regression models from the $R$
package $lme4$ to test whether LMs were also sensitive to reported
manipulations that affect human judgments. For each scoring metric, we took it
as the dependent variable and coded the manipulation condition representing a
certain hypothesis into the independent variable. We read the estimated
coefficient(s) of the tested condition variable(s) to infer whether LMs show
sensitivity to the effect of that condition manipulation on the scoring
metric. We evaluated language models in three broad aspects: acceptability
differentiation, illusion effect, and sensitivity to manipulations.

  * •

Acceptability differentiation We first asked whether language models could
distinguish acceptable sentences from unacceptable sentences that humans have
no trouble dealing with.555According to finer-grained linguistic criteria,
acceptable sentences are those that are grammatical, plausible, and felicious.
Please refer to Tonhauser and Matthewson (2015) for detailed definitions and
review. Models with relevant knowledge should assign lower
perplexity/surprisal to acceptable sentences versus unacceptable ones.

  * •

Illusion effect We took the results from the acceptability differentiation
task as the foundation to test the illusion sentences. Here, we hypothesized
that language models should either (i) align with humans’ illusionary
judgments, reflected by models’ generating a lower perplexity/surprisal for
illusion sentences than the unacceptable controls, or (ii) deviate from human
behavior and show hints of being a literal processor, reflected by models’
generating a higher or similar perplexity/surprisal score compared to the
unacceptable condition. If models behave like humans, then we expected (i) to
be the models’ consistent behavior. If models conform to (ii), we take this as
evidence of non-human-like behavior.

  * •

Sensitivity to manipulations Lastly, we assessed whether language models were
sensitive to illusion-specific linguistic manipulations that affect human
judgments. A greater degree of sensitivity indicates that the corresponding
linguistic knowledge and how the knowledge affects sentence acceptability
could be encoded in or learned by LMs. This allowed us to draw a fine-grained
comparison between humans and LMs. If language models are insensitive, that
indicates a difference between humans and LMs.

Figure 1: The $y$ axis shows the coefficient estimates which represent the
increase in perplexity/surprisal when the sentence is unacceptable compared to
the illusion case, crossing three language illusions and four LMs. “+” marks a
human-like behavior, in this case, an illusion effect where the unacceptable
condition receives significantly higher perplexity/surprisal values than the
illusion condition. “*” means that the estimated coefficient is significant.

##  4 Comparative illusion

A canonical comparative illusion surfaces in sentences like “More people have
been to Russia than I have”. People accept it at first glance but have trouble
pinning down the exact meaning Montalbetti (1984) one of which could be that
the number of the group of people who’ve been to Russia is greater than the
number of “me”. Potential rational nonliteral inference could be “people have
been to Russia more times than I have” or “people have been to Russia but I
haven’t” O’Connor (2015); Christensen (2016). Psycholinguistic research has
found that various factors modulate the strength of the illusion, including
the repeatability of the event described by the verb phrase, the subject form
of the than-clause subject (e.g. “… than the student has” vs. “…I have”), as
well as the number of that subject (e.g. “I have” vs. “we have”)Wellwood et
al. (2018). There is also a claim arguing that the processing mechanism
follows the noisy-channel predictions under an information-theoretic account
Zhang et al. (2023a).

We adapted the experimental materials with 32 items from Zhang et al.
(2023a).666See Table 3 in the Appendix for the full paradigm. An example is in
4 where 4 is the canonical comparative illusion, 4 is the acceptable control,
and 4 is the unacceptable one.777The repeatability of the verb phrase is
responsible for this contrast, as it is more natural to say “use Tiktok more
often or frequent” compared with “install Tiktok more often” when the action
typically takes place once (in a while).

. .(?) More teenagers have used Tiktok than I have. (illusion) .̱Many
teenagers have used Tiktok more than I have. (acceptable) .̧(#) Many teenagers
have installed Tiktok more than I have. (unacceptable)

###  4.1 Acceptability differentiation

We first ensured that LMs distinguish acceptable neighbors 4 of the illusion
sentence from unacceptable ones 4. We ran statistical mixed-effects linear
regression models on whole-sentence perplexity and the surprisal at the word
have for the four language models. Either the perplexity or the surprisal was
taken as the dependent variable with the condition “acceptability” as the
fixed effect (reference level = the unacceptable condition, with a
nonrepeatable verb phrase vs. the acceptable condition, with a repeatable verb
phrase) and the random intercept of each item as the random effect.888The
model syntax in $R$ was PPL/SURP $\sim$ acceptability + (1|item).

Table 1 shows the estimated coefficient for the main effect of each mixed-
effect model for each LM and each illusion phenomenon. A significant negative
estimated coefficient suggests that acceptable sentences received lower
perplexity/surprisal compared to the unacceptable ones, indicating that LMs
distinguish sentences based on acceptability. Except for surprisal values from
BERT and GPT-2, the other six statistical models indicate that the LMs capture
the acceptability difference of baseline sentences for the comparative
illusion.

###  4.2 Illusion effect

This task investigated whether language models pattern with humans in
demonstrating illusion effects with the basic comparative illusion
construction. The contrast involves the illusion condition 4 with existing
control conditions (4 and 4). The standardized metrics of the four LMs are
displayed in Figure 6 in the Appendix. To evaluate whether LMs capture an
illusion effect, we constructed another suite of statistical models across the
four LMs and two metrics where the main effect has three levels – the illusion
condition (reference), the acceptable condition, and the unacceptable
condition – and the random effect included a random intercept for items.999The
model syntax in R was PPL/SURP $\sim$ condition + (1|item) where condition had
three levels.

We analyzed the coefficient estimates of the main effect of the unacceptable
condition compared with the illusion condition.101010The coefficients for the
acceptable condition generate similar conclusions. Further, no illusion
sentences were rated better than acceptable ones. An illusion effect would
appear with higher perplexity/surprisal for the unacceptable condition
compared to the illusion case. In other words, the estimated coefficients for
the unacceptable condition should be significantly positive.

Figure 1 and Table 2 (in Appendix) display the estimated coefficients for the
unacceptable condition compared with the illusion condition. For the
comparative illusion, only BERT and RoBERTa measured by perplexity show a
human-like illusion effect. Other LM-metric combinations indicate that the
illusion condition was rated either the same or worse than the unacceptable
condition (contrary to humans).

###  4.3 Sensitivity to manipulations

In this step, we evaluated whether language models were sensitive to sentence
manipulations that affect human judgments. Three factors were investigated:
(1) than-clause subject structure (pronoun vs. NP), (2) subject number
(singular vs. plural), and (3) verb repeatability (repeatable vs.
nonrepeatable). For humans, plural than-clause subjects are more acceptable
than singular ones only in the NP case. Overall, repeatable verbs are more
acceptable than nonrepeatable ones O’Connor (2015); Wellwood et al. (2018);
Zhang et al. (2023a).

Figure 2: Estimated coefficients for critical linguistic manipulations in
comparative illusion. The $y$ axis shows the estimated coefficients for the
increase in perplexity/surprisal with respect to singular vs. plural than-
clause subjects, or nonrepeatable vs. repeatable verb phrases, respectively.
“*” means statistically significant contrasts; “+” means human-like results.

Figure 2 displays the estimated coefficients for the main effects from the
statistical models.111111More statistic model information: Iterating over LMs,
metrics, and the subject structure (NP vs. pronoun), we initiated statistical
models taking both repeatability (reference = repeatable) and subject number
(reference = plural) as the main effects with the random effect including a
random intercept for the items. As for the subject number, when the than-
clause subject was a pronoun, only BERT and GPT-2 (with perplexity) aligned
with human-like behavior: there is no difference between singular and plural
than-clause subjects. When it comes to NP subjects, all four LMs with both
metrics showed human-like behavior where the singular NP subject was more
unacceptable than the plural NP subject. As for repeatability, all four LMs
captured this distinction in the pronoun condition but in the NP condition,
only RoBERTa and GPT-3 achieved human-like results with perplexity.

In general, we only found partial overlap between LMs and humans. This
indicates that even though LMs show some knowledge of acceptability for
comparative structures, they might operate differently from humans when
processing more subtle differences. None of the language models fully captured
all the manipulations.

##  5 Depth-charge illusion

Consider the most famous depth-charge sentence No head injury is too trivial
to be ignored Wason and Reich (1979). People overwhelmingly interpret it as
meaning “no matter how trivial head injuries are, we should not ignore them”,
while the literal meaning is the opposite as “we should ignore them”.

To understand the depth-charge sentence requires knowing meaning composition
rules, multiple negation processing Wason and Reich (1979), adequate world
knowledge reasoning Paape et al. (2020), and the neighboring constructions of
too…to such as so…that, so…as to and enough to… Zhang et al. (2023b). Since
existing research already shows that language models are quite limited in
processing negation (e.g. Kassner and Schütze, 2019; Ettinger, 2020), we
speculate that LMs might encounter difficulty in the more complicated case of
depth-charge sentences.

The evaluation materials were adapted from Zhang et al. (2023b) with 32 items.
An example is 5 where we take the surprisal of the sentence-final word for
comparison.

. . (?) No head injury is too trivial to be ignored. (depth-charge sentence)
.̱ Some head injury is too severe to be ignored. (plausible, acceptable) .̧
(#) Some head injury is too trivial to be ignored. (implausible, unacceptable)

###  5.1 Acceptability differentiation

Utilizing the same methodology as the comparative illusion, we found, as
depicted in Table 1, that all combinations of LMs and metrics, except GPT-2
(perplexity), captured the acceptability difference between (5) and (5) with a
significantly lower perplexity/surprisal for the acceptable sentences like 5.

###  5.2 Illusion effect

Next, we studied if LMs “experience” the illusion effect by assigning lower
perplexity/surprisal scores to the depth-charge sentence 5 compared to the
unacceptable one 5.

Our statistical results show, in Figure 1 and Table 2 (Appendix), that only
RoBERTa and GPT-3 demonstrated an illusion effect (for surprisal) by assigning
a significantly higher score to the unacceptable control sentences. This means
that it is not easy to “trick” LMs with the depth-charge illusion. Similar
results have led concurrent work to suggest that LMs are better at deriving
the literal meaning of a sentence, which is in sharp contrast with the
overwhelming illusion effect from humans (Paape, 2023, a.o.).

###  5.3 Sensitivity to manipulations

This task tested LMs’ sensitivity to the plausibility contrast of three near-
neighbor pairs of the depth-charge sentence. These pairs differ by the degree
quantifier construction (too…to vs. so…as to vs. too…to not).121212The full
suite of paradigms is shown in Table 4 in the Appendix. Competent language
models should differentiate plausible sentences from implausible ones.

Figure 3 displays estimated coefficients of statistical models’ main effect.
We expect implausible sentences to receive higher perplexities/surprisals when
the illusion occurs.131313Iterating over sentence pairs, LMs, and metrics, we
ran mixed-effects linear regression models on scores over the plausibility
contrast (reference = plausible). We find that LMs captured some of the
distinctions in the too…to condition and the so…as to condition. However,
implausible sentences with too…to not were rated as more acceptable than their
plausible counterparts, which flouts what linguistic rules predict.141414The
sentence No head injury is too trivial to not be ignored should be plausible
because compositionally, “too trivial to not be ignored” means “too trivial to
be treated” which yields a plausible sentence given the sentential negation.
The fact that “No head injury is too trivial to be treated” and “No head
injury is too trivial to not be ignored” generate opposite results while
having the same meaning suggests LMs still struggled with negation, antonyms,
and meaning composition Kim and Linzen (2020); She et al. (2023); Truong et
al. (2023).

Figure 3: Estimated coefficients for the plausibility contrast (reference =
plausible) in depth-charge illusions. The $y$ axis shows the increase in
perplexity/surprisal when the sentence is implausible vs. plausible. “*” means
statistically significant contrasts; “+” means human-like behavior. While we
see differences among LMs and metrics in the “no…so…as to” and the “no…too…to”
conditions, the condition of “no…too…to not” yielded completely opposite
results to humans.

##  6 NPI illusion

Negative polarity items and their licensing conditions have been investigated
in prior work with language models. For a canonical NPI (e.g. ever, any) to be
acceptable, it has to be in the scope of negation.151515The licensing
conditions of negative polarity items are far more than in the scope of
negation. We focus on the classic licensing condition and refer to Giannakidou
et al. (2019) for a review. Existing computational research has shown that the
syntactic dependency between the licensor and the NPI is captured by language
models Jumelet and Hupkes (2018); Jumelet et al. (2021); Shin et al. (2023)
but with more difficulty as compared to subject-verb agreement or other
syntactic dependencies Marvin and Linzen (2018); Warstadt et al. (2019, 2020).
In this task, we expanded the suite of LMs and metrics and explored
sensitivities to four types of licensors.

Our materials came from Orth et al. (2021) with 32 items. The essential triad
is 6 where the illusion condition has the NPI ever not in the scope of the
negation word no.

. . (?) The hunter who no villager believed to be trustworthy will ever shoot
a bear. (NPI illusion) .̱ No hunter who the villager believed to be
trustworthy will ever shoot a bear. (Matrix No, acceptable) .̧ (*) The hunter
who the villager believed to be trustworthy will ever shoot a bear. (Licensor
Absent, unacceptable)

###  6.1 Acceptability differentiation

Table 1 shows that all the four LMs could capture the acceptability difference
of control sentences 6 and 6 (with both metrics).

###  6.2 Illusion effect

Figure 1 and Table 2 show that only in the case of surprisal did we see an
illusion effect where the unacceptable sentences (e.g. 6) received
significantly higher surprisals than the illusion sentence (e.g., 6). This
finding replicates Shin et al. (2023) in that, for the illusion condition (6)
where no linearly precedes ever but is in an unlicensing position, ever incurs
higher surprisal. It is interesting to see the sharp discrepancy between
surprisal and perplexity, which we leave to Section 7.4 for discussion.

Figure 4: Estimated coefficients for the illusion effect (unacceptable vs.
illusion = reference) in NPI illusions. The $y$ axis shows the increase in
perplexity/surprisal when the sentence is ungrammatical vs. is in the illusion
condition. “+” marks an illusion effect while none of the three licensors
should trigger an illusion effect according to human behavior; “*” means a
significant contrast.

###  6.3 Sensitivity to variations

The linguistic manipulations we explored concern the illusion effect in the
illusion condition with different NPI licensors. Among the ones we tested,
didn’t, did not, and never,161616Please refer to Table 5 for the full
experimental conditions. human research shows that none of these triggers
illusion effects (Orth et al., 2021; Vasishth et al., 2008, cf.).

Iterating over licensors, LMs, and metrics, we ran statistical models with the
same structure in Section 6.2. We plotted the estimated coefficients of the
unacceptable main effect in Figure 4 and predicted that a significantly
positive coefficient indicates an illusion effect. Contrary to human-like
behavior, for all three licensors there were some LM-metric combinations that
indicate an illusion effect: for the licensor did not, RoBERTa (perplexity)
and GPT-2 (perplexity) show an illusion effect; for didn’t, all four LMs with
perplexity show an illusion effect; for never, all four LMs with surprisal,
plus RoBERTa with perplexity, show an illusion effect. This pattern shows that
with NPI illusions, LMs are more easily tricked than humans.

##  7 Discussion

###  7.1 Illusion effect

Successful language processing requires a dynamic integration of lexical
knowledge, grammatical knowledge, logical reasoning, and world knowledge,
among other cognitive abilities and sources of knowledge. An illusion effect
in humans where unacceptable sentences receive unexpectedly high acceptability
presents a unique case where the comprehender might prioritize different
processing mechanisms or linguistic constraints for meaning inference over
those employed for common processing. Studying how language models process
language illusions helps us understand (1) from a superficial level, whether
LMs appear to be human-like – circumventing some grammatical facts and
reaching a good-enough sentence representation, and (2) from a deeper level,
whether LMs employ the same set of resources and abilities to process a
sentence (i.e. whether they can serve as cognitive models).

In this research, we aim for the first level of understanding. By studying
four language models’ acceptability judgments of three language illusions, we
found that LMs were good at the basic acceptability differentiation task and
yet no LMs showed consistent human-like illusion effects among three illusion
phenomena by any metric (Figure 5). We conclude from this result that LMs
might not be a good cognitive model of human language processing. With this
said, we do observe a divergence between the comparative/depth-charge illusion
and the NPI illusion – it seems more likely for LMs to be tricked by the NPI
illusion compared to the former two. Since the NPI illusion is more relevant
to the hierarchical structure of language whereas both the comparative
illusion and depth-charge illusion emphasize semantic nuances, we tentatively
conclude that LMs are more easily tricked by syntactic illusion rather than
semantic illusions.

Figure 5: Language models’ performance on all three illusions. ✓means LMs show
human-like behavior.

###  7.2 Human-like behaviors & Potential processing mechanisms

For both the comparative illusion and depth-charge illusion, the illusion
effect test did not show human-like behavior. This could either mean that LMs
strictly abide by linguistic rules to compose the language literally or that
LMs have trouble understanding this complicated set of sentences overall. For
the comparative illusion, the sensitivity task (Section 4.3) suggests that
they might have some capacity to process comparative structures. For the
depth-charge illusion, that LMs seem to have trouble understanding the literal
contrast between plausible/implausible pairs (Section 5.3) suggests sentences
involving multiple negations could pose a challenge to LMs. The two cases
indicate we still need to develop more robust evaluations to gauge LMs’
semantic capabilities in various semantic domains.

For the NPI illusion, the interpretation could be more complicated. On one
hand, the illusion test for the licensor no yields human-like results (with
surprisal) but other licensors also elicit non-human-like illusion effect (cf.
Orth et al., 2021). On the other hand, the discrepancy between sentence
perplexity and surprisal makes it difficult to conclude to what degree LMs and
humans overlap (cf. Shin et al., 2023).

Ultimately, we want to address whether LMs are like humans that utilize not
only grammatical rules but also contexts, frequencies, and semantic priors to
rationally process language, or LMs are like grammarians that interpret string
inputs in a strict compositional manner. Our investigation does not yield
consistent results given the three language illusions but the behavioral
inconsistency suggests that language models are far from being a cognitive
model of human language.

###  7.3 Language models’ performance in general

All four language models performed on par with each other. If we tallied the
number of tests where LMs reported expected results from Figure 5 and averaged
between perplexity and surprisal, we have a ranking order from RoBERTa (N=10)
and GPT-3 (N=9), to BERT (N=8.5) and GPT-2 (N=8). The successors of both the
masked language model and the autoregressive model perform better than their
predecessors.

###  7.4 Perplexity & Surprisal

It is surprising to see that the two widely used probability-based metrics can
generate different results for a given hypothesis and a given language model.
Future work should (i) investigate both mathematically and practically why the
difference could occur and (ii) check if better definitions for the critical
regions exist to capture surprisals. Future evaluation work that utilizes one
metric should be mindful of the intrinsic limitations of that metric.

###  7.5 Limitations

Considering the research methodology, acceptability judgment tasks (even with
carefully controlled minimal pairs) are indirect measures of language
comprehension and it is hard to infer the exact interpretation based on
probability-based measures. Further studies should work on direct
comprehension measures (e.g. generating paraphrases) that reveal LMs’ hidden
knowledge.

##  8 Conclusion

We tested four language models’ ability to process three language illusions
and asked (1) whether they judge unacceptable illusion sentences to be more
acceptable as humans (termed an illusion effect) and (2) whether they are
sensitive to linguistic manipulations that modulate human judgments. Our
results are based on whole-sentence perplexity and critical word surprisal. We
show that none of the LMs demonstrated consistent illusion effects or
exhibited overall human-like judgment behaviors. We conclude that given the
case of language illusions, language models neither behave like humans with
full sets of cognitive abilities and error-prone behavior nor possess the
necessary linguistic knowledge for error-free, literal sentence processing.
Language models cannot be viewed as cognitive models of language processing,
which makes understanding them even more intriguing.

## Acknowledgements

We thank the three anonymous reviewers for their helpful feedback. We thank
Ankana Saha, Carina Kauf and Hayley Ross for the helpful discussion about the
project.

## References

  * Arehalli et al. (2022) Suhas Arehalli, Brian Dillon, and Tal Linzen. 2022.  Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities.  In _Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)_ , pages 301–313, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Bock and Miller (1991) Kathryn Bock and Carol A Miller. 1991.  Broken agreement.  _Cognitive psychology_ , 23(1):45–93.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Christensen (2016) Ken Ramshøj Christensen. 2016.  The dead ends of language: The (mis) interpretation of a grammatical illusion.  _Let us have articles betwixt us–Papers in Historical and Comparative Linguistics in Honour of Johanna L. Wood. Aarhus: Aarhus University_ , pages 129–160.
  * Davis (2022) Forrest Davis. 2022.  Incremental Processing of Principle B: Mismatches Between Neural Models and Humans.  In _Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)_ , pages 144–156, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Davis and van Schijndel (2020) Forrest Davis and Marten van Schijndel. 2020.  Discourse structure interacts with reference but not syntax in neural language models.  In _Proceedings of the 24th Conference on Computational Natural Language Learning_ , pages 396–407, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  ArXiv:1810.04805 [cs].
  * Ettinger (2020) Allyson Ettinger. 2020.  What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48.  Place: Cambridge, MA Publisher: MIT Press.
  * Ferreira et al. (2002) Fernanda Ferreira, Karl G.D. Bailey, and Vittoria Ferraro. 2002.  Good-Enough Representations in Language Comprehension.  _Current Directions in Psychological Science_ , 11(1):11–15.
  * Futrell et al. (2020) Richard Futrell, Edward Gibson, and Roger P. Levy. 2020.  Lossy-Context Surprisal: An Information-Theoretic Model of Memory Effects in Sentence Processing.  _Cognitive Science_ , 44(3).
  * Futrell et al. (2019) Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019.  Neural language models as psycholinguistic subjects: Representations of syntactic state.  _arXiv preprint arXiv:1903.03260_.
  * Giannakidou et al. (2019) Anastasia Giannakidou, Klaus von Heusinger, Claudia Maienborn, and Paul Portner. 2019.  Negative and positive polarity items.  _Semantics—Sentence and information structure_ , pages 69–134.
  * Gibson et al. (2013) Edward Gibson, Leon Bergen, and Steven T Piantadosi. 2013.  Rational integration of noisy evidence and prior semantic expectations in sentence interpretation.  _Proceedings of the National Academy of Sciences_ , 110(20):8051–8056.
  * Gross (1983) Herbert S. Gross. 1983.  Errors in Linguistic Performance: Slips of the Tongue, Ear, Pen, and Hand.  _The journal of nervous and mental disease_ , 171(12):753–.
  * Hahn et al. (2022) Michael Hahn, Richard Futrell, Roger Levy, and Edward Gibson. 2022.  A resource-rational model of human processing of recursive linguistic structure.  _Proceedings of the National Academy of Sciences_ , 119(43):e2122602119.  Publisher: Proceedings of the National Academy of Sciences.
  * Hu et al. (2020) Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020.  A systematic assessment of syntactic generalization in neural language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1725–1744, Online. Association for Computational Linguistics.
  * Jumelet et al. (2021) Jaap Jumelet, Milica Denic, Jakub Szymanik, Dieuwke Hupkes, and Shane Steinert-Threlkeld. 2021.  Language Models Use Monotonicity to Assess NPI Licensing.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4958–4969, Online. Association for Computational Linguistics.
  * Jumelet and Hupkes (2018) Jaap Jumelet and Dieuwke Hupkes. 2018.  Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Items.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 222–231, Brussels, Belgium. Association for Computational Linguistics.
  * Kassner and Schütze (2019) Nora Kassner and Hinrich Schütze. 2019.  Negated LAMA: birds cannot fly.  _CoRR_ , abs/1911.03343.
  * Kauf and Ivanova (2023) Carina Kauf and Anna Ivanova. 2023.  A Better Way to Do Masked Language Model Scoring.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 925–935, Toronto, Canada. Association for Computational Linguistics.
  * Kauf et al. (2022) Carina Kauf, Anna A Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan S She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. 2022.  Event knowledge in large language models: the gap between the impossible and the unlikely.  _arXiv preprint arXiv:2212.01488_.
  * Kim and Linzen (2020) Najoung Kim and Tal Linzen. 2020.  COGS: A compositional generalization challenge based on semantic interpretation.  _CoRR_ , abs/2010.05465.
  * Lau et al. (2017) Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017.  Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge.  _Cognitive Science_ , 41(5):1202–1241.
  * Lee and Schuster (2022) Soo-Hwan Lee and Sebastian Schuster. 2022.  Can language models capture syntactic associations without surface cues? a case study of reflexive anaphor licensing in English control constructions.  In _Proceedings of the Society for Computation in Linguistics 2022_ , pages 206–211, online. Association for Computational Linguistics.
  * Levy (2008) Roger Levy. 2008.  A noisy-channel model of rational human sentence comprehension under uncertain input.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP ’08_ , page 234, Honolulu, Hawaii. Association for Computational Linguistics.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  ArXiv:1907.11692 [cs].
  * Mahowald et al. (2023) Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023.  Dissociating language and thought in large language models: a cognitive perspective.  ArXiv:2301.06627 [cs].
  * Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018.  Targeted syntactic evaluation of language models.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics.
  * Mikhailov et al. (2022) Vladislav Mikhailov, Tatiana Shamardina, Max Ryabinin, Alena Pestova, Ivan Smurov, and Ekaterina Artemova. 2022.  RuCoLA: Russian corpus of linguistic acceptability.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5207–5227, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Montalbetti (1984) Mario M. Montalbetti. 1984.  _After binding: On the interpretation of pronouns_.  PhD Thesis, Massachusetts Institute of Technology.
  * Nair et al. (2020) Sathvik Nair, Mahesh Srinivasan, and Stephan Meylan. 2020.  Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge.  _arXiv:2010.13057 [cs]_.  ArXiv: 2010.13057.
  * O’Connor (2015) Ellen O’Connor. 2015.  _Comparative illusions at the syntax-semantics interface_.  Ph.D. thesis, University of Southern California.
  * Orth et al. (2021) Wesley Orth, Masaya Yoshida, and Shayne Sloggett. 2021.  Negative polarity item (NPI) illusion is a quantification phenomenon.  _Journal of Experimental Psychology: Learning, Memory, and Cognition_ , 47(6):906–947.  Publisher: American Psychological Association.
  * Paape (2023) Dario Paape. 2023.  When Transformer models are more compositional than humans: The case of the depth charge illusion.  _Experiments in Linguistic Meaning_ , 2(0):202–218.  Number: 0.
  * Paape et al. (2020) Dario Paape, Shravan Vasishth, and Titus von der Malsburg. 2020.  Quadruplex negatio invertit? The on-line processing of depth charge sentences.  _Journal of Semantics_ , 37(4):509–555.
  * Pannitto and Herbelot (2020) Ludovica Pannitto and Aurélie Herbelot. 2020.  Recurrent babbling: Evaluating the acquisition of grammar from limited input data.  In _Proceedings of the 24th Conference on Computational Natural Language Learning_ , pages 165–176, Online. Association for Computational Linguistics.
  * Parker and Phillips (2016) Dan Parker and Colin Phillips. 2016.  Negative polarity illusions and the format of hierarchical encodings in memory.  _Cognition_ , 157:321–339.
  * Phillips et al. (2011) Colin Phillips, Matthew W. Wagers, and Ellen F. Lau. 2011.  Grammatical illusions and selective fallibility in real-time language comprehension.  _Experiments at the Interfaces_ , 37:147–180.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Salazar et al. (2020) Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020.  Masked language model scoring.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2699–2712, Online. Association for Computational Linguistics.
  * Schrimpf et al. (2021) Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2021.  The neural architecture of language: Integrative modeling converges on predictive processing.  _Proceedings of the National Academy of Sciences_ , 118(45):e2105646118.
  * She et al. (2023) Jingyuan Selena She, Christopher Potts, Samuel R Bowman, and Atticus Geiger. 2023\.  Scone: Benchmarking negation reasoning in language models with fine-tuning and in-context learning.  _arXiv preprint arXiv:2305.19426_.
  * Shin et al. (2023) Unsub Shin, Eunkyung Yi, and Sanghoun Song. 2023.  Investigating a neural language model’s replicability of psycholinguistic experiments: A case study of NPI licensing.  _Frontiers in Psychology_ , 14:937656.
  * Someya and Oseki (2023) Taiga Someya and Yohei Oseki. 2023.  JBLiMP: Japanese benchmark of linguistic minimal pairs.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1581–1594, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Song et al. (2022) Yixiao Song, Kalpesh Krishna, Rajesh Bhatt, and Mohit Iyyer. 2022.  SLING: Sino linguistic evaluation of large language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4606–4634, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Stengel-Eskin and Van Durme (2022) Elias Stengel-Eskin and Benjamin Van Durme. 2022.  The curious case of control.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11065–11076, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Tonhauser and Matthewson (2015) Judith Tonhauser and Lisa Matthewson. 2015.  Empirical evidence in research on meaning.
  * Truong et al. (2023) Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. 2023.  Language models are not naysayers: An analysis of language models on negation benchmarks.  _arXiv preprint arXiv:2306.08189_.
  * van Schijndel and Linzen (2021) Marten van Schijndel and Tal Linzen. 2021.  Single-Stage Prediction Models Do Not Explain the Magnitude of Syntactic Disambiguation Difficulty.  _Cognitive Science_ , 45(6):e12988.
  * Vasishth et al. (2008) Shravan Vasishth, Sven Brüssow, Richard L. Lewis, and Heiner Drenhaus. 2008.  Processing polarity: How the ungrammatical intrudes on the grammatical.  _Cognitive Science_ , 32(4):685–712.  Publisher: Wiley Online Library.
  * Warstadt et al. (2019) Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretič, and Samuel R. Bowman. 2019.  Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs.  ArXiv:1909.02597 [cs].
  * Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020.  BLiMP: The Benchmark of Linguistic Minimal Pairs for English.  _Transactions of the Association for Computational Linguistics_ , 8:377–392.
  * Wason and Reich (1979) Peter C. Wason and Shuli S. Reich. 1979.  A Verbal Illusion.  _Quarterly Journal of Experimental Psychology_ , 31(4):591–597.
  * Wei et al. (2021) Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021.  Frequency Effects on Syntactic Rule Learning in Transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 932–948, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Weissweiler et al. (2023) Leonie Weissweiler, Taiqi He, Naoki Otani, David R. Mortensen, Lori Levin, and Hinrich Schütze. 2023.  Construction grammar provides unique insight into neural language models.  In _Proceedings of the First International Workshop on Construction Grammars and NLP (CxGs+NLP, GURT/SyntaxFest 2023)_ , pages 85–95, Washington, D.C. Association for Computational Linguistics.
  * Wellwood et al. (2018) Alexis Wellwood, Roumyana Pancheva, Valentine Hacquard, and Colin Phillips. 2018\.  The Anatomy of a Comparative Illusion.  _Journal of Semantics_ , 35(3):543–583.
  * Wilcox et al. (2018) Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018.  What do RNN Language Models Learn about Filler-Gap Dependencies? In _Proceedings of BlackboxNLP 2018_.  ArXiv: 1809.00042.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xiang et al. (2009) M Xiang, B Dillon, and C Phillips. 2009.  Illusory licensing effects across dependency types: ERP evidence.  _Brain and Language_ , 108(1):40–55.
  * Zhang et al. (2022) Yuhan Zhang, Wenqi Chen, Ruihan Zhang, and Xiajie Zhang. 2022.  Representing affect information in word embeddings.
  * Zhang et al. (2023a) Yuhan Zhang, Carina Kauf, and Edward Gibson. 2023a.  A noisy-channel explanation of the comparative illusion.  _Architectures and Mechanisms for Language Processing_.
  * Zhang et al. (2023b) Yuhan Zhang, Rachel Ryskin, and Edward Gibson. 2023b.  A noisy-channel approach to depth-charge illusions.  _Cognition_ , 232:105346.

Illusion type | BERT | RoBERTa | GPT-2 | GPT-3
---|---|---|---|---
| PPL | Surp | PPL | Surp | PPL | Surp | PPL | Surp
Comparative | 0.43 | -0.07 | 0.45 | -0.22 | -0.33 | -0.08 | 0.15 | -0.04
Depth-charge | -0.61 | -0.01 | -0.20 | 0.28 | -0.41 | -0.01 | 0.12 | 0.90
NPI | -0.87 | 0.27 | -0.21 | 0.54 | -0.79 | 0.48 | -0.70 | 0.41
| Illusion sentences are more acceptable than unacceptable sentences.
| The unacceptable sentences are more acceptable than illusion sentences.
| No significant difference between the two conditions.
Table 2:  Estimates of the main effect (unacceptable sentences vs. illusion
sentences) for each statistical model. Positive estimates mean larger
perplexity or word surprisals for the unacceptable condition which indicates
an illusion effect. Negative estimates mean the unacceptable condition is more
acceptable than the illusion condition, which is opposite to the prediction.
Bolded estimates represent statistical significance ($p<.05$). We mark the
cell in green if there is an illusion effect; in orange for no illusion
effect.  COMPARATIVE ILLUSION
---
Number | VP | Examples
When the than-clause subject is noun phrase:
Singular | Repeatable | More students have been to Russia than the teacher has.
Singular | Non-repeatable | More students have escaped from Russia than the teacher has.
Plural | Repeatable | More students have been to Russia than the teachers have.
Plural | Non-repeatable | More students have escaped from Russia than the teachers have.
Control | Repeatable | More students have been to Russia than teachers have. (Good)
Control | Non-repeatable | More students have escaped from Russia than teachers have. (Good)
When the than-clause subject is pronoun:
Singular | Repeatable | More students have been to Russia than I have.
Singular | Non-repeatable | More students have escaped from Russia than I have.
Plural | Repeatable | More students have been to Russia than we have.
Plural | Non-repeatable | More students have escaped from Russia than we have.
Control | Repeatable | Many students have been to Russia more than I have. (Good)
Control | Non-repeatable | Many students have escaped from Russia more than I have. (Bad)
Table 3:  Full manipulation for the Comparative illusion  DEPTH CHARGE
ILLUSION
---
Conditions | Examples
Canonical depth-charge | No head injury is too trivial to be ignored.
Plausible control | Some head injury is too severe to be ignored.
Implausible control | Some head injury is too trivial to be ignored.
too…to | plausible | No head injury is too trivial to be treated.
too…to | implausible | No head injury is too trivial to be ignored.
too…to not | plausible | No head injury is too trivial to not be ignored.
too…to not | implausible | No head injury is too trivial to not be treated.
so…as to | plausible | No head injury is so trivial as to be ignored.
so…as to | implausible | No head injury is so trivial as to be treated.
Table 4:  Full manipulation for the Depth-charge illusion  NPI ILLUSION
---
Conditions | Examples
Matrix No |  No hunter who the villager believed to be trustworthy will ever shoot a bear.
Licensor Absent | The hunter who the villager believed to be trustworthy will ever shoot a bear.
Relative No | The hunter who no villager believed to be trustworthy will ever shoot a bear.
Relative Didn’t | The hunter who didn’t believe the villager to be trustworthy will ever shoot a bear.
Relative Did not | The hunter who did not believe the villager to be trustworthy will ever shoot a bear.
Relative Never | The hunter who never believed the villager to be trustworthy will ever shoot a bear.
Table 5:  Full manipulation for the NPI illusion Figure 6: Standardized scores
of the Perplexity & Surprisal for sentences in three conditions crossing LMs
and language illusion types. If the illusion effect appears, the illusion
condition should be rated more acceptable (thus lower in the graph) than the
unacceptable condition and therefore has lower perplexity/surprisal. (Error
bars are 95% bootstrapped confidence intervals).


In SymPy [51], we use the recursive formula above to compute the order-by-
order corrections to the modified Hamiltonian $H_{k}$. These are then used to
compute the values of $C_{j,k}=\Omega_{j}(H_{k})$ for each fixed value of
$j+k-1=N$ to check Conjecture 1 at each order in $N$. These are also used to
compute the remainders for fixed values of $j+k=N+1$.

We essentially use the formula (163) to compute the action of $\Psi_{j}$ on
each $H_{k}$ as needed. However, we do this by expanding both exponentials as
power series in $\eta$ up to sufficiently high order, from which we can
compute the derivatives symbolically with respect to $\eta$ and then evaluate
the result at $\eta=0$.

The necessary computations to check Conjecture 1 when $N=11$ can in principle
be run in Python. However, due to the explosion in the number of terms, the
program ran into memory and rounding issues when trying to cancel out all of
the fractions in the resultant expressions and show that the terms vanished.
We leave an investigation of a more efficient program for the verification of
the above conjecture as future work.

###  F.3 Concluding the proof

In the prior subsections of Appendix F, we confirmed (167) for orders
$N\in\\{0,1,2,3\\}$ by direct computation for arbitrry dimension $d$; we also
confirmed the cases $N\in\\{0,1,\dots,10\\}$ by computer for dimension $d=1$.
However, (167) only bounds the difference in $\widetilde{H}_{\eta}^{(N)}$
after one iteration while Conjecture 1 gives the same for multiple iterations.
To derive a bound after $k$ iterations, we apply (167) recursively and then
the triangle inequality as follows:

$\displaystyle\left|\widetilde{H}_{\eta}^{\left(N\right)}\left(p_{k},q_{k}\right)-\widetilde{H}_{\eta}^{\left(N\right)}\left(p_{0},q_{0}\right)\right|$
$\displaystyle\leq\sum_{j=0}^{k-1}{\left|\widetilde{H}_{\eta}^{\left(N\right)}\left(p_{j+1},q_{j+1}\right)-\widetilde{H}_{\eta}^{\left(N\right)}\left(p_{j},q_{j}\right)\right|}$
$\displaystyle\leq k\Psi\left(N\right)L^{N+3}\eta^{N+2},$

where in the first inequality above we use triangle inequality. This concludes
the proof.

##  Appendix G Characterization of $\Phi$

The coefficient function $\Phi:\mathbb{N}_{0}\rightarrow\mathbb{Q}$ from
Conjecture 1 has the following known values for $N=0,1,\dots,10$:

$N$ | $0$ | $1$ | $2$ | $3$ | $4$ | $5$ | $6$ | $7$ | $8$ | $9$ | $10$
---|---|---|---|---|---|---|---|---|---|---|---
$\Phi\left(N\right)$ (fraction) | $1$ | $3$ | $\frac{49}{12}$ | $\frac{197}{36}$ | $\frac{139}{20}$ | $\frac{1049}{108}$ | $\frac{850271}{60480}$ | $\frac{10046117}{453600}$ | $\frac{16791911}{453600}$ | $\frac{73961467}{1088640}$ | $\frac{31567569067}{239500800}$
$\Phi\left(N\right)$ (rounded) | $1.000$ | $3.000$ | $4.083$ | $5.472$ | $6.950$ | $9.713$ | $14.059$ | $22.148$ | $37.019$ | $67.939$ | $131.806$

It is not immediately obvious as to why $\Phi$ should be bounded. However, up
to $N=10$, we note that
$\Phi\left(N\right)=2\frac{\gamma(N)}{\sigma\left(N+2\right)}$, where
$\gamma:\mathbb{N}\rightarrow\mathbb{N}$ returns an unknown sequence of
positive integers

$N$ | $0$ | $1$ | $2$ | $3$ | $4$ | $5$ | $6$ | $7$ | $8$ | $9$ | $10$
---|---|---|---|---|---|---|---|---|---|---|---
$\gamma\left(N\right)$ | $1$ | $18$ | $49$ | $1970$ | $5004$ | $293720$ | $850271$ | $40184468$ | $134335288$ | $16271522740$ | $63135138134$

and $\sigma:\mathbb{N}\rightarrow\mathbb{N}$ returns the Hirzebruch numbers,
which can be defined as follows [57]:

$\displaystyle\sigma\left(n\right):=\prod_{\begin{subarray}{c}2\leq p\leq
n+1\\\ p\textrm{
prime}\end{subarray}}{p^{\lfloor\frac{n}{p-1}\rfloor}}\quad\forall
n\in\mathbb{N}.$

As proven in [11, Corollary 9],
$\sigma\left(n\right)=\mathcal{O}\left(n^{n}\right)$ for $n$ large.
Furthermore, $\gamma\left(N\right)\leq\frac{2}{3}\left(N+2\right)^{N+2}$ is
true up to $N=10$ and only saturates at $N=1$. Thus, assuming that
$\mathcal{O}\left(\gamma\left(N\right)\right)=\mathcal{O}\left(N^{N}\right)$
remains true for $N>10$, it remains plausible (although hitherto unproven)
that $\mathcal{O}\left(\Phi\left(N\right)\right)=\mathcal{O}\left(1\right)$
for large $N$. Hence, we suspect that $\Phi$ is monotonic increasing and
bounded as claimed in Conjecture 1.

## References

  * AB [12] Rüdiger Achilles and Andrea Bonfiglioli.  The early proofs of the theorem of Campbell, Baker, Hausdorff, and Dynkin.  Archive for History of Exact Sciences, 66(3):295–358, April 2012.
  * AKN [06] Vladimir I. Arnold, Valery V. Kozlov, and Anatoly I. Neishtadt.  Mathematical Aspects of Classical and Celestial Mechanics: Third Edition.  Springer Berlin Heidelberg, 2006.
  * ALW [21] Jacob Abernethy, Kevin A. Lai, and Andre Wibisono.  Fast convergence of fictitious play for diagonal payoff matrices.  In Proceedings of the Thirty-Second Annual ACM-SIAM Symposium on Discrete Algorithms, SODA ’21, page 1387–1404, USA, 2021. Society for Industrial and Applied Mathematics.
  * ANN [18] Shami A. Alsallami, Jitse Niesen, and Frank W. Nijhoff.  Closed-form modified Hamiltonians for integrable numerical integration schemes.  Nonlinearity, 31(11):5110–5146, 2018.
  * AS [65] Milton Abramowitz and I. A. Stegun, editors.  Handbook of mathematical functions.  Dover Books on Mathematics. Dover Publications, Mineola, NY, June 1965.
  * Atk [88] Kendall E. Atkinson.  An Introduction to Numerical Analysis.  Wiley, New York, 1988.
  * BBM [18] Stefano Biagi, Andrea Bonfiglioli, and Marco Matone.  On the Baker-Campbell-Hausdorff theorem: Non-convergence and prolongation issues.  Linear and Multilinear Algebra, 68(7):1310–1328, November 2018.
  * BC [04] Sergio Blanes and Fernando Casas.  On the convergence and optimization of the Baker-Campbell-Hausdorff formula.  Linear Algebra and its Applications, 378:135–158, February 2004.
  * Bet [16] Michael Betancourt.  Identifying the optimal integration time in Hamiltonian Monte Carlo.  arXiv:1601.00225, 2016.
  * Bet [19] Michael Betancourt.  The convergence of Markov chain Monte Carlo methods: from the Metropolis method to Hamiltonian Monte Carlo.  Annalen der Physik, 531(3):1700214, 2019.
  * BF [23] Abdelmalek Bedhouche and Bakir Farhi.  On some products taken over prime numbers.  Journal of Integer Sequences, 26(1), January 2023.
  * BGP [20] James P. Bailey, Gauthier Gidel, and Georgios Piliouras.  Finite regret and cycles with fixed step-size via alternating gradient descent-ascent.  In Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 391–407. PMLR, July 2020.
  * BJW [18] Michael Betancourt, Michael I Jordan, and Ashia C Wilson.  On symplectic optimization.  arXiv preprint arXiv:1802.03653, 2018.
  * Bos [86] A. J. Bosch.  The factorization of a square matrix into two symmetric matrices.  The American Mathematical Monthly, 93(6):462–464, 1986.
  * BP [19] James P Bailey and Georgios Piliouras.  Multi-agent learning in network zero-sum games is a Hamiltonian system.  In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 233–241, 2019.
  * BRM+ [18] David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel.  The mechanics of $n$-player differentiable games.  In International Conference on Machine Learning, pages 354–363. PMLR, 2018.
  * BRSS [18] Nawaf Bou-Rabee and Jesús Maria Sanz-Serna.  Geometric integrators and the Hamiltonian Monte Carlo method.  Acta Numerica, 27:113–206, 2018.
  * CDWY [20] Yuansi Chen, Raaz Dwivedi, Martin J. Wainwright, and Bin Yu.  Fast mixing of Metropolized Hamiltonian Monte Carlo: Benefits of multi-step gradients.  JMLR, 2020.
  * CE [14] G.C. Calafiore and L. El Ghaoui.  Optimization Models.  Control systems and optimization series. Cambridge University Press, October 2014.
  * CGH+ [17] Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus A Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell.  Stan: A probabilistic programming language.  Journal of statistical software, 76, 2017.
  * CM [09] Fernando Casas and Ander Murua.  An efficient algorithm for computing the Baker–Campbell–Hausdorff series and some of its applications.  Journal of Mathematical Physics, 50(3):033513, March 2009.
  * CS [90] P. J. Channell and C. Scovel.  Symplectic integration of Hamiltonian systems.  Nonlinearity, 3(2):231–259, May 1990.
  * CV [19] Zongchen Chen and Santosh S Vempala.  Optimal convergence rate of Hamiltonian Monte Carlo for strongly logconcave distributions.  International Conference on Randomization and Computation (RANDOM), 2019.
  * DJ [21] Jelena Diakonikolas and Michael I Jordan.  Generalized momentum-based methods: A Hamiltonian perspective.  SIAM Journal on Optimization, 31(1):915–944, 2021.
  * DKPR [87] Simon Duane, A. D. Kennedy, Brian J. Pendleton, and Duncan Roweth.  Hybrid Monte Carlo.  Physics Letters B, 1987.
  * DL [22] Valentin Duruisseaux and Melvin Leok.  Accelerated optimization on Riemannian manifolds via discrete constrained variational integrators.  Journal of Nonlinear Science, 32(4):42, 2022.
  * DLGS [23] G Bruno De Luca, Alice Gatti, and Eva Silverstein.  Improving energy conserving descent for machine learning: Theory and practice.  arXiv preprint arXiv:2306.00352, 2023.
  * ESD [05] Robert D. Engle, Robert D. Skeel, and Matthew Drees.  Monitoring energy drift with shadow Hamiltonians.  Journal of Computational Physics, 206(2):432–452, July 2005.
  * Eva [10] Lawrence C. Evans.  Partial differential equations.  American Mathematical Society, Providence, R.I., 2010.
  * Fel [68] William Feller.  An Introduction to Probability Theory and Its Applications, Volume 1.  Wiley Series in Probability and Statistics. John Wiley & Sons, Nashville, TN, 3 edition, January 1968.
  * FN [03] Chris M. Field and Frank W. Nijhoff.  A note on modified Hamiltonians for numerical integrations admitting an exact invariant.  Nonlinearity, 16(5):1673–1683, 2003.
  * GHP+ [19] Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Rémi Le Priol, Gabriel Huang, Simon Lacoste-Julien, and Ioannis Mitliagkas.  Negative momentum for improved game dynamics.  In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 1802–1811. PMLR, 16–18 Apr 2019.
  * GKP [94] Ronald L. Graham, Donald E. Knuth, and Oren Patashnik.  Concrete mathematics.  Addison Wesley, Boston, MA, 2 edition, February 1994.
  * Hal [15] Brian C. Hall.  Lie groups, Lie algebras, and representations: An elementary introduction.  Springer, 2015.
  * HG [14] Matthew D Hoffman and Andrew Gelman.  The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo.  Journal of Machine Learning Research, 15:1351–1381, 2014.
  * HH [91] Yoopyo Hong and Roger A. Horn.  The Jordan canonical form of a product of a Hermitian and a positive semidefinite matrix.  Linear Algebra and its Applications, 147:373–386, 1991.
  * Hof [96] Josef Hofbauer.  Evolutionary dynamics for bimatrix games: A Hamiltonian system?  Journal of mathematical biology, 34:675–688, 1996.
  * HWL [06] Ernst Hairer, Gerhard Wanner, and Christian Lubich.  Geometric Numerical Integration.  Springer, 2006.
  * Jac [79] Nathan Jacobson.  Lie Algebras.  Dover Books on Mathematics. Dover Publications, Mineola, NY, December 1979.
  * JS [98] Jorge V. Jose and Eugene J. Saletan.  Classical dynamics.  Cambridge University Press, Cambridge, England, August 1998.
  * KBN [21] Zsófia Kereskényi-Balogh and Gábor Nyul.  Fubini numbers and polynomials of graphs.  Mediterranean Journal of Mathematics, 18(6), September 2021.
  * KLSV [22] Yunbum Kook, Yin-Tat Lee, Ruoqi Shen, and Santosh Vempala.  Sampling with Riemannian Hamiltonian Monte Carlo in a constrained space.  Advances in Neural Information Processing Systems, 35:31684–31696, 2022.
  * Kry [96] N. Krylov.  Lectures on elliptic and parabolic equations in Hölder spaces.  Graduate studies in mathematics. American Mathematical Society, Providence, RI, September 1996.
  * Lan [99] Serge Lang.  Complex Analysis.  Springer New York, 1999.
  * Lax [02] Peter D. Lax.  Functional Analysis.  Pure and Applied Mathematics: A Wiley Series of Texts, Monographs and Tracts. John Wiley & Sons, Nashville, TN, March 2002.
  * Laz [93] Vladimir F. Lazutkin.  KAM Theory and Semiclassical Approximations to Eigenfunctions.  Springer Berlin Heidelberg, 1993.
  * LSC+ [19] Songtao Lu, Rahul Singh, Xiangyi Chen, Yongxin Chen, and Mingyi Hong.  Alternating gradient descent ascent for nonconvex min-max problems in robust learning and GANs.  In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pages 680–684, 2019.
  * LST [20] Yin Tat Lee, Ruoqi Shen, and Kevin Tian.  Logsmooth gradient concentration and tighter runtimes for Metropolized Hamiltonian Monte Carlo.  In Conference on learning theory, pages 2565–2597. PMLR, 2020.
  * LV [18] Yin Tat Lee and Santosh S Vempala.  Convergence rate of Riemannian Hamiltonian Monte Carlo and faster polytope volume computation.  In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pages 1115–1121, 2018.
  * Mil [73] Willard Miller.  Symmetry Groups and Their Applications.  ISSN. Elsevier Science, 1973.
  * MSP+ [17] Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, Amit Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Štěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz.  SymPy: Symbolic computing in Python.  PeerJ Computer Science, 3:e103, January 2017.
  * MV [18] Oren Mangoubi and Nisheeth Vishnoi.  Dimensionally tight bounds for second-order Hamiltonian Monte Carlo.  Advances in Neural Information Processing Systems, 31, 2018.
  * Nea [11] Radford M Neal.  MCMC using Hamiltonian dynamics.  Handbook of Markov Chain Monte Carlo, 2(11):2, 2011.
  * Nem [04] Arkadi Nemirovski.  Prox-method with rate of convergence o(1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems.  SIAM Journal on Optimization, 15(1):229–251, 2004.
  * NT [87] Morris Newman and Robert C. Thompson.  Numerical values of Goldberg’s coefficients in the series for $\log(e^{x}e^{y})$.  Mathematics of Computation, 48(177):265–271, 1987.
  * NY [83] A. S. Nemirovsky and D. B. Yudin.  Problem complexity and method efficiency in optimization.  Wiley-Interscience Series in Discrete Mathematics. John Wiley & Sons, Chichester, England, April 1983.
  * OEI [24] OEIS Foundation Inc.  A091137. From The On-Line Encyclopedia of Integer Sequences, 2024.  Last visited on 01/29/2024.
  * Ohs [23] Tomoki Ohsawa.  Preservation of quadratic invariants by semiexplicit symplectic integrators for nonseparable Hamiltonian systems.  SIAM Journal on Numerical Analysis, 61(3):1293–1315, 2023.
  * OM [19] Brendan O’Donoghue and Chris J Maddison.  Hamiltonian descent for composite objectives.  Advances in Neural Information Processing Systems, 32, 2019.
  * OvS [11] Georg Ostrovski and Sebastian van Strien.  Piecewise linear Hamiltonian flows associated to zero-sum games: Transition combinatorics and questions on ergodicity.  Regular and Chaotic Dynamics, 16:128–153, 2011.
  * PW [21] Georgios Piliouras and Xiao Wang.  Constants of motion: The antidote to chaos in optimization and game dynamics.  arXiv preprint arXiv:2109.03974, 2021.
  * QG [15] Jocelyn Quaintance and Henry W. Gould.  Combinatorial identities for Stirling numbers: The unpublished notes of H W Gould.  World Scientific Publishing, Singapore, December 2015.
  * Roc [96] Ralph Tyrell Rockafellar.  Convex analysis.  Princeton Landmarks in Mathematics and Physics. Princeton University Press, Princeton, NJ, December 1996.
  * Ros [13] Kenneth A. Ross.  Sequences and Series of Functions, pages 187–221.  Springer New York, New York, NY, 2013.
  * RS [81] Michael Reed and Barry Simon.  I: Functional Analysis: Volume 1.  Methods of Modern Mathematical Physics. Academic Press, San Diego, CA, 2 edition, February 1981.
  * SAF [02] Yuzuru Sato, Eizo Akiyama, and J Doyne Farmer.  Chaos in learning a simple two-person game.  Proceedings of the National Academy of Sciences, 99(7):4748–4751, 2002.
  * SBC [16] Weijie Su, Stephen Boyd, and Emmanuel J Candes.  A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights.  Journal of Machine Learning Research, 17(153):1–43, 2016.
  * SH [01] Robert D. Skeel and David J. Hardy.  Practical construction of modified Hamiltonians.  SIAM Journal on Scientific Computing, 23(4):1172–1188, 2001.
  * Sur [89] Yuri B. Suris.  Integrable mappings of the standard type.  Functional Analysis and Its Applications, 23(1):74–76, 1989.
  * Suz [77] Masuo Suzuki.  On the convergence of exponential operators—the Zassenhaus formula, BCH formula and systematic approximants.  Communications in Mathematical Physics, 57(3):193–200, October 1977.
  * SWF [16] John Salvatier, Thomas V Wiecki, and Christopher Fonnesbeck.  Probabilistic programming in Python using PyMC3.  PeerJ Computer Science, 2:e55, 2016.
  * Tan [75] Stephen M. Tanny.  On some numbers related to the Bell numbers.  Canadian Mathematical Bulletin, 17(5):733–738, 1975.
  * Tho [82] Robert C. Thompson.  Cyclic relations and the Goldberg coefficients in the Campbell-Baker-Hausdorff formula.  Proceedings of the American Mathematical Society, 86(1):12–14, 1982.
  * TPL [19] Andrew R Teel, Jorge I Poveda, and Justin Le.  First-order optimization algorithms with resets and Hamiltonian flows.  In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 5838–5843. IEEE, 2019.
  * Tuy [95] G. M. Tuynman.  The derivation of the exponential map of matrices.  The American Mathematical Monthly, 102(9):818–820, 1995.
  * Var [84] V. S. Varadarajan.  Lie Groups, Lie Algebras, and Their Representations.  Springer New York, 1984.
  * vN [28] J. v. Neumann.  Zur theorie der gesellschaftsspiele.  Mathematische Annalen, 100(1):295–320, December 1928.
  * vS [11] Sebastian van Strien.  Hamiltonian flows with random-walk behaviour originating from zero-sum games and fictitious play.  Nonlinearity, 24(6):1715, 2011.
  * Wan [24] Jun-Kun Wang.  Hamiltonian descent and coordinate Hamiltonian descent.  arXiv preprint arXiv:2402.13988, 2024.
  * [80] Eric W. Weisstein.  Kronecker delta. From MathWorld—A Wolfram Web Resource, 2023.  Last visited on 10/25/2023.
  * [81] Eric W. Weisstein.  Polylogarithm. From MathWorld—A Wolfram Web Resource, 2023.  Last visited on 10/25/2023.
  * WMW [19] Ashia C Wilson, Lester Mackey, and Andre Wibisono.  Accelerating rescaled gradient descent: Fast optimization of smooth functions.  Advances in Neural Information Processing Systems, 32, 2019.
  * Wol [23] Wolfram Research, Inc.  Mathematica, Version 13.3, 2023.  Champaign, IL, 2023.
  * WRJ [21] Ashia C Wilson, Ben Recht, and Michael I Jordan.  A Lyapunov analysis of accelerated methods in optimization.  The Journal of Machine Learning Research, 22(1):5040–5073, 2021.
  * WTP [22] Andre Wibisono, Molei Tao, and Georgios Piliouras.  Alternating mirror descent for constrained min-max games.  In Advances in Neural Information Processing Systems, volume 35, pages 35201–35212. Curran Associates, Inc., 2022.
  * WWJ [16] Andre Wibisono, Ashia C Wilson, and Michael I Jordan.  A variational perspective on accelerated methods in optimization.  Proceedings of the National Academy of Sciences, 113(47):E7351–E7358, 2016.
  * Yos [90] Haruo Yoshida.  Construction of higher order symplectic integrators.  Physics Letters A, 150(5):262–268, 1990.
  * Yos [93] Haruo Yoshida.  Recent progress in the theory and application of symplectic integrators.  Celestial Mechanics and Dynamical Astronomy, 56(1-2):27–43, 1993.
  * Zou [18] Qing Zou.  The log-convexity of the Fubini numbers.  Transactions on Combinatorics, 7(2):17–23, 2018.
  * ZWLG [22] Guodong Zhang, Yuanhao Wang, Laurent Lessard, and Roger B. Grosse.  Near-optimal local convergence of alternating gradient descent-ascent for minimax optimization.  In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 7659–7679. PMLR, March 2022.


# Early publications about nonzero cosmological constant

I. Horvath11affiliation: Dept. of Physics, Bolyai Military University,
Budapest, POB 15, H-1581, Hungary
Dept. of Physics, Bolyai Military University, Budapest, POB 15, H-1581,
Hungary<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In 2011 the Nobel Prize in Physics was awarded for the 1998 discovery of the
nonzero cosmological constant. This discovery is very important and surely
worth to receive the Nobel Prize. However, years earlier several papers had
been published (Paál, Horváth, & Lukács 1992; Holba et al. 1992, Holba et al.
1994) about a very similar discovery from observational data.

History of astronomy; Cosmology: cosmological parameters — dark energy — large
scale structure of the Universe

ieeetr

##  1 2011 Nobel Prize Winners in Physics

”The Nobel Prize in Physics 2011 was divided, one half awarded to Saul
Perlmutter, the other half jointly to Brian P. Schmidt and Adam G. Riess for
the discovery of the accelerating expansion of the Universe through
observations of distant supernovae.” the nobelprize.org wrote.
111http://www.nobelprize.org/nobelprizes/physics/laureates/2011/

Two research teams, Supernova Cosmology Project (SCP) lead by Saul Perlmutter
and High-z Supernova Search Team (HZT) headed by Brian Schmidt, raced to map
the Universe by locating the most distant supernovae. The two research teams
found over 50 distant supernovae which were dimmer than expected - this was a
sign that the expansion of the Universe was accelerating [1], [2].

Theoretically, this was not new idea since Albert Einstein came out the idea
of cosmological constant (often marked Lambda). Einstein did that because he
was guided by the paradigm of the day that the Universe was static. When he
heard the Edwin Hubble discovery that the Universe was actually expanding he
declared that the inclusion of the cosmological constant was his ”biggest
blunder”. Since there was no observation for cosmological constant after that
most scientists assumed that Lambda is zero. A series of papers were published
in Astrophysics and Space Science in the early 90’s years before the supernova
publications calculated $\Omega_{\Lambda}$ from observed data.

##  2 Earlier Publications About Positive Cosmological Constant

Paál et al. [3] used the so called pencil beam survey [4] to find out whether
the regularity found in the galaxy distribution is quasiperiodical or not. It
was found that $q_{0}$ was preferably negative [3]. Therefore a nonzero Lambda
term was needed. The preferred value was $\Omega_{\Lambda}$ equal 2/3. This is
very close to the value that later was observed by the Nobel Prize winners.

Figure 1: This figure was published in Holba et al. (1994) (figure 8. in that
article). The red line represents the flat cosmological model.

In the second paper [5] a two parameter fit was made. The positive
cosmological constant (negative $q_{0}$) found to be still needed. In the
third paper [6] optical and radio quasars were also used to find the preferred
cosmological parameters. Figure 8. in that paper [6] showed the results (see
figure). As it was written the contour meant 80% confidence level. The
preferred region is similar that the supernovae analisys later suggested. For
comparison please see this www page
http://vizion.galileowebcast.hu/HOI/Comparation.jpg Therefore those earlier
suggestions also support the Nobel Prize winning results.

The author thanks for his supervisors, B. Lukács and G. Paál, to being part of
these researches. Unfortunately, G. Paál died in 1992. This was surely
affected the fact that these results were almost unrecognized.
REFERENCES [1] Perlmutter, S. et al., 1999, ApJ, 517, 565 [2] Riess, A. G. et
al., 1998, AJ, 116, 1009 [3] Paál, G., Horváth, I., & Lukács, B. 1992, Ap&SS,
191, 107 [4] Broadhurst, T. J. et al., 1990, Nature, 343, 726 [5] Holba, A.,
Horváth, I., Lukács, B., & Paál, G. 1992, Ap&SS, 198, 111 [6] Holba, A.,
Horváth, I., Lukács, B., & Paál, G. 1994, Ap&SS, 222, 65